Layout

[//]: # jutt sellest, kuidas ja kuhu panna failid, et tervik omaks mõtet

Git-project and types of files in it

Let's assume that we have nicely separated code for loading data and it sits in dedicated project that is unreachable (in sense on code) to developers who write loading instructions. Assuming previous we can think instructions as separate project maintained by git (eg. GitLab)

├── deployfiles
│   ├── kubernetes
│   │   ├── .gitlab-ci.yml
│   │   ├── docker-compose.yml
│   │   └── dapu.dockerfile
│   ├── virtual_machine
│   │   ├── .gitlab-ci.yml
│   │   └── some_other.file
│   └── targets_conf_prepare.py
└── src // this folder will be deployed, maybe change of name during deploy to project name -> eg. "projects/dev1_tux24"
    ├── routes
    │   ├── from_asjur5
    │   │   ├── ods_asjur5
    │   │   │   ├── adm_grupp
    │   │   │   │   ├── haulwork.yaml
    │   │   │   │   ├── input.sql
    │   │   │   │   └── ver
    │   │   │   │       └── 001.sql
    │   │   │   └── adm_grupp_lubatud
    │   │   │       ├── haulwork.yaml
    │   │   │       ├── input.sql
    │   │   │       └── ver
    │   │   │           └── 001.sql
    │   │   └── route.yaml
    │   ├── inner
    │   │   ├── codes
    │   │   │   └── month
    │   │   │       ├── haulwork.yaml
    │   │   │       ├── insert.sql
    │   │   │       └── ver
    │   │   │           ├── 001.sql
    │   │   │           └── 002.sql
    │   │   └── route.yaml
    ├── conf
    │   ├── agents.yaml // if needed to override core behaviour or to add custom agents
    │   ├── api.yaml // includes API connections named profiles
    │   ├── sql.yaml // includes DBS connections named profiles (both target (named as "target") and all needed sources for routes)
    │   └── tasks_to_delete.txt // lines like "from_asjur5.ods_asjur5.adm_oldie" (keeping it in "conf" may seem strange, but...)
    ├── ver // see "target versioning"
    │   ├── changes.yaml // contains ordered references to SQL files in folder (usually SQL DDL commands)
    │   ├── v001_codes.sql
    │   └── v002_asjur5.sql
    └── scripts
        ├── runall.sh // bash script (as well here can be similar runall.bat for windows)
        ├── run_target_full.py // run this target/project, all commands in row
        └── create_cron.sh

Folder "deployfiles" can contain some templates for different deploy techniques and systems.

Currently used technique files must be copied to right place. Eg. for GitLab CI/CD file ".gitlab-ci.yml" must be in root folder of git project.