Layout

DataPuller is Python based solution. Just now python 3.12 is minimal requirement.

Make new project. Assuming contineous delivery You may want make 2-3 important root folders:

• src -- where all pulling instructions and controlling data go (for that project it can be called as source code)
• deploy(files) or something similar -- where to keep deployment instructions (gitlab enforces you keep main file (.gitlab-ci.yml) in project root, but other files you can put nicely here)
• docs or similar -- to keep your business rules in machine readable manner

Under src:

• routes -- pulling instructions in very fixed way, read more in Routes and sources and Definition
• ver -- SQL DDL command for preparing target database structure (like schemas, grants, indices), read more in Versioning
• conf -- where you can define resources for your sources (in case you have CD/CI awailable You can do it in there), neither way don't put database passwords nor API tokens into code repository!
• scripts -- your own cron script to run starter script, read more in Starter

GitLab project sample structure

├── deployfiles
│   ├── kubernetes
│   │   ├── .gitlab-ci.yml       # copy of root file for that way to deploy
│   │   ├── docker-compose.yml   # helper file 
│   │   └── dapu.dockerfile      # lazy ones may call it "Dockerfile." or ".dockerfile"
│   ├── virtual_machine
│   │   ├── .gitlab-ci.yml       # copy if root file for that way to deploy
│   │   └── some_other.file
│   ├── win_xcopy
│   │   └── imitate.bat          # yes, you can imitate deploy coping src to some other folder and run from there
│   └── targets_conf_prepare.py  # any kind of deploy helper in python level to avoid bash level difficulties
└── src                          # folder to be deployed, eg. into /srv/dapu or /opt/something/dapu etc
    ├── routes
    │   ├── from_asjur5          # one external source, prefix "from_" is advise
    │   │   ├── ods_asjur5       # schema name. prefix "ods_" is advise
    │   │   │   ├── adm_group    # table name 
    │   │   │   │   ├── haulwork.yaml     # instructions
    │   │   │   │   ├── input.sql         # details for instructions
    │   │   │   │   └── ver               # 3-digit numbering files for this target table
    │   │   │   │       │── 001.sql       # usually create table command
    │   │   │   │       └── 002.sql       # alter, comment, index and other improvements 
    │   │   │   └── adm_policy   # unother table
    │   │   │       ├── haulwork.yaml
    │   │   │       ├── input.sql
    │   │   │       └── ver
    │   │   │           └── 001.sql
    │   │   └── route.yaml       # definition of source
    │   ├── inner                # source is our target database
    │   │   ├── codes            # schema (eg. analytical domain, transformation result)
    │   │   │   └── month
    │   │   │       ├── haulwork.yaml
    │   │   │       ├── insert.sql
    │   │   │       └── ver
    │   │   │           ├── 001.sql
    │   │   │           └── 002.sql
    │   │   └── route.yaml
    ├── conf
    │   ├── agents.yaml          # only if needed to override core behaviour or to add custom agents
    │   ├── sql.yaml             # includes DBS connections named profiles (both target (named as "target") and all needed sources for routes)
    │   └── tasks_to_delete.txt  # lines like "from_asjur5.ods_asjur5.adm_oldie" (keeping it in "conf" may seem strange, but...)
    ├── ver                      # target versioning
    │   ├── changes.yaml         # contains ordered references to SQL files in folder (usually SQL DDL commands)
    │   ├── v001_codes.sql       # pattern "vXXX" gives in all op.systems visually ordered list
    │   └── v002_asjur5.sql
    └── scripts
        ├── runall.sh            # bash script (as well here can be similar runall.bat for windows)
        ├── run_target_full.py   # run this target/project, all commands in row
        └── create_cron.sh       #