Pulling by definition
Every data loading (pulling) task is defined using a definition file named haulwork.yaml.
See: Layout
The definition file must include task id and file itself must reside in folder which path corresponds to task id. Double-check.
# segment from file .../mydefs/from_arno/ods_arno/school/haulwoark.yaml
id: from_arno.ods_arno.school
Task ID
Task id is composed from three parts:
- route code
- target schema
- taget table
Parts are separated with dot (point).
All parts must be lowercase and without special characters (cannot include whitespace)!
Warning
Last 2 parts together must be unique for target database
Samples of task_id
Sample and explanation follows.
from_arno.education.dim_school
This points out that data will come from route with code "from_armo" ("arno" is education-management system) and data will be put into schema education into table dim_school (where dim_ stands for dimension).
from_arno.ods_arno.school
Example above shows that data will come from route "from_arno" and will be put into schema "ods_arno" where ods_ stands for "operational data storage" which points out that data will (almost) 1:1 to source. Just such name convension.
inner.education.dim_school
Example above shows that data will be pulled from our target database (it doesn't say from where directly -- sql-file is for that purpose) and data will be put into "education"."dim_school".
By desing You cannot use all 3 examples for same target database! Look previous warning! Advisable is to use last two (and not first one).
Sections
- id
- run
- actions
- upgrade_nondata
Section "run"
A task can be started independently by schedule or after another task has finished. Use run for a scheduled task and depends for a dependent task.
Scheduled task
Section run defines during which parts of day the task can run and how long must be kept between two runs.
run:
keep_pause: 1 hour
morning: true
workhours: false
evening: false
keep_pause is the minimum interval between two runs. It is written as a time interval, for example 30 minutes, 1 hour or 2 days.
Day is divided into three non-overlapping parts:
- morning -- from 00:00 until 07:59
- workhours -- from 08:00 until 17:59
- evening -- from 18:00 until 23:59
Use true to allow running in a part of day and false to disallow it.
Warning
At least one part of day must be defined and at least one should be true for the task to run by schedule.
Dependent task
Use depends when the task must start after another task has finished successfully. The dependent task will be added to the queue immediately after that.
depends:
- inner.codes.month
More than one task id can be listed when needed.
Section "actions"
Section actions contains the work which will be done by the task. It is a list of agents and their settings.
Every action must have do with the name of agent to use. Other fields depend on that agent.
actions:
- do: stream
file: input.sql
- do: shadow
input: shadow.sql
pkcol:
- id
Actions are run in defined order. Next action starts only after previous action has finished successfully. If an action fails, following actions are not run.
Section "upgrade_nondata"
Section upgrade_nondata is an optional list of version numbers which contain only non-data changes.
Non-data changes do not change contents or loading order of data. For example:
- adding or changing comments
- creating an index
- increasing maximum length of a text column
upgrade_nondata:
- 2 # varchar pikkuse muutus
This helps versioning handle technical changes without treating them as changes to data.
Warning
Add a version number only when its changes do not change data or data loading logic.
See: Versioning