Pulling by definition

Every data loading (pulling) task is defined using a definition file named haulwork.yaml.

See: Layout

The definition file must include task id and file itself must reside in folder which path corresponds to task id. Double-check.

# segment from file .../mydefs/from_arno/ods_arno/school/haulwoark.yaml
id: from_arno.ods_arno.school

Task ID

Task id is composed from three parts:

  • route code
  • target schema
  • taget table

Parts are separated with dot (point).

All parts must be lowercase and without special characters (cannot include whitespace)!

Warning

Last 2 parts together must be unique for target database

Samples of task_id

Sample and explanation follows.

from_arno.education.dim_school

This points out that data will come from route with code "from_armo" ("arno" is education-management system) and data will be put into schema education into table dim_school (where dim_ stands for dimension).

from_arno.ods_arno.school

Example above shows that data will come from route "from_arno" and will be put into schema "ods_arno" where ods_ stands for "operational data storage" which points out that data will (almost) 1:1 to source. Just such name convension.

inner.education.dim_school

Example above shows that data will be pulled from our target database (it doesn't say from where directly -- sql-file is for that purpose) and data will be put into "education"."dim_school".

By desing You cannot use all 3 examples for same target database! Look previous warning! Advisable is to use last two (and not first one).

Sections

  • id
  • run
  • actions
  • upgrade_nondata

Section "run"

A task can be started independently by schedule or after another task has finished. Use run for a scheduled task and depends for a dependent task.

Scheduled task

Section run defines during which parts of day the task can run and how long must be kept between two runs.

run:
  keep_pause: 1 hour
  morning: true
  workhours: false
  evening: false

keep_pause is the minimum interval between two runs. It is written as a time interval, for example 30 minutes, 1 hour or 2 days.

Day is divided into three non-overlapping parts:

  • morning -- from 00:00 until 07:59
  • workhours -- from 08:00 until 17:59
  • evening -- from 18:00 until 23:59

Use true to allow running in a part of day and false to disallow it.

Warning

At least one part of day must be defined and at least one should be true for the task to run by schedule.

Dependent task

Use depends when the task must start after another task has finished successfully. The dependent task will be added to the queue immediately after that.

depends:
  - inner.codes.month

More than one task id can be listed when needed.

Section "actions"

Section actions contains the work which will be done by the task. It is a list of agents and their settings.

Every action must have do with the name of agent to use. Other fields depend on that agent.

actions:
  - do: stream
    file: input.sql

  - do: shadow
    input: shadow.sql
    pkcol:
      - id

Actions are run in defined order. Next action starts only after previous action has finished successfully. If an action fails, following actions are not run.

See: List of built-in agents

Section "upgrade_nondata"

Section upgrade_nondata is an optional list of version numbers which contain only non-data changes.

Non-data changes do not change contents or loading order of data. For example:

  • adding or changing comments
  • creating an index
  • increasing maximum length of a text column
upgrade_nondata:
  - 2 # varchar pikkuse muutus

This helps versioning handle technical changes without treating them as changes to data.

Warning

Add a version number only when its changes do not change data or data loading logic.

See: Versioning