Agent: restapi

Calls configured REST API endpoint and saves response as JSON file.

Used for getting data from a REST API and saving the response into a file.

Typical flow:

restapi -> saves JSON file
json    -> loads JSON file into target table

For repeated API calls:

restapi -> creates response files + index file
json    -> reads index file, loads response files, deletes processed files

Current limitation

restapi currently expects API response to be valid JSON.

The agent parses the response as JSON before saving it into a file. Because of that, non-JSON responses such as plain text or XML are not supported yet.

This may change later if restapi is extended to work as a more general response/content downloader.

API definition

API connection information is kept in route-level route.yaml.

File location:

routes/{route}/route.yaml

Example route.yaml:

api_alias:
  url: "https://www.example.com/" # root url
  method: GET # optional; default: GET
  auth: # optional; default: public auth
    method: public # public | basic | bearer; default: public
    username: "%BASIC_API_USER%" # for basic
    password: "%BASIC_API_PASSWORD%" # for basic
    token: "%API_TOKEN%" # for bearer
  headers:
    Accept: "application/json"

Minimal:

api_alias:
  url: "https://www.example.com/"
  headers:
    Accept: "application/json"

If method is missing, GET is used.

If auth is missing, API is treated as public:

auth:
  method: public

route.yaml uses top-level keys as API aliases. In haulwork.yaml, api_source.api references one of those aliases.

Single request

- do: restapi
  api_source:
    api: public_request
    path: "/haridusasutused"
  single_destination:
    type: task
    file: haridusasutused.json

Single request saves one API response into single_destination.

TODO: POST and request body support exists, but docs should be expanded later when session/token extraction and dependent requests are implemented.

If values need to be inserted into the URL path or saved for later JSON processing, use repeated requests with repeat or repeat_sql.

Repeated requests

Repeated requests are used when the same API endpoint must be called several times with different parameters.

- do: restapi
  api_source:
    api: public_request
    path: "/haridusasutused/{{jrk}}"

  repeat:
    - params:
        jrk: 17
    - params:
        jrk: 18

  repeat_index:
    type: task
    file: unprocessed_files.json
    mode: append

This creates separate response files and writes their paths with used parameters into unprocessed_files.json.

Example index file:

[
  {
    "file": "work/routes/test_route/public/schools/restapi_20260721_134739_579796.json",
    "params": {
      "jrk": 17
    }
  }
]

Continue on error

By default, repeated requests stop on the first failed API request.

If some repeat params may fail but the next ones should still be processed, use continue_on_error.

- do: restapi
  api_source:
    api: public_request
    path: "/haridusasutused/{{jrk}}"

  repeat:
    - params:
        jrk: 17
    - params:
        jrk: 18

  continue_on_error: true

  repeat_index:
    type: task
    file: unprocessed_files.json
    mode: append

If continue_on_error is true, failed repeat params are logged and the agent continues with the next params.

Only successful responses are saved as response files and added to repeat_index.

If continue_on_error is missing or false, the first failed repeated request stops the action.

Repeated requests from SQL

repeat_sql can be used when repeat parameters should come from target database.

- do: restapi
  api_source:
    api: public_request
    path: "/haridusasutused/{{jrk}}"

  repeat_sql:
    file: repeat.sql
    mapping:
      jrk: 1
      aeg: 2

  repeat_index:
    type: task
    file: unprocessed_files.json
    mode: append

repeat_sql.file is read from current task directory.

repeat_sql.mapping maps parameter names to SQL result column positions. Column positions start from 1.

Example repeat.sql:

SELECT 17, 'hommik'
UNION
SELECT 18, 'lõuna'

Repeat index mode

repeat_index controls where repeated response files are tracked.

repeat_index:
  type: task
  file: unprocessed_files.json
  mode: append

Default values:

repeat_index.type -> task
repeat_index.file -> unprocessed_files.json
repeat_index.mode -> append

Supported modes:

append   - keep old index entries and append new ones
override - delete files from old index, clear index, then write new entries

In override mode, files listed in the previous index file are deleted before the new index is created. This prevents old response files from becoming orphaned when the index is overwritten.

Single destination

For non-repeated requests, response is saved to single_destination.

single_destination:
  type: task
  file: response.json

File pointers

single_destination and repeat_index use file pointers. See File pointers.

Normal usage is to save files into current task folder:

type: task
file: response.json

Filesystem folders can be used with type: fs.

type: fs
dir: "/tmp/somepath"
file: response.json

Required and optional settings

Required in haulwork.yaml for single request:

do
api_source
api_source.api
single_destination
single_destination.type
single_destination.file

Required in haulwork.yaml for repeated requests:

do
api_source
api_source.api
repeat or repeat_sql

Optional:

api_source.path
api_source.method
api_source.params
api_source.headers
api_source.body
api_source.timeout
api_source.retry
continue_on_error
repeat_index
repeat_index.type
repeat_index.file
repeat_index.mode
single_destination

Default values:

api_source.method       -> route.yaml method or GET
api_source.timeout      -> route.yaml timeout or 30
repeat_index.type   -> task
repeat_index.file   -> unprocessed_files.json
repeat_index.mode   -> append

Path parameters

Repeat params can be used in api_source.path.

api_source:
  api: public_request
  path: "/schools/{{jrk}}"

repeat:
  - params:
      jrk: 17

This creates request path:

/schools/17

Both forms are supported:

{{jrk}}
{{ jrk }}

Authentication

Supported auth methods:

public
none
basic
bearer
token

public and none mean that no authentication is used.

Basic auth:

auth:
  method: basic
  username: "%API_USER%"
  password: "%API_PASSWORD%"

Bearer auth:

auth:
  method: bearer
  token: "%API_TOKEN%"

token is treated as bearer token auth.

Environment variables are written as %ENV_VAR%.

Retry

Retry can be configured in route.yaml or action api_source.

retry: 3

Retry is used for temporary request failures and selected HTTP status codes:

429
500
502
503
504

Full pipeline sample

actions:
  - do: restapi
    api_source:
      api: bearer_request
      path: "/schools/{{jrk}}"
    repeat:
      - params:
          jrk: 17
    repeat_index:
      type: task
      file: unprocessed_files.json
      mode: override

  - do: json
    origin:
      type: task
      file: unprocessed_files.json
      file_is_index: true

    cols:
      - name: jrk
        from: params
        data: jrk
        type: int

      - name: arno_id
        from: data
        data: arno_id
        type: int

      - name: nimi
        from: data
        data: name
        type: text

In this example, restapi calls API once for every repeat param set, creates response files, and writes them to unprocessed_files.json.

Then json reads unprocessed_files.json, imports response files into target table, removes processed response files from index, and deletes the index file when it becomes empty.

Remarks

• Current implementation expects API response to be valid JSON.
• Non-JSON response support is not implemented yet.
• GET and POST methods are supported.
• Public, basic and bearer auth are supported.
• Sessioned auth and response extraction are not implemented yet.
• Use single_destination for one response file.
• Use repeat or repeat_sql for multiple API calls.
• continue_on_error: true allows repeated requests to continue after failed params.