Agent: filecopy

Copies one file from configured origin location to configured destination location.

Typical use case:

restapi  -> downloads JSON file
filecopy -> copies JSON file to another task directory
json     -> loads copied JSON file into target table

This is useful when one downloaded origin file is needed by several loading tasks. Each target task can process its own copy independently.

Origin and destination

filecopy uses common file pointers for both origin and destination. See File pointers.

origin defines where the file is copied from.

destination defines where the file is copied to.

Syntax and samples

Copy from current task directory

If dir is missing for task, current task directory is used.

- do: filecopy
  origin:
    type: task
    file: people.json

  destination:
    type: task

In this example, the file is copied as people.json.

Copy between task directories

- do: filecopy
  origin:
    type: task
    dir: test_route.public.origin_task
    file: people.json

  destination:
    type: task
    dir: test_route.public.people_jobs

This copies people.json from one task directory to another and keeps the same file name.

Copy from filesystem directory

- do: filecopy
  origin:
    type: fs
    dir: "/tmp/somepath"
    file: people.json

  destination:
    type: task

This copies people.json from filesystem directory into current task directory.

Rename while copying

If destination.file is defined, the copied file gets that name.

- do: filecopy
  origin:
    type: fs
    dir: "/tmp/somepath"
    file: people.json

  destination:
    type: task
    file: people_jobs.json

This copies people.json and saves it as people_jobs.json.

Copy to filesystem directory

- do: filecopy
  origin:
    type: task
    file: response.json

  destination:
    type: fs
    dir: "/tmp/somepath"

If destination.file is missing, the origin file name response.json is used.

Required keys

Required for origin:

origin
origin.type
origin.file

For origin.type: fs, origin.dir is also required.

For origin.type: task, origin.dir is optional. If missing, current task directory is used.

Required for destination:

destination
destination.type

For destination.type: fs, destination.dir is also required.

For destination.type: task, destination.dir is optional. If missing, current task directory is used.

destination.file is optional. If missing, origin file name is used.

Remarks

• filecopy copies one file.
• Origin file is not deleted.
• File content is not changed.
• The agent does not read or validate JSON, CSV, SQL or other file content.
• The agent does not write anything into the target database.
• Use filecopy when several tasks need the same origin file.
• Use type: fs for shared or external filesystem locations.
• Use type: task for task-specific working files.
• For pipelines, put filecopy before the agent that needs the copied file.