Components
Components are created by application developers and others to be used within an EAGLE workflow. The components themselves exist outside of EAGLE, and may be Application Components such as command line (shell) code, in-line Python and C/C++ dynamic libraries, and MPI code; or they can be Data Components, such as Memory, File, S3 and NGAS. Each of these provide different underlying functionality and integration.
To create a workflow, EAGLE only needs to access JSON representations of each component, which contain enough information to translate the workflow into a graph and then execute it correctly. These JSON files are referred to as component descriptions.
During the execution of a workflow, the executables and data wrapped by a component description have to be available to the execution engine; however, during the development of a Palette or translation into a Physical Graph Template, these don’t need to be accessible by EAGLE.
In this documentation, a distinction isn’t made between the component’s external code and its description unless it is necessary for clarity.
Each component has a set of inputs and outputs, as well as exposed parameters. Executable code called by an Application Component may range from the most simple – for example, just a single mathematical operation in Python or C, to a complete and complex workflow all by itself. The inner workings of the Application Component are not handled within EAGLE.
In combination, components allow the parallel reduction of many individual data sets.
Creating Components for Docker Images
The process for generating component descriptions for applications contained in Docker images is as follows:
Locate the image you wish to use on Docker Hub. For example, the ICRAR images are stored at https://hub.docker.com/u/icrar
Create a new graph and then create one Docker node from the Template Palette
Click the node to modify its attributes:
The “Image” field should contain the name of the image, for example, icrar/leap_cli
The “Tag” field should contain the image tag, for example, 0.8.1.
The “Digest” field should contain the hexadecimal hash of that version.
The “Command”
The “User”
The “Ensure User And Switch”
The “Remove Container”
The “Additional Bindings”
Important Notes on Docker Images
DALiuGE can only execute applications from Docker containers that satisfy the following requirements:
pack a Bash shell (/bin/bash)
pack /usr/bin/cat
pack /etc/passwd
It is also recommended to pack /usr/bin/ls.
Linking Components with Edges
Within EAGLE, an output port from one component may be connected to the input port of another component via an edge. This is illustrated graphically by an arrow linking the two. An edge represents an event triggered by one component that in turn triggers other components to be processed.
It is only possible to link components that meet certain criteria, and some edges are inadvisable as they may affect performance. EAGLE provides error and warning messages when these edges are created.
Environment Variables
DALiuGE and, by extension, EAGLE support globally accessible environment variables in the form of EnvironmentVars
components.
These components act as a globally available key-value store.
Other drops’ parameters can reference parameters specified in this component. The translator and runtime engine handles filling these values in during workflow execution.
Importantly, each EnvironmentVars
component in a graph needs a unique name to avoid variable aliasing.
Reference a store’s variable in another component using the following syntax:
$store_name.var_name
For example, consider a store with the name ‘environment_vars’ and parameter ‘scratch_dir: ‘/users/me/scratch’’.
A second drop could reference this value in the parameter ‘working_dir’ by setting the parameter field to $environment_vars.scratch_dir
Dynamic getting and setting of such variables are currently unsupported; they remain static variables, an editor accessible replacement for commonly used configuration files.