This document provides an overview of the Directed Acyclic Graph (DAG) used in this project for managing workflows in AWS.
A DAG is a collection of tasks organized in a way that defines their execution order based on dependencies. This ensures that tasks are executed in a sequence that respects their dependencies.
- Nodes: Represent individual tasks in the workflow.
- Edges: Define the dependencies between tasks.
- Root Node: The starting point of the DAG.
- Leaf Node: The final task in the sequence.
- Parallel Execution: Tasks without dependencies can run concurrently.
- Failure Handling: Easy to identify and handle failures in specific tasks.
- Scalability: Suits complex workflows with many interdependent tasks.
- Define tasks in the
tasks.pyfile. - Set up dependencies in the
dag.pyfile. - Execute the DAG using the command:
python run_dag.py.
For more details, refer to the AWS DAG documentation.