The Skill Explained
What Is Airflow or Dbt?
Apache Airflow and dbt are two widely used data engineering tools. Airflow schedules and orchestrates data workflows, running each step in the right order. dbt (data build tool) lets teams write data transformations as tested, versioned SQL inside a data warehouse.
Why AI Employers Ask for It
Modern data teams build their pipelines with tools like these, so job ads name them directly. Knowing them means you can join a data platform quickly: schedule pipelines that feed models and dashboards, and transform data in a way others can trust and maintain.
How It Works
In Airflow, a workflow is a DAG (directed acyclic graph) defined in Python: a set of tasks with dependencies, such as "extract, then transform, then train". The scheduler runs DAGs on a timetable or when triggered, retries failed tasks and shows every run in a web interface.
dbt works inside the warehouse. Each model is a SQL SELECT file; dbt works out the order from the references between models, builds tables or views, and runs tests such as "this column has no nulls" or "this id is unique". It generates documentation and a lineage graph showing how data flows.
The two are often used together: Airflow (or another orchestrator) triggers dbt runs after new data has been loaded, then starts downstream jobs such as model training.
What You Do With It at Work
- Write Airflow DAGs that run and monitor daily pipelines.
- Model warehouse data in dbt with tested, documented SQL.
- Add data tests that stop bad data from reaching dashboards or models.
- Debug failed runs from logs and rerun them safely.
- Review data model changes like code, in Git.
Tools and Terms You Will Meet
- DAG
- A graph of tasks with dependencies and no loops.
- Operator
- An Airflow building block that performs one kind of task.
- dbt model
- A SQL file that defines one table or view.
- dbt test
- A rule a column or table must satisfy.
- Lineage graph
- A diagram of how tables are built from one another.
Industry Example: Pharma
A commercial data team uses Airflow to load sales and market data each night and dbt to build tested tables for forecasting models, so analysts never work from half-loaded data.
A Common Mix-Up
Airflow does not transform data itself; it orchestrates tasks that do. dbt does not move data between systems; it transforms data already in the warehouse.
How Deep Do You Need to Go?
- Know itExplain what orchestration and transformation tools do.
- Use itWrite Airflow DAGs and dbt models with tests.
- Master itDesign data platforms with these tools, with standards for testing and deployment.