Integrations
Apache Airflow
and
ZenML
Streamline ML Workflows with Apache Airflow Orchestration in ZenML
Apache Airflow
All integrations

Apache Airflow

Streamline ML Workflows with Apache Airflow Orchestration in ZenML
Add to ZenML
Category
Orchestrator
COMPARE
related resources
No items found.

Streamline ML Workflows with Apache Airflow Orchestration in ZenML

Seamlessly integrate the robustness of Apache Airflow with the ML-centric capabilities of ZenML pipelines. This powerful combination simplifies the orchestration of complex machine learning workflows, enabling data scientists and engineers to focus on building high-quality models while leveraging Airflow's proven production-grade features.

Features with ZenML

  • Native execution of ZenML pipelines as Airflow DAGs
  • Simplified management of complex ML workflows
  • Enhanced efficiency and scalability for MLOps pipelines
  • Compatibility with both local and remote Airflow deployments

Main Features

  • Robust workflow orchestration for data pipelines
  • Extensive library of pre-built operators and sensors
  • Intuitive web-based user interface for monitoring and managing workflows
  • Scalable architecture for running workflows on distributed systems
  • Strong focus on extensibility, allowing custom plugins and operators

How to use ZenML with
Apache Airflow

from zenml import step, pipeline
from zenml.integrations.airflow.flavors.airflow_orchestrator_flavor import AirflowOrchestratorSettings

@step
def my_step():
    print("Running in Airflow!")

airflow_settings = AirflowOrchestratorSettings(
    operator="airflow.providers.docker.operators.docker.DockerOperator",
    operator_args={}
)

@pipeline(settings={"orchestrator.airflow": airflow_settings})
def my_airflow_pipeline():
    my_step()

if __name__ == "__main__":
	my_airflow_pipeline()]

This code snippet demonstrates how to configure a ZenML pipeline to run on Apache Airflow. The AirflowOrchestratorSettings allow specifying the Airflow operator (in this case, the DockerOperator) and any additional arguments. Each step of the pipeline will run in a separate Docker container orchestrated by Airflow

Additional Resources
Read the documentation
Read the full ZenML Airflow integration documentation
Learn more about Apache Airflow

Streamline ML Workflows with Apache Airflow Orchestration in ZenML

Seamlessly integrate the robustness of Apache Airflow with the ML-centric capabilities of ZenML pipelines. This powerful combination simplifies the orchestration of complex machine learning workflows, enabling data scientists and engineers to focus on building high-quality models while leveraging Airflow's proven production-grade features.
Apache Airflow

Start Your Free Trial Now

No new paradigms - Bring your own tools and infrastructure
No data leaves your servers, we only track metadata
Free trial included - no strings attached, cancel anytime
Alt text: "Dashboard displaying a list of machine learning models with details on versioning, authors, and tags for insights and predictions."

Connect Your ML Pipelines to a World of Tools

Expand your ML pipelines with Apache Airflow and other 50+ ZenML Integrations
XGBoost
Kubernetes
Discord
BentoML
Elastic Container Registry
Evidently
WhyLabs whylogs
Google Cloud Vertex AI Pipelines
Seldon
Deepchecks
Modal