Skypilot VM

Simplify the process of running machine learning pipelines on the cloud by integrating SkyPilot VM orchestration with ZenML. This integration empowers you to provision and manage virtual machines (VMs) on AWS, GCP, Azure, or Lambda Labs, offering cost savings, high GPU availability, and managed execution for your ML workloads.

Features with ZenML

• Seamless provisioning and scaling of VMs for ZenML pipelines
• Automatic selection of cost-optimized VM/zone/region/cloud configurations
• Fine-grained resource allocation for each pipeline step
• Autostop feature to clean up idle clusters and prevent unnecessary costs
• Compatibility with ZenML's Dashboard for pipeline and artifact monitoring

Main Features

• Support for AWS, GCP, Azure, and Lambda Labs cloud platforms
• Automatic provisioning of spot and on-demand VMs
• Built-in cost optimization for selecting VM/zone/region/cloud
• Configurable VM types and resources for each workload
• Autostop functionality to terminate idle clusters

How to use ZenML with Skypilot VM

 # Install the necessary requirements locally
 # For AWS
  pip install "zenml[connectors-aws]"
  zenml integration install aws skypilot_aws 

  # for GCP
  pip install "zenml[connectors-gcp]"
  zenml integration install gcp skypilot_gcp # for GCP

  # for Azure
  pip install "zenml[connectors-azure]"
  zenml integration install azure skypilot_azure # for Azure
  
  # for Azure
  pip install "zenml[connectors-azure]"
  zenml integration install skypilot_lambda # for Lambda Labs

# easily configure your runtime
skypilot_settings = {
    cpus="2",
    memory="16",
    accelerators="V100:2",
    ...
}


@pipeline(
    settings={
        "orchestrator.vm_aws": skypilot_settings,
        # "orchestrator.vm_gcp": skypilot_settings,
        # "orchestrator.vm_azure": skypilot_settings,
        # "orchestrator.vm_lambda": skypilot_settings
    }
)

Additional Resources

• SkyPilot blog