ZenML
Google Cloud Storage (GCS)
All integrations

Google Cloud Storage (GCS)

Seamlessly Store your pipeline step outputs with Google Cloud Storage (GCS)

Add to ZenML

Seamlessly Store your pipeline step outputs with Google Cloud Storage (GCS)

Integrate Google Cloud Storage (GCS) with ZenML to leverage a scalable and reliable artifact store for your ML workflows. This integration enables you to store and share pipeline artifacts, making it ideal for collaboration, remote execution, and production-grade MLOps.

Features with ZenML

  • Seamlessly store and retrieve ZenML pipeline artifacts using GCS.
  • Share artifacts across teams and enable remote pipeline execution.
  • Scale storage effortlessly to handle production-grade ML workflows.
  • Ensure secure access to artifacts with GCP authentication methods.
  • Easily integrate GCS with other ZenML stack components.
Google Cloud Storage (GCS) integration screenshot

Main Features

  • Scalable and durable object storage for any amount of data
  • Highly available and performant storage infrastructure
  • Secure access control and encryption for data protection
  • Seamless integration with other GCP services and tools
  • Cost-effective storage with flexible pricing options

How to use ZenML with Google Cloud Storage (GCS)

zenml integration install gcp
zenml stack set ...
from typing_extensions import Annotated

from zenml import pipeline, step
from zenml.client import Client

@step
def my_step(input_dict: dict) -> Annotated[dict, "dict_from_aws_cloud_storage"]:
    output_dict = input_dict.copy()
    output_dict["message"] = "Store this in cloud storage"
    return output_dict

@pipeline
def my_pipeline(input_dict: dict):
    my_step(input_dict)

if __name__ == "__main__":
    input_data = {"key": "value"}
    my_pipeline(input_data)

		# access the remote file from local code
		data = Client().get_artifact_version(name_id_or_prefix="dict_from_aws_cloud_storage").load()
		
    print(
        "The artifact value you saved in the `my_pipeline` run is:\n "
        f"{data}"
    )

Additional Resources

Connect Your ML Pipelines to a World of Tools

Expand your ML pipelines with more than 50 ZenML Integrations

  • Amazon S3
  • Apache Airflow
  • Argilla
  • AutoGen
  • AWS
  • AWS Strands
  • Azure Blob Storage
  • Azure Container Registry
  • AzureML Pipelines
  • BentoML
  • Comet