Software Engineering

What's New in v0.13: Spark, Custom Code Deployment, Stack Recipes, and More

Dickson Neoh Tze How
Sep 21, 2022
•
7 Mins Read
All posts
ZenML
Contents
Related Posts
ChequeEasy: Banking with Transformers
What's New in v0.5.7
Transforming Vanilla PyTorch Code into Production Ready ML Pipeline - Without Selling Your Soul
This is also a heading
This is a heading

Last updated: November 3, 2022.

This release blog describes the changes for three releases v0.13.0 (major release), v0.13.1 and v0.13.2 (minor releases). v0.13.0 brings the first iteration of our Apache Spark integration. This integration opens up the possibility of running large-scale workloads on single-node machines or clusters. Additionally, this release also allows you to run custom code along with your models using KServe or Seldon. Lastly, we introduce the Stack Recipe as a convenient way to spin up perfectly configured infrastructure with ease. v0.13.1 and v0.13.2 includes several bugfixes and quality of life improvements for ZenML users.

Version 0.13.0 is chock-full of exciting features:

  • Spark Integration - You can now run Spark jobs within ZenML with the long-awaited Spark integration.
  • Custom Code Deployment - It’s now possible to run custom code alongside your model with KServe and Seldon.
  • Stack Recipes - We introduce a convenient way to spin up infrastructures using Stack Recipes and how you can extend them to your needs.

View the full release notes here.

Version 0.13.1 comes with several quality of life improvements:

  • Specify the exact order in which your pipelines steps should be executed, e.g., via step_b.after(step_a)
  • It’s now possible to use TensorBoard with PyTorch and other modeling frameworks.
  • You can now configure the Evidently integration to ignore specific columns in your datasets.

View the full release notes here.

Version 0.13.2 comes with a new local Docker orchestrator and many other improvements and fixes:

  • You can now run your pipelines locally in isolated Docker containers per step. This is useful to test whether the dockerization process will work in remote orchestrators like Kubeflow.
  • @gabrielmbmb updated our MLflow experiment tracker to work with Databricks deployments.
  • Documentation updates for cloud deployments and multi-tenancy Kubeflow support.

View the full release notes here.

As always, we’ve also included various bug fixes and lots of improvements to the documentation and our examples.

⚡ Spark Integration

To date, Spark has been the most requested feature on our Roadmap.

We heard you! And in this release, we present to you the long-awaited Spark integration!

With Spark, this release brings distributed processing into the ZenML toolkit. You can now run heavy data-processing workloads across machines/clusters as part of your MLOps pipeline and leverage all the distributed processing goodies that come with Spark.

We showcased how to use it in our community meetup on 17th August 2022 👇

Run the Spark integration example here.

🎯 Custom Code Deployment

We continue our streak in supporting model deployment in ZenML by introducing a feature that allows you to deploy custom code alongside your models on KServe or Seldon.

With this, you can now ship the model with the pre-processing and post-processing code to run within the deployment environment.

We showcased how to deploy custom code with a model during our community meetup on 24th August 2022

Run the example here.

🥘 Spin Up Infrastructure with Stack Recipes

Spinning up and configuring infrastructure is a difficult part of the MLOps journey and can easily become a barrier to entry.

Worry not! Now you don’t need to get lost in the infrastructure configuration details.

Using our mlops-stacks repository, it is now possible to spin up perfectly-configured infrastructure with the corresponding ZenML stack using the ZenML CLI.

View the demo recorded during our community meetup on 31st August 2022 👇

Check out all the Stack Recipes here.

💔 Breaking Changes

This release introduces a breaking change to the CLI by adjusting the access to the stack component-specific resources for secrets managers and model deployers to be more explicitly linked to the component.

Here is how:


# `zenml secret register ...` becomes 
zenml secrets-manager secret register ...

# `zenml served_models list` becomes 
zenml model-deployer models list


➕ Other Updates, Additions, and Fixes

0.13.0

Model Deployment -

  • Update kserve installation to 0.9 on kserve deployment example by @safoinme in #823.
  • Custom deployment with KServe and Seldon Core by @safoinme in #841.
  • Fix served models logs formatting error by @safoinme in #836.

Spark Integration -

Tekton Orchestrator -

Materializer -

  • Pillow Image materializer by @strickvl in #820.
  • Implement Recursive Built-In Container Materializer by @fa9r in #812.

CLI Improvement -

  • Unify CLI concepts (removing secret, feature and served-models) by @strickvl in #833.
  • Add zenml stack recipe CLI commands by @wjayesh in #807.

Secrets -

README page improvements -

  • Update Readme with latest info from docs page by @dnth in #810.
  • Typo on Readme by @dnth in #821.
  • Put Slack call to action at the top of README page. by @dnth in #846.

Link checker and broken links -

  • Add automated link check github actions by @dnth in #828.
  • Link checker by @dnth in #818.
  • Put link checker as part of CI by @dnth in #838.
  • Fix broken links from link checker results by @dnth in #835.

Misc -

0.13.1

0.13.2

🙌 Community Contributions

👩‍💻 Contribute to ZenML!

Join our Slack to let us know if you have an idea for a feature or something you’d like to contribute to the framework.

We have a new home for our roadmap where you can vote on your favorite upcoming feature or propose new ideas for what the core team should work on. You can vote without needing to log in, so please do let us know what you want us to build!

‍

Start your new ML Project today with ZenML Cloud

Join 1,000s of members already deploying models with ZenML.
Learn more
Request Demo