Structuring projects

Packaging code and its dependencies for reproducible execution can be a complex task. Metaflow and Anaconda Platform simplify this process. By following the best practices established on this page, you can ensure your projects are reproducible, ready for production, and structured for rapid iteration during development.

Packaging software

A typical Metaflow project consists of five layers of software:

A diagram showing the five software layers of a Metaflow project and which layers Metaflow packages automatically

A Metaflow flow defined in a Python file.
Custom Python modules and packages that contain the project-specific code.
The Metaflow library itself and any installed Metaflow extensions.
Third-party libraries and frameworks used by the project.
The underlying operating system and hardware drivers, especially CUDA for GPUs.

Metaflow automatically packages layers 1 through 3, the parts that change most frequently during development. Layers 4 and 5, external libraries and low-level concerns such as device drivers, are managed separately. For details, see Managing dependencies.

Structuring a project

To demonstrate a typical project structure, the following example creates a flow that visualizes a fractal using two off-the-shelf libraries, pyfracgen and matplotlib. Follow Python best practices when designing your project: use Python modules and packages to modularize your code into logical components. For instance, it makes sense to create a dedicated module for all the logic related to fractal generation. Save the following in a file named myfractal.py:

def make_fractal():
    import pyfracgen as pf
    from matplotlib import pyplot as plt

    string = "AAAAAABBBBBB"
    xbound = (2.5, 3.4)
    ybound = (3.4, 4.0)
    res = pf.lyapunov(
        string, xbound, ybound, width=4, height=3, dpi=300, ninit=2000, niter=2000
    )
    pf.images.markus_lyapunov_image(res, plt.cm.bone, plt.cm.bone_r, gammas=(8, 1))
    return plt.gcf()

Why separate modules and packages?

Creating a separate module, or a package composed of multiple modules, offers several benefits:

You can develop each module independently. For example, you can test make_fractal in a notebook by adding a cell:
import myfractal myfractal.make_fractal()
You can use standard Python testing tools, such as pytest, to unit test the module.
You can share the module between multiple flows and other systems, encouraging reusability and consistent business logic across projects.

Using a custom module in a flow

Save this flow in fractalflow.py, in the same directory as myfractal.py:

from metaflow import FlowSpec, card, pypi, step, current
from metaflow.cards import Image

class FractalFlow(FlowSpec):
    @step
    def start(self):
        self.next(self.plot)

    @pypi(python="3.11.9", packages={"pyfracgen": "0.0.11", "matplotlib": "3.9.0"})
    @card(type="blank")
    @step
    def plot(self):
        import myfractal
        img = myfractal.make_fractal()
        current.card.append(Image.from_matplotlib(img))
        self.next(self.end)

    @step
    def end(self):
        pass


if __name__ == "__main__":
    FractalFlow()

The highlighted lines show the two things that make this work:

The @pypi decorator declares the environment for the plot step: the Python version and the packages it needs.
The myfractal import works because Metaflow packages the flow file and everything in its directory (and subdirectories) automatically.

For details about this packaging logic, see Structuring projects in the Metaflow documentation.

To see exactly which files Metaflow includes in its code package, run:

python fractalflow.py --environment=pypi package list

To guarantee consistent execution across environments, Metaflow includes the metaflow library itself and all installed extensions.

Including libraries and frameworks

The myfractal.py module only works if it can import the pyfracgen and matplotlib packages. You could install them manually with pip install pyfracgen matplotlib, but that approach has problems:

Nothing in the code records which packages it needs. A colleague, or you on a new laptop, cannot reproduce the project without outside knowledge.
Cloud executions cannot rely on your local installation. Each run needs its own environment.
Production deployments are exposed to upstream changes. A new matplotlib release can break the code at any time.

The @pypi decorator on the plot step addresses all three: the dependencies are declared in the code, built into an isolated environment for every run, and pinned to specific versions. For the full range of dependency management options, see Managing dependencies. To run the flow locally:

python fractalflow.py --environment=pypi run

The first run takes a few minutes while Metaflow builds and caches the environment. When the run completes, open it in the platform’s Runs view to see the fractal rendered on the @card. To run the same flow on cloud compute, no code changes are needed:

python fractalflow.py --environment=pypi run --with kubernetes

The @pypi decorator does not pip install the packages individually at run time. It creates and caches a stable, production-ready execution environment, isolated from any changes in the upstream libraries.

Anaconda Platform

Getting Started

Platform Guides

Structuring projects

Packaging software

Structuring a project

Why separate modules and packages?

Using a custom module in a flow

Including libraries and frameworks

​Packaging software

​Structuring a project

​Why separate modules and packages?

​Using a custom module in a flow

​Including libraries and frameworks

Packaging software

Structuring a project

Why separate modules and packages?

Using a custom module in a flow

Including libraries and frameworks