Access parent directories from a flow

To import a shared package from a parent directory in a Metaflow flow, add a symlink in the directory containing your flow script, pointing at the shared package. Metaflow dereferences symlinks when packaging your code (supported since Metaflow 2.5.2), so the package is included whether the flow runs locally or remotely.

flow_type_1

flow1.py

common_utils

flow_type_2

flow2.py

common_utils

__init__.py

some_module.py

In the tree above, flow_type_1/common_utils and flow_type_2/common_utils are symlinks pointing to the shared common_utils package.

Define the shared package

Define the functionality you want to reuse in a Python package:

common_utils/some_module.py

import os

def general_function():
    return os.getcwd()

Export it from the package’s initialization module:

common_utils/__init__.py

from .some_module import general_function

Create the symlink

Add a symlink from the flow’s directory to the shared package:

ln -s ../common_utils flow_type_1/common_utils

Import the package in a flow

flow_type_1/flow1.py

from metaflow import FlowSpec, step

class Flow1(FlowSpec):

    @step
    def start(self):
        from common_utils import general_function
        self.result = general_function()
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    Flow1()

Run the flow

python flow_type_1/flow1.py run

The run completes successfully, with the shared package included:

Workflow starting (run-id 1666836481231665):
[1666836481231665/start/1 (pid 81716)] Task is starting.
[1666836481231665/start/1 (pid 81716)] Task finished successfully.
[1666836481231665/end/2 (pid 81719)] Task is starting.
[1666836481231665/end/2 (pid 81719)] Task finished successfully.
Done!

Anaconda Platform

Getting Started

Platform Guides

Access parent directories from a flow