Configuring secrets with AWS Secrets Manager

This guide assumes you have read Configuring and using secrets, which covers how the platform manages secrets and how to access them in your flows.

This page specifically details storing secrets in AWS Secrets Manager on your own AWS account.

With this approach, your secrets live in AWS Secrets Manager in an account you control, and the platform retrieves them at task execution time. The platform’s access is tag-based: tasks can read any secret tagged for your deployment.

Granting the platform access to your secrets

Admin only

Add secrets in AWS Secrets Manager

Add secrets to AWS Secrets Manager as you normally would. Store each secret as a JSON object of key-value pairs:

{
	"SECRET_KEY1": "secret_value1",
	"SECRET_KEY2": "secret_value2"
}

You can choose any name for the secret. Your flows reference the secret by this name.

Tag the secret for your deployment

In AWS Secrets Manager, add the following tag to each secret you want the platform to access:

Key: outerbounds.com/accessible-by-deployment
Value: Your deployment name

The tag grants your deployment’s tasks read access to the secret. Anaconda provides your deployment name during onboarding. If you do not know it, contact Anaconda support.

Using secrets

Access tagged secrets in your flows with the @secrets decorator, exactly as described in Configuring secrets. The difference is the source string: AWS Secrets Manager secrets are referenced by their secret name directly, without the outerbounds. prefix. For example, to retrieve the two keys stored in a secret named basic-secret-kv:

from metaflow import FlowSpec, step, secrets

class SecretsFlow(FlowSpec):
    @secrets(sources=["basic-secret-kv"])
    @step
    def start(self):
        import os
        assert os.environ.get("SECRET_KEY1") == "secret_value1"
        assert os.environ.get("SECRET_KEY2") == "secret_value2"
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    SecretsFlow()

Using plaintext secrets

In some cases, you might not be able to store a secret as a JSON object. For a secret that contains an arbitrary string, expose it through an environment variable by setting the json option to False:

from metaflow import FlowSpec, step, secrets

class SpecialSecretFlow(FlowSpec):
    @secrets(sources=[{"id": "my-secret-plain", "options": {"json": False}}])
    @step
    def start(self):
        import os
        print(os.environ.get("my_secret_plain"))
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == "__main__":
    SpecialSecretFlow()

For plaintext secrets, the environment variable name is derived from the secret name by replacing special characters with underscores. For example, a secret named my-secret-plain becomes my_secret_plain.

Anaconda Platform

Getting Started

Platform Guides

Configuring secrets with AWS Secrets Manager

Granting the platform access to your secrets

Using secrets

Using plaintext secrets

​Granting the platform access to your secrets

​Using secrets

​Using plaintext secrets

Granting the platform access to your secrets

Using secrets

Using plaintext secrets