Setting up notifications for deployments

When you deploy a flow to run on a schedule or in response to events, you can configure the deployment to send a notification when the flow succeeds or fails, so you can stay informed about production runs without watching the UI. Anaconda Platform has built-in support for Slack and PagerDuty notifications. For email, send the notification from a step in your flow.

Slack
PagerDuty
Email

Slack notifications use an incoming webhook: the platform posts a message to a Slack channel of your choice when the flow succeeds or fails.

Create an incoming webhook in Slack

Follow Slack’s incoming webhooks guide to create a webhook for your workspace. At the end of the setup, Slack gives you a webhook URL that looks like this:

Example Slack webhook URL

https://hooks.slack.com/services/T0XXXXXXXXX/B0XXXXXXXXX/qZXXXXXX

Deploy the flow with notifications enabled

Enable notifications either for a single deploy or by default for all deploys:

Per deploy
By default

Deploy the flow with the --notify-on-error and --notify-on-success flags and the webhook URL:

python flow.py argo-workflows create --notify-on-error --notify-on-success --notify-slack-webhook-url <SLACK_WEBHOOK_URL>

To enable notifications by default without passing the flags on every deploy, set the environment variable:

export METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_SLACK_WEBHOOK_URL=<SLACK_WEBHOOK_URL>

Verify the notification

The next time the deployed flow succeeds or fails, a notification appears in the Slack channel:

Slack notifications showing a deployed flow run that failed and then succeeded after a retry

PagerDuty notifications use the Events API V2: the platform creates an event in PagerDuty when the flow succeeds or fails. You can connect the event to your on-call policies.

Create an Events API integration in PagerDuty

Follow PagerDuty’s services and integrations guide to add an Events API V2 integration to your PagerDuty service.

Find the integration key

Open the service in PagerDuty and find the integration key in the Events API V2 section.

Deploy the flow with notifications enabled

Enable notifications either for a single deploy or by default for all deploys:

Per deploy
By default

Deploy the flow with the --notify-on-error and --notify-on-success flags and the integration key:

python flow.py argo-workflows create --notify-on-error --notify-on-success --notify-pager-duty-integration-key <PAGERDUTY_INTEGRATION_KEY>

To enable notifications by default without passing the flags on every deploy, set the environment variable:

export METAFLOW_ARGO_WORKFLOWS_CREATE_NOTIFY_PAGER_DUTY_INTEGRATION_KEY=<PAGERDUTY_INTEGRATION_KEY>

After the flow runs, PagerDuty shows a new event under Incidents when the flow fails, or under Changes when it succeeds.

To get an email alert when your flow finishes, send the email from a step in your flow. The flow formats its results as a Metaflow card—a quick way to visualize flow artifacts as HTML—and emails the card as an attachment. This procedure uses Postmark as the email provider, but any provider with an HTTP API works the same way; replace the send_email function with one that calls your provider.

Get an API token from your email provider

Set up a Postmark server and set its API token as an environment variable:

export PM_SERVER_API_TOKEN=<YOUR_TOKEN>

Add the email helper

Save the following as emailer.py. The flow calls send_email to send the card’s HTML as an attachment:

emailer.py

import os
import requests
import base64

def send_email(from_email, to_email, subject, html):
    headers = {
        "Accept": "application/json",
        "Content-Type": "application/json",
        "X-Postmark-Server-Token": os.environ.get('PM_SERVER_API_TOKEN')
    }
    payload = {
        "From": from_email,
        "To": to_email,
        "Subject": subject,
        "HtmlBody": html,
        "Attachments": [
            {
                "Name": "card.html",
                "Content": base64.b64encode(html.encode('utf-8')).decode('utf-8'),
                "ContentType": "text/html"
            }
        ]
    }
    response = requests.post("https://api.postmarkapp.com/email", json=payload, headers=headers)
    if response.status_code == 200:
        print("Email sent successfully!")
    else:
        print("Failed to send email:", response.json())

Run the flow

Save the following as send_email_from_flow.py. The start step builds a card with a chart image, and the end step emails the card:

send_email_from_flow.py

import os
from metaflow import FlowSpec, step, current, card, Flow
from metaflow.cards import Markdown, Image, get_cards

CAT = 'https://upload.wikimedia.org' + \
      '/wikipedia/commons/b/b9/CyprusShorthair.jpg'

class EmailCardFlow(FlowSpec):

    @card(type='blank')
    @step
    def start(self):
        import requests
        resp = requests.get(CAT, headers = {'user-agent': 'metaflow-example'})
        current.card.append(Markdown("# Meow mail"))
        current.card.append(Image(resp.content))
        self.next(self.end)

    @step
    def end(self):
        from emailer import send_email
        send_email(
            '<FROM_EMAIL>',
            '<TO_EMAIL>',
            f'Card from {current.flow_name}/{current.run_id}',
            get_cards(
                Flow(current.flow_name)[current.run_id]['start'].task
            )[0].get()
        )

if __name__ == '__main__':
    EmailCardFlow()

Run the flow:

python send_email_from_flow.py run

The run prints a confirmation when the email sends:

Workflow starting (run-id 6752)
[6752/start/26219 (pid 97152)] Task is starting.
[6752/start/26219 (pid 97152)] Task finished successfully.
[6752/end/26220 (pid 97421)] Task is starting.
[6752/end/26220 (pid 97421)] Email sent successfully!
[6752/end/26220 (pid 97421)] Task finished successfully.
Done!

Verify the card content

To confirm the email’s attachment matches the card, view the card locally:

python send_email_from_flow.py card view start

Anaconda Platform

Getting Started

Platform Guides

Setting up notifications for deployments