How We Made It Easy for Anyone at Our Company to Trigger Alerts
by Calvin Huang on August 20th 2018
Scale relies on Mode Analytics for internal SQL and Python based reporting and analytics, helping power our operations, insights and business decisions. Many of these reports we set to run on a schedule, so that we can always go to a report and see data that’s no more than fifteen minutes to an hour out of date. However, Mode doesn’t natively support a method of notifying people when something needs attention, e.g. when our margins on a project take a sharp dive or many new tasks are suddenly created. We’ve set up a system to utilize these reports to easily alert us when something’s unusual, so that someone can take action.
The flow for alerting goes as follows: when a report is run (either manually or on a schedule), it sends a webhook to an AWS Lambda function, which then pulls the report data from the Mode API, and retrieves a list of alert definitions from our database. It then looks through the alert definitions and sees if any are applicable to the current report being run, and if so, creates an alert for each row in the report that satisfies conditions set out in the matching alert definition. These alerts are then sent to a Slack webhook, which generates an alert in a channel specified in the alert definition.
Alerts are defined in our database, and are easily viewable/creatable from our corp dashboard.
Alerts are simply defined with a handful of fields:
- Report URL: the URL of the Mode report that this alert can fire for.
- Alert Title
- Alert Condition: Python expression that is evaluated with every row of a report to determine if that row is alert-worthy.
- Filter-max Column: name of a column to only take rows where the value of that column is maximal. Useful for reports with time-series data, usually you only want to alert on rows with the most recent data.
- Alert Text Format String: Python format string, used to generate the message text in the Slack alert.
- Color: sidebar color of the alert message
- Alert Channel: Slack channel to post to
The combination of the easy-to-use dashboards, and the simple definitions of the alerts, makes creating alerts based on reports very easy to non-technical people—all that’s required to make fully-featured custom alerts is very basic Python knowledge—essentially basic math operations—and SQL if one is creating their own reports from scratch.