Alert Integrations

We offer several Integrations to route Alerts to various third-party services. Before you can create an Alert, you will need to create an Integration.

To create an Integration, first make sure you are in one of your Environments (Integrations, like Alerts, belong to an individual Environment). Then click “Settings,” and under Environment Settings choose “Integrations” and “Create an integration.”

You can test an Integration at any time by clicking ‘Test Now.’ The Status icon will indicate whether the last attempt to send a notification to that Integration succeeded, whether from a test or a live Alert. A red icon indicates that there is a problem sending messages to the Integration and it needs your attention.

In addition, though not an Alert integration, we integrate with Grafana through our Grafana plugin.

Email

Our email integration allows you to route alerts to one or more email addresses. You can provide a comma-separated list of recipient email addresses.

PagerDuty

Our PagerDuty integration allows you to route alerts to PagerDuty for further escalation and routing. You will need a name for the Integration as well as your PagerDuty integration key. To get your key and create a PagerDuty integration, follow these instructions:

  1. Log in to your PagerDuty account.
  2. Select “Configuration” from the top navigation bar.
  3. Select “Services” from the dropdown.
  4. Click on “+ Add New Service” button in the upper right.
  5. From “Integration Type” select “Use our API directly”.
  6. Configure rest as you see fit. At the bottom of the page click the “Add Service” button.
  7. Copy the “Integration Key” from the table.
  8. Select “PagerDuty” on the Integrations page in DPM, and paste the key into the “Secret Key” field.
  9. Click “Create Integration.”
Microsoft Teams

By integrating DPM with Microsoft Teams, you can send alerts from DPM to a channel that you choose in Microsoft Teams. Here’s how to set it up.

In the Microsoft Teams desktop app:

  1. Click the ellipses button (…) next to the channel where you want DPM alerts to be sent.
  2. Select Connectors.
  3. Find “Incoming Webhook” and click Configure.
  4. Enter a name for the integration and click Create.
  5. Copy the generated URL to your clipboard.
  6. Click Done.

In DPM:

  1. Navigate to Settings > Integrations
  2. Click Create new integration.
  3. Select Microsoft Teams.
  4. In the box next to “MSTEAMS CHANNEL URL”, paste the URL that you previously copied to your clipboard.
  5. Enter a name for the integration.
  6. Click Create integration.
Slack

Our Slack integration allows you to send alerts to a Slack room. You will need a name for the Integration as well as your Slack incoming webhook URL. This is not the room’s URL address in your browser’s address bar if you use the Slack web application. To find the webhook URL and create a Slack integration, follow these instructions:

  1. Log in to your Slack account’s admin page.
  2. Go to Configure Apps, select Custom Integrations, and choose Incoming WebHooks.
  3. Click “Add Configuration.”
  4. Choose a channel from the dropdown and click “Add Incoming Webhooks integration” button.
  5. Copy the “Webhook URL”. The correct URL should look something like https://hooks.slack.com/services/A123B4CDE/F1GH2I345/ABCDEf6gH7IjklMd8nOp9QRS.
  6. Select “Slack” on the Integrations page in DPM, and paste the Slack URL into the WebHook URL field.
  7. Click “Create Integration.”
OpsGenie

Our OpsGenie integration allows you to send DPM alerts to your OpsGenie account so your team can handle them appropriately. You will need a name for the Integration as well as your OpsGenie API Key. To get your key and create an OpsGenie integration, follow these instructions:

  1. Log in to your OpsGenie account.
  2. Go to “Settings,” and then “Integrations”.
  3. Choose “API”.
  4. Name the integration, and get the API key.
  5. Ensure the integration has Read Access as well as Create and Update Access.
  6. Select “OpsGenie” on the Integrations page in DPM, paste the key into the “API Key” field, and name the Integration.
  7. Click “Create Integration.”
VictorOps

Our VictorOps integration allows you to route alerts to VictorOps for further escalation and routing. You will need a name for your Integration, your VictorOps secret key, and your VictorOps routing key. To get your VictorOps keys and create an integration, follow these instructions:

  1. Log in to your VictorOps account.
  2. From the top menu bar select “Settings”.
  3. From the top right menu bar select “Integrations”.
  4. From the right-hand menu select “REST Endpoint”.
  5. From the “Post URL” copy the secret key and the routing key: https://alert.victorops.com/integrations/generic/20131114/alert/{secret_key}/$routing_key
  6. Select “VictorOps” on the Integrations page in DPM, and paste the secret key and routing key into the appropriate fields.
  7. Click “Create Integration.”
Generic Webhook

DPM can send a POST request with a JSON payload of alert information to a destination URL of your choice. The configuration has two mandatory fields, and one optional field:

  • Integration Name: Name of the integration used to differentiate from other integrations.
  • Hook URL: The endpoint where the alert request should be sent. This must begin with http:// or https://.
  • Secret Token: Optional field used to compute a unique X-VividCortex-Signature header to verify request origins. The signature is calculated by hex-encoding the SHA1 hash of the payload and token concatenated together. See example code snippets for examples on calculating the signature to verify message authenticity.

The following attributes are sent as part of the JSON POST request. If you are integrating with a third-party webhook API, you will need to first create a proxy to translate this POST request into a format the third-party API can ingest.

Attribute Description Type Example
alert_id The internal ID of the alert. number 2
alert_name Alert name required when creating the alert string “Test Threshold Alert”
alert_url URL pointing to the alert configuration page. string https://app.vividcortex.com/env-test/settings/alerts?aid=1
environment_name Name of the environment. string “Default”
event_level This is the event severity. string “info”, “warn”, or “crit”.
event_metric Name of the metric that triggered the event. string “os.cpu.user_us”
event_message This is the message related to the event triggered. string “trigger threshold greater than 40.00 for 1s”
event_type Indicates whether a Threshold or Event alert. string “Threshold”
event_uuid Universal unique identifier (UUID) of the event.
128 bits represented in 32 hex alphanumeric characters.		 string “068fc74be914f0eaf7092d33585e0b33”
event_url URL pointing to the event list page filtered by the host that triggered the alert. string https://app.vividcortex.com/env-test/events
host_arch Host architecture. string “amd64”
host_description Host description. string “Linux Lubuntu 16.04 64 bits”
host_name Name of the host. string “dev-lubuntu-1604-64”
host_os The operating system installed on the host. string “linux”
host_type Host type. string Possible options: “os” / “mysql” / “mongo” / “pgsql” / “redis”
integration_id Each integration has an unique internal destination ID. number 2
integration_name Name of the integration, as it was completed on the field Integration Name when creating the alert. string “My Webhook integration”
integration_url URL pointing to the integration configuration page. string https://app.vividcortex.com/env-test/settings/integrations?duid=2
resolved Identifies the threshold resolution event. boolean true to indicate the resolution of a previously triggered alert.
org_name Organization name. string “VividCortex”
org_nick Organization nickname. string “vividcortex”
timestamp The time stamp when the event happened. number 1525540729

DPM displays the status of an Integration on the Integrations page. This webhook integration will update its status to ‘healthy’ when it receives a HTTP 200 response to its POST. The status codes we use to treat an endpoint as ‘unhealthy’ are BadRequest (400), Unauthorized (401), NotFound (404), and UnsupportedMediaType (415).

Here is an example threshold alert:

{
"environment_name":"Default",
"event_metric":"os.cpu.total_us",
"event_type":"Threshold Alert",
"event_url":"https://app.vividcortex.com/Default/events?from=1526303681\u0026filterFrom=1526303856\u0026filterUntil=1526303981\u0026hosts=id=1",
"event_uuid":"d694d8b072e7c895ec11531ed141c088",
"event_level":"warn",
"host_arch":"amd64",
"host_description":"Server lubuntu-1604 version 4.10.0-42-generic",
"host_name":"lubuntu-1604",
"host_os":"linux",
"host_type":"os",
"org_name":"VividCortex",
"org_nick":"vividcortex",
"event_message":"Metric os.cpu.total_us at 280.0K, greater than trigger threshold of 10.0 for 1 second",
"timestamp":1526303881,
"alert_id":2,
"alert_name":"Threshold Alert",
"alert_url":"https://app.vividcortex.com/Default/settings/alerts?aid=2",
"integration_id":2,
"integration_name":"Test Webhook",
"integration_url":"https://app.vividcortex.com/Default/settings/integrations?duid=2"
}

Here is an example event alert:

{
"environment_name":"Default",
"event_type":"Host stopped sending data",
"event_url":"https://app.vividcortex.com/Default/events?from=1526302186\u0026filterFrom=1526302361\u0026filterUntil=1526302486\u0026hosts=id=3",
"event_uuid":"914635964ce520cfe54c7623d409d248","event_level":"warn",
"host_arch":"amd64",
"host_description":"MySQL Community Server (GPL) version 5.6.39 on lubuntu-1604 listening on [::]:3306",
"host_name":"lubuntu-1604",
"host_os":"linux",
"host_type":"mysql",
"org_name":"VividCortex",
"org_nick":"vividcortex",
"event_message":"Host has recovered and is sending data again",
"timestamp":1526302386,
"alert_id":1,
"alert_name":"Ubuntu - All",
"alert_url":"https://app.vividcortex.com/Default/settings/alerts?aid=1",
"integration_id":2,"integration_name":"Test Webhook",
"integration_url":"https://app.vividcortex.com/Default/settings/integrations?duid=2"
}