Setting the Priority of Pagerduty Incidents with Service Event Rules in Terraform

My last outstanding PagerDuty Terraform question I wanted to answer, is on using pagerduty_service_event_rule to set the Priority of an incident.

As I expected from doing some work previously on global event rules for routing alerts to services, it’s really easy and luckily also possible on the free plan (as my trial expired a few weeks ago now). The first step is manual to enable Incident Priorities under Account Settings in the WebUI (needs admin permissions), as there is only an API list and not to enable / update priorities.

So now we have enabled Incident Priorities and chosen our names (I’ve used the defaults), it’s possible to start working on the Terraform. To create a service event rule which sets a priority, first we’ll need to add some data sources for mapping the priory names to the PagerDuty ID, so the can reference them in the event rules:

data "pagerduty_priority" "serv_1" {
  name = "SERV-1"
}

data "pagerduty_priority" "serv_2" {
  name = "SERV-2"
}

data "pagerduty_priority" "serv_3" {
  name = "SERV-3"
}

So if we wanted to say every critical severity incident the service received was a SERV-1 it would be as simple as creating the following resource:

resource "pagerduty_service_event_rule" "blog_lambdas_critical_is_serv_1" {
  service = pagerduty_service.blog_lambdas.id
  position = 0

  conditions {
    operator = "and"
    subconditions {
      operator = "equals"
      parameter {
        path = "severity"
        value = "critical"
      }
    }
  }
  actions {
    priority {
      value = data.pagerduty_priority.serv_1.id
    }
  }
}

The path under subconditions is from the CEF payload (see events-api-v2 for more details), and the operator support both and and or for the subconditions so you can make more complex conditions so that you classify based on more complex conditions.

Also support for doing a suppress on an matching alert is also possible (BTW the paid plans offer way more complex options):

resource "pagerduty_service_event_rule" "blog_lambdas_critical_is_serv_1" {
  # Stuff removed
  actions {
    suppress {
      value = true
    }
  }
}

If you’re setting these event rules on every service, I would suggest that you create a module so reduce the copy-pasta. This would also allow you to create a nice simple way to add dependencies between techincal / business services.

• Pagerduty
• Terraform