> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pgw.sipmicro.services/llms.txt
> Use this file to discover all available pages before exploring further.

# Set up alerts for pgw

> Alert on failing pushes, unreachable PBXs, password guessing, and an expiring certificate with Prometheus rules.

A phone that doesn't ring reports nothing, so without alerts you learn of a failure only when a user complains. The following rules alert on the usual causes before a user notices. Each row is a PromQL expression that you can paste into a Prometheus rule. The thresholds suit a small install, so tune them to your traffic.

Start with **Pushes fail**, **A PBX stopped answering**, and **The certificate expires within 14 days**.

## Pushes and calls

| Condition | Expression |
| - | - |
| Pushes fail | `sum by (provider) (rate(pgw_pushes_total{result="error"}[5m])) > 0` |
| Push services report dead tokens | `sum by (provider) (increase(pgw_pushes_total{result="dead"}[15m])) > 10` |
| A push service fails for an app, so pgw holds its pushes back | `max(pgw_push_breakers_open) > 0` |
| Apps are slow to wake | `histogram_quantile(0.95, sum by (le) (rate(pgw_hold_seconds_bucket[10m]))) > 10` |
| pgw refuses calls because it holds too many | `increase(pgw_invites_refused_total{reason="global_cap"}[15m]) > 0` |
| pgw doesn't store push events | `increase(pgw_push_events_dropped_total[15m]) > 0` |

## PBXs and user sources

| Condition | Expression |
| - | - |
| Registrations fail at the PBX | `rate(pgw_registrations_total{result="upstream_error"}[5m]) > 0` |
| A PBX stopped answering | `pgw_upstream_up == 0` |
| A PBX refuses pgw's stored credentials | `increase(pgw_upstream_auth_failures_total[15m]) > 0` |
| A PBX doesn't keep pgw's Path | `increase(pgw_upstream_path_missing_total[10m]) > 0` |
| A PBX that isn't configured sends calls | `increase(pgw_invite_outcomes_total{result="forbidden"}[15m]) > 0` |
| Kamailio's database doesn't answer | `sum by (upstream) (rate(pgw_user_source_lookups_total{result="unavailable"}[5m])) > 0` |
| Kamailio's database is slow | `histogram_quantile(0.95, sum by (upstream, le) (rate(pgw_user_source_lookup_seconds_bucket[10m]))) > 0.5` |
| pgw can't read Kamailio's domains | `increase(pgw_kamailio_domain_load_failures_total[15m]) > 0` |
| pgw can't fetch a token issuer's keys | `increase(pgw_bearer_key_fetch_failures_total[15m]) > 0` |

## Devices and security

| Condition | Expression |
| - | - |
| Devices disappear | `delta(pgw_bindings[15m]) < -0.2 * (pgw_bindings offset 15m)` |
| pgw banned a source address for refused credentials | `increase(pgw_register_bans_total[15m]) > 0` |
| Many addresses guess passwords | `rate(pgw_register_auth_failures_total[10m]) > 1` |
| Too many addresses guess passwords for pgw to ban them all | `increase(pgw_register_auth_failures_dropped_total[15m]) > 0` |

## The gateway

| Condition | Expression |
| - | - |
| The certificate expires within 14 days | `pgw_tls_cert_not_after_timestamp_seconds - time() < 14 * 86400` |
| App connections are near the limit | `sum(pgw_app_connections) > 0.9 * 250000` |
| pgw recovered from an internal error | `increase(pgw_panics_total[15m]) > 0` |
| The SIP message log drops messages | `increase(pgw_sip_log_dropped_total[15m]) > 0` |

In the connections expression, replace `250000` with your `app_max_connections`.

## Write an alerting rule

Put each expression that you want in a Prometheus rule file.

```yaml theme={null}
groups:
  - name: pgw
    rules:
      - alert: PgwPushesFail
        expr: sum by (provider) (rate(pgw_pushes_total{result="error"}[5m])) > 0
        for: 5m
      - alert: PgwCertificateExpires
        expr: pgw_tls_cert_not_after_timestamp_seconds - time() < 14 * 86400
```

## Next steps

> [card] [Metric reference](https://docs.pgw.sipmicro.services/metrics.md#metric-reference)
>
> Look up what each metric and label value means.

> [card] [Read the logs](https://docs.pgw.sipmicro.services/logs.md)
>
> Find which device or upstream set off an alert.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.