/metrics on the admin listener, with no API key.
The admin listener accepts connections only from the pgw host by default, so run Prometheus or a scraping agent on that host.
Start with these four metrics:
| Metric | What it tells you |
|---|---|
pgw_bindings | How many devices pgw can wake. A sudden drop means that apps can’t register. |
pgw_pushes_total | Pushes by provider and result. A growing error count means that a push service refuses or fails your pushes. |
pgw_invite_outcomes_total | How calls end: delivered, timed out, or cancelled. |
pgw_upstream_up | Whether each PBX answers pgw’s keepalive. |
Scrape the metrics
1
Check the endpoint
curl -s http://127.0.0.1:8080/metrics | grep ^pgw_
pgw_app_connections{transport="tls"} 0
pgw_app_connections{transport="wss"} 0
pgw_app_connections_refused_total 0
pgw_bindings 0
...
pgw_invite_outcomes_total{result="delivered"} 1
...
pgw_registrations_total{result="ok"} 1
pgw_registrations_total{result="rejected"} 0
pgw_registrations_total{result="upstream_error"} 0
...
pgw_tls_cert_not_after_timestamp_seconds 1.826322758e+09
...
pgw_upstream_up{upstream="hTA160K5dFVJf-42dGDCrA"} 1
...
pgw_pushes_total, appear only after pgw counts their first event.2
Add a scrape job
Add a job for pgw to the Prometheus configuration. If you changed
admin_listen, use its address as the target.scrape_configs:
- job_name: pgw
static_configs:
- targets: ["127.0.0.1:8080"]
Metric reference
Besides the following metrics, pgw exports the Go runtime metrics (go_*), the process metrics (process_*), and go_build_info.
Registrations
| Metric | Type | Labels | Description |
|---|---|---|---|
pgw_registrations_total | Counter | result: ok, rejected, upstream_error | REGISTER requests from apps. upstream_error means that the PBX didn’t answer or answered 503 or 408, that in managed mode it refused pgw’s stored credentials or rejected the registration, or that it dropped pgw’s Path on an upstream with strict_path. A 401 challenge, or another PBX refusal that pgw relays to the app unchanged, isn’t counted. |
pgw_bindings | Gauge | Registered devices. | |
pgw_register_auth_failures_total | Counter | reason: pbx_refused, wrong_password, unknown_user, disabled_user, username_mismatch, ambiguous_credentials, bearer_invalid, bearer_subject_mismatch | Refused credentials that count toward a ban of their source address. pbx_refused means the PBX refused an app’s REGISTER in passthrough mode. The other reasons come from managed mode: a wrong password, a user that the upstream doesn’t have or has turned off, a digest username that isn’t the To user, more than one set of credentials, a Bearer token that fails its checks, or a valid token for another user. The app gets the same answer for each. A refresh over a registered connection isn’t counted. |
pgw_register_bans_total | Counter | Source addresses that pgw banned for too many refused credentials. | |
pgw_register_banned_total | Counter | REGISTER requests that pgw answered with 503 because their source address was banned. | |
pgw_register_auth_failures_dropped_total | Counter | Refused credentials that pgw couldn’t count toward a ban, because it already tracks 65,536 source addresses. |
Calls
| Metric | Type | Labels | Description |
|---|---|---|---|
pgw_invite_outcomes_total | Counter | result: delivered, forbidden, not_found, merged, busy, timeout, cancelled, answered_elsewhere, unavailable, push_failed, error | Calls from the PBX, by how they ended. Each device counts on its own, so a call to a user with three devices counts three times. forbidden means the source address isn’t in any upstream’s invite_source_cidrs. answered_elsewhere means another device of the user answered, or the PBX cancelled the call with Reason: SIP;cause=200. |
pgw_invites_refused_total | Counter | reason: per_binding_cap, global_cap, call_push_limit | Calls that pgw refused because it already held max_held_per_binding calls for the device or max_held_invites calls in total, or because the device reached call_push_rate_per_binding. |
pgw_bucket_depth | Gauge | Calls held now while their apps wake. | |
pgw_hold_seconds | Histogram | outcome: delivered, timeout, cancelled, push_failed, shutdown, push_limited | How long pgw held a call, by how the hold ended. push_limited means the device reached call_push_rate_per_binding, so pgw answered 486 Busy Here without a push. |
pgw_app_requests_total | Counter | method: INVITE, MESSAGE, SUBSCRIBE, OPTIONS, REFER, PUBLISH, INFO, NOTIFY, BYE, UPDATE, PRACK, or other; result: relayed, forbidden, rate_limited, error | Requests that apps sent toward their PBX, such as calls, messages, and requests inside a call. ACK isn’t counted, nor a request refused with 483 or 420 for its Max-Forwards or Proxy-Require. forbidden means the connection had no registration, a request inside a call came on another connection than the registration’s, or the Request-URI carried pn-* parameters. |
Pushes
| Metric | Type | Labels | Description |
|---|---|---|---|
pgw_pushes_total | Counter | provider: apns, fcm, webpush; type: call, missed, refresh, test; result: ok, dead, error, skipped, backoff, limited | Push attempts. See Push results. |
pgw_push_latency_seconds | Histogram | provider | Time until the push service answered a push. |
pgw_push_queue_depth | Gauge | provider; lane: call, refresh | Pushes waiting for a free worker. The call lane holds call, missed-call, and test pushes. Each lane holds up to push_queue_size pushes. A call lane that stays near push_queue_size means the push service is slow or push_workers is too low. |
pgw_push_queue_skipped_total | Counter | provider; lane | Queued pushes that pgw dropped because the caller had given up or the call had already ended. |
pgw_push_breakers_open | Gauge | provider: apns, fcm, webpush | Apps whose push service failed 5 times in a row, so pgw holds their pushes back. For webpush, each pair of a web app and a browser push service counts once. |
pgw_push_event_queue_depth | Gauge | Push events waiting to be stored. A value that stays high means the database is slow. | |
pgw_push_events_dropped_total | Counter | reason: queue_full, store_error, shutdown | Push events that pgw didn’t store. The pushes themselves were sent. queue_full means the database fell behind, store_error that a write failed, and shutdown that pgw stopped before it could store them. |
Push results
| Result | Meaning |
|---|---|
ok | The push service accepted the push. |
dead | The push service reported the push token as invalid. |
error | The push failed. This includes a call push that pgw couldn’t queue because the call lane was full. |
skipped | pgw didn’t send the push: a non-call push to an iPhone that registered no remote token, a missed-call or refresh push to a browser, which gets only call and test pushes, or a call push whose call ended while the push waited in the queue. |
backoff | pgw held the push back because the push service was failing, was rate-limiting the app or the device, or had asked pgw to wait with Retry-After, or because half of push_workers were already waiting on the same browser push service. |
limited | pgw didn’t send a missed-call or test push because the device reached call_push_rate_per_binding. |
Upstreams
| Metric | Type | Labels | Description |
|---|---|---|---|
pgw_upstream_requests_total | Counter | method: REGISTER; class: 1xx through 6xx, or other | Final responses from the PBX to app REGISTER requests that pgw forwarded in passthrough mode. pgw’s own managed mode registrations aren’t counted. |
pgw_upstream_register_seconds | Histogram | Time the PBX took to answer an app’s REGISTER that pgw forwarded in passthrough mode. | |
pgw_upstream_up | Gauge | upstream: the upstream’s ID | 1 when the PBX answered pgw’s recent OPTIONS, 0 after it missed 3 in a row. Any answer counts, even an error status. Only upstreams whose keepalive_seconds isn’t 0 have it. |
pgw_upstream_keepalive_failures_total | Counter | upstream | OPTIONS to the PBX that got no answer within 5 seconds. |
pgw_upstream_auth_failures_total | Counter | upstream | Managed registrations that the PBX refused after pgw sent the stored password. The cause is a changed password, or an AOR with no room for another contact, which some PBXs answer with 403. |
pgw_upstream_path_missing_total | Counter | upstream | Registrations whose answer from the PBX didn’t keep pgw’s Path. Such a PBX sends calls straight to the app, past pgw, so the app gets no call push. |
Managed users and Bearer tokens
| Metric | Type | Labels | Description |
|---|---|---|---|
pgw_user_source_lookups_total | Counter | upstream; result: found, missing, unavailable, cached | Lookups of a managed upstream’s users, in pgw’s users or in Kamailio’s database. missing means the user doesn’t exist or is turned off. unavailable means pgw couldn’t read the user, so the app got 500 with Retry-After: 5. cached counts Kamailio subscribers that pgw read in the last 60 seconds and didn’t query again. |
pgw_user_source_lookup_seconds | Histogram | upstream | Time Kamailio’s database took to answer a subscriber query. A query gives up after 2 seconds. |
pgw_kamailio_domains | Gauge | upstream | Domains in the last list that pgw read from Kamailio’s domain table. |
pgw_kamailio_domain_load_failures_total | Counter | upstream | Failed reads of Kamailio’s domain table. pgw keeps the last list that it read. |
pgw_bearer_verifications_total | Counter | upstream; result: valid, invalid, subject_mismatch, keys_unavailable | Bearer tokens on app REGISTER requests. keys_unavailable means pgw couldn’t fetch the issuer’s keys, so the app got 500 with Retry-After: 5 and nothing counted toward a ban. |
pgw_bearer_key_fetch_failures_total | Counter | upstream | Failed fetches of the token issuer’s keys from bearer.jwks_url. pgw keeps the keys that it last fetched, and after a failure fetches again at most once a minute. |
Connections and internals
| Metric | Type | Labels | Description |
|---|---|---|---|
pgw_app_connections | Gauge | transport: tls, wss | App connections open now. |
pgw_app_connections_refused_total | Counter | App connections that pgw closed at once because app_max_connections connections were open. | |
pgw_tls_cert_not_after_timestamp_seconds | Gauge | When the app listener’s certificate expires, in Unix seconds. | |
pgw_sip_log_dropped_total | Counter | SIP messages that the SIP message log left out because writing fell behind, for example when a log collector reads standard output slowly. pgw never delays SIP handling to wait for the log. | |
pgw_panics_total | Counter | Internal errors that pgw recovered from. The affected request or push failed, with 500 when a request still expected an answer, and pgw kept running. |
Next steps
Set up alerts
Copy PromQL expressions that alert on these metrics.
Read the logs
Find the log line behind a counter that moved.