Observability¶

RPC Plane exposes Prometheus metrics and a health endpoint out of the box. No configuration needed.

Prometheus metrics¶

Metrics are served on :9401/metrics (configurable via server.metrics_listen).

curl http://localhost:9401/metrics

Metrics reference¶

Metric	Type	Labels	Description
`rpc_plane_requests_total`	Counter	`method`, `provider`, `status`	Total JSON-RPC calls handled. A batch request increments by its call count, so a 1000-call batch counts as 1000.
`rpc_plane_request_duration_seconds`	Histogram	`method`, `provider`	Request latency. One observation per request (a batch records its single round-trip latency once).
`rpc_plane_provider_health_score`	Gauge	`provider`	Current health score (0.0–1.0)
`rpc_plane_provider_slot_height`	Gauge	`provider`	Last observed `processed` slot height
`rpc_plane_slot_drift`	Gauge	`provider`	Slots behind the network tip (worst of `processed`/`confirmed`)
`rpc_plane_provider_slot_height_commitment`	Gauge	`provider`, `commitment`	Slot height per commitment (`processed`/`confirmed`/`finalized`)
`rpc_plane_slot_drift_commitment`	Gauge	`provider`, `commitment`	Slots behind the per-commitment tip
`rpc_plane_circuit_breaker_state`	Gauge	`provider`	0=closed, 1=half-open, 2=open
`rpc_plane_failover_total`	Counter	`from_provider`, `to_provider`	Failover events
`rpc_plane_rate_limited_total`	Counter	`provider`	Requests shed from a provider because its `max_rps` token bucket was empty. A rising rate means the provider is regularly at its configured cap.

Batch request labels

For a JSON-RPC batch, the method label is normalized to keep cardinality bounded: distinct method names are deduplicated and capped (up to 5, with a +N suffix for the rest). A homogeneous batch collapses to the bare method name (e.g. a batch of 1000 getTransaction is labeled getTransaction), so it groups with single calls — while the counter still reflects all 1000 calls.

The status label

rpc_plane_requests_total uses four status values:

ok — 2xx response with no JSON-RPC error body.
error — a provider-attributable failure: a transport error, a 5xx, a retryable JSON-RPC error, or an auth failure (401/403). These count against the provider's health score and circuit breaker.
rate_limited — an HTTP 429. The request fails over to the next provider, and the throttled provider's score is demoted (so traffic sheds to peers) but its circuit is not opened — a rate limit means the provider is capped, not broken. See health scoring.
client_error — a client-attributable 4xx (400, 404, 405, 413, 415, 422): a malformed body, unknown route, or unsupported payload. The status is passed through to the caller, but it is not counted against provider health — a buggy client loop can't open every provider's circuit.

Only error feeds provider error-rate panels; rate_limited and client_error are tracked separately so a throttle or a caller bug doesn't read as a provider fault. (rpc_plane_probe_requests_total gains a matching rate_limited status when a health probe is itself throttled.)

Transaction (`sendTransaction`) analytics¶

RPC Plane decodes the compute-budget settings of every sendTransaction it forwards — the compute unit limit, the compute unit price, and the resulting requested priority fee — without any configuration. Decoding runs entirely off the request path: it starts only after the client response has already been selected, on a bounded background queue serviced by a single worker, so a slow or oversized transaction can never add latency to the forwarded call. If that queue is ever saturated, the response is still returned normally and the drop is counted (rpc_plane_tx_decode_total{result="queue_dropped"}) rather than blocking.

Metric	Type	Labels	Description
`rpc_plane_tx_submissions_total`	Counter	`provider`, `outcome` (`accepted`/`rejected`)	`sendTransaction` calls by RPC-acceptance outcome. `provider` is the first acknowledging provider (2xx with no JSON-RPC error member); `provider="none"` when every provider rejected the request.
`rpc_plane_tx_decode_total`	Counter	`provider`, `result`	Decode outcome for each submission: `parsed`, `unparsed`, `unsupported` (a transaction version other than legacy or v0), `invalid_budget` (a malformed or duplicated compute-budget instruction), `batch_unsupported` (a JSON-RPC batch containing `sendTransaction` — batch bodies aren't decoded), or `queue_dropped`.
`rpc_plane_tx_compute_unit_limit`	Histogram	`provider`	Submitted (or runtime-defaulted) compute unit limit.
`rpc_plane_tx_compute_unit_price_micro_lamports`	Histogram	`provider`	Submitted compute unit price, in micro-lamports per compute unit.
`rpc_plane_tx_requested_priority_fee_lamports`	Histogram	`provider`	`ceil(compute_unit_limit × compute_unit_price / 1_000_000)` — the total priority fee the transaction requests, in lamports.

RPC-accepted, not landed

These metrics observe RPC acceptance — at least one provider returned a 2xx with no JSON-RPC error member — not whether the transaction later landed on-chain or paid a fee. The histograms above only ever receive a sample for an accepted submission; a rejected submission still increments rpc_plane_tx_submissions_total{outcome="rejected"} and its decode-result counter, but contributes no CU/price/fee observation. simulateTransaction is excluded entirely — it never pays a fee, so it would only add noise.

Prometheus scrape config¶

# prometheus.yml
scrape_configs:
  - job_name: rpc-plane
    static_configs:
      - targets: ["localhost:9401"]

For Prometheus Operator (Kubernetes):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rpc-plane
spec:
  endpoints:
    - port: metrics
      interval: 15s
  selector:
    matchLabels:
      app: rpc-plane

Health endpoint¶

curl http://localhost:9401/health | jq

Returns a JSON snapshot of each provider's current state: health score, slot height, slot drift, circuit state, latency, availability, plus a per-commitment commitments object:

{
  "name": "provider-a",
  "score": 0.912,
  "slot": 341892471,
  "slot_drift": 0,
  "is_drifting": false,
  "circuit": "closed",
  "available": true,
  "rate_limited": false,
  "commitments": {
    "processed": { "slot": 341892471, "drift": 0,  "is_drifting": false },
    "confirmed": { "slot": 341892469, "drift": 2,  "is_drifting": false },
    "finalized": { "slot": 341892439, "drift": 0,  "is_drifting": false }
  }
}

finalized drift is measured against the finalized tip (not processed), so a healthy provider shows ~0 finalized drift despite trailing the processed tip by ~32 slots.

available is false when the provider is out of rotation — either its circuit is open (broken) or it is currently at its max_rps cap. rate_limited distinguishes the second case: true means the provider is healthy but at its configured per-second cap, so new requests are shedding to peers.

When an external slot reference is configured ([health] reference_url), the response also carries a top-level reference object with the reference's last observed slot per commitment — the checkpoint the tips are pinned to. It is null when no reference is configured. Reference probe outcomes are counted under rpc_plane_probe_requests_total{type="reference"}, so you can alert on the reference itself going unreachable.

Live status CLI¶

rpc-plane status
#   NAME          SCORE          SLOT   DRIFT     LATENCY  CIRCUIT
#   ----------  -------  ------------  ------  ----------  -------
#   provider-a    0.912   341892471       0      23.4ms     closed
#   provider-b    0.841   341892469       2      31.1ms     closed
#   provider-c    0.724   341892468       3      38.7ms     closed

The proxy must be running. rpc-plane status reads from the health endpoint.

Grafana dashboard¶

The repository ships grafana/dashboard.json — import it directly into any Grafana instance.

Import steps¶

In Grafana, go to Dashboards → Import.
Click Upload JSON file and select grafana/dashboard.json from the release archive or the repo.
Select your Prometheus datasource.
Click Import.

Dashboard panels¶

Panel	Description
Provider health score	Score over time per provider (multi-line chart)
Slot height	Absolute slot height per provider
Slot drift	Slots behind network tip — gauge and line chart
Circuit breaker state	Current state per provider (table)
Request rate by method	Top 10 methods by request rate
Latency p50/p95/p99	Percentile breakdown over time
Failovers	Failover events as timeline annotations
Error rate	Per-provider error rate
Period totals	Stat cards — today / 7d / 30d (follows dashboard time picker)