Health scoring¶

Each provider gets a score in [0.0, 1.0] recomputed every probe cycle. The router selects providers based on these scores.

Score formula¶

score = w_latency * latency_score
      + w_error   * (1 − error_rate)
      + w_slot    * slot_freshness_score
      + w_success * recent_success_rate

Weights are normalised before use — they don't need to sum to 1.

Component	Default weight	What it measures
`latency_score`	0.4	Round-trip time. ~20ms → near 1.0; ~500ms → near 0.0.
`1 − error_rate`	0.3	Fraction of probes that succeeded in the last `window_secs` (default 60s).
`slot_freshness_score`	0.2	1.0 when at the network tip; `threshold / (threshold + drift)` after that, so 0.5 at `slot_drift_threshold` (default 10). Uses the worst drift across the probed commitments (see below).
`recent_success_rate`	0.1	Fraction of the last N probes that succeeded. Fast-reaction signal.

Background probes¶

Health probe (default: every 1s)¶

Each cycle sends one concurrent getSlot request per probed commitment — probe_commitments, processed and finalized by default. It measures round-trip latency (from the processed call), tracks success/failure, updates the rolling error rate and consecutive-failure counter, records the per-commitment slot heights, and triggers circuit breaker state transitions.

Individual requests are used rather than a JSON-RPC batch so the probe stays on the same connection-pool path as live traffic and doesn't depend on provider-side batch handling. On metered plans the default is 2 getSlot calls/provider/second (~173k/provider/day) — raise interval_ms to reduce it, or probe ["processed"] only.

Every provider is probed on one schedule. Probes fire together every interval_ms, and each slot is compared against the tip of the same round. Loops that each slept interval_ms after their own probe used to drift out of phase by their latency difference, until a provider's slot was compared against another's sampled up to a whole interval later — at 200 ms slots, up to interval_ms / 200 slots of drift that wasn't there (150 at a 30 s interval). A provider added by hot reload joins the schedule at its next round.

Commitment isolation¶

Health is tracked per commitment, not from processed alone. A provider whose finalization pipeline stalls keeps advancing processed while finalized freezes — a single processed probe would see a perfectly fresh node while every finalized (or confirmed) read served increasingly stale state.

Drift is measured against a per-commitment network tip (the max slot across providers at that commitment). This matters because finalized trails processed (about one slot since Alpenglow, ~32 slots under TowerBFT before it); comparing it to the processed tip would flag every provider.
The slot component of the score uses the worst drift across the probed commitments, so a stalled finalized view demotes the provider even when processed looks fresh.
confirmed isn't probed by default. Since Alpenglow, Agave sets confirmed and finalized from the same bank, and Solana's upgrade guide marks confirmed for removal in a later release, so a finalized probe measures both. Add "confirmed" to probe_commitments if you want it read separately.

Per-commitment slot and drift are surfaced on the /health endpoint (commitments object) and as the rpc_plane_provider_slot_height_commitment / rpc_plane_slot_drift_commitment metrics.

External slot reference¶

The network tip is the max slot across your configured providers — which has a blind spot: if every provider stalls together (a shared upstream incident, a cluster-wide slowdown), the tip slides down with them and drift stays at 0, so the stall is invisible. A single-provider or own-node deploy has the same gap by definition: the one node is the tip, so it can never show drift against itself.

The optional [health] reference_url closes that gap. Set it to an external endpoint (e.g. a public cluster RPC) and RPC Plane probes it for its slot on the same cadence as provider health probes, then folds that slot into the network tip. Now an all-providers-stale stall shows up as every provider drifting behind the reference, and a lone node finally has a checkpoint to measure against.

[health]
reference_url = "https://api.mainnet-beta.solana.com"

Probe-only, never routed. The reference is never added to the routing pool — no client request is ever sent to it. It contributes a slot number and nothing else.
Default-off. Unset by default, so the proxy keeps its promise of zero outbound connections beyond your configured providers. Enabling it is an explicit opt-in to one extra getSlot probe stream.
Safe when it lags or fails. The fold is a per-commitment max, so a reference that falls behind (or stops answering) simply stops pinning the tip — it can never raise a provider's drift falsely. The only misconfiguration that reports false drift is pointing it at the wrong cluster: use a devnet/testnet reference for a devnet/testnet provider set, mainnet for mainnet.
Observability. The reference slot appears on /health as the top-level reference object, and its probe outcomes are counted under rpc_plane_probe_requests_total{type="reference"}.

Circuit breaker¶

Closed ──(N failures or error_rate > threshold)──▶ Open
   ▲                                                │
   └───── probe succeeds ──── HalfOpen ◄────────────┘
                                  (after cooldown_secs)

Closed — normal routing. All requests are eligible.

Open — provider excluded from routing. Opens when either:

Consecutive probe failures ≥ circuit_open_failures (default 5), or
Rolling error rate ≥ circuit_error_threshold (default 0.5)

HalfOpen — after circuit_cooldown_secs (default 30s), one probe is sent. Success closes the circuit; failure reopens it and resets the cooldown.

Note

When all providers have open circuits, the proxy routes to all of them anyway — degraded behaviour is better than refusing to attempt.

What counts as an error

Transport failures, 5xx responses (except 501), the retryable JSON-RPC errors -32005 and -32603, other JSON-RPC errors not listed below, and auth failures (401/403 — a revoked or wrong key makes the provider genuinely unusable) all count against health and can open the circuit. The full table is in Routing — Retries.

These categories are treated differently:

Rate limits (429) — the request fails over to the next provider, and the throttled provider's score is demoted so best-score/weighted routing sheds its traffic to peers with headroom. But a 429 is a load signal, not a fault, so it is excluded from the circuit breaker (both the consecutive-failure and error-rate triggers). A rate-limited provider stays eligible and reabsorbs traffic the moment the burst passes — opening its circuit would exclude it for the full cooldown and cascade its load onto everyone else, exactly when you can least afford it.
Client errors — a client-attributable 4xx (400, 404, 405, 413, 415, 422), or a JSON-RPC error the request itself caused (bad params, a failed preflight, -32015 for a client still sending maxSupportedTransactionVersion: 0 now that v1 transactions are on mainnet). It is the caller's fault and every provider would say the same, so it is passed through to the client and recorded as client_error without touching provider health at all. This stops one buggy client loop from serially opening every provider's circuit.
A node that is behind (-32016 minContextSlot not reached, -32014 block status not yet available) — the request fails over to a fresher provider and is recorded as behind. Drift is the probes' job, so it doesn't count as a failure.
Method not implemented (HTTP 501, e.g. Firedancer's RPC, which serves only a subset) — fails over and is recorded as unsupported.
History misses — a null or error (other than a client or behind error) from a history = "recent" provider to a getTransaction or getBlock call, or a history-gap code from any provider, fails over to a full-history provider and is recorded as miss without touching health. The provider is working; it just doesn't keep that far back. Without this, a shallow provider's fast nulls would score it as the healthiest provider for history lookups it can't serve.

Tuning for latency-sensitive workloads¶

For trading bots and latency-critical paths, tighten the probe interval and weight slot freshness heavily:

[health]
interval_ms            = 500    # probe every 500ms
window_secs            = 30     # react faster to errors
slot_drift_threshold   = 3      # aggressively deprioritise stale nodes (0.6 s at 200 ms slots)
circuit_open_failures  = 3      # fail fast
circuit_error_threshold = 0.3   # open at 30% error rate
circuit_cooldown_secs  = 10     # recover quickly

w_latency = 0.3
w_error   = 0.2
w_slot    = 0.4   # slot freshness is critical for transaction landing
w_success = 0.1

See the trading-bot example in the repo.

Checking scores live¶

See Observability — Live status for rpc-plane status output and the /health JSON endpoint.