The Solana RPC failure modes no status page shows you¶
Every RPC provider publishes an uptime number in the high nines, and every one of them is honest. The number measures whether the endpoint answered. It does not measure whether the answer was right, and the failures that take your application down are almost all in the second category.
Here are the ones worth instrumenting, what each looks like from the client, and how to detect it without waiting for a status page to catch up.
1. Slot drift — a correct-looking answer from behind the tip¶
A provider's node falls behind the chain. It keeps serving. Every response is HTTP 200 with well-formed JSON. The data is just from fourteen slots ago.
What it looks like from the client: a balance that reads zero for an account that has funds. An indexer writing an empty block. A bot that misses a fill because the account it read was pre-trade. No errors anywhere.
Why the status page is green: it is. The endpoint is up. Nothing about "responded successfully" is false.
How to detect it: you need a second opinion. A single provider is its own definition of the chain tip — if it says slot 300,000,000 and nothing contradicts it, it is right by construction. Compare each provider's reported slot against the highest slot any provider has reported, and treat the difference as a health signal:
[health]
interval_ms = 1000
slot_drift_threshold = 10 # slots behind tip before freshness score hits 0
w_slot = 0.2 # weight in the composite score
If you run one provider, that comparison has nothing to compare against. Point reference_url at
an external endpoint — probed for its slot, never routed to — so a stall shared by your only
provider still surfaces:
It has to be the matching cluster. A mainnet reference against a devnet provider set reports drift that is not there.
2. Write-path degradation — accepted, never landed¶
sendTransaction returns a signature. That is not a promise the transaction landed; it is an
acknowledgement that the provider took it. The two come apart during congestion, and they come
apart provider by provider.
What it looks like: signatures that never confirm. Retries that pile up. A success rate on your own metrics that looks fine because you are counting the 200, not the confirmation.
Why it is invisible: the read path and the write path can degrade independently, and every
health check anyone runs is a read. A provider can be answering getSlot in 12 ms while its
transaction forwarding is broken.
What to do about it: stop treating writes as something you route. Broadcast them. Solana deduplicates by signature, so sending the same transaction to every provider at once is harmless and the first success wins:
This costs real money — it bills your write volume once per provider — so it is a deliberate trade, not a default. It is off by default for that reason. See routing for the mechanics and trading bots for what the multiplier works out to on a real mix.
A related trap: sendTransaction is a normal billable call on every provider in
the dataset. It is not free on Helius — it costs 1
credit, the same as getAccountInfo. Any write strategy built on "writes are free" is built on
nothing.
3. Partial rollout — healthy and broken at the same address¶
A provider deploys a config change across a pool. Some nodes have it, some do not. Your requests land on both. You get a stream of 503s or malformed responses interleaved with perfectly good answers, from one hostname, for twenty minutes.
Why a status page misses it: aggregate error rate across the fleet stays under whatever threshold triggers an incident. Your slice of it does not.
How to detect it: rolling error rate over a short window, not a consecutive-failure count. A consecutive counter never trips when one request in three fails.
[health]
window_secs = 60
circuit_error_threshold = 0.5 # rolling error rate that opens the circuit
circuit_open_failures = 5 # consecutive failures — the other trigger
circuit_cooldown_secs = 30
Two independent triggers matter here. Consecutive failures catch a hard down; a rolling rate catches a partial one. A breaker with only the first will sit closed through an entire partial rollout.
4. Per-method rate limits you did not know existed¶
Providers publish one headline number and enforce several. Helius's free tier is 10 req/s overall
but 1 sendTransaction/s, 2 DAS/s and 5 getProgramAccounts/s. The Solana Foundation's public
endpoint is roughly 10 req/s overall but 4 req/s for any single method.
What it looks like: 429s at a request rate comfortably under the limit you were told about.
How to handle it: since a rate cap applies per provider entry and only the name has to be unique, register the same URL twice and give the constrained method its own bucket:
[[providers]]
name = "helius-writes"
url = "https://mainnet.helius-rpc.com/?api-key=${HELIUS_API_KEY}"
methods = ["sendTransaction"]
max_rps = 1
[[providers]]
name = "helius-reads"
url = "https://mainnet.helius-rpc.com/?api-key=${HELIUS_API_KEY}"
max_rps = 10
The two entries are independent buckets, so writes cannot eat the read budget. Full detail in configuration.
The common thread¶
Three of the four failures above return HTTP 200. That is the whole problem with uptime as a metric: it is measured at the layer where nothing interesting goes wrong.
What you want instead is a small number of signals taken continuously, per provider, from the same place your traffic goes: reported slot versus the fleet's best, rolling error rate over a window, observed latency, and probe success. RPC Plane combines exactly those four into one score and routes on it — see health scoring for the formula, and observability for the Prometheus metrics to alert on.
None of this requires a proxy. It requires that something in your stack holds more than one provider's answer at the same time and notices when they disagree.