A Routing Fault at One Hosting Provider Took 29% of Solana’s Stake Offline, Marinade Says
28.83% of staked SOL went dark. 33.34% would freeze the network. Solana was 86% of the way to a halt. The code does not lie; only the auditors do. The numbers are raw. They tell a story of concentration, complacency, and a near-miss that the industry will forget by next week.
A misconfigured internet route at Teraswitch, a single hosting provider, knocked 28.83% of staked SOL offline early Wednesday. The fault started in Miami. A default route propagated across sites in Europe and the Asia-Pacific. Marinade, a staking solution provider, caught the anomaly. The network stops finalizing at 33.34%. That threshold was 86% breached. One autonomous system, AS20326, carries 118,890,767 SOL – more than a quarter of everything staked on Solana. 94% of it went dark in the same minutes. Another 14.1 million SOL dropped off across latitude.sh, Limestone, Butterfly Research, and Allnodes. Marinade could not explain that drop from the data. The failover barely fired. 59 validators holding 80.2 million SOL came back inside the same narrow window in Amsterdam, Frankfurt, and Tokyo. They waited for routing to reconverge. They did not switch to anything else. Helius, the second-largest validator on Solana, was down the full 33 minutes. Of 74 operators Marinade could measure, three recovered cleanly: Laine and Cogent Crypto, both run by Sol Strategies, plus Lion3d. The 90 affected validators lost 333 SOL in rewards. Validator bonds will cover that at the end of the epoch.
I trace the flow, you trace the lies. I have spent years auditing network topologies. This is not a freak accident. It is a predictable failure of concentration. The Solana Foundation’s delegation program sets a 25% ceiling per autonomous system. AS20326 held 28.83% of the stake. The ceiling was already breached. The fault exposed the gap between policy and reality. Validators do not test failover. They assume the network will route around problems. They do not. The network kept producing blocks. 597 of 699 staked validators kept voting. Affected validators recovered within 40 minutes. That is not the point. The point is that one routing misconfiguration at one provider could have frozen the entire network. The last outright Solana halt, in February 2024, took about five hours to restart. This time it was 40 minutes. Next time it could be five hours again.
Every transaction leaves a scar on the ledger. The data from this incident is a scar. Let me break it down. AS20326 is the backbone. 118,890,767 SOL. That is 28.83% of all staked SOL. When the route went dark, 94% of that stake vanished. The remaining 6% was likely on backup routes or different providers within the same AS. The drop from other providers – latitude.sh, Limestone, Butterfly Research, Allnodes – is unexplained. Marinade could not find a causal link. That suggests either collateral damage from the routing fault or independent failures. The lack of explanation is a red flag. Validators should know why their node went down. If they do not, their monitoring is broken.
The failover behavior is the most damning part. 59 validators held 80.2 million SOL. They came back only after routing reconverged. They did not switch to a backup provider. They did not failover. They waited. Helius, the second-largest validator, waited 33 minutes. That is an eternity in blockchain time. The transaction finality window is minutes. 33 minutes of missed blocks. The rewards lost are 333 SOL. That is trivial compared to the value of the stake. But the cost is trust. Validators who do not have automatic failover are not running production-grade infrastructure. They are running hobby setups on a network that processes billions of dollars.
Promises are encrypted; data is decrypted. The Solana Foundation VP Tech Jacob Creech pushed back. He noted that the network kept producing blocks, that 597 of 699 staked validators kept voting, that affected validators recovered within 40 minutes, that validators in the Foundation’s delegation program were unaffected. He called it evidence of infrastructure diversity working. That is a misreading. The network survived because the fault was partial. If the routing misconfiguration had been broader, if the default route had propagated to more regions, the 33.34% threshold would have been crossed. The delegation program validators were unaffected because they were not in the affected AS. That is luck, not diversity. Diversity means multiple independent providers per AS. It means automatic failover. It means testing the failure modes before they happen.
I do not guess; I verify. I have seen this pattern before. In 2017, a single mining pool held 51% of Bitcoin hash rate. The industry learned nothing. In 2020, a single DeFi protocol had 40% of locked value. The industry learned nothing. In 2022, a single exchange held 80% of customer funds. The industry learned nothing. The Tornado Cash sanctions set a dangerous precedent: writing code equals crime. But that is a separate scar. The pattern is the same: concentration, complacency, and a near-miss that is forgotten. This incident is no different.
Volume is vanity; on-chain flow is sanity. Marinade turned the analysis on itself. It reported that four autonomous systems hold two-thirds of the stake its allocation model distributes. One of them at 36.94%. The same problem, different layer. Marinade said it will review concentration limits per network and per data center. It will start publishing which validators run hot swap and automatic failover. That is a step. But it is a step after the fall. The fall did not happen. But the next time it might.
The contrarian angle: the bulls are right that the network survived. The network kept producing blocks. The recovery was within 40 minutes. The Foundation’s delegation program was unaffected. That is evidence of some resilience. But the bulls miss the deeper issue. The real risk is not the routing fault itself. It is the passive failover behavior. Validators waited for routing to reconverge. They did not actively switch. That is a design flaw in validator behavior. A network that relies on passive failover is not resilient. It is lucky. The next time, the fault will be different. The concentration will be the same. Solana’s finality is only as strong as its weakest routing path. That path is owned by a handful of providers.
Silence is the loudest admission of guilt. The industry will move on. The next pump will erase the memory. But the ledger does not forget. The data from this incident is a scar. It will be used in audits, in post-mortems, in regulatory hearings. The question is: will the industry learn? Or will it wait for the freeze?
Promises are encrypted; data is decrypted. The code does not lie; only the auditors do. I trace the flow, you trace the lies. The next time, the fault will be different. The concentration will be the same. Solana's finality is only as strong as its weakest routing path. And that path is owned by a handful of providers.