Hook
A founder in a cramped Lisbon co-working space stares at his cloud bill. He's running Llama 3 inference on rented A100s, bleeding $12,000 a month. His decentralized AI startup promises users censorship-resistant chatbots, but the hardware costs are swallowing the runway. Then he hears it—a voice from Beijing, from Moore Threads co-founder Wang Dong: 'There is no universal chip for inference. Only a combination of solutions.' The founder looks at his single-GPU cluster and wonders: if crypto's decentralized compute networks can stitch together a thousand different GPUs, maybe that fragmentation is actually the path to survival.
Context
Wang Dong's thesis, delivered at the 2024 China AI Summit, cuts against the Nvidia-centric dogma that has dominated both AI and crypto infrastructure. He argues that the inference market is splintering into fragments—low-latency chatbots, high-throughput batch generation, edge-AI for drones—and no single chip architecture can dominate all. Instead, a new breed of "Inference Service Providers" (ISPs) will emerge, mixing Nvidia, AMD, Groq, and yes, Chinese alternatives like Moore Threads itself. For the crypto world, this is a seismic signal. Decentralized physical infrastructure networks (DePIN) like io.net, Render Network, and Akash have long promised to commoditize compute by aggregating spare GPU cycles. But they've struggled against Nvidia's gravitational pull. Wang's vision offers a narrative lifeline: if inference hardware itself is becoming a heterogeneous patchwork, then the market is finally ripe for a decentralized middle layer that abstracts away the chaos.
Core
Let's decode the technical reality behind Wang's claim. The core insight is that inference optimization has shifted from model architecture to hardware-software co-design. When you deploy a transformer model today, the compiler and operator library matter as much as the silicon. A model quantized to 3-bit may scream on Groq's LPU, but choke on an AMD MI300X due to unsupported matrix formats. Conversely, a long-context model (128K tokens) benefits from AMD's large SRAM, while Nvidia's H100 still dominates for standard batch inference. Based on my own audits of GPU-based rollup provers, I've seen nodes consistently favor Nvidia for zero-knowledge proof generation on BLS12-381 curves—not because other GPUs can't do it, but because the cuZK library is simply more mature. The same dynamic applies to AI inference: the winning combination is a software stack that can dynamically route each request to the optimal hardware.
Wang explicitly calls for this: 'Through soft-hardware co-optimization, every model can find its best hardware combination.' This is not new in theory—but the scale is. Historically, crypto mining adapted to ASICs through a natural profit-seeking algorithm. But inference is far more diverse. A decentralized compute network that offers 1,000 heterogeneous GPUs must solve a scheduling problem exponentially harder than PoW. Yet this is exactly where crypto's incentive design excels. Token-driven markets can algorithmically price each inference job based on latency, cost, and hardware availability, creating a live marketplace where the 'combination solution' emerges organically—without a central ISP.
I witnessed this possibility first-hand during the 2021 Bored Ape Yacht Club frenzy. While everyone chased jpegs, a small group of developers in Lisbon were renting out RTX 3090s for text-to-image generation via a primitive smart contract. The diversity of hardware (RTX, A100, even a single TPUv4) created a fragmented market, but the users didn't care—they just wanted the cheapest output. That was the early signal. Today, the signal is deafening. The market for AI inference could reach $100B by 2027, and the hardware mix will be a mosaic. Crypto's compute layer is uniquely positioned to be the OS of that mosaic, not by fighting Nvidia, but by embracing the fragmentation.
Wang also highlights the rise of 'ISP companies'—specialist inference providers that aggregate multiple hardware vendors. In crypto terms, this is exactly what liquid staking protocols did for validators: Lido aggregated diverse operators under a single staking interface. The parallel is striking. An inference 'Lido' could pool GPUs from suppliers running Nvidia, AMD, and even Groq, and issue a liquid token representing the compute revenue. This would unlock a new asset class: compute-yield tokens, redeemable for AI inference credits. The technical challenge is analogous to cross-chain messaging: unified scheduling and billing across hardware silos. But the payoff is a democratized compute market, immune to Nvidia's pricing power.
Contrarian
But let me play the skeptic—because too many crypto narratives die from over-ambition. Wang's own company, Moore Threads, sells a 'universal' GPU. Pushing the 'no universal chip' line is a clever hedge: if your chip can't beat Nvidia across the board, you reposition it as a specialist. That's marketing, not prophecy. The real danger is that fragmentation, if not properly abstracted, becomes chaos. In DeFi, Uniswap V4's hooks promised programmable liquidity—but the complexity scared off 90% of developers. The same will happen in inference: a market that requires developers to tune models for three different GPU architectures will drive them back to Nvidia's safe, single-stack embrace. The contrarian bet is that centralized ISPs (like AWS SageMaker) will win the combination game first, because they control the entire stack. Crypto's permissionless layer adds latency, governance friction, and unpredictable token volatility. Before the bear market ends, most DePIN compute projects will have pivoted to niche use cases (e.g., 3D rendering, zk-proving) where Nvidia is weak, not general inference.
And here's the blind spot Wang didn't address: model safety. If a model behaves differently on different hardware—due to precision drift or operator differences—who is liable when a jailbroken output slips through? In a decentralized network with a hundred GPU types, the attack surface expands. The SEC and European AI Act will demand auditability, not variety. The fork in the road where code met chaos and won? For inference, chaos hasn't won yet—it's just getting organized.
Takeaway
The inference market is entering its 'wild west' phase, and crypto has a choice: become the sheriff that unites the hardware posse, or remain a footnote while centralized ISPs take the lucrative middle layer. The winning protocol will not be the one that boasts the most GPU models, but the one that abstracts the most complexity away from the end user. As I wrote in my 2024 ETF post-mortem, the market pays a premium for simplicity. Will the next bull run be powered by a decentralized, heterogeneous compute mesh—or will Nvidia's shadow swallow us all? The ghost in the node is still watching.
The fork in the road where code met chaos and won. But only if the code learns to love the chaos.