Four Startups Building the Next Generation of Language Models

The Transformer Ceiling

Nine years after the landmark paper that introduced transformers, the architecture underpinning ChatGPT, Claude, and Gemini is showing cracks. OpenAI will spend $50 billion on compute this year, according to its president Greg Brockman. Data center electricity consumption is on track to double by 2030, the International Energy Agency projects. Yet the core technology has barely changed.

The bottleneck is dense attention, the mechanism that lets transformers understand text by comparing every word to every other word through multiplication. A 10,000-word document can require 50 million such operations. That mathematical explosion is why inference costs remain stubbornly high and why context windows - the amount of text a model can hold in memory - stay constrained even as labs throw more silicon at the problem.

Reasoning models and agents make the crunch worse. They generate internal scratch pads, chain-of-thought traces that balloon the data a transformer must track. If the next wave of AI is supposed to digest entire codebases or orchestrate multi-model workflows, the current architecture is in the way.

At DailyTechWire, we've tracked four startups that believe the transformer era is ending. They're not abandoning language models - they're rebuilding the engine underneath.

Sparse Attention Gets a Second Look

Subquadratic, a Miami-based startup, claims it has cracked a problem researchers have chased for years: a sparse attention mechanism that rivals dense attention in quality. Instead of comparing all word pairs, sparse attention runs calculations on a subset. The savings are enormous, but earlier attempts sacrificed too much accuracy.

Subquadratic's model, SubQ, decides on the fly which word pairings matter for a given piece of text. The company says thousands have joined its waitlist and that SubQ matches top-tier LLMs on search and coding tasks. Industry skepticism persists - sparse attention has overpromised before - but if the benchmarks hold, the implications for inference cost are hard to overstate.

Manifest AI, based in San Francisco, takes a different tack. It replaces attention entirely with a mechanism called power retention. Where attention forces a model to track everything in its context window, power retention maintains a rolling summary, discarding less relevant information as new data arrives. The company adapted an open-source coding model, StarCoder, into PowerCoder using the technique, and released Brumby, a model it positions against Alibaba's Qwen.

Cofounder and CTO Carles Gelada sees the use case in long-duration tasks: analyzing hours of video, or agents that run for weeks without losing the thread. The question is whether summarization - by definition, a lossy process - can preserve enough nuance for the complex reasoning tasks labs are chasing.

Hybrid Architectures and Biological Inspiration

Liquid AI, an MIT spinout in Cambridge, Massachusetts, pairs transformers with liquid neural networks, a technology inspired by the nervous systems of nematode worms. The result is what CEO Ramin Hasani calls liquid foundation models, or LFMs.

Liquid neural networks are an evolution of convolutional networks, the architecture that dominated computer vision before transformers. Their defining feature is adaptability: they can adjust behavior in response to new information without retraining. Transformers, once trained, are static. Liquid AI's early models flew drones and drove cars. Now the company is scaling the approach to compete with mainstream LLMs.

Its latest LFMs are 20 percent transformer, 80 percent liquid neural network. That ratio was determined by another AI system Liquid AI built - a designer model that searches combinations of architectures for optimal performance-per-watt. According to Hasani, the hybrid models match rivals four times their size, including versions of Qwen and Google's Gemma.

Liquid AI's models run on the chips inside Mercedes vehicles and on Raspberry Pi boards that cost $50. The company offers them free to organizations with annual revenue below $10 million and has logged nearly 34 million downloads. The pitch is efficiency: the human brain runs on 20 watts, Hasani notes, and transformers are nowhere close to that frontier.

Diffusion Comes for Text

Inception, a Palo Alto startup, is applying diffusion - the technology behind Midjourney and Sora - to language. Diffusion models generate images by iteratively refining a grid of random pixels into coherent scenes. Inception trains its LLMs to do the same with text, starting with a random string of words and sculpting it into grammatical, meaningful output.

The advantage is parallelism. Traditional LLMs produce one token at a time; diffusion models generate entire blocks simultaneously. That makes transformers, which Inception still uses for encoding, work harder per forward pass. CEO Stefano Ermon, who is also a researcher at Stanford, says the company's latest model, Mercury 2, performs on par with some GPT-4 variants but runs 10 times faster.

The technical hurdle was adapting a continuous process - pixel values can transition smoothly from blue to red - to the discrete domain of language, where there's no natural intermediate between "cat" and "dog." Ermon and Stanford colleagues solved the math in 2024, first matching GPT-2 performance at 10x speed. Inception has since raised the bar.

Google is experimenting with the same approach through a prototype called Diffusion Gemma. Ermon views the validation as confirmation. "The currency is going to be intelligence per dollar," he says, and diffusion is the path to scale.

Beyond Words Entirely

Pathway, also in Palo Alto, has built Dragon Hatchling, an LLM that aims to escape the constraints of language altogether. The model's standout result is a 97 percent solve rate on a benchmark of more than 250,000 difficult sudoku puzzles. Leading models from major labs solved zero.

Sudoku is not a language task. It's a logic puzzle that requires spatial reasoning and constraint satisfaction - capabilities that transformers, optimized for sequential text, struggle with. Pathway's thesis is that despite their versatility, LLMs remain brittle outside the domain they were designed for. Dragon Hatchling processes problems in a format closer to how humans think about them: as structured constraints, not sentences.

The approach is early. Pathway has not disclosed the architecture or training regime in detail, and sudoku performance does not guarantee success on real-world tasks. But the result underscores a broader frustration: transformers are phenomenal at language, yet many of the problems we want AI to solve - planning, reasoning over graphs, multi-step optimization - don't map cleanly to text.

The Economics of Disruption

These startups are betting that the labs with the most to lose will be slowest to abandon transformers. OpenAI, Anthropic, and Google have invested tens of billions in infrastructure optimized for the current architecture. Retraining from scratch is expensive; maintaining compatibility with existing tooling and customer expectations is a drag on experimentation.

Newcomers face the opposite incentive structure. They can't outspend incumbents, so they have to outmaneuver them. If sparse attention, diffusion, or hybrid models deliver even 3x better efficiency at comparable quality, enterprises will route inference traffic accordingly. The hyperscalers will follow the workload.

The risk is that none of these approaches scale. Transformers won the last decade because they were simple, parallelizable, and stable to train. Sparse attention has failed before. Diffusion in text is unproven at frontier scale. Liquid neural networks are elegant in theory but complex in practice. Dragon Hatchling's sudoku result, while striking, is a narrow benchmark.

Still, the pressure is real. The International Energy Agency's projection that data center power demand will double by 2030 is not a forecast - it's a constraint. Utilities in Virginia and Singapore are already turning away new capacity requests. If the industry can't make models cheaper to run, the growth story stalls.

What Comes After Attention

Transformers will not vanish. They remain the most reliable way to train large-scale language models, and the installed base of infrastructure, tooling, and expertise is vast. But the next generation of models - what MIT Technology Review calls LLMs+ - will likely be hybrid, mixing transformers with newer techniques that handle long context, parallel generation, or non-linguistic reasoning more gracefully.

The question is which techniques survive contact with production. Sparse attention needs to prove it can maintain quality across diverse tasks. Diffusion must show it can scale to models with hundreds of billions of parameters. Liquid neural networks need to demonstrate they can be trained as reliably as transformers. And approaches like Pathway's need to generalize beyond benchmarks.

The startups racing to answer those questions have narrow windows. If they succeed, they rewrite the economics of AI. If they fail, the transformer incumbents consolidate further, and the next architectural shift waits for another decade. Either way, the assumption that dense attention is the only way to build a language model is no longer holding.