These startups are in search of the next big AI innovation (Transformers may lose ground)

Author: Will Douglas Heaven | Source: MIT Technology Review

Back in the summer of 2017, AI researchers at Google published a paper called “Attention Is All You Need,” in which they described a new type of neural network called a transformer. It proved to be highly efficient at processing long sequences of data, especially text.

Nine years later, transformers are the engines behind all major large language models (LLMs) available on the market. “The entire AI industry is built on transformers,” says Justin Dangel, co-founder and CEO of the AI startup Subquadratic. “They are one of the most important innovations in the history of computer science and have changed the world.”

But transformers are starting to show their age. Many of the recent advances in LLMs, such as the development of so-called reasoning models and their ability to handle large volumes of input information at once, are not natural extensions of this core technology, but rather workarounds that attempt to bypass some fundamental flaws.

A growing number of scientists and engineers are now wondering what comes next. LLMs are not going away, but the way they are built is up in the air (in the list MIT Technology Review made this year of ten things that matter in AI, these models were dubbed “LLMs+”).

It is against this backdrop that a wave of startups emerges, hoping to push the limits of this booming technology. Some of them will undoubtedly fail, but they have everything to gain and much less to lose than the companies currently leading this race.

The strength of numbers

But first, the problem. The main strength of transformers lies in a mechanism called dense attention, which encodes the meaning of a block of text into a series of numbers. The process involves comparing every word, or part of a word (known as a token), in that text with all other words through a form of multiplication.

Dense attention can capture the meaning of the text with extraordinary precision. But as the size of this text increases, the amount of calculations required to process it grows rapidly. A 10,000-word document might require a transformer to perform 50 million multiplications. This is the main reason why LLMs consume so much energy.

The costs are enormous. OpenAI is expected to spend $50 billion on computing this year, according to the company’s president, Greg Brockman. And the International Energy Agency predicts that the total amount of electricity consumed by data centers will double by 2030.

Furthermore, transformers struggle precisely with what many of the newer models were designed to do. Because of the way they process text word by word, transformers are not very good at keeping track of a large amount of information at the same time. In other words, what is known as the context window cannot get too large. However, for LLMs to perform harder tasks, they will need to absorb larger volumes of data: an entire library of documents, a complete codebase, or, in the case of agents, results produced by other LLMs.

As for reasoning models, they work by writing notes to themselves in a sort of scratchpad known as a chain of thought, and then re-reading those notes, which again increases the amount of data that needs to be tracked.

As LLMs get bigger and better, transformers become a bottleneck. The core strength of the technology is now also a limitation.

Here are four new ideas to solve the transformer problem, innovations that could definitively transform LLMs, making them faster, much more efficient, and perhaps even smarter.

01- Rethinking attention

An obvious way to make LLMs faster and cheaper is to attack the problem directly and change the way attention works. Replacing dense attention with a mechanism called sparse attention, which performs calculations only on some pairs of words in a block of text rather than all of them, could radically reduce the amount of processing LLMs need to perform.

Over the years, researchers have developed various sparse attention mechanisms. The problem is that none of them were as good as dense attention at capturing meaning.

This may have changed. Subquadratic, a Miami-based startup, claims to have invented the first sparse attention mechanism capable of rivaling top conventional LLMs in some tasks, including search and programming. It is an ambitious claim, and some in the industry remain skeptical.

Subquadratic says its model, SubQ, works by dynamically determining, for each piece of incoming text, which words matter and which do not. The company also states that thousands of people have signed up for the waitlist and plans to make the model widely available soon.

Meanwhile, Manifest AI, a startup based in San Francisco, is approaching the problem from another angle. Instead of changing how attention works, it is replacing this mechanism with something else.

The company has developed a mechanism called power retention, which stores only the information most relevant to a given task and ensures that the amount of data an LLM needs to track does not grow out of control.

Attention mechanisms force LLMs to track everything in their context window. A sparse attention model like SubQ discards many individual words but still maintains an approximate view of everything it has encountered. In contrast, power retention works by providing the model with a continuously updated summary of its context window. As new information is added, less relevant information is discarded.

The basic principle of retention has been around for a decade. Manifest AI claims to have updated these techniques to build models that, for the first time, can compete with transformer-based LLMs.

The company says it is possible to adapt a transformer model to turn it into a power retention model with minimal retraining. To demonstrate this, they transformed an existing open-source coding LLM called StarCoder into a version that uses power retention, called PowerCoder. They also released a model called Brumby, which the company says rivals some versions of Alibaba’s popular open-source model Qwen.

Carles Gelada, co-founder of Manifest AI, stated in a video announcing the company’s technology in 2025 that there are many useful applications: from analyzing multi-hour videos to building agents capable of staying focused on a task for weeks.

02- Making models smaller and more flexible

Liquid AI, an MIT spin-off based in Cambridge, Massachusetts, has not modified or completely abandoned transformers, but combines them with its own technology, liquid neural networks, to build what co-founder and CEO Ramin Hasani calls LFMs (liquid foundation models).

Liquid AI’s models are much smaller and consume less energy than most LLMs. The company develops models for automakers, including Mercedes, that run on small chips installed inside vehicles. Its latest models can run on a Raspberry Pi, a low-power computer aimed at enthusiasts that costs $50.

Its models are available for free to any organization with annual revenue of less than $10 million. And they have proven popular: the company has already accumulated nearly 34 million downloads, according to Hasani.

Liquid neural networks were inspired by the brains of worms. They are an extension of another type of neural network that predates transformers, called a convolutional network. The main innovation is a mechanism that allows a model to adapt its behavior to new information, so it can learn as it operates. This is not possible with transformers: once a model is trained, its behavior is fixed.

Liquid AI’s first models were fairly basic but could pilot drones or drive vehicles. With LFMs, the company is trying to scale its technology to compete with conventional LLMs. Its new models achieve performance equivalent to competitors four times their size, including versions of Alibaba’s Qwen and Google’s open-source LLM, Gemma.

A typical LLM is built from a stack of connected transformers. Liquid AI’s recent LFMs are hybrid models composed of 20% transformers and 80% liquid neural networks.

This ratio was found by another AI system developed by Liquid AI, which the company uses to help design all its models. “That is the core technology of our company right now,” says Hasani. This designer AI examines various combinations of neural networks (liquid, convolutional, and others), along with transformers, and creates architectures that connect different types to reach a sweet spot between performance and efficiency.

Hasani believes transformers were just the beginning: “Your brain is an Artificial General Intelligence (AGI) system and runs on 20 watts of power. How is that possible? We can innovate much more.”

03- Generating text all at once

Almost all LLMs produce their output one word at a time. This makes sense because it is how people speak and write. But for computers, it is very inefficient.

It is faster and cheaper for LLMs to generate text all at once, producing entire sentences or paragraphs in a single step. This is the approach taken by Inception, a startup based in Palo Alto, California, which is building LLMs using a technique called diffusion.

Diffusion is best known as the technology powering most image and video generation models. Diffusion models are trained to take a random grid of pixels, similar to the static of an old television, and turn it into an image. They do this by working on all the pixels at once, identifying which ones need to be changed so that the static starts to look more like a high-definition photograph.

It turns out this process also works with text. Inception trained its LLMs to take a random sequence of words and turn it into sentences that make sense. Diffusion LLMs still use transformers to encode meaning, but by producing entire blocks of text at once, they make transformers deliver more with less. “You still use a large transformer model, but you can predict many tokens at the same time,” says Stefano Ermon, co-founder and CEO of Inception. “That is why these models are much faster and more cost-efficient compared to what most other people are building today.”

The challenge was taking a technology developed for image generation and applying it to text. With images, if you need to change a blue pixel to red, you can gradually move through intermediate colors, Ermon explains. That doesn’t work with text: “When you have ‘cat’ and ‘dog’, there isn’t really anything in between.”

Ermon is also a researcher at Stanford University. In 2024, he and two Stanford colleagues figured out the math needed to make diffusion models work with text. They trained a diffusion model that matched the performance of GPT-2, an LLM OpenAI developed in 2019, but was ten times faster. That was enough for Ermon to turn the technology into a company.

Today, he is targeting the industry elite. Inception claims its latest model, Mercury 2, performs on par with some of OpenAI’s GPT-4 models released in 2023, but, again, is ten times faster. “We are very confident in this approach because it is what will scale,” says Ermon.

The only things that matter are speed and cost, he adds: “At the end of the day, the currency will be intelligence per dollar.”

Inception is not the only company betting on diffusion. Google is also experimenting with this approach and has built an LLM prototype called Diffusion Gemma. But Ermon isn’t worried about the competition. “I think it validates our approach,” he says. “This is the future.”

04- Going beyond words

Pathway, another startup based in Palo Alto, is perhaps the most radical of this new group. It wants to free LLMs from the limitations of language.

The company built a type of LLM called Dragon Hatchling, named after the dragons in Terry Pratchett’s novel The Color of Magic, which materialize if you think about them hard enough. Its highest-profile result so far is a high score on a benchmark test that sets LLMs against more than 250,000 extremely difficult sudoku puzzles. Dragon Hatchling solved over 97% of the puzzles. Several of the top LLMs from the biggest labs failed to solve any.

The point Pathway wants to make is that, despite the extraordinary success of LLMs in many different tasks, there are still crucial classes of problems where they fail. Sudoku is just one example. If we want LLMs to find genuine, novel solutions to real problems, we need to go beyond transformers, says Zuzanna Stamirowska, co-founder and CEO of Pathway.

This is because transformers force LLMs to do everything through text. But language is not the best tool for certain types of reasoning. “It is very difficult to represent a sudoku board word by word,” says Stamirowska.

Pathway’s solution is to change the math behind the transformer, replacing the attention mechanism with a mathematical structure called a state space. Instead of encoding information word by word, state spaces compress it into a more abstract representation. Using this technique, Dragon Hatchling can still process and produce text, but it can also mimic forms of reasoning that do not involve sequences of words. This not only makes Pathway’s model more efficient but, in theory, allows it to tackle tasks that other LLMs cannot perform.

Think of chess or math: these types of puzzles are not held in your mind as a long sentence, Stamirowska says: “The eureka moment that pops into your brain is not necessarily in language. We would argue that if you have to reason through language, you are somewhat constrained.”

Stamirowska admits that a conventional LLM could read a book on how to solve sudoku and then write code to do it. But we want to build models with more than book-acquired knowledge, she says: “The hope for AI is not to solve sudoku, it is to cure cancer. There is no book for that.”

“Transformers are an engineering convenience that we stumbled into,” she adds. “This gave rise to a religion, but it is absurd to think that a new breakthrough will not happen.”