For nearly a decade, the AI industry operated on a simple assumption. Bigger is always better. Scale the model, and its intelligence follows. They had their own reasons, though. In 2020, OpenAI’s GPT-3, with over 175 billion parameters, demonstrated that increasing size, data, and compute produced astonishing gains. The logic held the test of time, so the industry followed suit. By the time GPT-4 entered the mainstream, developers had already reached an extreme level of financial success. According to estimates cited in the 2024 AI Index by Stanford Institute of Human-Centred AI, training GPT-4 costs roughly $78 million, while Gemini Ultra costs a whopping $191 million.Â
Retrieval Changing the Nature of Intelligence
This is where Retrieval-Augmented Generation flips the script. It’s the same lesson generative AI’s actual value turned out to hinge on application, not sheer size the model stopped being a warehouse for facts and started being a reasoning engine instead. It was no longer responsible for storing knowledge. It became responsible for reasoning over the provided context. That shift reduces the complexity of the task. And when the task becomes simpler, the need for massive scale disappears. This is the turning point. Smaller models are not succeeding because they are cheaper. They are succeeding because the problem has been decomposed
Smaller Models Changing the Game Forever
This is not a fickle claim based on theory, but a claim backed by data from industry leaders. Models in the 7B to 13B range, including those released by Meta, are demonstrating strong performance in domain-specific tasks when paired with retrieval systems and fine-tuning. In constrained environments, they can match or often outperform significantly larger models. At the same time, efficiency gains are unparalleled. And this is something not a lot of people emphasise. The cost of generating outputs. No matter how big an organisation is, someone, somewhere, is paying the price.Â
However, now, the cost of generating an output at the GPT-3.5 level dropped from $20 per million tokens in 2022 to $0.07 by late 2024, a 280x reduction. Latency has also improved by a mile. A response that used to take around 2 or 3 seconds now works instantaneously.Â
Real Systems Don’t Use “Big Models”, They Use Smart, Concise Models
The shift becomes even clearer when you look at how AI is deployed in production. In 2023, Morgan Stanley rolled out an AI assistant for its 16,000+ financial advisors, built around a retrieval system indexing more than 100,000 internal documents. The model is not treated as a knowledge base. It is used to interpret and generate responses based on retrieved context from those 100,000 internal documents. This makes the responses extremely fast. Concurrently, it makes responses more accurate.
Frequently Asked Questions
How does Retrieval-Augmented Generation actually make models cheaper to run?
By shifting the model’s job from “memorize everything” to “reason over what’s given.” That’s a simpler task, which means it doesn’t need nearly as much scale to perform well.
Is Morgan Stanley’s AI assistant unusual, or is this becoming standard practice?
It’s becoming the norm. More companies are building retrieval-based systems on smaller or mid-sized models rather than defaulting to the biggest model available.
Will inference costs keep dropping this fast?
Hard to say for certain, but the trend so far a 280x drop in about two years suggests efficiency gains are accelerating, not slowing down.
