Assessing AI, when you cannot see inside it
In one product we measured, the data advantage was spent within five examples.
The number of ways to build an AI product has outrun the market's ability to evaluate them
A single capability, contract review, can now be built in more distinct ways than a diligence team could test in a quarter. The model underneath can be any of a dozen frontier systems. Each can be aimed through prompting, retrieval, fine-tuning, or tool use. Each sits inside a product that wraps it in workflow. The combinations run into the thousands, and almost none carry an independent measurement of what they do.
This is the practical problem behind most AI assessment today. A buyer, an investor, or an operator is not choosing between two clear options. They are looking at one point in a large and shifting space, described by the party with the strongest incentive to describe it well.
Independent benchmarks exist, but they measure models in isolation, not the product a buyer actually pays for. The distance between the two is where advantage is won and lost.
Two forces set an AI product's value, and only one of them can be read from the outside
Every reported measure of an AI business blends two different things. One is the market it sells into: demand, competition, switching costs, regulation. The other is the product itself: what it can actually do, and how hard that is to reproduce.
The market side is well served. Customer conversations, channel checks, and comparable analysis read it reliably, and the methods are mature.
The product side is the harder one, and it has become harder still. What a model can and cannot do is now changing on a timescale of months. The people closest to the product, its own users, cannot see this clearly, because the capability sits below the surface they interact with.
So the intrinsic strength of the product, the part most exposed to AI, is also the part least visible to a conventional read. One half of the assessment has no reliable instrument.
Any software product is a combination of four components
Reading the product side becomes tractable once the product is broken into parts. Whatever the application, the same four components are present, and each can be measured on its own.
This frame carries everything that follows. It locates, for a specific product, where each part contributes and how reproducible that part is. The advantage can sit in any one of these four components, or in several at once.
Units. The functional blocks that do the work (a model, a classifier, a solver). A model is now the baseline unit, available to rent.
Data and knowledge. Everything the product carries: raw data, structured records, encoded know-how (templates, labelled sets, domain rules).
Connections. How the parts are wired together and to outside systems (retrieval, integrations, APIs, workflow steps).
Logic. How the model is aimed and orchestrated (prompts, decomposition, tool routing, control logic).
To understand a product, isolate each component, test it, and read it on its own
A compound, a product's overall performance, tells you nothing about its individual parts. To read one component, hold everything else fixed, move only that one component, and observe the change in outcome.
This is the standard method of experimental science. It is the one thing that assessment based only on the outward result can never do, because from outside you observe only the combined effect.
The moment a component can be isolated and varied independently, its contribution stops being a matter of opinion and becomes a measured fact.
Four components, four dimensions, one principle
Each component has its own way of being tested, but the method is the same: isolate it, vary it, read the result against independent ground truth. What is measured for each is set out below.
Units: how much the unit adds over the best available rented baseline, and how quickly that baseline is catching up.
Data and knowledge: the marginal value of each additional increment of data, typically steep at first, then bending, then flat.
Connections: how dependent the outcome is on the wiring, and how easily that wiring could be reconstructed by a competitor.
Logic: how sensitive the outcome is to how the model is aimed, holding the model and the data fixed.
The first component, the unit: the model has become the baseline, commoditising the price of basic capability
A base model is now the default unit of capability. It is the basic competence a product gets out of the box, before any wiring, data, or tuning. For most tasks, that baseline is already strong.
Two things followed. The quality of the baseline unit rose sharply, and the price for equivalent quality fell by roughly ten times a year. A capability that cost about sixty dollars per million tokens in 2021 costs a few cents today.
When the baseline unit is rentable by anyone at a collapsing price, owning a generic unit stops being an advantage. A unit regains value only when it stops being the generic baseline.
The companies repriced hardest during the SaaS repricing were built closest to the baseline unit
Figma's strong unit was a design-surface capability, a browser-native multiplayer canvas for interface design. As the baseline model became able to generate and manipulate design surfaces directly, that capability lost proximity to the frontier.
Atlassian's strong unit was a workflow capability, issue tracking and knowledge structuring for engineering teams. As the baseline model absorbed the drafting and reasoning steps in those workflows, the standalone workflow layer sat closer to the baseline than to a defensible frontier.
Adobe's strong unit was a file-format and editing capability, mastery of image, vector, and document formats and the tools to edit them. Baseline generative models began producing and editing those artefacts directly, compressing the value of the editing tool. Adobe's forward multiple fell from roughly thirty times to twelve.
These were well-engineered companies with strong products and real customers. What they shared was proximity to the baseline unit, and that is what drove the repricing. A product whose advantage sits in data or connections is a separate case.
A unit earns value back when it stops being generic
The collapse in the baseline price does not make units worthless. It moves the value to units that a competitor cannot simply rent.
A unit regains durable value in three main ways. Each turns the unit from a shared commodity into something specific to the product, and each is measurable.
Holds unique data in itself. A unit trained on data no one else has stops being generic. The weights carry the advantage, not just the architecture.
Is specialised or fine-tuned. A model tuned for a narrow domain can beat a larger general one on that domain, at lower cost. Small, domain-tuned models are the clearest case.
Runs where the general model cannot. On-device, in a secure environment, or under latency and cost limits a frontier model cannot meet.
The question for any product built on a model is simple: is its unit still the generic one, or has it become one of these? The answer sets how exposed the product is to the next model release.
The second component, data and knowledge: a data advantage is real, but it runs out at a different point for every product
Data and knowledge is the second component. It is widely treated as a strong layer of protection. The real question is the point at which additional data stops adding value.
Compounding case. An email-deliverability product keeps improving as volume grows, because genuine infrastructure and feedback learning turn each new send into signal. Scale keeps compounding.
Flat case. A product without that infrastructure effect flattens early, because the underlying advantage was shallow. More data buys almost nothing past the first climb.
The plateau location is specific to each product and has to be measured, not assumed.
The Caliper Lab builds the independent standard for measuring what AI products actually do.
