Build by matching, not specifying Turn the tools you trust into reference oracles

There's a kind of software nobody sells you. You want one tool that does two jobs well, and the whole market does one job well and fakes the other. Pick the tool that's great at the first job and the second feels stapled on. Pick the one that's great at the second and you've got the same problem pointed the other way.

The usual move is to write a spec and build the thing yourself. I want to describe a different move, because it changes what a spec even is.

Don't write one. Take the two tools you already trust, the ones that each nail half of what you want, and use them as the answer key. Then tell a coding assistant to build a single system whose output matches those two references and runs at least as fast. Let it grind until it matches, and keep it running until it's done.

The assistant isn't the interesting bit here. The references are.

The expensive part of testing was never running the test. It's deciding what the test should have returned in the first place. That's the part you do by hand, inventing expected outputs, arguing with yourself about edge cases, writing assertions one at a time.

Two tools you already trust just hand you that. One of them already knows the correct output for everything on its side of the problem. The other knows it for the rest. You never write down what "correct" means, because correct is just whatever those two spit out. Each trusted tool is a reference oracle, something you can ask for the right answer without ever defining it, and you've got two of them. They're a spec that runs. The reason written specs go stale is that they drift from the code. These can't drift, because they are the code.

Correctness stops being a judgment call and becomes a number that has to reach zero.

So building becomes a loop, a dumb one in the good way. Throw some work at the references and at your build, compare the results, hand back every place they disagree, repeat. Correctness stops being a judgment call and turns into a number that has to reach zero. That's exactly what an assistant is good at. It'll close a gap it can measure forever. It just can't tell you what "good" means when nobody's defined it, and here nobody has to.

“

They're a spec that runs. Written specs go stale because they drift from the code. These can't drift, because they are the code.

Here's the catch, and since it's the whole reason you started, it's worth sitting with.

You're building the combined tool for the combination. The behavior you want is the stuff that only happens when both jobs run at once, the operations that straddle both halves. None of that is in either answer key, because neither tool does both. The one region you care about most is exactly where the references go quiet.

The region you care about most is the one place the references go quiet - so you manufacture its answers yourself.

Sounds fatal. It isn't, and the workaround is where it gets fun.

You can usually manufacture the missing answers by running both references yourself. Do the first half in tool A, the second half in tool B, then glue the results together in your test harness the way the combined operation is meant to work. Neither tool can do the whole thing alone, but your harness can stitch the two outputs into the answer the combined operation owes you. That stitched result is your ground truth for the seam.

Now look at the speed side. The thing you're trying to beat is two separate tools with glue in between, and that glue is slow. Two round trips, data marshalled out and back twice, a join sitting in the middle in whatever scripting language you wrote the harness in. One tool doing all of it in one place should walk past that. So the speed bar is easiest to clear right where the correctness bar was hardest to define.

For the plain operations it's backwards, which surprised me the first time I thought it through. Anything one tool already does by itself is trivial to verify, because a single reference just answers it. But matching that specialist's speed, on its own home turf, from inside your do-everything tool, is the hard part. The simple stuff is easy to check and a pain to keep up with. The fancy stuff is a pain to check and easy to outrun. Most of the job is keeping track of which of those two you're fighting at any given moment.

The simple stuff is easy to check and a pain to keep up with. The fancy stuff is a pain to check and easy to outrun.

"Run until the outputs match" sneaks in an assumption: that the references are deterministic. Plenty of software isn't. Anything with approximation, randomness, threads landing in a different order, floating point, an unstable sort. Any of those can give you two correct answers to the same question, and your reference is free to pick a different one tomorrow. Demand an exact match against that and you're not testing correctness anymore. You're testing whether you reproduced one run's coin flips, which gets you a worse tool than the one you set out to match.

The reference is still the answer key. You're just grading the way you'd grade two answers that are both allowed to be right.

So for those parts you loosen the check. Results within a tolerance. Overlap above some threshold. The right order except where things tie. The reference is still the answer key. You're just grading against it the way you'd grade two answers that are both allowed to be right. How loose to go is a real decision, and the assistant can't make it for you, because it comes down to how much imprecision the people using your tool will put up with.

This is the part I keep turning over.

It feels like you got away without writing a spec. You didn't. The loop converges on exactly one thing: your build matches the references on the work you happened to feed it. Every input shape your generator never produced is undefined. Your tool can do whatever it likes there and the loop still flashes green, because the loop only ever saw what you showed it.

Which means the spec is the generator. What it covers, how big it goes, how hard it pushes, which nasty edge cases it bothers to make. That's the real definition of your software now. The work of saying what the thing should do didn't go away. It moved out of a document nobody reads and into a test generator nobody's checking, and that's the harder of the two jobs, not the easier one. "Let it run until it's done" was always just "let it match the references on the inputs I was clever enough to dream up."

I'm not telling you to skip it. I'm telling you the effort goes into the generator and barely any into prose, which is a fine trade as long as you know that's the trade you're making. The people selling this as "no spec needed" have buried the spec in the test rig and stopped looking.

Make two tools your ground truth and you inherit them whole, ugly bits included. Their rounding. Whatever they do with empty input. Their off-by-one at the boundary. The strange default when you leave an argument out. Your build reproduces all of it faithfully. Sometimes that's the point, because being a drop-in match is the product. Sometimes you're carefully recreating a bug the original authors would happily kill next week.

Speed hides the same trap. "At least as fast" was never one number. Typical case, worst case. Cold cache, warm cache. First run, settled state. One user, a thousand users. Tiny input, enormous input. Tune against the one number you actually measured and you'll get faster there and quietly slower everywhere you didn't look. The benchmark is a second spec, and it leaks the same way the first one did.

The core trick still holds up. Turning tools you trust into a live answer key takes a vague, mushy build and gives it something sharp to aim at, and an assistant earns its keep against a target like that. Stitching the references together to grade the operations neither one can do alone is the move that makes the whole thing real instead of a neat idea you bring up at lunch.

But the one-liner, build something that matches both and run it till it's done, skips the three things that actually decide it. The operations you care about most have no answer until you build one by hand. Half your system can't be graded by equality at all. And "done" never means more than "matched on the inputs I thought to generate," which leaves the real spec sitting in a generator nobody's watching. Get those right and the assistant chewing through the work overnight is the easy part. Get them wrong and you ship something that passes every test you wrote, then falls over on the first thing a real user types.

Zoom out and this stops being a build trick and turns into a market force.

The thing that made a specialist tool worth paying for was that it was correct and fast at its one job, and that correctness took years to grind out. The grind was the moat. The moment a tool's behavior can be used as a reference oracle, the moat doubles as the blueprint. Anyone can point an assistant at it, match it, set it next to a second tool, and ship something broader for a fraction of what the original cost to build. Correctness stops being a thing you own. It becomes a thing you leak every time someone queries you.

“

Correctness stops being a thing you own. It becomes a thing you leak every time someone queries you.

Two pressures land on whoever got matched. The first is price. Reaching parity with a mature tool is cheap now, and a buyer looking at a fused alternative that does the same job plus the thing next to it has a lot less reason to pay the premium. The second is velocity. The time it takes a competitor to reach parity just fell through the floor, so sitting on a finished, well-understood product for a year stopped being safe. The lead you can hold shrinks to however long your behavior stays un-oracled.

And it isn't only scrappy startups running the play. Incumbents can run it on each other. The vector specialist can absorb the graph specialist by oracling it, the graph specialist can do the reverse, and every pure single-purpose tool is suddenly under pressure to swallow its neighbors before somebody fuses past it. The market drifts toward broader tools and squeezes the narrow ones, unless the narrow one lives in a domain still moving fast enough that there's no stable answer to match yet.

Which is where the defensible value actually went. It was never in being the best implementation of a known thing, because known things have oracles now, and oracles get matched. It's in being the one defining behavior that has no reference yet. The frontier, where nobody can grade your output against an existing tool, is the only ground where a lead survives. Everything behind that line gets cheaper, faster, and better for the people buying it, and a good deal more uncomfortable for the people who used to charge for it.

If the shape of this feels familiar, machine learning has been running it for years. The name over there is distillation. You take a big, capable model, the teacher, and you train a smaller, cheaper one, the student, to reproduce what it says. You never tell the student what's correct. You show it the teacher's answers and push it to produce the same ones. The teacher is a reference oracle. The student is your matched, faster build.

Walk the correspondence and it holds the whole way down.

Walk the correspondence and it holds the whole way down. You generate a pile of inputs, run them through the teacher, and keep its answers as targets, which is just the harness making work and reading the references. You match the teacher's full output instead of only its top answer, because the spread of probabilities carries more signal than the single label, the same reason you grade with tolerances and overlap rather than demanding an exact hit. The student only learns behavior on the inputs you distilled over and goes vague everywhere else, so once again the real spec is hiding in the generator. And it soaks up the teacher's biases and confident wrong answers right next to the useful parts, which is inheriting the references whole, ugly bits and all.

The fusion move is in there too. Distill from one teacher and you get a smaller version of it. Distill from several at once, a handful of specialists, and you get a single student carrying what all of them knew. That's the two databases folded into one tool, except the tool is a network and the seam gets learned instead of stitched together by hand.

And this is the incumbent squeeze in its rawest form, which is why the model labs are living it right now. A frontier model's entire value is its behavior, and the moment it sits behind an API that behavior is queryable. Every call is a look at the oracle. The value walks out with the responses. Cheaper models get trained on those outputs, land close enough for most of what people actually do, and undercut the thing they learned from. The labs know it, which is why a frontier lead now gets counted in months instead of years, and why the rules about what you can do with model outputs have turned into a live fight.

The limit is the same limit too, and it's the one worth holding onto. Distillation compresses what the teacher already knew. It doesn't invent anything the teacher couldn't already do. A genuinely new capability still has to come from somewhere with real ground truth, the actual data or the reward signal, the thing itself rather than an echo of an echo. Matching a known behavior keeps getting cheaper. Making the new one that nothing can grade yet is still the whole game.

I've leaned on one word the law is far pickier about than I've been, so let me clean it up. I'm not a lawyer and this isn't advice, just the lay of the land. There's a wide gap between matching what a tool does and lifting how it does it, and nearly everything here lives on the safe side of that gap, but only if you stay deliberate about which side you're on.

Copyright protects expression, not function. The source code is expression, and taking it is infringement, full stop. But what a tool does, the mapping from input to output, the behavior you're treating as an oracle, is function, and function mostly isn't copyrightable. You're allowed to watch what a program does and write your own that does the same. That isn't a loophole, it's the basis of having a competitive software market at all. The condition is that you actually write your own. The moment your assistant has the original's code in front of it and starts paraphrasing, you've crossed from matching behavior into copying expression, and running it through a model doesn't launder that.

This is why the old hands did clean-room reimplementation. One group studies the original and writes down only what it does, never how. A second group, who never saw the original, builds from that description. The wall between them is the proof that you reproduced behavior and not code. A reference oracle is a clean room by construction, as long as you only ever feed it inputs and read its outputs, and never let the source itself leak into the build.

Copyright also isn't the only thing that can bite, which is easy to forget. A patent covers the method rather than the wording, so a spotless from-scratch reimplementation can still infringe one, because the method is exactly what your behavior matching reproduces. Trade-secret law is fine with reverse engineering something you obtained fairly and not at all fine with internals you got through an NDA or a leak. And the constraint that shows up most often isn't intellectual property at all. It's the contract you clicked to use the thing. Plenty of terms of service forbid reverse engineering, forbid benchmarking, or forbid using the product to build a competitor at all. Trip over one of those and the claim is breach of contract, which doesn't care how clean your room was.

Four different things can bite, and only the first is about the code itself.

The model version has its own twist. It's unsettled whether a model's raw outputs are copyrightable in the first place, and the going position in some places is that purely machine-generated text isn't. If that holds, training a student on a teacher's answers isn't copyright infringement, because there's no protected work being copied. That sounds like a clearance right up until you notice the leash just moved: the provider's terms forbid training on their outputs, so your exposure slides from copyright over to contract, exactly like it does with ordinary software. The fight over distillation is mostly being had there, not in copyright court.

Then there's the word the whole subject attracts, theft. I'd go easy with it. Lifting code is theft in any sense that counts. Studying a rival's public behavior and building something better is just competition, and the law has spent more than a century refusing to hand anyone a monopoly on function only because the function was expensive to work out. The hard cases sit in between, where you've taken nothing protected and still free-ridden on years of someone else's investment. That can be fully legal and still feel like something. People land in different places on it, and where they land tends to track whether they're the one building or the one being matched more than any clean principle does.

So keep the line in sight while you do this. Match behavior, don't ingest code. Read the terms before you point an oracle at something you pay to access. And if you catch yourself reaching for the word "copy" to describe what you built, stop and work out which side of the gap you actually landed on. If it ever comes to it, someone else will.