The Cheaper, More Accurate Classifier That Broke in Production

A field note on why two environments that look identical are not identical until every dependency is pinned to say so - in your requirements.txt, your Dockerfile, and everywhere else.

We had a document classification problem, and the fashionable answer was to throw an LLM at it. We tried that. We also tried the unglamorous approach: TF-IDF features feeding a traditional machine learning classifier.

The traditional model won on both axes that matter. It was more accurate than the LLM-based classifier, and it ran at a fraction of the cost.

Near-perfect numbers are a red flag, and we treated them like one. We spent many hours pouring over the results, checking for leakage, mislabeled data, and the dozen other ways a model can look great for the wrong reasons. The diligence held up. The development results were genuinely correct. The model had learned the right thing.

Then we shipped it to production, and the accuracy quietly fell apart.

We trained the model in a Strongly workspace and deployed it to our model registry, where the serving container loads it for inference. The same documents that classified correctly in the workspace were getting the wrong label once the model ran inside the deployed container. Not randomly wrong. Consistently wrong.

“

A scanned contract that the workspace classified as Contract was coming back from production as BillOfSale. Every time.

The instinct in this situation is to suspect the usual culprits. Was the model nondeterministic? Was concurrency corrupting state under load? Was the wrong model artifact deployed? We ruled those out one at a time, and the answer turned out to be none of them.

The model was fine. The training was fine. We had already proven that. What differed between development and production was not the model or the data or the way it was trained. It was the PDF extraction libraries installed in each environment.

The pymupdf and pdfplumber drift was minor. The pymupdf4llm jump was not. Going from 0.0.27 to 1.27.2.3 is not a patch bump. It is effectively a different library, a whole generation apart, with different behavior baked into the same function names.

The model artifact we deployed was byte-identical to what we trained. The difference was the two environments that ran around it. The model registry handed both the same model, but the text that model received was produced by two different libraries. How did this happen? The requirements.txt used loose lower-bound pins.

With a >= pin and no upper bound and no lockfile, the Docker build was free to resolve to whatever the latest published version happened to be. So the serving container floated all the way up to 1.27.2.3, while the workspace where we trained and validated still used 0.0.27. Nothing about the model changed. Two environments that were supposed to be identical were not, because the requirements file never said they had to be.

Knowing the versions differed is not the same as knowing why it changed the prediction. Here is the actual chain of events for an image-only (scanned) PDF, and why the same document took two different paths in two environments.

We did not want to ship a theory. We wanted a controlled result. Here is what the tests showed.

The conclusions fall out cleanly. The last one is the answer.

The last row of the results table is the clincher. A document with a real text layer, which never needs OCR at all, classifies correctly everywhere. The failure only appears precisely where the new library's OCR behavior gets a chance to act.

The remedy is boring, which is how you know it is right. Pin the extraction libraries to the exact versions used in the workspace, rebuild the serving image, and stop using loose lower bounds for anything that touches feature extraction.

Replace every >= with an exact ==, commit a lockfile so the build is reproducible, and treat the dependency set as part of the model, because it is. The features the model sees are only as stable as the libraries that produce them. And requirements.txt is only the most obvious offender. The same discipline has to extend to every layer that defines the environment.

A pinned requirements.txt sitting on top of a base image that floats is still a moving target. A >=0.0.10 pin is an open invitation for the serving environment to drift away from the one you validated in - and so is every unpinned line anywhere else in the build. The fix is to remove that invitation wherever it appears.

Two environments that are supposed to be identical are not identical until you force them to be, at every layer. A requirements.txt full of >= pins says "any version from here up will do," and that sentence is almost never true for code that extracts features. But the same trap lives in the Dockerfile base image, the system packages, and the transitive dependencies you never named directly. The versions that did fine in the workspace are the only ones you have any evidence about. Everything above them is untested, and one of those untested versions will eventually land in your build.

“

The model is usually the part that works, and it worked here. The hard part is everything around it.

We love coding assistants, and they make all of us faster. But every data scientist and engineer at Strongly still knows how to read a stack trace, reason about a dependency graph, and reproduce a bug under controlled conditions. No assistant flagged a >= pin as the thing that would silently degrade a production model. Finding this took someone who understood that the model, the OCR fallback logic, the word-count threshold, and the build's dependency resolution were all one system - and who could hold that whole chain in their head while running the experiments to isolate it.

AI and ML are not easy in production. The model is usually the part that works. The hard part is everything around it: the extraction libraries, the build process, and the quiet assumption that the container you ship is running the same packages you trained against.