I built a natural-language product search engine over a marketplace catalogue in twelve languages. The interesting failures were not in the model's language understanding — that part mostly worked. They were in the seams: a price filter that had never once filtered, an LLM confidently inventing valid-looking category IDs, a substring match that turned "newborn" into "born" for years, and the finding that a six-second cold search lost more shoppers than an imperfect result ever did.

Two years ago, searching a large marketplace catalogue in anything other than English or Chinese was close to useless. Type a plain description of what you want in Hebrew, Arabic or Polish and you get a handful of results, most of them irrelevant. The catalogue has hundreds of millions of listings. The problem is not that they are missing — it is that the words a shopper uses and the words a seller writes in a listing title are different words, and the gap gets wider the further you move from English.

That is a translation problem in the loose sense and an LLM is obviously good at it. So I built a search engine that takes a plain-language query in any of twelve languages, converts it into the terms sellers actually list under, queries the marketplace API, and ranks what comes back by real rating and order volume rather than ad spend.

The language part worked roughly as expected. Everything around it did not. Here are the failures that taught me something.

1. The price filter had never once filtered anything

The marketplace API takes a min_sale_price parameter. I had been passing it for months — a floor to strip out the two-dollar junk that pollutes every category.

Results kept coming back below the floor. I assumed the parameter was advisory, shrugged, and filtered client-side as a patch.

The parameter is in cents. Passing 15 had been asking for a minimum of fifteen cents. Every price filter in the system, on every surface, had been a no-op since the day it was written.

The lesson is not "read the docs." I had read them. The lesson is that a filter that silently does nothing looks identical to a filter that is working on data that happens not to need it, and I had no test that would have told the difference. A parameter you cannot observe failing is a parameter you should assert on: pass an absurd floor, and assert the result set becomes empty. That test would have caught it in five minutes at any point in those months.

2. The model invented category IDs, and they looked completely valid

To narrow results, queries get routed to a category before hitting the API. I asked the model to pick the category ID.

It did. The IDs were plausible — right length, right numeric range, returned without hesitation. Some of them did not exist. Others existed and were the wrong department entirely.

This is the failure mode people mean when they say hallucination, but the specific danger here is different from a wrong sentence. A wrong sentence is visible. A wrong ID is not. The API accepted them, returned a perfectly well-formed result set from some other corner of the catalogue, and everything downstream looked healthy. No error, no empty result, no log line. Just quietly wrong results for a subset of queries.

The fix was boring: fetch the real category tree, and validate every ID the model produces against it before it is used. If the model's pick is not in the tree, fall back to the keyword mapping.

The general rule I took away: never let a model's output be used as an identifier without validating it against the authoritative list. Free text can be wrong and a human notices. An identifier can be wrong and nothing notices.

3. A substring match that had been mutilating queries for years

Queries can carry a sort intent — "cheap running shoes", "popular kitchen gadgets". The parser stripped those words out and turned them into a sort order.

It stripped them with a substring match.

So newborn baby clothes had the word "new" removed from inside "newborn" and searched for born baby clothes. cheaper earbuds searched for er earbuds. ratings monitor searched for s monitor.

This had been running for years. It never threw. It never logged. It just quietly returned bad results for an entire class of query — and "newborn" is not an exotic word in a catalogue full of baby products.

The fix is a whole-word boundary, with one wrinkle worth writing down: \b in JavaScript regex is defined on ASCII word characters and does not do what you expect against Hebrew or Arabic. I had to match on whitespace-or-string-edge instead:

const re = new RegExp("(^|\\s)" + escaped + "(?=\\s|$)", "i");
If your text processing has to work outside the Latin alphabet, audit every \b you have.

4. Latency cost more than relevance did

This is the one that changed how I prioritise.

I had assumed the thing to optimise was result quality. Better prompts, better ranking, better category routing. I spent months there.

Then I looked at where people actually left. The overwhelming majority of drop-off was not at checkout and not after a bad result — it was on the results page, before anything had rendered. Cold searches were taking six to eight seconds, because a cache miss means a live call to the marketplace API and that call is slow and outside my control. The people leaving had never seen the results at all, good or bad.

Caching every search for 24 hours took a warm query from around seven seconds to around 0.2. That single change did more for conversion than every ranking improvement I had shipped.

Related, and slightly embarrassing: a couple of category pages were paying for an LLM call before checking the cache. The cache was there. It worked. It was just sitting behind an expensive call that ran unconditionally. Moving the check in front of it took those pages from 2.4 seconds to 0.2.

If you put a model in a request path, draw the actual sequence of what happens on a cache hit. Not the intended sequence — the real one.

5. Image search is accurate about the category and unreliable about the model

Photograph a product, get similar listings. It works, but not in the way people expect.

It reliably tells you the kind of thing you are looking at. It rarely finds the exact item. Photograph a specific gemstone ring and you get "women's gemstone ring" and forty listings that are not that ring.

I treated this as a shortfall for a while and tried to close the gap. Then I noticed that the failure is more useful than the success would have been. On a marketplace, the same item is typically listed by dozens of sellers at very different prices. A user who wanted "that ring" is usually better served by forty near-identical alternatives, sorted by rating, than by one exact match at whatever price that particular seller chose.

Sometimes the honest thing is to describe what your feature does rather than keep pushing it toward what you assumed users wanted.

6. Sometimes the retrieval is right, and the catalogue is wrong

A pet category returned the query "cat toy" as a stream of cat-themed merchandise for humans: mugs with cats on them, cat-print socks, cat cushions. The retrieval was doing its job. The seller listings genuinely use those words.

There is no prompt that fixes this, because nothing is wrong upstream. The category simply contains a large volume of items that legitimately match the words and do not match the intent. It needs a domain-specific negative filter — and building one taught me its own lesson, because my first pass filtered out "plush", "puzzle" and "simulation" as obvious human-merchandise words. Those are all real, common dog toy descriptors. I had to put them back.

Junk filters are domain knowledge, not string lists. Write them with someone who knows the domain, or measure them before shipping.

What I would tell myself at the start

Most of what broke was not the model. The model did the linguistically hard part adequately from early on. What broke was everything at the boundary between the model and the rest of the system: unvalidated identifiers, a unit mismatch in an API parameter, an encoding assumption in a regex, and an ordering mistake that put an expensive call in front of a cache.

And the biggest single win was not intelligence at all. It was making the page appear before people gave up on it.

The engine is at onefindme.com if you want to try breaking it — it is free and there is no signup. I am still interested in queries it gets wrong, particularly in languages I do not read.