← All posts

Reranking search results with AI for three hundredths of a cent

My board game app's search blends exact-word matching with search by meaning, and every so often it returns a result no person would pick. A decision model now double-checks each result before anyone sees it. I built a test harness first, to prove it was worth adding.

Roll Play is an iOS app I'm building for board game collectors. It keeps track of the games you own and helps you find new ones. You can search for a title or a designer, but also for a theme, a mood, or a description of the evening you want: "a cozy nature game," "a heavy two-player war game," "a quick co-op for the family."

Roll Play search results for "cozy farming": Agricola: Family Edition, Stardew Valley: The Board Game, Agricola: All Creatures Big and Small, Lowlands, and Agricola (Revised Edition)

A search for "cozy farming" in Roll Play, with the reranker on: farming games, best fit first.

Under the hood, search blends two methods:

  • Keyword search looks for the exact words you typed in game titles, themes, and mechanics. It's fast and predictable, but it only knows spelling.
  • Semantic search looks for meaning instead of exact words. An AI model turns each game's description into a list of numbers that captures what it's about (called an embedding), and does the same to your search. Games whose numbers land close to yours come back, even if they share no words with what you typed. That's how "a cozy nature game" can find a game whose description never says "cozy."

Most of the time the blend works. But both methods are purely mechanical. Neither one knows what you actually want, and every so often a result fails the smell test:

  • Searching "farming" surfaced Terraforming Mars, a game about Mars that happens to contain the word.
  • Searching "engine builder" turned up Kingdom Builder and Kanban, which match the words but not the kind of game.
  • Searching "scheduling a dentist appointment", which has nothing to do with board games, returned Operation. Close in meaning, technically.

A person would never make these mistakes. So I added one step that works a little more like a person.

Build the test before the feature

Before changing anything, I built a test harness, so I'd know whether a change helped or just looked like it did. Adding an AI step to every search adds cost and delay, and I wanted proof it was worth both, not a hunch.

It's a list of 31 search queries, each labeled with what a good answer looks like, across every kind of search people make:

Kind of search Example What counts as right
Title "brass birmingham" That game comes back first
Designer "uwe rosenberg" His best-known games show up
Theme word "farming" Farming games, and no Terraforming Mars in the top 5
Genre "horror" Well-known horror games show up
Description "a cozy nature game" Games like Wingspan and Cascadia show up
Junk "filing quarterly income taxes" Nothing comes back at all

The harness runs each query through the real search code, on a local copy of the server loaded with production data, so it measures what users actually get. It saves the model's answers, so rerunning it after a code change costs nothing.

It also settled design questions before I wrote any production code. Asking about all the candidates in one request matched the accuracy of one request per candidate, at about 60% of the tokens and a thirtieth of the requests. And short game summaries (players, play time, complexity, themes, mechanics) did as well as longer ones, so search doesn't need an extra database read to build them.

One question per result

The new step uses Jev, a decision model from TypeSafe. Jev doesn't write text. It answers yes-or-no and multiple-choice questions about whatever you give it, with a probability for each answer.

After the blended search finds its top 40 candidates, they all go to Jev in a single request, with one yes-or-no question about each: is this a game the person wants for this search? Here are the criteria, in plain English:

const CRITERIA = {
  true:
    "The game is the one the search names, is designed by the person the search names, or " +
    "fits the searched theme, genre, mechanic, or description without contradicting any " +
    "stated constraint (player count, length, complexity, audience).",
  false:
    "The game only shares a word or a loose association with the search (for example, the " +
    "searched word appears inside its title but the game is not about it), contradicts a " +
    "stated constraint, or the search is not about board games at all.",
}

The "false" answer names the exact failures from the list above: a shared word, a broken constraint, a search that isn't about games.

The results are then ranked by Jev's probability. That fixes a deeper problem too. Before, the final order came from blending scores on scales that don't compare, like how closely the words matched and how closely the meaning matched, held together with hand-tuned constants. Now every candidate has one number on one scale.

The rules around the model stay in code:

  • An exact title match always comes first. If you type a game's name, you get that game, no judgment needed.
  • Anything under 50% is dropped. In testing, the top result for a real search typically scored around 92%, and no candidate for a junk search scored above 28%, so the cutoff has plenty of room. If nothing clears it, search returns nothing, which is the right answer for "dentist appointment."
  • If Jev takes longer than 1.2 seconds or fails, search falls back to the blended order. A slow model never breaks search.
  • It only runs when it's needed. Typing a title as you go uses plain keyword search and never calls the model.

The harness found the next fix, too

The first run showed the reranker helping descriptions a lot and big genres like "sci fi" barely at all. The reason: a genre like sci-fi has hundreds of games, and the best-known ones weren't making the top 40 for Jev to promote. Terraforming Mars came in at #47.

But "is this a sci-fi game?" doesn't depend on who's searching or how they worded it. So instead of asking at search time, I asked once, ahead of time, for every game in every large genre: 3,803 questions, for about 3 cents. The answers ship with the server as a data file, so a genre search makes no model call at all. "Sci fi" now puts Terraforming Mars at #1. I then extended the same approach to every theme, mechanic, and category, 18,402 questions in all, for about 14 cents.

What changed, and what it costs

Here's the harness run against the real search code, with the model's step turned off and then on:

Kind of search Before After
Title: right game first 11 of 11 11 of 11
Designer: expected games found 100% 100%
Theme word: expected games in the top 10 48% 81%
Genre: expected games in the top 10 27% 60%
Description: expected games in the top 10 39% 65%
Junk: nothing returned 2 of 3 3 of 3

For each theme, genre, and description search, I listed a few games almost any board gamer would expect to see, like Wingspan and Cascadia for "a cozy nature game." The percentages are the average share of those that made the top 10. Title and designer searches were already right, and stayed right.

Some swaps from the testing: for "engine builder," Jev replaced Kingdom Builder and Kanban with real engine builders like Gizmos and Fantastic Factories. And "scheduling a dentist appointment" now returns nothing, instead of Operation.

At search time, each reranked search sends Jev about 7,400 tokens, and Jev charges $0.042 per million. That's about $0.0003 a search, or 30 cents per thousand. In my first test searches through the server, the median call took 175 milliseconds.

Where this pattern fits

The pattern is simple: let fast, cheap, mechanical search cast a wide net, then have a decision model make the "would a person pick this?" call on what it caught. It's a small, testable addition to search you already have, not a rebuild. The same model labels my GitHub issues and ranks my to-do list.

Most businesses have a search box that returns things a person wouldn't. Some ideas:

  • A product catalog: a shopper searching "waterproof boots for kids" shouldn't see adult rain jackets just because the words overlap.
  • An internal knowledge base: an employee asking about the travel policy gets the current policy, not every document that mentions travel.
  • Support: match an incoming ticket to the help articles that actually answer it, so the person replying starts with the right ones.
  • Past work: pull up the earlier quotes or proposals that really resemble a new request, not just the ones that share a few words.

In each case the reranker suggests and a person decides. And the place to start is the same as mine: write down a few dozen searches and what good answers look like, before changing anything.

Questions or thoughts? Discuss this post on LinkedIn.

Ready to stop missing calls?

Book a short introduction call and ask us anything, or just email us to sign up.

Prefer the phone? Call Warren, our AI receptionist, at (737) 265-8825.