Giving my smartest AI a veto, not a vote

I expected the smarter AI to fix my model’s mistakes. It overruled 38 right answers and fixed one.

My wildlife discovery app has to name the animal in a photo, and say “Not sure” when it can’t tell. A specialized vision model does the naming well: it’s right 98.9% of the time on the 40 species the app knows. Its weak spot is lookalikes. Animals that aren’t on the list, like a different species of rabbit or dragonfly, sometimes get confidently mislabeled as the nearest listed species.

I measured this with a test of 763 photos: 632 of the 40 species, and 131 of animals that aren’t on the list. I’ll call those the strangers, since the right answer for them is “Not sure.” The obvious fix was to bring in a bigger, general-purpose AI model to double-check the specialist. A smarter model should catch the mistakes, right?

The obvious plan failed

My first experiment gave the general model the photo plus the specialist’s top five candidates and let it pick the winner. On photos where the specialist was already right, the general model swapped in a wrong answer 38 times. It fixed a wrong answer once.

The general model had to choose between five near-identical hawks using only the photo. The specialist had scored all 40 species, and the general model never saw that ranking. Being smarter in general didn’t help it on a narrow job the specialist was built for. It also played it safe, sending more than half of the animals it should have recognized to “Not sure.”

But it was brilliant at one thing

The same experiment showed me something else. The general model caught every one of the strangers that had fooled the specialist, the wrong species of rabbit, dragonfly, or hummingbird. Overall, it rejected 98.5% of the 131.

It was bad at choosing and great at doubting. Those turn out to be different skills.

A veto, not a vote

So now the specialist always names the animal, and the general model only gets to say “hold on.” What happens next depends on how closely the photo matches anything on the list, which the specialist reports as a similarity score:

Weak match

Not sure

Looks like nothing on the list. No second opinion needed.

Ambiguous

Ask the second AI

It must agree with the specialist and be sure. Otherwise, "Not sure."

Strong match

Trust the specialist

A clear match. No second opinion needed.

The second AI can raise a flag, but it can never rename the animal. Only about a third of photos reach it, so most scans run on the specialist alone.

The results on my test photos:

• Labeled Confident on about 82% of photos, and 97.6% of those were right. The general model alone managed about 40%.
• Only 1.9% of the animals on the list sent to “Not sure.” The general model alone sent about half.
• 93.1% of the strangers caught, up from 88.5% for the specialist alone.

The catch

Nothing’s free. The specialist has to run inside the server, so the hybrid needs a bigger machine than a design that only calls the general model. I’m building for the hybrid and developing locally, where hosting costs nothing. If a bigger host doesn’t make sense later, the app falls back to the general model alone, and that’s a configuration switch, not a rewrite. A decision that’s easy to reverse is one I can make before I know all the answers.

The takeaway

“Which model is smarter?” turned out to be the wrong question. Better questions are what each one is actually good at, and what happens when it’s wrong. Here, a good judge with a veto beat a smarter one with a vote.

Nine of the 131 strangers still get through as a confident match, so there’s more to figure out. I’ll write that up when I do.