How We Cut Manual Review Time by 90%, With AI Trained on Your Own Bar

When Tata MCS came to us with 10 million customer success stories to evaluate, manual review wasn’t just slow — it was impossible at that scale. A team reading and scoring every entry by hand would have taken years, and even then, no two reviewers would agree on exactly where the bar sits.

We built a custom AI evaluation system instead — one trained not on generic rules, but on Tata’s own historical evaluation decisions. Here’s how it worked, and what it took to get there.

The problem with manual shortlisting

Every layer of human review introduces inconsistency. What passes one reviewer’s bar might not pass another’s, and at 10 million entries, that inconsistency compounds fast. By the time you’re down to a shortlist of 500, small early disagreements have already shaped who made the cut and who didn’t — and there’s no clean way to audit it after the fact.

There’s also the sheer time cost. Even a fast reviewer scoring 200 entries a day would need over 130 years to get through 10 million submissions alone. Scaling that with more reviewers just multiplies the inconsistency problem instead of solving it.

Training on past evaluation data

Instead of building rules from scratch, we trained the model on Tata’s own historical evaluation decisions — letting it learn the actual bar the team already used, rather than a generic definition of “good.” This mattered because “quality” in a customer success story isn’t just grammar or length — it’s about whether the story actually demonstrates the specific outcomes Tata cares about, and that judgment lives in years of institutional knowledge, not a style guide.

We fed the model thousands of previously reviewed and scored entries, along with the reasoning behind borderline calls, so it could learn not just the obvious yes/no cases but the nuanced ones in between.

Shortlisting in stages, not all at once

Rather than asking the model to go from 10 million straight to a top 10, we broke the process into stages that mirrored how a human team would naturally narrow the field: stage one cut the initial 10 million down to a top 500, filtering out entries that clearly didn’t meet the baseline bar. Stage two took that 500 down to a top 64, applying stricter comparative judgment. Stage three brought it to a final top 10, where the model’s confidence scores were closest and the differences most subtle.

This staged approach meant each pass could apply progressively finer judgment, instead of asking one model call to make an impossibly broad decision in one shot.

Zero manual evaluation, full auditability

The result: zero manual evaluation was needed anywhere in the pipeline, while every decision remained fully traceable back to the training data and scoring logic that produced it. That auditability turned out to matter as much as the speed — when stakeholders asked “why did this entry make the top 10,” there was a clear, consistent answer every time, something that’s much harder to produce from a room full of human reviewers working independently.

What this means beyond Tata MCS

The same pattern — training on an organization’s own past decisions instead of generic rules — applies anywhere manual review doesn’t scale: content moderation, application screening, quality audits, compliance checks. The technology isn’t the hard part. The hard part is capturing what “good” actually means for your specific bar, and that’s exactly the problem custom AI evaluation is built to solve.

FAQ