DiDi Doubled Intent-Verification Accuracy by Hiding the Alternatives

More context is often treated as an unqualified improvement in AI design. Give the model the full customer record, the complete policy, every category it could select, and it should make a better decision. DiDi’s contact-centre implementation points to a more useful rule: information required to repair a decision can corrupt the test of whether that decision was acceptable in the first place.

In a production intent-verification workflow, DiDi’s International Business Group reported raising validation accuracy from 38% to 86% after showing the model less at the first step. The result is not an independent benchmark: the public account does not provide validation-set size, class distribution, confidence intervals or an external audit. It is nevertheless unusually concrete operational evidence. A failed design, its replacement and the measured before-and-after result are all described.

The central lesson is not that models should receive less context in general. It is that verification and reclassification are different tasks. When the alternatives are visible too early, the second task can distort the first.

The original design asked two questions at once

DiDi built three production pipelines for Spanish- and Portuguese-language customer service across ride-hailing, food delivery and financial services: intent verification, compliance evaluation and Voice of Customer analysis. The intent-verification pipeline checks whether a human-assigned contact reason fits the conversation.

Its initial design placed the conversation, the assigned reason and the complete contact-reason tree into one model call. That seems efficient. The model has all possible labels in view and can decide whether the current one is right. But the implementation account says the model over-corrected. It found labels that were marginally more precise and treated those alternatives as grounds to reject labels that were still reasonable. Repeated prompt tuning did not resolve the behaviour.

This is a familiar product error dressed as a prompt problem. The workflow asked the model to perform a binary assessment—does the current label fit?—while exposing it to a comparative optimization problem—which available label is best? “Acceptable” and “best available” are not equivalent standards. A taxonomy makes that gap more acute because taxonomies often contain overlapping or increasingly specific categories.

“A remediation menu is not neutral evidence for an initial test. It changes the decision the model is being asked to make.

The consequence is practical for any team using an LLM to review human work: do not assume that supplying the correction space improves the quality of the review. It may instead encourage avoidable disagreement, increase exception volume and send correct-enough work back for rework.

The replacement preserves the first question

DiDi separated the workflow into verification and classification. In the first phase, the model sees the conversation and the current label, but not the full taxonomy. It evaluates that label on its own terms. Only if the label is rejected does a second phase receive the complete taxonomy and select a replacement. DiDi reports that production-validation accuracy rose from 38% to 86% with this redesign.

This is decision sequencing, not merely prompt decomposition. The first stage has a deliberately constrained choice set: retain or reject the current label. The second stage has the richer choice set required for remediation. Each stage sees the information needed for its own decision, rather than all information that might be useful somewhere in the overall process.

The distinction matters because the stages optimize different outcomes. Verification should avoid rejecting acceptable work. Classification should identify the most suitable alternative after rejection. Combining them makes the more discriminating classification standard silently become the acceptance standard. Separating them makes the standard explicit and gives each call a narrower, intelligible job.

Use the pattern where a current decision deserves a fair hearing

The pattern applies beyond contact reasons. A support-quality system may first ask whether an existing disposition is justified, then identify the better disposition only if it is not. A document-review system may first test whether a selected clause is applicable, then search alternative clauses after failure. A transaction-review system may first evaluate whether the stated exception meets its criteria, then route it to another exception category only when it does not.

The design test is simple: would showing every possible remedy cause the reviewer to judge the existing decision against an unnecessarily stronger standard? If yes, hide the remedy space until the verification result is settled.

That does not mean withholding relevant evidence. The conversation itself remains essential evidence for both stages. Nor does it mean the first stage should ignore applicable rules. The boundary is narrower: hold back alternatives whose main role is to choose a replacement, where their presence can turn verification into a search for marginal improvement.

Put deterministic work where it belongs

DiDi’s compliance pipeline reinforces the same discipline of task boundaries. It reportedly exceeded 90% average accuracy. Some facts, such as response-wait times, are calculated in code rather than inferred by a model. For rule-based findings such as spelling-error counts, a post-validation layer rechecks the finding against the source conversation.

The operating implication is not that an LLM cannot participate in compliance evaluation. It is that a model should not be asked to simulate a calculator, timer or exact counter when the system can compute those facts deterministically. And when a model produces a finding that can be checked against source material by a rule, that check should occur after generation. This narrows the model’s role to interpretation where interpretation is actually needed.

The same approach appears in DiDi’s Voice of Customer workflow. Conversations are processed independently; extracted issue labels are clustered using embeddings and statistical ranking; then a report is generated from the highest-frequency clusters. DiDi says work that previously required hours can be completed in minutes. The workflow does not ask one call to read the entire corpus, discover trends and write a report. It separates extraction, grouping and communication.

A better sequence is not a universal performance claim

DiDi’s numbers should be read carefully. They are company-reported production-validation results, not a general proof that staged prompting will produce the same gain elsewhere. The public account does not establish how the result varies by label, language, traffic type or evaluator. It also does not establish that Amazon Bedrock, rather than the change in task and information sequencing, caused the improvement.

What the account does establish is more actionable than a broad model comparison. Prompt tuning did not solve a structural conflict in the original task. Changing what the model was allowed to compare, and when, did. For product teams, that is a useful order of operations: before adding instructions, examples or a larger model, inspect whether one call has been given incompatible decision standards.

Structured output can support this design, but it should be described precisely. Amazon Bedrock supports strict structured outputs through its relevant mechanisms, including the documented strict setting for strict tool use. Simply forcing a tool selection should not be presented as a universal guarantee of strict JSON-schema compliance. Format enforcement is valuable, but it does not correct a poorly sequenced decision.

Design the correction path after the judgment path

Many AI review systems begin with a generous instinct: give the model every option so it can be helpful. DiDi’s result suggests a sharper instinct for verification workflows. First ask whether the work already done clears the stated bar. Then, and only then, reveal the space of possible corrections.

That sequence makes acceptance criteria legible, reduces pressure to find cosmetic improvements and confines reclassification to the cases that need it. It will not remove the need for validation, configuration management or human accountability. But it can stop a model from mistaking a slightly better answer for evidence that the current answer is wrong. In systems reviewing high volumes of human-labelled work, that is not a small prompt refinement. It is a change in the economics of disagreement.