Why predicting court case outcomes fails
Predicting court case outcomes breaks down not because of the model but because of the data.
Heterogeneous decision texts come from different sources: one court provides a structured export, another a scan without a text layer, and a third a fragment with a truncated reasoning part. After cleaning, only what survived remains, and the prediction is built on an incomplete picture.
Then the quality of the training sample comes into play. There is no unified labeling: in one decision the outcome is read from the operative part, in another only from the reasoning, while settlement agreements and withdrawals of claims are mixed with a loss. The share of contradictory examples grows, and the model learns on noise.
When labeling is noisy, the model takes a shortcut—it relies on formal features. The court number, composition of participants, template phrases, and text length begin to weigh more than the facts and legal position. From the outside it looks like a working prediction, but testing on new cases makes it fall apart.
A separate failure is errors on rare categories of disputes. Mass cases number in the thousands, rare ones in the tens, and an averaged score across the entire sample simply hides them.
Where there are few examples, the share of correct answers drops severalfold, and it is precisely such disputes that most often need to be assessed.
That is why the analysis begins not with the model but with the data: how heterogeneous decision texts are brought to a comparable form, how labeling is built, and how quality is checked separately for rare categories. Below is what we do at each step.
What a prediction model changes in the work of a legal team
A prediction model does not replace the lawyer's legal position—it gives the team a common reference point.
We measure the effect of the solution by the spread of assessments among lawyers, the time to the first assessment of prospects, and the share of disputes where the prediction was confirmed by the actual outcome.
A unified assessment of prospects for all lawyers
- Previously, for the same case, lawyers gave assessments ranging from “clearly losing” to “we will win”—the client chose a strategy based on someone else's intuition.
- Now the assessment of a dispute's prospects is calculated using a formalized set of features: the subject and basis of the claim, procedural violations, and the stability of practice on similar compositions.
- The spread of assessments among specialists is reduced to a few percentage points; the lawyer separately records what does not enter the data: the counterparty's behavior, readiness for settlement, and the course of negotiations.
Speed of initial analysis of materials
- Reviewing a volume of materials and selecting relevant practice happen in one pass—hours instead of days of waiting.
- The model labels the parties, chronology, and key procedural episodes: the lawyer checks a ready structure instead of assembling it manually.
- A unified assessment standard holds for new cases as well: the result on typical disputes does not depend on who took the case.
“We stopped discussing whose assessment was more correct and moved on to facts and strategy. Contested cases are now visible immediately—by the low confidence of the prediction.” — head of the legal team
The training dataset is assembled from open legal databases, including ConsultantPlus, and anonymized archives of practice: the client's data does not leave its perimeter.
Approaches to building a model: comparison of options
The choice of approach is determined not by the “strength” of the model but by what data it is trained on and who is responsible for an error in a specific case.
In practice, three working alternatives are compared—from a quick start on a ready-made solution to a model fine-tuned on an archive of one's own decisions.
| Option | When it fits | Limitations |
|---|---|---|
| External prediction service | A result is needed before accumulating your own corpus, a uniform flow of cases | Trained on someone else's practice, fine-tuning on your own decisions is unavailable, explanation of the conclusion is depersonalized |
| In-house model with fine-tuning on your own decisions | There is an archive of judicial acts and lawyers ready to label outcomes | A corpus and labeling are needed, quality grows gradually, monitoring of changes in practice is required |
| Hybrid scheme: model plus expert rules | Heterogeneous categories of disputes, high cost of error | More difficult to maintain: rules and model are managed together |
External service: what you pay for readiness
A ready-made prediction service gives an assessment immediately, without collecting a corpus and labeling. What you pay for this is control: model updates come from outside, it cannot be fine-tuned on your own practice, and quality drops on rare categories of cases. The issue of transferring case materials to a third party is resolved separately.
Hybrid scheme: model plus expert rules
Some decisions are made not by probabilities but by rules: procedural deadlines, jurisdiction, category, and scope of claims cut off clearly impossible outcomes, and the model ranks the remaining ones.
If confidence is below the threshold, the case goes to a lawyer—this way the model's limitations become explicit rather than hidden. Fine-tuning on your own decisions is ongoing: every closed case refines the weights.
Implementation stages: from data audit to handover of the perimeter
Implementation of a court case outcome prediction system begins not with the model but with a corpus of decisions and a pre-fixed metric. The implementation stages follow a standard engineering cycle: data, labeling, training, blind testing, handover. Below are seven steps completed before launch.
Corpus audit and target metric definition
We check the completeness of the corpus by instances, periods, and types of disputes, the quality of text extraction, duplicates, and gaps.
We fix the target metric before training: the share of predictions matching the final decision and recall for each class of outcomes—otherwise an increase in quality cannot be distinguished from overfitting.
- Corpus audit: composition, completeness, duplicates, text quality.
- Sample labeling: outcome, articles, claim amounts, procedural features.
- Feature schema and the link between the decision and case materials.
- Training with probability calibration by class.
- Testing on lawyers' blind examples.
- Analysis of discrepancies with experts and fine-tuning.
- Handover of the support perimeter: regulations, monitoring, rollback.
Testing on lawyers' blind examples
The sample is prepared by the client's lawyers: they provide an opinion before they see the prediction, and we compare both results.
“We consider the tool operational when it matches the lawyer's position in an agreed share of cases,” states a review of practice on corporate disputes.
This is exactly where it becomes clear where the model confuses outcomes and which features are missing from the schema.
What we hand over: data schemas, regulations, instructions
A kit for predicting court case outcomes cannot be handed over with the formula “the model is trained, accuracy is high”—it lives on closed case materials.
We hand over a set of documents that your lawyers and technical team use without us: what the system can do, what data it was trained on, and who updates it and how.
- Model card: where the prediction is stable, where it is biased, and at what confidence threshold it is not issued.
- Labeling regulation: how decisions are parsed, how disputed cases are interpreted, and how the control sample is formed.
- Data flow schema: the path from loading case materials to the prediction and the points where data does not leave the perimeter.
- Instructions for lawyers: how to submit a query, how to read the answer, and when manual verification is mandatory.
- Access matrix for case materials: roles, scope of visible data, and access log.
- Feature reference: which factors the model takes into account and shows in the explanation for the prediction.
- Quality monitoring procedure: which metrics are captured and what drift stops predictions from being issued.
Model card and boundaries of applicability
The card records the categories of cases where the prediction is stable and those where it is biased: an atypical instance, a rare set of claims, an incomplete corpus.
The confidence threshold is set so that below it the system returns not a conclusion but a list of factors with references to case fragments.
Retraining regulation and roles in the team
The regulation describes the update cycle: who prepares the sample, who checks the labeling, who approves the release and by which metrics, and how rollback is performed.
Roles are assigned to people on both sides—lawyers and the technical team—so that quality does not depend on a single specialist.
Implementation case study: how assessments of prospects leveled out
A legal practice with 40 lawyers, contractual disputes, and a flow of about 60 cases per month.
The initial assessment of prospects determined whether to go to court, enter negotiations, or refuse—so in the implementation case study we measured metrics before and after on the same flow of cases.
What it was like before implementation
The assessment was prepared by a junior lawyer in 3–5 business days: they pulled up practice on the composition of the dispute, searched for similar cases, and wrote a two-page conclusion.
The spread of lawyers' assessments was the main problem—on the same facts, a senior and a junior differed by 25–30 percentage points, and juniors more often assessed prospects more optimistically. Matching with the final decision across 300 closed cases was about 60%.
All the initial routine fell on junior lawyers: up to 70% of their time went to assessments instead of handling cases.
What resulted afterward
The model was trained on an internal archive of closed cases and texts of judicial acts; the foundation was CatBoost—it provides calibrated probability and runs in a closed perimeter, so documents do not leave the firm.
We added factor analysis: composition of the dispute, amount of claims, region, and stability of practice on similar cases.
Initial assessment time was reduced to 15–20 minutes per case—the lawyer uploads documents, receives a probability and explanation, and analyzes discrepancies.
The spread of assessments within the team fell to 5–7 percentage points, matching with the final decision rose to 84%, and junior lawyers returned to handling cases.
Previously, a disputed issue was decided by whoever argued more convincingly. Now the discussion revolves around the factors the system highlighted—head of practice.
How does a prediction on the merits of a dispute differ from risk assessment?
The difference between these tasks is in the target variable. A prediction on the merits of a dispute answers the question “how will the case end”: the variable is taken from the operative part—granted, denied, granted in part.
Such a variable is discrete, observable, and confirmed by a document. Therefore it is tested on a holdout sample without manual expertise, and the metric is the share of matches with the operative part and confidence calibration separately for each category.
Procedural risk assessment predicts not the outcome but the movement of the case: adjournment, suspension, appointment of expertise, leaving without consideration, appeal, and the time until the decision enters into force. The target variable here is an event and time, not the result of the dispute.
This leads to different labeling requirements. For the merits, a corpus of decisions with a clean operative part and labeling of dispute categories is needed—otherwise the model will mix contractual disputes with corporate ones and lose accuracy on rare classes. For risks, a case movement log is sufficient: dates, rulings, instances.
The two variables cannot be mixed into one: the model will start producing an averaged answer that is not suitable for either strategy or planning.
The working scheme is two layers on shared features from the text and case history, where risk assessment refines the prediction on the merits rather than replacing it.
Quality, explainability, and boundaries of model applicability
The model assesses the outcome of a case as a probability by class, not as a verdict. Acceptance of such a system comes down to one question: where statistics end and legal assessment begins, for which the lawyer is responsible. Below are answers to the doubts that most often arise when handing the system over for use.
How stable is the prediction?
The same claim with rephrasings and changed personal data should produce a similar probability. If the spread between runs grows, the model goes for fine-tuning, not into operation.
How can we understand why the model decided this way?
We attach to the prediction the factors that influenced the conclusion: fragments of the claim, legal norms, circumstances, and deadlines. The lawyer sees support for the position and can challenge the calculation instead of trusting an inexplicable number.
What to do about regional differences in practice?
The same legal norm works differently in different courts, so the calculation takes into account the region, instance, and type of dispute. Where there is little data for a region, the prediction is marked as less reliable instead of being averaged across the country.
How are settlement agreements accounted for in the sample?
Cases closed by settlement distort statistics on the merits of the dispute. We label them as a separate class so that they do not bias the prediction for cases that will reach a decision.
Who is responsible for the system's conclusions?
The prediction remains material for preparing a position; the lawyer makes the decision. The system stores the grounds and the calculation version, so the conclusion is reproducible and can be challenged within the company.
Let's discuss the task: what data and conclusions you need
Discussion of the task begins not with a model demonstration but with your subject area: which category of disputes, who uses the conclusion, and what decision is made based on it.
We need a real array of cases—it shows whether it contains enough features for stable prediction of court case outcomes. To return with an assessment and work plan rather than general words, we need from you:
- 300–500 closed cases of one category of disputes with full texts of acts;
- a list of features that you consider significant and those that must not be used;
- a description of the target conclusion: probability, range, or recommendation to a lawyer;
- requirements for the perimeter: where documents are stored and who gets access;
- examples of cases where the lawyer's intuition diverges from the expected outcome;
- a responsible person on your side who will accept the work according to acceptance criteria.
In response, we send a stage-by-stage work plan with approximate timelines and acceptance criteria fixed before the start.
The first stage is a pilot on one category of disputes: corpus labeling, assembly of a baseline model, comparison of its conclusions with your lawyers' decisions, and an accuracy report.
Such narrowing gives an honest picture: if quality is not confirmed in the pilot, you see this before scaling to related areas of law. Describe the task—we will return with an assessment and work plan.
