Pedestrian detection: where the model breaks down in traffic
Dense traffic breaks pedestrian detection not because the model “does not see a person,” but because there are 15–20 objects in the frame at the same time and most of them are occluded.
A pedestrian occluded by a parked car, a bush, or another pedestrian is recognized from a fragment of the silhouette — and confidence drops below the operating threshold.
As a result, missed objects are concentrated not somewhere on the periphery, but exactly where the cost of an error is highest: at intersections, near stops, and at parking lot exits.
Poor visibility finishes off what is left. At night, in rain, and under backlight from headlights, the features that recognition relies on — contour, pose, motion pattern — blur, and glare on wet asphalt produces false positives.
The model just as confidently “sees” a pedestrian in the shadow of a pole and loses a person in dark clothing against a dark background, because contrast is local, not global.
A separate category is non-standard scenarios: a person on a scooter or with a bicycle nearby, a child below hood level, a group merging into one blob, a sitting or lying person.
In training data, such cases account for fractions of a percent, and the model behaves unpredictably on them — not “makes mistakes,” but specifically unpredictably, which is worse for operation.
The root of the problem is a single confidence threshold. Lower it so as not to miss people — you get false braking on shadows and road markings; raise it to remove false positives — missed objects increase.
A single frame does not provide enough context to resolve this contradiction, so the model’s bottleneck lies not in recognition, but in how the decision is made over time — across a sequence of frames, taking into account the track and direction of movement.
For operation, this results in two classes of failures: a missed road user and braking on an empty road. The first is a safety issue, the second is a trust issue: an operator or dispatcher tired of false positives begins to ignore signals. We explain how this is addressed at the level of data, thresholds, and confirmation logic.
Measurable effect: fewer misses and false positives
Before improvement, recognition recall sagged in specific scenes: night, rain, backlight, a pedestrian behind parked vehicles, a cyclist at an intersection.
These were exactly the scenes that produced most of the misses, while in daytime scenes the metric looked healthy and masked the problem.
We labeled weak scenes separately, added them to training with increased weight for rare classes, and validated on a holdout set that did not participate in training — using a methodology close to nuScenes.
Pedestrian recognition recall on difficult scenes grew from ~0.82 to ~0.94 at the same triggering threshold.
The second half of the task is false positives: if the model reacts to glare or a reflection in a puddle, the vehicle brakes for no reason, the schedule slips, and operator trust drops.
We labeled such cases as a separate error class and added hypothesis verification across a sequence of frames — a single-frame anomaly is no longer considered an object. The false positive rate per hour of movement decreased from 6–8 to 1–2.
Error correction time became manageable: an incident is analyzed using logs and sensor recordings, the labeled case goes into additional training, the nightly build runs the regression set and rolls out the update first to part of the fleet. Previously it took weeks to fix, now a typical defect is closed in 2–3 days.
Together, this reduces vehicle downtime for investigation: most triggers are resolved remotely, without taking the vehicle out of service. The metrics — recognition recall, false positives, time to fix — are consolidated into a single dashboard and visible for each route.
Operating modes and integration options on vehicles
The operating mode is chosen not by the detection model, but by where computation can physically live: on board, on a server, or in a combination.
Then the decision is determined by three constraints — communication channel stability, acceptable response latency for braking, and privacy requirements for camera frames.
A separate story is integration with the on-board loop that is already installed on the vehicle. Here what matters more is not our stack, but the load headroom of the installed compute unit, access to video streams, and how the system sends a signal to the control bus.
We start with an inventory: half of projects break not on pedestrian detection, but on incompatibility with what is already installed.
| Option | When it fits | Limitations |
|---|---|---|
| On-board processing | Need for response without channel latency, operation in areas without coverage and in closed zones | Compute resources are limited: the number of cameras and resolution must be normalized, model updates are less frequent |
| Server-side processing | Heavy models, aggregate fleet analytics, frequent logic updates | Dependence on connectivity: if the connection drops, detection is unavailable; higher requirements for protecting transmitted frames |
| Hybrid scheme | Critical functions on board, analysis of disputed situations and training on the server | Two model branches and version coordination: more difficult maintenance and debugging of discrepancies |
| Upgrading the installed loop | Cameras and compute unit already exist, fleet cannot be changed | The model is trimmed to available resources; limitations remain on field of view angles and stream synchronization |
We fix the choice not based on datasheet promises, but on running real route recordings: we see how the option handles night, rain, and dense traffic. Then we compare maintenance cost, and only after that move to designing the loop.
How the project proceeds: from data audit to handover
The work proceeds not as “model training,” but as a managed sequence where each stage has a verifiable artifact. The stages are not reordered: a skipped data audit surfaces at acceptance, when rework is most expensive.
-
Data and scenario audit. We analyze recordings by conditions: lighting, weather, traffic density, road types. We record rare and dangerous cases — a pedestrian in dark clothing at night, a cyclist behind a large vehicle — and build a gap map with a plan to close them.
-
Solution design. We determine where detection fits into the overall perception loop, what comes in as input, and what goes out. Here we also set target metrics: acceptable miss rate and acceptable false positive rate.
-
Labeling and training. We agree on unified labeling rules so that class boundaries do not “drift” between annotators, and train the model on a set that covers the scenario map, not just frequent cases.
-
Validation on a holdout set. The holdout set is assembled by scenario, not randomly: it includes the same night, rainy, and dense sections. We look not at the average metric, but at the spread across conditions — that is what shows where the solution will fall apart.
-
Result acceptance. We run the solution on the agreed set of scenarios and compare against the thresholds from the second stage. Acceptance follows a protocol that the customer can reproduce independently.
-
Handover and support. We provide configurations, the procedure for reproducing checks, and a report on behavior in difficult cases. After that — a regulation for additional training on new data.
Project artifacts: labeling, diagrams, regulations
A pedestrian and cyclist detection project ends not with a demonstration run, but with a package that your team can use to continue working without us.
Everything developed during training and field runs is recorded in artifacts — from reference labeling to the incident analysis regulation.
- Reference labeling — a validated set of frames with pedestrians, cyclists, and edge cases: partial occlusion, night, rain, headlight glare, low contrast. With attribution rules and instructions for annotators, so that new data fits the same format without rebuilding the dataset.
- Scenario maps — a matrix of situations: jaywalking, a group of people, a child behind an obstacle, a scooter rider at speed. For each — whether it is covered by checks, on what data it was confirmed, and where the applicability boundary lies.
- Detection loop configurations — triggering thresholds, zones of interest, frame processing rate, and logic for merging results over time. With an explanation of why these particular values were chosen and what changes when they shift.
- Interface diagram with the on-board loop — exchange formats, latencies, behavior on signal loss, requirements for the compute unit, and the degradation procedure if data is insufficient.
- Incident analysis regulation — what to record during an incident, what minimum data to save for reproduction, and how to classify the error: miss, false positive, or delayed response.
- Model documentation — scope of applicability, metrics on the holdout set, known limitations, and conditions under which additional training on new data is needed.
- Reproduction checklist — environment, launch order, retraining and acceptance steps, so that not only the model author can reproduce the result.
The package makes quality verifiable: metrics are tied to specific scenarios, each limitation has a stated source, and analysis of a new failure follows a single regulation, not the developer’s memory.
Case study: reduced misses on night routes
The customer is a supplier of autonomous driving systems.
During the day, pedestrian detection remained at an acceptable level, but on night routes misses began: at unlit crossings, under backlight from headlights, and in rain, recognition recall dropped to 0.78 for pedestrians and 0.69 for cyclists, false positives reached 41 per thousand frames — the system reacted to glare, reflections, and roadside bushes.
A labeling audit explained the cause: dark silhouettes went under a single class “pedestrian,” partially occluded and edge objects were not labeled at all, and night fragments in the control set were less than one tenth.
The class scheme was reworked — separate categories for people in dark clothing, silhouettes at crossings, and cyclists behind vehicles; additional night recordings were collected and a thermal imaging channel FLIR was added, where the silhouette is readable regardless of lighting.
After relabeling and retraining, pedestrian recall rose to 0.94, cyclist recall to 0.92, and false positives fell to 8 per thousand frames.
The share of night scenarios in the control set grew to 40%: metrics began to be calculated where failures were previously simply not noticed. Rollback to the previous build is available via checkpoints — a drop on a specific section can be removed without stopping tests.
What transfers to other projects
The workflow is reproducible: error analysis by scenario, then the class scheme, dataset balancing, and only then thresholds separately for each class. On other projects, it delivers comparable recall gains through data, not by complicating the model.
What to do if pedestrian detection is already deployed?
If detection is already deployed, a restart is not always needed. We distinguish three scenarios: improving an existing model on your own data, migrating it to a new compute loop, and rebuilding the processing pipeline when the sensor set or triggering logic changes.
A benchmark for assessment is the Euro NCAP scenarios for vulnerable road users: they show exactly which cases the model lacks.
| Option | When it fits | Limitations |
|---|---|---|
| Improving the existing model | Lack of night and rain frames, cyclists in traffic | Limited by labeling quality and volume |
| Model migration | New vehicle or platform, same sensors | Requires recalibration and on-site acceptance |
| Pipeline rebuild | Sensor set or triggering logic changes | Validation is built from scratch |
Improvement without stopping the system is the most common path. We retrain on labeling of rare cases, refine triggering thresholds, and separately check partially occluded pedestrians.
Compatibility with the current loop is checked quickly: the same input format, the same output interface — the model is swapped on the fly, and the behavior of the old and new versions is compared on the same recording.
A rebuild is necessary when the processing order changes: calibration, channel synchronization, rules for combining camera and lidar data. This is no longer a weight replacement, but a new pipeline with separate validation.
We make the choice based on constraints and timelines: we compare options on your dataset — share of missed pedestrians, false braking, and labeling volume for each.
Concerns about data, weather, and rare scenarios
On the road, what matters is not average accuracy, but model behavior in a rare frame: a pedestrian in dark clothing at the roadside, a cyclist emerging from behind a parked van, wet road markings after rain. Below are the concerns most often raised when defending a solution, and how we address each.
We have no labeling for rare situations — will the model simply not see them?
We collect the long tail in three ways: select difficult frames from recordings based on model uncertainty, rearrange scene participants and change the viewpoint on already labeled data, and supplement with recordings from problematic route sections. During training, rare classes receive increased weight so they do not drown in the overall flow.
We keep separate validation slices — a pedestrian behind an obstacle, a group at the roadside, an object at the frame edge — otherwise growth in the average metric masks failure on a rare class.
How does detection behave in rain, snow, and at night?
Each condition is checked by its own subgroup: wet asphalt produces glare and reflections that the model mistakes for objects; snow blurs boundaries and changes the silhouette; at night contrast drops and the share of headlight overexposure grows.
At the training stage, we add controlled distortions of brightness, contrast, blur, and noise, and calculate acceptance by condition subgroups, not a single overall number.
This way degradation is visible immediately and is closed in a targeted manner by additional training on the needed slice.
What happens when we retrain the model ourselves after project handover?
We hand over a description of the labeling scheme, training and validation scripts, a checklist for accepting new data, and versioning rules. The dataset and model are versioned, the previous build is retained — rollback is possible to any step.
The new run is compared on the same validation slice as the previous one, so it is clear what exactly improved and what dropped. If needed, we support the first retraining cycles and review disputed frames together with your team.
Let’s discuss the task and come back with an estimate and plan
To come back with a project estimate rather than general considerations, we need a minimum of inputs about your driving scenario. Send what you already have: at this step, completeness matters more than precision.
- purpose: where the vehicle drives — yards, parking lots, highways, mixed traffic;
- sensor recordings, including difficult cases — twilight, rain, occlusions, scooters, children;
- requirements for pedestrian and cyclist detection recall and for the acceptable false positive rate;
- constraints of the on-board compute unit: what is already installed on the platform;
- integration scheme — where the signal goes: to the planner, braking subsystem, telemetry;
- availability and quality of labeling, requirements for storing recordings.
In response, we send a document, not a chat thread: it includes the scope of work, a stage-by-stage work plan, stage timelines, and acceptance points with metrics.
The project estimate is based on your data, so we mark disputed points separately — it is immediately clear where access to a test site is needed.
- which object classes we detect and at what distances;
- how we validate the result: acceptance scenarios and metric thresholds;
- what we do with rare classes and labeling for them;
- how additional training is organized when new types of road users appear in traffic.
Describe the task — we will come back with an estimate and plan.
