BeyondSight: Object Permanence for End-to-End Autonomous Driving

Overview

Object permanence is fundamental to human cognition, but still an open challenge for robotic systems. BeyondSight formalizes learning object permanence for end-to-end autonomous driving.

When actors are occluded, end-to-end driving models silently drop them — removing potentially critical agents from prediction and planning. BeyondSight treats object permanence as a first-class requirement: actor hypotheses persist through occlusion via temporal propagation and are fused with incoming observations. We also introduce nuScenes-Permanence, the first benchmark to supervise and evaluate occluded actors in end-to-end driving.

0 → 0.249
mAPunobs — unobservable actor detection, vs. SparseDrive
0.61 → 0.54
L2avg planning error — safer plans near occluded actors
+30%
more annotations in nuScenes-Permanence (932k → 1.33M boxes)

SparseDrive drops actor during occlusion

(a) Existing stacks drop actor hypotheses once occluded.

BeyondSight maintains persistent actor hypothesis

(b) BeyondSight maintains a persistent representation.

Figure 1. A pedestrian near a crosswalk becomes occluded. SparseDrive drops the actor; BeyondSight keeps it in the scene representation throughout.

Method

BeyondSight extends SparseDrive with a three-stage actor belief update that separates temporal propagation from observation updates — keeping unobservable actors in the scene without forcing the visual branch to hallucinate non-visible evidence. The extra decoders add <5% parameters.

①

Temporal Prior

Propagates actor queries using motion-conditioned self-attention — no image features. Keeps hypotheses alive during occlusion.

②

Observation Update

Combines the prior with new queries and attends to current BEV features. Supervised only on observable actors.

③

Posterior Fusion

Merges propagated and observation-conditioned hypotheses. The unified set feeds motion prediction and planning.

Figure 2. BeyondSight overview. The three-stage actor belief update mirrors Bayesian filtering: temporal prior → observation update → posterior fusion.

nuScenes-Permanence

Standard benchmarks penalize predictions of occluded actors as false positives — discouraging any persistence at all. nuScenes-Permanence fixes this with two contributions:

Annotation extension. Unobservable actor states are reconstructed offline via interpolation and short-horizon extrapolation, expanding nuScenes from 932k to 1.33M boxes.

Observability-conditioned evaluation. Ground truth is split into observable and unobservable subsets with symmetric ignore rules, reporting mAP_obs, mAP_unobs, and mAP_all — preventing cross-penalization.

Occlusion duration distribution

Ego distance at occlusion onset

Figure 4. Occlusion is common and close. Full-sensor absence occurs frequently across all distances — not just rare edge cases.

Figure 5. Generated annotations (zero-point, interpolated, extrapolated) across urban, intersection, and highway scenes. Boxes are colored by observability status.

Results

BeyondSight improves across the board on standard nuScenes — and dramatically on nuScenes-Permanence — without touching the planner.

Standard nuScenes

Method	mAP ↑	NDS ↑	AMOTA ↑	L2_avg ↓	CR_avg ↓
UniAD	—	—	—	1.03	0.31%
SparseDrive	0.415	0.526	0.372	0.61	0.08%
BridgeAD	0.423	0.534	0.398	0.58	0.08%
MomAD	0.423	0.531	0.391	0.60	0.09%
BeyondSight (ours)	0.427	0.536	0.401	0.54	0.07%

nuScenes-Permanence

Method	mAP_all ↑	mAP_obs ↑	mAP_unobs ↑	L2_avg ↓
SparseDrive	0.389	0.415	0.000	0.61
BeyondSight (ours)	0.413	0.421	0.249	0.54

Planning under occlusion

When an actor becomes fully occluded, SparseDrive drops it — and plans through it. BeyondSight maintains a persistent hypothesis and routes around it. On a high-occlusion subset, collision rate drops from 0.163 → 0.08 CR_avg.

Figure 3. Left: SparseDrive drops the hidden actor and plans through its future path. Right: BeyondSight maintains the hypothesis and produces a safer trajectory.

BibTeX

@inproceedings{papais2026beyondsight,
  title     = {BeyondSight: Object Permanence for End-to-End Autonomous Driving},
  author    = {Papais, Sandro and Wang, Letian and Jain, Mudit and Rezaei, Behnaz and Waslander, Steven L.},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}