ECCV 2026

PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving

Pin Tang1, Guoqing Wang1, Xiangxuan Ren1, Zhongdao Wang2, Guodongfang Zhao2, Bailan Feng2, Chao Ma1
1Shanghai Jiao Tong University, 2Huawei

PixelPilot decouples image-space planning from metric-space lifting, enabling scalable VLA training across heterogeneous driving datasets while grounding planning in visual reasoning.

Paper Coming Soon Code Coming Soon Video BibTeX
PixelPilot decoupled planning and lifting overview.
Abstract

2D-first planning for scalable driving VLAs

Vision-Language-Action Models leverage the reasoning capabilities of VLMs, but existing driving VLAs typically predict 3D trajectories directly from 2D images. This couples learning with camera parameters, limiting data scalability across heterogeneous driving datasets and encouraging trivial ego-status shortcuts.

PixelPilot introduces a decoupled planning and lifting paradigm. It reformulates scene understanding and trajectory prediction as sensor-agnostic 2D-to-2D tasks in the image plane, then deterministically lifts planned 2D trajectories to 3D only during inference. A knowledge-instilled policy learning strategy further applies dense intermediate rewards through GRPO, encouraging a causal chain from perception to reasoning, meta-action, and final spatial planning.

0.30m Avg. L2 error on nuScenes open-loop planning
79.14 Driving score on Bench2Drive closed-loop evaluation
58.87% Success rate on Bench2Drive in CARLA
54.2 2D object detection mAP on nuScenes
Method

Decoupled planning and lifting

PixelPilot keeps the learnable VLA policy in the image plane and moves sensor-specific geometry into a deterministic inference-time lifting step.

Sensor-agnostic planning

Perception, visual reasoning, meta-action prediction, and waypoint planning are optimized as 2D image-space tasks.

Deterministic lifting

Predicted 2D trajectories are projected into 3D with the target vehicle's camera parameters during inference.

Visual grounding

Reasoning with 2D bounding boxes anchors the final plan to visible agents and scene structure.

PixelPilot decoupled planning and lifting paradigm.
The planning phase is scalable because it operates entirely in image space. The unscalable 2D-to-3D geometry is handled only by deterministic lifting at inference time.
Local plane and image-space obstacle justification.
Under a short-horizon local-plane approximation, image-plane trajectories correspond to feasible road-plane trajectories, while image-space obstacles provide conservative visible collision cues.
Training

Knowledge-instilled policy learning

PixelPilot first learns foundational driving behavior through multi-task SFT, then refines the policy with GRPO using dense rewards on verifiable intermediate outputs.

PixelPilot training pipeline.
Dense rewards are assigned to format, perception, meta-action, and trajectory outputs, while free-form reasoning remains flexible.
Ego-centric consistency preprocessing for multi-view images.
Ego-centric consistency preprocessing produces a continuous multi-view planning surface by rotating the back view, avoiding lane misalignment and discontinuous trajectories.
Results

Open-loop and closed-loop performance

PixelPilot achieves state-of-the-art open-loop planning performance on nuScenes and the best overall driving score among compared methods on Bench2Drive.

Method nuScenes Avg. L2 ↓ Avg. Collision ↓ Avg. Intersection ↓ Bench2Drive Score ↑ Success Rate ↑
OmniDrive 0.33 0.30 3.00 - -
OpenDriveVLA 0.33 0.10 - - -
AutoVLA 0.40 0.20 - 78.84 57.73%
Orion - - - 77.74 54.62%
PixelPilot 0.30 0.25 1.77 79.14 58.87%

Open-loop metrics are reported on nuScenes. Closed-loop metrics are reported on Bench2Drive in CARLA.

Method Ego-Status Images Avg. L2 ↓
OmniDrive No Yes 1.98
OmniDrive Yes No 0.36
Imprompt-VLA No Yes 2.39
Imprompt-VLA Yes No 0.36
PixelPilot No Yes 0.71
PixelPilot Yes No 0.90

Unlike direct 3D prediction VLAs, PixelPilot degrades less when ego-status is removed than when images are removed, indicating stronger reliance on visual cues.

Demo

PixelPilot in motion

The demo video shows PixelPilot's image-space reasoning and trajectory planning behavior in autonomous driving scenes.

PixelPilot demo video. The browser controls can be used to pause, seek, or replay the clip.
Visualization

Image-space trajectories

PixelPilot produces reliable 2D trajectories that closely align with ground truth across turning and interaction-heavy scenes.

Qualitative trajectory comparison on nuScenes.
Qualitative comparison among Qwen2.5-VL-7B, OmniDrive, and PixelPilot. Predicted trajectories are shown in blue and ground-truth trajectories in green.

BibTeX

@inproceedings{tang2026pixelpilot,
  title     = {PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving},
  author    = {Tang, Pin and Wang, Guoqing and Ren, Xiangxuan and Wang, Zhongdao and Zhao, Guodongfang and Feng, Bailan and Ma, Chao},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}