D-Robotics Open Sources X-Lens, a 40 Million Parameter Model for Real Time Metric Depth Estimation
D-Robotics has open-sourced X-Lens, a compact metric depth model built for heterogeneous camera systems. Designed to handle both fisheye and pinhole cameras, X-Lens brings real-time, metric-scale 3D perception closer to practical robotic applications.
For a robot to understand the three dimensional world, it must first know how far away things are.
Humans estimate distance through binocular disparity. A robot can face a much more complex visual problem. To combine broad environmental awareness with long range detail, a single system may use several types of cameras at once. A fisheye camera can provide a field of view close to 180 degrees, but severe image distortion occurs near the edges. A pinhole camera preserves straight lines and distant detail, but covers a narrower field of view.
These cameras do not describe the world with the same projection geometry. The same wall can appear curved in a fisheye image and straight in a pinhole image. As a result, a depth model built around one camera type may struggle to align observations from the other.
Common solutions introduce difficult tradeoffs. Rectifying a fisheye image before inference can reduce both useful field of view and accuracy. Training a separate model for each camera type fragments perception across the robot. Large general geometry models can process a broader range of inputs, but their parameter counts and inference costs make real time deployment on a robot challenging.
D-Robotics has open sourced X-Lens to address this problem with a different design philosophy. Instead of using scale alone to absorb the differences among cameras, X-Lens represents their geometry explicitly.
Compact Metric Depth for Heterogeneous Camera Rigs
X-Lens is a compact feed forward model for metric depth estimation from a variable number of calibrated fisheye and pinhole views. It contains only 0.04 billion parameters and predicts dense depth together with a global metric scale in one pass.
This distinction matters. Many depth models estimate relative depth, which can determine that object A is farther away than object B but cannot state the physical distance between them. X-Lens produces metric depth. It can represent a wall as being 2.352 meters away, rather than simply placing it behind another object.
For robots, physical scale is essential. Manipulation requires the end effector to reach the correct location. Navigation requires the system to maintain a safe distance from obstacles. Spatial computing requires virtual content to remain anchored to real geometry. A depth map without reliable scale provides only part of the information these tasks need.
X-Lens accepts arbitrary mixtures of calibrated fisheye and pinhole views without requiring panorama conversion. According to deployment results from the D-Robotics team, the model processes a six camera mixed input at more than 20 frames per second on the D-Robotics RDK S600. A single fisheye input reaches 41 frames per second.
The model achieves this balance through three core design choices.
One Geometric Language for Different Cameras
Traditional image models reason primarily in pixel coordinates. This becomes problematic when different camera models project the same three dimensional point to very different image locations and shapes.
X-Lens moves the geometric reasoning into ray space. Each pixel is converted into a unit ray that describes its direction in three dimensional space. The network then reasons about geometry using these rays rather than assuming that pixel coordinates follow the same projection model.
For a pinhole camera, the input calibration can be derived from a standard 3 by 3 intrinsic matrix. For a fisheye camera, the system can use a lookup table that maps each pixel to a ray direction. The same formulation can also support cameras whose projection is not described by a simple closed form model.
This separation is important. Camera specific calibration remains at the input, while the network learns the relationship between rays and scene geometry. Changing a camera therefore changes the calibration input rather than the model architecture or its trained weights.
By preserving each view in its native image domain, X-Lens avoids the geometric resampling required by panoramic stitching. It can use the wide peripheral coverage of fisheye cameras and the distant detail of pinhole cameras within one shared metric representation.
Calibration Tokens for Fisheye Adaptation
A shared ray representation resolves part of the problem, but fisheye images still contain highly nonuniform distortion. Distortion is relatively mild near the center of the image and becomes much stronger near the boundary.
X-Lens introduces learnable calibration tokens to adapt fisheye features inside the Transformer. These tokens are injected into fisheye views at every layer, where they learn corrections associated with this type of projection. They remain local to the relevant view and are removed before the next layer receives a fresh set.
The tokens do not store the parameters of one particular lens. Those parameters are already represented by the input ray map. Instead, they learn a transferable method for adapting features affected by fisheye distortion.Pinhole views follow their original path through the network. This design prevents fisheye adaptation from contaminating the shared features used for pinhole geometry and helps the model retain strong performance across both camera types.
A Distortion Aware Prior Inside Attention
Visual similarity alone is not sufficient for matching features across heterogeneous cameras. A door frame near the edge of a fisheye image can look compressed and curved, while the same structure remains rectangular in a pinhole view.
X-Lens addresses this issue with a Jacobian parameterized distortion bias in the global cross view attention layers. The model estimates local projection behavior from the ray field. It considers ray orientation, local spatial expansion, and anisotropic stretching when calculating how strongly one patch should attend to another.
This geometric bias acts as an additional matching prior. The attention mechanism can consider both appearance and projection geometry, which improves correspondence between strongly distorted fisheye regions and perspective views.
The result is a compact architecture that models camera differences directly instead of asking a large backbone to memorize every possible lens configuration.
Progressive Training in Three Stages
The architecture is paired with a progressive training strategy that separates general geometric learning from fisheye adaptation and mixed camera fusion.
In the first stage, the base network is trained on multiview pinhole data. This establishes strong priors for depth, confidence, and metric scale before nonlinear fisheye effects are introduced.
In the second stage, the backbone, ray encoders, and prediction heads are frozen. Only the fisheye calibration tokens are optimized. This allows the model to learn lens specific corrections without overwriting the perspective geometry acquired during the first stage.
In the third stage, the complete network is trained on mixed fisheye and pinhole data. The Jacobian distortion bias is activated, and the model learns cross view interactions in the shared ray representation. Pure pinhole samples remain part of this stage to preserve multiview pinhole capability.
This sequence reduces interference between projection adaptation and cross camera fusion. Each stage has a clear objective, which helps a relatively small model use its capacity efficiently.
OmniScene Provides the Missing Training Data
Heterogeneous camera depth estimation also faces a data problem. Most existing fisheye datasets focus on driving scenes. Datasets that combine synchronized fisheye and pinhole views with dense metric depth are rare.
Real sensors cannot provide perfect supervision in every condition. LiDAR measurements are sparse. Depth cameras can fail under strong light or at long distances. Reconstruction based labels may also introduce small geometric errors in areas such as plain walls.
To provide accurate supervision at scale, the D-Robotics algorithm team created OmniScene, a synthetic dataset designed for heterogeneous camera perception.Its virtual camera rig contains four synchronized fisheye cameras with overlapping fields of view for full horizontal coverage, plus two pinhole cameras that capture longer range detail in the front and rear. Every frame includes calibration in the OpenCV convention, dense metric depth, validity masks, and sky indicators.
OmniScene contains approximately 266,000 synchronized six view frames, more than 1.7 million calibrated images, 564 motion sequences, and 103 indoor and outdoor scenes. The scenes cover residential, commercial, industrial, urban, and science fiction environments.
The dataset prioritizes geometric accuracy. Post processing removes degraded frames, including cases where the virtual camera intersects scene geometry, and applies consistency checks across views. Training and test data are separated by scene, so evaluation measures performance in environments not seen during training.
OmniScene is openly available on Hugging Face for research use.
Accuracy, Efficiency, and Real Time Performance
On OmniScene Full, the most demanding benchmark in the study, the input consists of four fisheye views and two pinhole views at a resolution of 504 by 798 pixels.
X-Lens achieved an absolute relative depth error of 0.1021. This is 25.4 percent lower than the strongest baseline reported in the paper. The model also achieved a scale absolute relative error of 0.1181 and an RMSE of 1.5993.
Compared with MapAnything on the same benchmark, X-Lens used 96.7 percent fewer parameters, reduced scale absolute relative error by 68.1 percent, and ran more than four times faster in the paper evaluation.
The model also remained competitive in homogeneous settings. It performed strongly on fisheye benchmarks including KITTI360 and on pinhole benchmarks including ETH3D and OmniOcc. Relative to DepthAnything3 Giant, X-Lens reduced the parameter count by 97.1 percent while directly predicting a global metric scale.
These results show that efficiency does not have to come from sacrificing camera coverage or metric consistency. A model designed around the geometry of the task can achieve a practical balance across accuracy, speed, and parameter count.
From Navigation to Robotic Manipulation
Any robot that must perceive distance while moving can benefit from metric depth. A cleaning robot needs the distance to furniture. A warehouse vehicle must navigate around obstacles. An augmented reality device must align digital objects with physical space. A surround perception system must maintain reliable coverage across wide and narrow fields of view.
The immediate role of X-Lens is real time perception for robots with heterogeneous camera inputs. The D-Robotics team has completed deployment with a VODP+ model, providing metric depth features for robotic manipulation and improving task fluency. In adaptation work with a Vision Action model, X-Lens also outperformed VGGT.
The next step is to support dynamic manipulation for humanoid robots, including loco manipulation in changing three dimensional environments.
The longer term roadmap extends beyond depth estimation. The team plans to add odometry output and explore new approaches to SLAM. The pretrained backbone may also serve as a geometry aware feature extractor for Vision Language Action models and other end to end manipulation systems.
A Design First Path to Spatial Intelligence
The current trend in three dimensional vision often favors larger models and broader reconstruction objectives. X-Lens takes a more focused path. It concentrates on the outputs that real time robotic perception needs, dense depth, confidence, and metric scale, then builds camera geometry into the architecture.
With 40 million parameters, native support for mixed calibrated cameras, real time inference, and openly released code and data, X-Lens offers a practical foundation for robots that need to see distance accurately as they move and act.