University of Wisconsin-Madison
LiDAR-based 3D sensors provide point clouds, a canonical 3D representation used in various 3D scene understanding tasks. Modern LiDARs face key challenges in various real-world scenarios such as long-distance or low-albedo objects, producing sparse or erroneous point clouds. These errors, which are rooted in the noisy raw LiDAR measurements, get propagated to downstream perception models, resulting in potentially severe loss of accuracy. This is because conventional 3D processing pipelines used to construct point clouds from raw LiDAR measurements do not retain the uncertainty information available in the raw sensor data. We propose a novel 3D scene representation called Probabilistic Point Clouds (PPC) where each point is augmented with a probability attribute that encapsulates the measurement uncertainty (confidence) in raw data. We further introduce inference approaches that leverage PPC for robust 3D object detection; these methods are versatile and can be used as computationally lightweight drop-in modules in 3D inference pipelines. We demonstrate, via both simulations and real captures, that PPC-based 3D inference methods outperform several baselines with LiDAR as well as camera-LiDAR fusion models, across challenging indoor and outdoor scenarios involving small, distant, and low-albedo objects, as well as strong ambient light.
. # MMdetection3d Code
.
.
├── tools/ppc_simulation/ # Code for Probabilistic Point Cloud Simulation
└── README.md- Follow the Installation steps for mmdetection3d framework.
matlabis required for PPC simulation.
- Follow the original dataset instructions to prepare clean point cloud dataset.
- Use
ppc_simulate.shto simulate 3D temporal waveforms.
cd tools/ppc_simulation
./ppc_simulate.sh 0 10- Use
gen_points.shto create probabilistic point clouds from the 3D waveforms.
./gen_points.sh 0 10- Use
create_pkl.pyto create label files for the whole dataset. It also creates a copy of clean point clouds to create ppc with probability 1.
python create_pkl.pyEdit the dataset field in the scripts to simulate for KITTI dataset. Increase 10 to the size of the dataset to simulate all scenes.
- Train PPC model using
ppc_train.shscript. Uncomment lines in the script to train all PPC models and baselines.
./ppc_train.sh- Evaluate PPC model using
ppc_test.shscript.
./ppc_test.sh| Method | AP@25 | Download | ||||
|---|---|---|---|---|---|---|
| Clean | 0.1 | 0.05 | 0.02 | 0.01 | ||
| Matched Filtering | 51.34 | 42.43 | 38.77 | 16.95 | 11.34 | model | log |
| Thresholding | 57.11 | 51.27 | 46.44 | 29.58 | 16.47 | model | log |
| PPC | 58.61 | 54.29 | 52.46 | 38.49 | 29.42 | model | log |
| Method | mAP | Download | ||||
|---|---|---|---|---|---|---|
| Clean | 0.05 | 0.02 | 0.01 | 0.005 | ||
| Matched Filtering | 60.11 | 55.76 | 50.03 | 47.06 | 37.01 | model | log |
| Thresholding | 58.63 | 57.72 | 54.80 | 49.23 | 38.62 | model | log |
| PPC | 60.62 | 59.12 | 59.04 | 55.39 | 49.51 | model | log |
Model weights will be updated soon.
📘Documentation | 🛠️Installation | 👀Model Zoo | 🆕Update News | 🚀Ongoing Projects | 🤔Reporting Issues
English | 简体中文
MMDetection3D is an open source object detection toolbox based on PyTorch, towards the next-generation platform for general 3D detection. It is a part of the OpenMMLab project.
The main branch works with PyTorch 1.8+.
Major features
-
Support multi-modality/single-modality detectors out of box
It directly supports multi-modality/single-modality detectors including MVXNet, VoteNet, PointPillars, etc.
-
Support indoor/outdoor 3D detection out of box
It directly supports popular indoor and outdoor 3D detection datasets, including ScanNet, SUNRGB-D, Waymo, nuScenes, Lyft, and KITTI. For nuScenes dataset, we also support nuImages dataset.
-
Natural integration with 2D detection
All the about 300+ models, methods of 40+ papers, and modules supported in MMDetection can be trained or used in this codebase.
-
High efficiency
It trains faster than other codebases. The main results are as below. Details can be found in benchmark.md. We compare the number of samples trained per second (the higher, the better). The models that are not supported by other codebases are marked by
✗.Methods MMDetection3D OpenPCDet votenet Det3D VoteNet 358 ✗ 77 ✗ PointPillars-car 141 ✗ ✗ 140 PointPillars-3class 107 44 ✗ ✗ SECOND 40 30 ✗ ✗ Part-A2 17 14 ✗ ✗
Like MMDetection and MMCV, MMDetection3D can also be used as a library to support different projects on top of it.
In version 1.4, MMDetecion3D refactors the Waymo dataset and accelerates the preprocessing, training/testing setup, and evaluation of Waymo dataset. We also extends the support for camera-based, such as Monocular and BEV, 3D object detection models on Waymo. A detailed description of the Waymo data information is provided here.
Besides, in version 1.4, MMDetection3D provides Waymo-mini to help community users get started with Waymo and use it for quick iterative development.
v1.4.0 was released in 8/1/2024：
- Support the training of DSVT in
projects - Support Nerf-Det in
projects - Refactor Waymo dataset
v1.3.0 was released in 18/10/2023:
- Support CENet in
projects - Enhance demos with new 3D inferencers
v1.2.0 was released in 4/7/2023
- Support New Config Type in
mmdet3d/configs - Support the inference of DSVT in
projects - Support downloading datasets from OpenDataLab using
mim
v1.1.1 was released in 30/5/2023:
- Support TPVFormer in
projects - Support the training of BEVFusion in
projects - Support lidar-based 3D semantic segmentation benchmark
Please refer to Installation for installation instructions.
For detailed user guides and advanced guides, please refer to our documentation:
User Guides
Advanced Guides
Results and models are available in the model zoo.
| Backbones | Heads | Features |
|
| LiDAR-based 3D Object Detection | Camera-based 3D Object Detection | Multi-modal 3D Object Detection | 3D Semantic Segmentation |
|
|
|
|
| ResNet | VoVNet | Swin-T | PointNet++ | SECOND | DGCNN | RegNetX | DLA | MinkResNet | Cylinder3D | MinkUNet | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SECOND | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| PointPillars | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| FreeAnchor | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| VoteNet | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| H3DNet | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| 3DSSD | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Part-A2 | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| MVXNet | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| CenterPoint | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| SSN | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| ImVoteNet | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| FCOS3D | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| PointNet++ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Group-Free-3D | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| ImVoxelNet | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| PAConv | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| DGCNN | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| SMOKE | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| PGD | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| MonoFlex | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ |
| SA-SSD | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| FCAF3D | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| PV-RCNN | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| Cylinder3D | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ |
| MinkUNet | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| SPVCNN | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| BEVFusion | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| CenterFormer | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| TR3D | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ |
| DETR3D | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| PETR | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| TPVFormer | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
Note: All the about 500+ models, methods of 90+ papers in 2D detection supported by MMDetection can be trained or used in this codebase.
Please refer to FAQ for frequently asked questions.
We appreciate all contributions to improve MMDetection3D. Please refer to CONTRIBUTING.md for the contributing guideline.
MMDetection3D is an open source project that is contributed by researchers and engineers from various colleges and companies. We appreciate all the contributors as well as users who give valuable feedbacks. We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their own new 3D detectors.
If you find this project useful in your research, please consider cite:
@misc{mmdet3d2020,
title={{MMDetection3D: OpenMMLab} next-generation platform for general {3D} object detection},
author={MMDetection3D Contributors},
howpublished = {\url{https://github.com/open-mmlab/mmdetection3d}},
year={2020}
}This project is released under the Apache 2.0 license.
- MMEngine: OpenMMLab foundational library for training deep learning models.
- MMCV: OpenMMLab foundational library for computer vision.
- MMEval: A unified evaluation library for multiple machine learning libraries.
- MIM: MIM installs OpenMMLab packages.
- MMPreTrain: OpenMMLab pre-training toolbox and benchmark.
- MMDetection: OpenMMLab detection toolbox and benchmark.
- MMDetection3D: OpenMMLab's next-generation platform for general 3D object detection.
- MMRotate: OpenMMLab rotated object detection toolbox and benchmark.
- MMYOLO: OpenMMLab YOLO series toolbox and benchmark.
- MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark.
- MMOCR: OpenMMLab text detection, recognition, and understanding toolbox.
- MMPose: OpenMMLab pose estimation toolbox and benchmark.
- MMHuman3D: OpenMMLab 3D human parametric model toolbox and benchmark.
- MMSelfSup: OpenMMLab self-supervised learning toolbox and benchmark.
- MMRazor: OpenMMLab model compression toolbox and benchmark.
- MMFewShot: OpenMMLab fewshot learning toolbox and benchmark.
- MMAction2: OpenMMLab's next-generation action understanding toolbox and benchmark.
- MMTracking: OpenMMLab video perception toolbox and benchmark.
- MMFlow: OpenMMLab optical flow toolbox and benchmark.
- MMagic: OpenMMLab Advanced, Generative and Intelligent Creation toolbox.
- MMGeneration: OpenMMLab image and video generative models toolbox.
- MMDeploy: OpenMMLab model deployment framework.