3D Object Detection

Table of Contents

• Overview
• Baselines
• Task Definition
• 3D Object Detection Taxonomy
  • Metrics
    • Average Precision
    • True Positive Metrics
      • Average Translation Error (ATE)
      • Average Scale Error (ASE)
      • Average Orientation Error (AOE)
    • Composite Detection Score (CDS)
• Evaluation

Overview

The Argoverse 3D Object Detection task differentiates itself with its 26 category taxonomy and long-range (150 m) detection evaluation. We detail the task, metrics, evaluation protocol, and detailed object taxonomy information below.

Baselines

• https://github.com/neeharperi/LT3D

Task Definition

For a unique tuple, (log_id, timestamp_ns), produce a ranked set of predictions $\mathcal{P}$ that describe an object’s location, size, and orientation in the 3D scene:

$$\begin{array}{rl}\mathcal{P}& ={\left\{x_{\text{ego}}^i,y_{\text{ego}}^i,z_{\text{ego}}^i,l_{\text{obj}}^i,w_{\text{obj}}^i,h_{\text{obj}}^i,{\theta }_{\text{obj}}^i,c^i,o^i\right\}}_{i=1}^N\text{where},\\ & \\ x_{\text{ego}}^i& :\text{Location along the x-axis in the ego-vehicle reference frame.}\\ y_{\text{ego}}^i& :\text{Location along the y-axis in the ego-vehicle reference frame.}\\ z_{\text{ego}}^i& :\text{Location along the z-axis in the ego-vehicle reference frame.}\\ l_{\text{obj}}^i& :\text{Extent along the x-axis in the object reference frame.}\\ w_{\text{obj}}^i& :\text{Extent along the y-axis in the object reference frame.}\\ h_{\text{obj}}^i& :\text{Extent along the z-axis in the object reference frame.}\\ {\theta }_{\text{obj}}^i& :\text{Counter clockwise rotation from the x-axis in the object reference frame.}\\ c^i& :\text{Predicted likelihood.}\\ o^i& :\text{Categorical object label.}\end{array}$$

3D Object Detection Taxonomy

Metrics

All of our reported metrics require assigning predictions to ground truth annotations written as $a_{\text{pd},\text{gt}}$ to compute true positives (TP), false positives (FP), and false negatives (FN).

Formally, we define a true positive as:

$${\text{TP}}_{\text{pd},\text{gt}}=\left\{a_{\text{pd},\text{gt}}:\Vert v_{\text{pd}}-v_{\text{gt}}{\Vert }_2\le d\right\},$$

where $d$ is a distance threshold in meters.

Average Precision

Average precision measures the area underneath the precision / recall curve across different true positive distance thresholds.

$$\begin{array}{rl}\text{AP}& =\frac{1}{100}\sum _{d\in \mathcal{D}}\sum _{r\in \mathcal{R}}{\text{p}}_{\text{interp}}(r)\text{where}\\ \mathcal{D}& =\left\{0.5\text{ m},1.0\text{ m},2.0\text{ m},4.0\text{ m}\right\}\\ \mathcal{R}& =\left\{0.01,0.02,\dots ,1.00\right\}\end{array}$$

True Positive Metrics

All true positive metrics are at a $2\text{ m}$ threshold.

Average Translation Error (ATE)

ATE measures the distance between true positive assignments.

$$\begin{array}{r}\text{ATE}=\Vert t_{\text{pd}}-t_{\text{gt}}{\Vert }_2\text{where}{t}_{\text{pd}}\in {\mathbb{R}}^3,t_{\text{gt}}\in {\mathbb{R}}^3.\end{array}$$

Average Scale Error (ASE)

ASE measures the shape misalignent for true positive assignments.

$$\begin{array}{r}\text{ASE}=1-\prod _{d\in \mathcal{D}}\frac{\min (d_{\text{pd}},d_{\text{gt}})}{\max (d_{\text{pd}},d_{\text{gt}})}.\end{array}$$

Average Orientation Error (AOE)

AOE measures the minimum angle between true positive assignments.

$$\begin{array}{r}\text{AOE}=|{\theta }_{\text{pd}}-{\theta }_{\text{gt}}|\text{where}{\theta }_{\text{pd}}\in [0,\pi )\text{ and }{\theta }_{\text{gt}}\in [0,\pi ).\end{array}$$

Composite Detection Score (CDS)

CDS measures the overall performance across all previously introduced metrics.

$$\begin{array}{rl}\text{CDS}& =\text{AP}\cdot \sum _{x\in \mathcal{X}}1-x,\\ \mathcal{X}& =\{{\text{ATE}}_{\text{unit}},{\text{ASE}}_{\text{unit}},{\text{AOE}}_{\text{unit}}\},\end{array}$$

where $\{{\text{ATE}}_{\text{unit}},{\text{ASE}}_{\text{unit}},{\text{AOE}}_{\text{unit}}\}$ are the normalized true positive errors.

Evaluation

The 3D object detection evaluation consists of the following steps:

1. Partition the predictions and ground truth objects by a unique id, (log_id: str, timestamp_ns: uint64), which corresponds to a single sweep.

2. For the predictions and ground truth objects associated with a single sweep, greedily assign the predictions to the ground truth objects in descending order by likelihood.

3. Compute the true positive, false positive, and false negatives.

4. Compute the true positive metrics.

5. True positive, false positive, and false negative computation.