metrics

Accuracy of a confusion matrix.

Confusion matrices add up, so the matrix may describe one batch or the sum of confusion_matrix over a whole split.

Parameters:

• cm (Tensor) –

  Confusion matrix with true classes as rows, shape \((C, C)\) (see confusion_matrix).

• average (Literal['micro', 'macro', 'none'], default: 'micro' ) –

  "micro" returns the overall accuracy (the fraction of points on the diagonal); "macro" returns the mean class accuracy (the mean of the per-class recalls); "none" returns the per-class accuracy.

• ignore_index (Union[int, Sequence[int], None], default: None ) –

  Class index, or indices, to ignore: points whose true class is ignored are dropped, and the ignored classes are left out of the mean. Indices outside \([0, C)\) have no effect.

• zero_division (float, default: 0.0 ) –

  Accuracy given to a class without any point (and to an empty matrix with "micro").

• class_names (Optional[Sequence[str]], default: None ) –

  Name of each class index; with average="none" the per-class accuracy comes back as a {name: accuracy} dict instead of a tensor.

Returns:

• Union[float, Tensor, Dict[str, float]] –

  The accuracy as a float with average="micro" or "macro", or the per-class accuracy, shape \((C,)\),

• Union[float, Tensor, Dict[str, float]] –

  with average="none" (a {name: accuracy} dict when class_names is given).

>>> cm = confusion_matrix(torch.tensor([0, 1, 1, 1]), torch.tensor([0, 1, 0, 1]), num_classes=2)
>>> accuracy(cm), accuracy(cm, average="macro")
(0.75, 0.75)
>>> accuracy(cm, average="none")
tensor([0.5000, 1.0000])
>>> accuracy(cm, average="none", class_names=["wall", "floor"])
{'wall': 0.5, 'floor': 1.0}

Compute the confusion matrix.

Parameters:

• preds (Tensor) –

  Predicted class indices, shape \((N,)\).

• target (Tensor) –

  Ground truth class indices, shape \((N,)\).

• num_classes (int) –

  Total number of classes.

• ignore_index (Optional[int], default: None ) –

  Class index to exclude from computation.

Returns:

• Tensor –

  Confusion matrix of shape \((\text{num\_classes}, \text{num\_classes})\) where

• Tensor –

  cm[i, j] is the number of points with true class i

• Tensor –

  predicted as class j.

3D detection average precision (AP) of matched box predictions.

Dataset- and model-agnostic: box_matches reduces any detector's packed (boxes, scores, labels, batch) output to the same record, one per batch, and the records of a whole split are scored together. Within a class, predictions claim their best ground-truth box in descending score order: the first one above the IoU threshold is a true positive, the others are false positives, and the AP is the area under the resulting precision-recall curve.

With one iou_threshold for every class, the classes that have ground truth are scored. With one per class index, exactly those classes are scored, and one without ground truth scores \(0\) (its predictions are all false positives).

Parameters:

• matches (Sequence[BoxMatches]) –

  The box_matches record of every evaluated batch.

• iou_threshold (Union[float, Mapping[int, float]], default: 0.5 ) –

  IoU a match must exceed: one value for every class (e.g. 0.25), or one per class index (e.g. KITTI's {0: 0.7, 1: 0.5, 2: 0.5}).

• average (Literal['macro', 'none'], default: 'macro' ) –

  "macro" returns the mean AP (mAP) over the scored classes; "none" returns the per-class AP.

• num_classes (Optional[int], default: None ) –

  Number of classes, i.e. the length of the average="none" output; defaults to the number of class_names, else to the largest class index met plus one.

• class_names (Optional[Sequence[str]], default: None ) –

  Name of each class index; with average="none" the per-class AP comes back as a {name: ap} dict instead of a tensor.

• interpolation (Interpolation, default: 'all' ) –

  AP interpolation: "all" integrates the full precision-recall curve; "r11" / "r40" sample the KITTI 11- / 40-point recall grids.

Returns:

• Union[float, Tensor, Dict[str, float]] –

  The mAP as a float with average="macro" (\(0\) when no class is scored), or the per-class AP, shape

• Union[float, Tensor, Dict[str, float]] –

  \((C,)\) float64, with average="none" (a {name: ap} dict when class_names is given), holding NaN for

• Union[float, Tensor, Dict[str, float]] –

  the classes that are not scored.

>>> boxes = torch.tensor([[0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 0.0], [9.0, 9.0, 9.0, 1.0, 1.0, 1.0, 0.0]])
>>> labels, batch = torch.tensor([0, 1]), torch.tensor([0, 0])
>>> preds = {"boxes": boxes, "scores": torch.tensor([0.9, 0.4]), "labels": labels, "batch": batch}
>>> matches = [box_matches(preds, {"boxes": boxes[:1], "labels": labels[:1], "batch": batch[:1]})]
>>> average_precision3d(matches, iou_threshold=0.25)
1.0
>>> average_precision3d(matches, iou_threshold={0: 0.7, 1: 0.5}, average="none")
tensor([1., 0.], dtype=torch.float64)
>>> average_precision3d(matches, average="none", class_names=["Car", "Cyclist"])
{'Car': 1.0, 'Cyclist': nan}

Match one batch of box predictions to its ground truth, the record scored by average_precision3d.

Which ground-truth box a prediction overlaps most, and by how much, depends on neither the IoU threshold nor on what the other predictions matched. The oriented IoU is therefore paid here, once per batch and on the device of the inputs, and nothing box-sized outlives the call: a whole validation split accumulates as a few flat tensors per batch, so keep one record per batch in a list and score the list.

Target boxes flagged by ignore_mask (see Boxes3D) are ignore regions rather than ground truth; their labels entry names the class they excuse. Predictions flagged by their own ignore_mask are left out of the record, so they can neither match a box nor count as a false positive (the KITTI min-height rule).

Parameters:

• preds (Detection3D) –

  Packed predictions (one decode output), {"boxes", "scores", "labels", "batch"} with an optional ignore_mask.

• target (Boxes3D) –

  Packed ground truth aligned to preds, {"boxes", "labels", "batch"} with an optional ignore_mask.

Returns:

• BoxMatches –

  The BoxMatches record of the batch, as CPU tensors.

>>> boxes = torch.tensor([[0.0, 0.0, 0.0, 2.0, 2.0, 2.0, 0.0], [9.0, 9.0, 9.0, 1.0, 1.0, 1.0, 0.0]])
>>> index = torch.zeros(2, dtype=torch.long)
>>> preds = {"boxes": boxes, "scores": torch.tensor([0.9, 0.4]), "labels": index, "batch": index}
>>> match = box_matches(preds, {"boxes": boxes[:1], "labels": index[:1], "batch": index[:1]})
>>> match["pred_iou"].tolist(), match["pred_gt"].tolist()
([1.0, 0.0], [0, 0])
>>> average_precision3d([match], iou_threshold=0.5)
1.0

Point-mask instance-segmentation average precision (AP) of matched instance predictions.

Follows the standard indoor instance-segmentation protocol (the ScanNet benchmark): per class and IoU threshold, ground-truth instances greedily consume overlapping predicted masks above the threshold, duplicates on a matched instance count as false positives with the lower score, and an unmatched prediction whose void / small-instance point fraction exceeds the threshold is excused. The AP integrates the score-swept precision-recall curve with centered recall steps, and is averaged over the IoU thresholds: the default sweep \(0.5, 0.55, \ldots, 0.9\) is the benchmark's headline AP, iou_threshold=0.5 and 0.25 its AP50 and AP25. A class is scored when it has ground truth.

Parameters:

• matches (Sequence[InstanceMatches]) –

  The instance_matches record of every evaluated scene.

• iou_threshold (Union[float, Sequence[float]], default: (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9) ) –

  IoU a match must exceed, or a sequence of them to average the AP over.

• average (Literal['macro', 'none'], default: 'macro' ) –

  "macro" returns the mean AP (mAP) over the scored classes; "none" returns the per-class AP.

• num_classes (Optional[int], default: None ) –

  Number of instance classes, i.e. the length of the average="none" output; defaults to the number of class_names, else to the largest class index met plus one.

• class_names (Optional[Sequence[str]], default: None ) –

  Name of each class index; with average="none" the per-class AP comes back as a {name: ap} dict instead of a tensor.

• min_points (int, default: 100 ) –

  Minimum point count for a prediction or ground-truth instance to be scored; smaller ground-truth instances count as ignore regions.

Returns:

• Union[float, Tensor, Dict[str, float]] –

  The mAP as a float with average="macro" (\(0\) when no class is scored), or the per-class AP, shape

• Union[float, Tensor, Dict[str, float]] –

  \((C,)\) float64, with average="none" (a {name: ap} dict when class_names is given), holding NaN

• Union[float, Tensor, Dict[str, float]] –

  for the classes without ground truth.

>>> masks = torch.tensor([[True, True, True, False], [False, False, False, True]])
>>> match = instance_matches(
...     masks,
...     torch.tensor([0, 1]),
...     torch.tensor([0.9, 0.8]),
...     torch.tensor([0, 0, 0, 1]),
...     torch.tensor([0, 0, 0, 1]),
... )
>>> instance_average_precision([match], min_points=1)
1.0
>>> instance_average_precision([match], iou_threshold=0.5, average="none", class_names=["chair", "table"], min_points=1)
{'chair': 1.0, 'table': 1.0}

Reduce one scene's instance predictions to the compact match record scored by instance_average_precision.

Predicted masks are dense \((K, N)\) booleans: a few hundred masks over a \(\sim 50\text{k}\)-point scene is only tens of MB and intersections reduce to bincounts, while index lists would be ragged and no smaller. The returned record holds per-instance counts and pairwise intersections only, so nothing mask-sized outlives the call and a whole validation split can be accumulated scene by scene.

Ground truth instances are the unique gt_instance ids among points with a non-negative id and a valid semantic label; points whose gt_label equals ignore_index are void, and predictions overlapping them are excused accordingly during scoring. Each instance must carry a single semantic label. Intersections are recorded for same-class (prediction, instance) pairs only.

Parameters:

• pred_masks (Tensor) –

  Per-instance point masks, shape \((K, N)\) bool.

• pred_labels (Tensor) –

  Per-instance class indices, shape \((K,)\).

• pred_scores (Tensor) –

  Per-instance confidences, shape \((K,)\).

• gt_instance (Tensor) –

  Per-point ground-truth instance ids, shape \((N,)\); negative marks no instance.

• gt_label (Tensor) –

  Per-point semantic labels in the instance-class space, shape \((N,)\).

• ignore_index (int, default: -1 ) –

  Semantic label marking void points.

Returns:

• InstanceMatches –

  The InstanceMatches record of the scene, as CPU tensors.

>>> masks = torch.tensor([[True, True, True, True]])
>>> match = instance_matches(
...     masks, torch.tensor([0]), torch.tensor([0.9]), torch.tensor([0, 0, 1, -1]), torch.tensor([0, 0, 0, -1])
... )
>>> match["gt_counts"].tolist(), match["pair_inter"].tolist(), match["pred_void"].tolist()
([2, 1], [2, 1], [1])

The nuScenes detection metrics: per-class AP, mAP, the five TP errors and the NDS.

Follows the official protocol of the nuScenes benchmark (nuScenes: A Multimodal Dataset for Autonomous Driving). Predictions are matched per sample and class by BEV center distance: in descending score order each prediction greedily takes the closest still-unmatched ground-truth box strictly below the threshold. AP interpolates precision at 101 recall points \(0.00, 0.01, \ldots, 1.00\), drops recalls up to min_recall, subtracts min_precision, clamps at \(0\), averages and rescales by the remaining precision span; mAP averages over class_names and dist_thresholds. The TP errors ATE (BEV center distance), ASE ($1 - $ IoU of center- and yaw-aligned boxes), AOE (absolute yaw difference, modulo \(\pi\) for barrier), AVE (L2 xy-velocity difference) and AAE ($1 - $ attribute accuracy) average the cumulative-mean error curve of the tp_threshold matches from min_recall to the highest achieved recall; a class without matches scores the full error of \(1\). The officially excluded pairs (traffic_cone: AOE/AVE/AAE, barrier: AVE/AAE) are left out of the per-metric means, and \(\text{NDS} = (5 \cdot \text{mAP} + \sum_\text{tp} (1 - \min(1, \text{err}))) / 10\).

Boxes are filtered before scoring: each sample keeps its max_boxes_per_sample highest-scoring predictions, boxes farther from the sensor origin (BEV) than their class range are dropped on both sides, and ground-truth boxes with gt_num_points == 0 are removed. When velocity columns or attributes are absent (on either side), AVE / AAE fall back to the full penalty of \(1.0\) per class.

Parameters:

• pred_boxes (Tensor) –

  Predicted boxes \((M, 7)\) of \((c_x, c_y, c_z, d_x, d_y, d_z, \theta)\), or \((M, 9)\) with \((v_x, v_y)\) velocity columns appended.

• pred_scores (Tensor) –

  Per-box confidence, shape \((M,)\).

• pred_labels (Tensor) –

  Per-box class index into class_names, shape \((M,)\).

• pred_batch (Tensor) –

  Per-box sample index, shape \((M,)\).

• gt_boxes (Tensor) –

  Ground-truth boxes \((K, 7)\) or \((K, 9)\), like pred_boxes.

• gt_labels (Tensor) –

  Per-box class index into class_names, shape \((K,)\).

• gt_batch (Tensor) –

  Per-box sample index, shape \((K,)\).

• class_names (Sequence[str]) –

  Class name per label index; barrier and traffic_cone get their official special handling by name.

• gt_num_points (Optional[Tensor], default: None ) –

  Optional per-box point count, shape \((K,)\); boxes with exactly \(0\) points are removed (unknown counts of \(-1\) are kept).

• pred_attributes (Optional[Tensor], default: None ) –

  Optional per-box attribute id, shape \((M,)\). Without it AAE is \(1.0\).

• gt_attributes (Optional[Tensor], default: None ) –

  Optional per-box attribute id, shape \((K,)\); a negative id marks a box without an attribute, which is skipped in the AAE mean. Without it AAE is \(1.0\).

• class_ranges (Optional[Mapping[str, float]], default: None ) –

  Maximum BEV evaluation range per class name; defaults to the official ranges (50 m car/truck/bus/trailer/construction_vehicle, 40 m pedestrian/motorcycle/bicycle, 30 m traffic_cone/barrier). A name missing from the mapping is not range-filtered.

• dist_thresholds (Sequence[float], default: (0.5, 1.0, 2.0, 4.0) ) –

  Matching thresholds in meters the AP is averaged over.

• tp_threshold (float, default: 2.0 ) –

  Matching threshold in meters of the TP-error metrics.

• max_boxes_per_sample (int, default: 500 ) –

  Per-sample cap on scored predictions (highest scores kept).

• min_recall (float, default: 0.1 ) –

  Recall up to which the AP and TP-error curves are clipped.

• min_precision (float, default: 0.1 ) –

  Precision subtracted before the AP mean.

Returns:

• Dict[str, float] –

  A flat dict with AP/<class> (averaged over dist_thresholds), mAP, mATE, mASE, mAOE,

• Dict[str, float] –

  mAVE, mAAE and NDS.

>>> zero = torch.tensor([0])
>>> pred_boxes = torch.tensor([[0.25, 0.0, 0.0, 4.0, 2.0, 1.5, 0.0]])
>>> gt_boxes = torch.tensor([[0.0, 0.0, 0.0, 4.0, 2.0, 1.5, 0.0]])
>>> metrics = nuscenes_detection_metrics(
...     pred_boxes, torch.tensor([0.9]), zero, zero, gt_boxes, zero, zero, class_names=["car"]
... )
>>> f"{metrics['AP/car']:.2f} {metrics['mATE']:.2f} {metrics['NDS']:.3f}"
'1.00 0.25 0.775'

Intersection over Union (IoU, the Jaccard index) of a confusion matrix.

Confusion matrices add up, so the matrix may describe one batch or the sum of confusion_matrix over a whole split. With average="macro" a class absent from the whole matrix (zero union) counts as zero_division, matching sklearn's jaccard_score(zero_division=0); toolboxes that average only over present classes report a higher value on splits missing a class, so compare published numbers accordingly.

Parameters:

• cm (Tensor) –

  Confusion matrix with true classes as rows, shape \((C, C)\) (see confusion_matrix).

• average (Literal['macro', 'none'], default: 'macro' ) –

  "macro" returns the mean IoU (mIoU) over the classes; "none" returns the per-class IoU.

• ignore_index (Union[int, Sequence[int], None], default: None ) –

  Class index, or indices, to ignore: points whose true class is ignored are dropped, and the ignored classes are left out of the mean. Indices outside \([0, C)\) have no effect.

• zero_division (float, default: 0.0 ) –

  IoU given to a class with zero union (and to the ignored classes with average="none").

• class_names (Optional[Sequence[str]], default: None ) –

  Name of each class index; with average="none" the per-class IoU comes back as a {name: iou} dict instead of a tensor.

Returns:

• Union[float, Tensor, Dict[str, float]] –

  The mean IoU as a float with average="macro", or the per-class IoU, shape \((C,)\), with average="none"

• Union[float, Tensor, Dict[str, float]] –

  (a {name: iou} dict when class_names is given).

>>> cm = torch.tensor([[1, 1], [0, 1]])
>>> intersection_over_union(cm)
0.5
>>> intersection_over_union(cm, average="none")
tensor([0.5000, 0.5000])
>>> intersection_over_union(cm, average="none", class_names=["wall", "floor"])
{'wall': 0.5, 'floor': 0.5}

Per-shape IoU averaged over the parts of the shape's category (the ShapeNetPart protocol).

Each shape is scored only over the part labels its category owns (e.g. ShapeNetPart's Airplane owns parts \([0, 1, 2, 3]\)); a part absent from both the prediction and the target counts as IoU \(1\).

Parameters:

• preds (Tensor) –

  Predicted part indices, shape \((N,)\).

• target (Tensor) –

  Ground truth part indices, shape \((N,)\).

• part_ids (Sequence[Sequence[int]]) –

  Part labels owned by each category, e.g. ShapeNetPart.seg_ids.values().

• category (Tensor) –

  Per-shape category index into part_ids, shape \((B,)\).

• batch (Optional[Tensor], default: None ) –

  Optional per-point shape index, shape \((N,)\); when omitted, all the points belong to one shape.

Returns:

• Tensor –

  Per-shape IoU tensor of shape \((B,)\).

>>> preds, target = torch.tensor([0, 1, 1, 1]), torch.tensor([0, 1, 0, 1])
>>> part_intersection_over_union(preds, target, part_ids=[[0, 1], [2, 3]], category=torch.tensor([0]))
tensor([0.5833])

Mean IoU of per-shape IoUs (the ShapeNetPart instance and class mIoU).

A shape's IoU does not depend on the other shapes of its batch, so the IoUs may come from one batch or be the part_intersection_over_union of every batch of a split, concatenated.

Parameters:

• ious (Tensor) –

  Per-shape IoUs (see part_intersection_over_union), shape \((B,)\).

• category (Tensor) –

  Per-shape category index, shape \((B,)\).

• average (Literal['micro', 'macro', 'none'], default: 'micro' ) –

  "micro" returns the instance mIoU (the mean over all shapes); "macro" returns the class mIoU (the mean, over the categories that have a shape, of their per-category means); "none" returns the per-category mean IoU.

• num_classes (Optional[int], default: None ) –

  Number of categories, i.e. the length of the average="none" output; defaults to the number of class_names, else to the largest category index met plus one.

• class_names (Optional[Sequence[str]], default: None ) –

  Name of each category index; with average="none" the per-category mean IoU comes back as a {name: iou} dict instead of a tensor.

Returns:

• Union[float, Tensor, Dict[str, float]] –

  The mIoU as a float with average="micro" or "macro" (NaN without any shape), or the per-category mean

• Union[float, Tensor, Dict[str, float]] –

  IoU, shape \((C,)\), with average="none" (a {name: iou} dict when class_names is given), holding NaN

• Union[float, Tensor, Dict[str, float]] –

  for the categories without any shape.

>>> ious, category = torch.tensor([1.0, 0.5, 0.0]), torch.tensor([0, 0, 1])
>>> part_mean_intersection_over_union(ious, category)
0.5
>>> part_mean_intersection_over_union(ious, category, average="macro")
0.375
>>> part_mean_intersection_over_union(ious, category, average="none", class_names=["Airplane", "Bag"])
{'Airplane': 0.75, 'Bag': 0.0}