pointrcnn

PointRCNN detection model.

First page of PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud

1812.04244 · December 2018

Classes:

PointRCNNOutput –

Inference-mode PointRCNN output: refined boxes with stage-2 confidences (packed layout).
PointRCNNTrainOutput –

Training-mode PointRCNN output: stage-1 point predictions plus sampled-ROI stage-2 tensors.
PointHeadBox –

Stage-1 per-point foreground head + bin-free box proposal generation (PointHeadBox).
PointRCNNRefinementHead –

Stage-2 ROI refinement head (PointRCNNHead): point ROI pooling + canonical transform + PointNet++.
PointRCNNDetection –

PointRCNN two-stage point-based 3D object detector (packed point format).

Functions:

rotate_points_along_z –

Rotate point sets about the \(+z\) axis (angle increases \(x \to y\)).
decode_point_residuals –

Decode per-point box residuals with class mean-size anchors.

PointRCNNOutput [source] ¶

Bases: TypedDict

Inference-mode PointRCNN output: refined boxes with stage-2 confidences (packed layout).

Attributes:

rcnn_cls (Tensor) –

Stage-2 confidence logit per ROI, shape \((M, 1)\).
boxes (Tensor) –

Refined boxes \((c_x, c_y, c_z, d_x, d_y, d_z, \theta)\), shape \((M, 7)\).
roi_labels (Tensor) –

Stage-1 ROI class per box (\(1\)-based), shape \((M,)\).
roi_scores (Tensor) –

Stage-1 sigmoid proposal score per box, shape \((M,)\).
batch (Tensor) –

Per-ROI scene index, shape \((M,)\).

PointRCNNTrainOutput [source] ¶

Bases: TypedDict

Training-mode PointRCNN output: stage-1 point predictions plus sampled-ROI stage-2 tensors.

Attributes:

point_cls_preds (Tensor) –

Stage-1 per-point class logits, shape \((N, \text{num\_classes})\).
point_box_preds (Tensor) –

Stage-1 per-point box residuals, shape \((N, 8)\).
point_pos (Tensor) –

Per-point coordinates, shape \((N, 3)\).
point_batch (Tensor) –

Per-point scene index, shape \((N,)\).
rcnn_cls (Tensor) –

Stage-2 confidence logit per sampled ROI, shape \((M, 1)\).
rcnn_reg (Tensor) –

Stage-2 raw ROI box residuals, shape \((M, 7)\).
rcnn_boxes (Tensor) –

Stage-2 refined boxes in the lidar frame, shape \((M, 7)\).
rois (Tensor) –

Sampled proposal boxes (generated without gradient), shape \((M, 7)\).
gt_of_rois (Tensor) –

ROI-canonical matched ground-truth box, shape \((M, 7)\).
gt_of_rois_src (Tensor) –

Lidar-frame matched ground-truth box, shape \((M, 7)\).
roi_ious (Tensor) –

Per-ROI max IoU with the matched box, shape \((M,)\).

PointHeadBox [source] ¶

PointHeadBox(
    in_channels: int,
    num_classes: int,
    *,
    cls_channels: Sequence[int],
    reg_channels: Sequence[int],
    mean_sizes: Tensor,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

Stage-1 per-point foreground head + bin-free box proposal generation (PointHeadBox).

Two MLPs over the per-point backbone features predict a per-point class logit and an 8-D box residual. At inference every point becomes a proposal: the class score is the sigmoid of the max class logit and the box is decoded by decode_point_residuals against the point's predicted class mean size.

Parameters:

in_channels (int) –

Backbone feature channels per point.
num_classes (int) –

Number of foreground classes.
cls_channels (Sequence[int]) –

Hidden channels of the classification MLP.
reg_channels (Sequence[int]) –

Hidden channels of the box-regression MLP.
mean_sizes (Tensor) –

Per-class mean box size \((d_x, d_y, d_z)\), shape \((\text{num\_classes}, 3)\).
act (Union[str, Callable, None], default: 'relu' ) –

Activation type or callable.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Extra activation arguments.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization type or callable.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Extra normalization arguments.

Methods:

forward –

Predict per-point class scores, raw box residuals and decoded proposal boxes.

forward [source] ¶

forward(
    x: Tensor, pos: Tensor
) -> Tuple[Tensor, Tensor, Tensor, Tensor]

Predict per-point class scores, raw box residuals and decoded proposal boxes.

Parameters:

x (Tensor) –

Per-point backbone features, shape \((N, C)\).
pos (Tensor) –

Per-point coordinates, shape \((N, 3)\).

Returns:

Tensor –

A tuple (point_scores, cls_preds, box_preds, boxes) of the sigmoid foreground score \((N,)\),
Tensor –

the raw class logits \((N, \text{num\_classes})\), the raw box residuals \((N, 8)\) (the stage-1
Tensor –

regression targets are formed against these), and the decoded boxes \((N, 7)\).

Shape

x: \((N, C)\)
pos: \((N, 3)\)
output: \((N,)\), \((N, \text{num\_classes})\), \((N, 8)\), \((N, 7)\)

PointRCNNRefinementHead [source] ¶

PointRCNNRefinementHead(
    in_channels: int,
    *,
    sa_channels: Sequence[Sequence[int]],
    sa_num_points: Sequence[int],
    sa_radii: Sequence[float],
    sa_num_neighbors: Sequence[int],
    xyz_up_channels: Sequence[int],
    cls_channels: Sequence[int],
    reg_channels: Sequence[int],
    num_sampled_points: int = 512,
    pool_extra_width: Sequence[float] = (0.0, 0.0, 0.0),
    depth_normalizer: float = 70.0,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

Stage-2 ROI refinement head (PointRCNNHead): point ROI pooling + canonical transform + PointNet++.

For each proposal it pools a fixed number of input points inside the (optionally enlarged) box, appends the per-point foreground score and a depth feature, canonically transforms the pooled points (translate to the ROI center, rotate by \(-\theta\)), lifts the canonical xyz with an MLP, fuses it with the pooled point features, and runs a small PointNet++ to produce a confidence logit and a 7-D box refinement.

Parameters:

in_channels (int) –

Pooled point-feature channels (the stage-1 backbone feature dim).
sa_channels (Sequence[Sequence[int]]) –

Per-SA-block MLP channel lists.
sa_num_points (Sequence[int]) –

Per-SA-block sample counts; -1 groups all remaining points.
sa_radii (Sequence[float]) –

Per-SA-block ball-query radii.
sa_num_neighbors (Sequence[int]) –

Per-SA-block neighbor caps.
xyz_up_channels (Sequence[int]) –

Channels of the canonical-xyz lifting MLP.
cls_channels (Sequence[int]) –

Hidden channels of the confidence MLP.
reg_channels (Sequence[int]) –

Hidden channels of the box-refinement MLP.
num_sampled_points (int, default: 512 ) –

Points pooled per ROI.
pool_extra_width (Sequence[float], default: (0.0, 0.0, 0.0) ) –

Per-axis enlargement of the pooling box.
depth_normalizer (float, default: 70.0 ) –

Divisor for the point-depth feature.
act (Union[str, Callable, None], default: 'relu' ) –

Activation type or callable.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Extra activation arguments.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization type or callable.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Extra normalization arguments.

Methods:

reset_parameters –

Xavier-normal weights and zero biases on every layer, with the last regression weights near zero.
roipool –

Pool a fixed number of in-box points per ROI and canonically transform them.
forward –

Refine proposals into confidence logits, raw box residuals and refined boxes.

reset_parameters [source] ¶

reset_parameters() -> None

Xavier-normal weights and zero biases on every layer, with the last regression weights near zero.

roipool [source] ¶

roipool(
    pos: Tensor, x: Tensor, rois: Tensor
) -> Tuple[Tensor, Tensor]

Pool a fixed number of in-box points per ROI and canonically transform them.

Mirrors the reference roipoint_pool3d CUDA kernel: for every (optionally enlarged) box, the input points are scanned in order and the first num_sampled_points that fall inside are kept (cyclically duplicated when fewer are found, zeroed when none). Pooled points are translated to the ROI center and rotated by \(-\theta\) into the box-canonical frame.

Parameters:

pos (Tensor) –

Per-point coordinates of one scene, shape \((N, 3)\).
x (Tensor) –

Per-point pooled features (score + depth + backbone), shape \((N, 5 + C)\).
rois (Tensor) –

Proposal boxes \((x, y, z, d_x, d_y, d_z, \theta)\), shape \((M, 7)\).

Returns:

Tensor –

A tuple (pooled, empty) of the pooled features \((M, S, 3 + (5 + C))\) in canonical xyz and a
Tensor –

boolean ROI-empty flag \((M,)\).

Shape

pos: \((N, 3)\)
x: \((N, 5 + C)\)
rois: \((M, 7)\)
output: \((M, S, 3 + 5 + C)\), \((M,)\)

forward [source] ¶

forward(
    pos: Tensor,
    x: Tensor,
    point_scores: Tensor,
    batch: Tensor,
    rois: Tensor,
    roi_batch: Tensor,
) -> Tuple[Tensor, Tensor, Tensor]

Refine proposals into confidence logits, raw box residuals and refined boxes.

Parameters:

pos (Tensor) –

Per-point coordinates, shape \((N, 3)\).
x (Tensor) –

Per-point backbone features, shape \((N, C)\).
point_scores (Tensor) –

Per-point stage-1 foreground score, shape \((N,)\).
batch (Tensor) –

Per-point scene index, shape \((N,)\).
rois (Tensor) –

Proposal boxes, shape \((M, 7)\).
roi_batch (Tensor) –

Per-ROI scene index, shape \((M,)\).

Returns:

Tensor –

A tuple (rcnn_cls, rcnn_reg, refined_boxes) of the confidence logit \((M, 1)\), the raw ROI box
Tensor –

residuals \((M, 7)\) (the stage-2 regression targets are formed against these) and the refined
Tensor –

boxes \((M, 7)\) in the lidar frame.

Shape

pos: \((N, 3)\), x: \((N, C)\), point_scores: \((N,)\)
rois: \((M, 7)\)
output: \((M, 1)\), \((M, 7)\), \((M, 7)\)

PointRCNNDetection [source] ¶

PointRCNNDetection(
    in_channels: int = 4,
    num_classes: int = 3,
    *,
    mean_sizes: Union[Tensor, Sequence[Sequence[float]]],
    sa_channels: Sequence[Sequence[Sequence[int]]],
    sa_num_points: Sequence[int],
    sa_radii: Sequence[Sequence[float]],
    sa_num_neighbors: Sequence[Sequence[int]],
    fp_channels: Sequence[Sequence[int]],
    point_cls_channels: Sequence[int] = (256, 256),
    point_reg_channels: Sequence[int] = (256, 256),
    roi_sa_channels: Sequence[Sequence[int]],
    roi_sa_num_points: Sequence[int],
    roi_sa_radii: Sequence[float],
    roi_sa_num_neighbors: Sequence[int],
    roi_xyz_up_channels: Sequence[int] = (128, 128),
    roi_cls_channels: Sequence[int] = (256, 256),
    roi_reg_channels: Sequence[int] = (256, 256),
    num_sampled_points: int = 512,
    pool_extra_width: Sequence[float] = (0.0, 0.0, 0.0),
    depth_normalizer: float = 70.0,
    proposal_pre_maxsize: int = 9000,
    proposal_post_maxsize: int = 100,
    proposal_nms_thresh: float = 0.85,
    proposal_nms_rotated: bool = True,
    train_proposal_post_maxsize: int = 512,
    train_proposal_nms_thresh: float = 0.8,
    roi_per_image: int = 128,
    fg_ratio: float = 0.5,
    reg_fg_thresh: float = 0.55,
    cls_fg_thresh: float = 0.6,
    cls_bg_thresh_lo: float = 0.1,
    hard_bg_ratio: float = 0.8,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: DetectionModel

PointRCNN two-stage point-based 3D object detector (packed point format).

Reference: PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud (Shi et al., 2019). Reference implementation: open-mmlab/OpenPCDet.

Stage 1 runs a multi-scale PointNet++ U-Net (PointNet2Encoder + PointNet2Decoder) over the raw point cloud, then a per-point head (PointHeadBox) predicts foreground scores and one box proposal per point. The top proposals (after class-agnostic NMS) become ROIs that stage 2 (PointRCNNRefinementHead) pools points around, canonically transforms, and refines into a confidence and a box correction.

Parameters:

in_channels (int, default: 4 ) –

Raw point feature channels including xyz (e.g. \(4\) for \(x, y, z, \text{intensity}\)).
num_classes (int, default: 3 ) –

Number of foreground classes.
mean_sizes (Union[Tensor, Sequence[Sequence[float]]]) –

Per-class mean box size \((d_x, d_y, d_z)\), shape \((\text{num\_classes}, 3)\).
sa_channels (Sequence[Sequence[Sequence[int]]]) –

Stage-1 per-SA-block, per-scale MLP channel lists.
sa_num_points (Sequence[int]) –

Stage-1 per-SA-block sample counts.
sa_radii (Sequence[Sequence[float]]) –

Stage-1 per-SA-block, per-scale ball-query radii.
sa_num_neighbors (Sequence[Sequence[int]]) –

Stage-1 per-SA-block, per-scale neighbor caps.
fp_channels (Sequence[Sequence[int]]) –

Stage-1 per-FP-block MLP channel lists, ordered from the coarsest skip level to the finest (PointNet2Decoder order).
point_cls_channels (Sequence[int], default: (256, 256) ) –

Stage-1 classification MLP hidden channels.
point_reg_channels (Sequence[int], default: (256, 256) ) –

Stage-1 box-regression MLP hidden channels.
roi_sa_channels (Sequence[Sequence[int]]) –

Stage-2 per-SA-block MLP channel lists.
roi_sa_num_points (Sequence[int]) –

Stage-2 per-SA-block sample counts (-1 groups all).
roi_sa_radii (Sequence[float]) –

Stage-2 per-SA-block ball-query radii.
roi_sa_num_neighbors (Sequence[int]) –

Stage-2 per-SA-block neighbor caps.
roi_xyz_up_channels (Sequence[int], default: (128, 128) ) –

Stage-2 canonical-xyz lifting MLP channels.
roi_cls_channels (Sequence[int], default: (256, 256) ) –

Stage-2 confidence MLP hidden channels.
roi_reg_channels (Sequence[int], default: (256, 256) ) –

Stage-2 box-refinement MLP hidden channels.
num_sampled_points (int, default: 512 ) –

Points pooled per ROI in stage 2.
pool_extra_width (Sequence[float], default: (0.0, 0.0, 0.0) ) –

Per-axis enlargement of the stage-2 pooling box.
depth_normalizer (float, default: 70.0 ) –

Divisor for the stage-2 point-depth feature.
proposal_pre_maxsize (int, default: 9000 ) –

Proposals kept before stage-1 NMS.
proposal_post_maxsize (int, default: 100 ) –

ROIs kept after stage-1 NMS at inference (the stage-2 batch size per scene).
proposal_nms_thresh (float, default: 0.85 ) –

Stage-1 proposal NMS IoU threshold at inference.
proposal_nms_rotated (bool, default: True ) –

Run the stage-1 NMS on the rotated BEV IoU (the reference nms_gpu) rather than the axis-aligned 3D IoU.
train_proposal_post_maxsize (int, default: 512 ) –

Proposals kept after stage-1 NMS during training (before ROI sampling).
train_proposal_nms_thresh (float, default: 0.8 ) –

Stage-1 proposal NMS IoU threshold during training.
roi_per_image (int, default: 128 ) –

ROIs sampled per scene for stage-2 training.
fg_ratio (float, default: 0.5 ) –

Target fraction of foreground ROIs in the sampled set.
reg_fg_thresh (float, default: 0.55 ) –

ROI-to-GT IoU at or above which a sampled ROI is foreground (box regression valid).
cls_fg_thresh (float, default: 0.6 ) –

ROI-to-GT IoU used with reg_fg_thresh to define the foreground sampling threshold.
cls_bg_thresh_lo (float, default: 0.1 ) –

ROI-to-GT IoU below which a ROI is easy background (else hard background).
hard_bg_ratio (float, default: 0.8 ) –

Fraction of the sampled background ROIs drawn from hard (higher-IoU) background.
act (Union[str, Callable, None], default: 'relu' ) –

Activation type or callable.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Extra activation arguments.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization type or callable.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Extra normalization arguments.

Methods:

configure_encoder –

Build the stage-1 multi-scale PointNet++ encoder.
configure_decoder –

Build the stage-1 PointNet++ feature-propagation decoder.
configure_point_head –

Build the stage-1 per-point foreground and box-proposal head.
configure_roi_head –

Build the stage-2 ROI refinement head.
forward –

Run both stages; in train mode the ground truth drives the stage-2 ROI sampling.
decode –

Decode a forward output into raw per-ROI detections (no score threshold or NMS).

Attributes:

num_features (int) –

Channel count \(C\) of the per-point backbone features entering the heads.

num_features `property` ¶

num_features: int

Channel count \(C\) of the per-point backbone features entering the heads.

configure_encoder [source] ¶

configure_encoder() -> PointNet2Encoder

Build the stage-1 multi-scale PointNet++ encoder.

configure_decoder [source] ¶

configure_decoder() -> PointNet2Decoder

Build the stage-1 PointNet++ feature-propagation decoder.

configure_point_head [source] ¶

configure_point_head() -> PointHeadBox

Build the stage-1 per-point foreground and box-proposal head.

configure_roi_head [source] ¶

configure_roi_head() -> PointRCNNRefinementHead

Build the stage-2 ROI refinement head.

forward [source] ¶

forward(
    x: OptTensor,
    pos: Tensor,
    batch: Tensor,
    gt_boxes: OptTensor = None,
    gt_labels: OptTensor = None,
    gt_batch: OptTensor = None,
) -> Union[PointRCNNTrainOutput, PointRCNNOutput]

Run both stages; in train mode the ground truth drives the stage-2 ROI sampling.

Stage-2 training samples its ROIs by matching stage-1 proposals to ground-truth boxes at forward time, so train mode requires the packed ground truth. The GT arguments default to None and are omitted at inference (a training pipeline passes box / label / batch_box after the point inputs, e.g. via input_keys).

Parameters:

x (OptTensor) –

Per-point features including reflectance, shape \((N, \text{in\_channels} - 3)\).
pos (Tensor) –

Per-point coordinates, shape \((N, 3)\).
batch (Tensor) –

Per-point scene index, shape \((N,)\).
gt_boxes (OptTensor, default: None ) –

Ground-truth boxes \((K, 7)\), required in train mode.
gt_labels (OptTensor, default: None ) –

Ground-truth \(0\)-based classes, shape \((K,)\), required in train mode.
gt_batch (OptTensor, default: None ) –

Per-box scene index, shape \((K,)\), required in train mode.

Returns:

Union[PointRCNNTrainOutput, PointRCNNOutput] –

A PointRCNNTrainOutput in train mode (stage-1 point predictions plus sampled-ROI stage-2
Union[PointRCNNTrainOutput, PointRCNNOutput] –

tensors for the loss), otherwise a PointRCNNOutput (refined boxes with confidences).

Shape

x: \((N, \text{in\_channels} - 3)\), pos: \((N, 3)\), batch: \((N,)\)
gt_boxes: \((K, 7)\), gt_labels / gt_batch: \((K,)\)

decode [source] ¶

decode(out: PointRCNNOutput) -> Detection3D

Decode a forward output into raw per-ROI detections (no score threshold or NMS).

Scores each refined box by its stage-2 confidence (sigmoid of rcnn_cls) and labels it by the stage-1 ROI label (shifted to 0-indexed). The full per-ROI set is returned; the evaluation pipeline applies class-agnostic 3D NMS then score thresholding via the torch_pointcloud.ops.box3d utilities (see the benchmark example).

Parameters:

out (PointRCNNOutput) –

A forward output {"rcnn_cls", "boxes", "roi_labels", "roi_scores", "batch"}.

Returns:

Detection3D –

Packed per-ROI detections {"boxes": (R, 7), "scores": (R,), "labels": (R,), "batch": (R,)}
Detection3D –

(PyG layout).

rotate_points_along_z [source] ¶

rotate_points_along_z(
    points: Tensor, angle: Tensor
) -> Tensor

Rotate point sets about the \(+z\) axis (angle increases \(x \to y\)).

Parameters:

points (Tensor) –

Point sets, shape \((B, N, 3 + C)\); only the first three channels are rotated.
angle (Tensor) –

Per-set yaw, shape \((B,)\).

Returns:

Tensor –

The rotated point sets, shape \((B, N, 3 + C)\).

Shape

points: \((B, N, 3 + C)\)
angle: \((B,)\)
output: \((B, N, 3 + C)\)

decode_point_residuals [source] ¶

decode_point_residuals(
    encodings: Tensor,
    points: Tensor,
    classes: Tensor,
    mean_sizes: Tensor,
) -> Tensor

Decode per-point box residuals with class mean-size anchors.

Decodes a stage-1 prediction \((x_t, y_t, z_t, d_{x,t}, d_{y,t}, d_{z,t}, \cos, \sin)\) at a foreground point into an oriented box \((c_x, c_y, c_z, d_x, d_y, d_z, \theta)\), using the predicted class mean size as the anchor for the size residuals.

Parameters:

encodings (Tensor) –

Box residuals, shape \((N, 8)\).
points (Tensor) –

Anchor point coordinates, shape \((N, 3)\).
classes (Tensor) –

Predicted class index per point (\(1 \ldots \text{num\_classes}\)), shape \((N,)\).
mean_sizes (Tensor) –

Per-class mean box size \((d_x, d_y, d_z)\), shape \((\text{num\_classes}, 3)\).

Returns:

Tensor –

Decoded boxes \((c_x, c_y, c_z, d_x, d_y, d_z, \theta)\), shape \((N, 7)\).

Shape

encodings: \((N, 8)\)
points: \((N, 3)\)
output: \((N, 7)\)

pointrcnn

PointRCNNOutput [source] ¶

PointRCNNTrainOutput [source] ¶

PointHeadBox [source] ¶

forward [source] ¶

PointRCNNRefinementHead [source] ¶

reset_parameters [source] ¶

roipool [source] ¶

forward [source] ¶

PointRCNNDetection [source] ¶

num_features property ¶

configure_encoder [source] ¶

configure_decoder [source] ¶

configure_point_head [source] ¶

configure_roi_head [source] ¶

forward [source] ¶

decode [source] ¶

rotate_points_along_z [source] ¶

decode_point_residuals [source] ¶

num_features `property` ¶