Skip to content

PointRCNN

PointRCNN detection model.

First page of PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud

1812.04244 · December 2018

Classes:

  • PointRCNNOutput –

    Inference-mode PointRCNN output: refined boxes with stage-2 confidences (packed layout).

  • PointRCNNTrainOutput –

    Training-mode PointRCNN output: stage-1 point predictions plus sampled-ROI stage-2 tensors.

  • PointHeadBox –

    Stage-1 per-point foreground head + bin-free box proposal generation (PointHeadBox).

  • PointRCNNRefinementHead –

    Stage-2 ROI refinement head (PointRCNNHead): point ROI pooling + canonical transform + PointNet++.

  • PointRCNNDetection –

    PointRCNN two-stage point-based 3D object detector (packed point format).

Functions:

PointRCNNOutput

Bases: TypedDict

Inference-mode PointRCNN output: refined boxes with stage-2 confidences (packed layout).

Attributes:

  • rcnn_cls (Tensor) –

    Stage-2 confidence logit per ROI, shape \((M, 1)\).

  • boxes (Tensor) –

    Refined boxes \((c_x, c_y, c_z, d_x, d_y, d_z, \theta)\), shape \((M, 7)\).

  • roi_labels (Tensor) –

    Stage-1 ROI class per box (\(1\)-based), shape \((M,)\).

  • roi_scores (Tensor) –

    Stage-1 sigmoid proposal score per box, shape \((M,)\).

  • batch (Tensor) –

    Per-ROI scene index, shape \((M,)\).

PointRCNNTrainOutput

Bases: TypedDict

Training-mode PointRCNN output: stage-1 point predictions plus sampled-ROI stage-2 tensors.

Attributes:

  • point_cls_preds (Tensor) –

    Stage-1 per-point class logits, shape \((N, \text{num\_classes})\).

  • point_box_preds (Tensor) –

    Stage-1 per-point box residuals, shape \((N, 8)\).

  • point_pos (Tensor) –

    Per-point coordinates, shape \((N, 3)\).

  • point_batch (Tensor) –

    Per-point scene index, shape \((N,)\).

  • rcnn_cls (Tensor) –

    Stage-2 confidence logit per sampled ROI, shape \((M, 1)\).

  • rcnn_reg (Tensor) –

    Stage-2 raw ROI box residuals, shape \((M, 7)\).

  • rcnn_boxes (Tensor) –

    Stage-2 refined boxes in the lidar frame, shape \((M, 7)\).

  • rois (Tensor) –

    Sampled proposal boxes (generated without gradient), shape \((M, 7)\).

  • gt_of_rois (Tensor) –

    ROI-canonical matched ground-truth box, shape \((M, 7)\).

  • gt_of_rois_src (Tensor) –

    Lidar-frame matched ground-truth box, shape \((M, 7)\).

  • roi_ious (Tensor) –

    Per-ROI max IoU with the matched box, shape \((M,)\).

PointHeadBox 

PointHeadBox(
    in_channels: int,
    num_classes: int,
    *,
    cls_channels: Sequence[int],
    reg_channels: Sequence[int],
    mean_sizes: Tensor,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

Stage-1 per-point foreground head + bin-free box proposal generation (PointHeadBox).

Two MLPs over the per-point backbone features predict a per-point class logit and an 8-D box residual. At inference every point becomes a proposal: the class score is the sigmoid of the max class logit and the box is decoded by decode_point_residuals against the point's predicted class mean size.

Parameters:

  • in_channels (int) –

    Backbone feature channels per point.

  • num_classes (int) –

    Number of foreground classes.

  • cls_channels (Sequence[int]) –

    Hidden channels of the classification MLP.

  • reg_channels (Sequence[int]) –

    Hidden channels of the box-regression MLP.

  • mean_sizes (Tensor) –

    Per-class mean box size \((d_x, d_y, d_z)\), shape \((\text{num\_classes}, 3)\).

  • act (Union[str, Callable, None], default: 'relu' ) –

    Activation type or callable.

  • act_kwargs (Optional[Dict[str, Any]], default: None ) –

    Extra activation arguments.

  • norm (Union[str, Callable, None], default: 'batch_norm' ) –

    Normalization type or callable.

  • norm_kwargs (Optional[Dict[str, Any]], default: None ) –

    Extra normalization arguments.

Methods:

  • forward –

    Predict per-point class scores, raw box residuals and decoded proposal boxes.

forward 

forward(
    x: Tensor, pos: Tensor
) -> Tuple[Tensor, Tensor, Tensor, Tensor]

Predict per-point class scores, raw box residuals and decoded proposal boxes.

Parameters:

  • x (Tensor) –

    Per-point backbone features, shape \((N, C)\).

  • pos (Tensor) –

    Per-point coordinates, shape \((N, 3)\).

Returns:

  • Tensor –

    A tuple (point_scores, cls_preds, box_preds, boxes) of the sigmoid foreground score \((N,)\),

  • Tensor –

    the raw class logits \((N, \text{num\_classes})\), the raw box residuals \((N, 8)\) (the stage-1

  • Tensor –

    regression targets are formed against these), and the decoded boxes \((N, 7)\).

Shape
  • x: \((N, C)\)
  • pos: \((N, 3)\)
  • output: \((N,)\), \((N, \text{num\_classes})\), \((N, 8)\), \((N, 7)\)

PointRCNNRefinementHead 

PointRCNNRefinementHead(
    in_channels: int,
    *,
    sa_channels: Sequence[Sequence[int]],
    sa_npoints: Sequence[int],
    sa_radii: Sequence[float],
    sa_num_neighbors: Sequence[int],
    xyz_up_channels: Sequence[int],
    cls_channels: Sequence[int],
    reg_channels: Sequence[int],
    num_sampled_points: int = 512,
    pool_extra_width: Sequence[float] = (0.0, 0.0, 0.0),
    depth_normalizer: float = 70.0,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

Stage-2 ROI refinement head (PointRCNNHead): point ROI pooling + canonical transform + PointNet++.

For each proposal it pools a fixed number of input points inside the (optionally enlarged) box, appends the per-point foreground score and a depth feature, canonically transforms the pooled points (translate to the ROI center, rotate by \(-\theta\)), lifts the canonical xyz with an MLP, fuses it with the pooled point features, and runs a small PointNet++ to produce a confidence logit and a 7-D box refinement.

Parameters:

  • in_channels (int) –

    Pooled point-feature channels (the stage-1 backbone feature dim).

  • sa_channels (Sequence[Sequence[int]]) –

    Per-SA-block MLP channel lists.

  • sa_npoints (Sequence[int]) –

    Per-SA-block sample counts; -1 groups all remaining points.

  • sa_radii (Sequence[float]) –

    Per-SA-block ball-query radii.

  • sa_num_neighbors (Sequence[int]) –

    Per-SA-block neighbor caps.

  • xyz_up_channels (Sequence[int]) –

    Channels of the canonical-xyz lifting MLP.

  • cls_channels (Sequence[int]) –

    Hidden channels of the confidence MLP.

  • reg_channels (Sequence[int]) –

    Hidden channels of the box-refinement MLP.

  • num_sampled_points (int, default: 512 ) –

    Points pooled per ROI.

  • pool_extra_width (Sequence[float], default: (0.0, 0.0, 0.0) ) –

    Per-axis enlargement of the pooling box.

  • depth_normalizer (float, default: 70.0 ) –

    Divisor for the point-depth feature.

  • act (Union[str, Callable, None], default: 'relu' ) –

    Activation type or callable.

  • act_kwargs (Optional[Dict[str, Any]], default: None ) –

    Extra activation arguments.

  • norm (Union[str, Callable, None], default: 'batch_norm' ) –

    Normalization type or callable.

  • norm_kwargs (Optional[Dict[str, Any]], default: None ) –

    Extra normalization arguments.

Methods:

  • roipool –

    Pool a fixed number of in-box points per ROI and canonically transform them.

  • forward –

    Refine proposals into confidence logits, raw box residuals and refined boxes.

roipool 

roipool(
    pos: Tensor, x: Tensor, rois: Tensor
) -> Tuple[Tensor, Tensor]

Pool a fixed number of in-box points per ROI and canonically transform them.

Mirrors the reference roipoint_pool3d CUDA kernel: for every (optionally enlarged) box, the input points are scanned in order and the first num_sampled_points that fall inside are kept (cyclically duplicated when fewer are found, zeroed when none). Pooled points are translated to the ROI center and rotated by \(-\theta\) into the box-canonical frame.

Parameters:

  • pos (Tensor) –

    Per-point coordinates of one scene, shape \((N, 3)\).

  • x (Tensor) –

    Per-point pooled features (score + depth + backbone), shape \((N, 5 + C)\).

  • rois (Tensor) –

    Proposal boxes \((x, y, z, d_x, d_y, d_z, \theta)\), shape \((M, 7)\).

Returns:

  • Tensor –

    A tuple (pooled, empty) of the pooled features \((M, S, 3 + (5 + C))\) in canonical xyz and a

  • Tensor –

    boolean ROI-empty flag \((M,)\).

Shape
  • pos: \((N, 3)\)
  • x: \((N, 5 + C)\)
  • rois: \((M, 7)\)
  • output: \((M, S, 3 + 5 + C)\), \((M,)\)

forward 

forward(
    pos: Tensor,
    x: Tensor,
    point_scores: Tensor,
    batch: Tensor,
    rois: Tensor,
    roi_batch: Tensor,
) -> Tuple[Tensor, Tensor, Tensor]

Refine proposals into confidence logits, raw box residuals and refined boxes.

Parameters:

  • pos (Tensor) –

    Per-point coordinates, shape \((N, 3)\).

  • x (Tensor) –

    Per-point backbone features, shape \((N, C)\).

  • point_scores (Tensor) –

    Per-point stage-1 foreground score, shape \((N,)\).

  • batch (Tensor) –

    Per-point scene index, shape \((N,)\).

  • rois (Tensor) –

    Proposal boxes, shape \((M, 7)\).

  • roi_batch (Tensor) –

    Per-ROI scene index, shape \((M,)\).

Returns:

  • Tensor –

    A tuple (rcnn_cls, rcnn_reg, refined_boxes) of the confidence logit \((M, 1)\), the raw ROI box

  • Tensor –

    residuals \((M, 7)\) (the stage-2 regression targets are formed against these) and the refined

  • Tensor –

    boxes \((M, 7)\) in the lidar frame.

Shape
  • pos: \((N, 3)\), x: \((N, C)\), point_scores: \((N,)\)
  • rois: \((M, 7)\)
  • output: \((M, 1)\), \((M, 7)\), \((M, 7)\)

PointRCNNDetection 

PointRCNNDetection(
    in_channels: int = 4,
    num_classes: int = 3,
    *,
    mean_sizes: Union[Tensor, Sequence[Sequence[float]]],
    sa_channels: Sequence[Sequence[Sequence[int]]],
    sa_npoints: Sequence[int],
    sa_radii: Sequence[Sequence[float]],
    sa_num_neighbors: Sequence[Sequence[int]],
    fp_channels: Sequence[Sequence[int]],
    point_cls_channels: Sequence[int] = (256, 256),
    point_reg_channels: Sequence[int] = (256, 256),
    roi_sa_channels: Sequence[Sequence[int]],
    roi_sa_npoints: Sequence[int],
    roi_sa_radii: Sequence[float],
    roi_sa_num_neighbors: Sequence[int],
    roi_xyz_up_channels: Sequence[int] = (128, 128),
    roi_cls_channels: Sequence[int] = (256, 256),
    roi_reg_channels: Sequence[int] = (256, 256),
    num_sampled_points: int = 512,
    pool_extra_width: Sequence[float] = (0.0, 0.0, 0.0),
    depth_normalizer: float = 70.0,
    proposal_pre_maxsize: int = 9000,
    proposal_post_maxsize: int = 100,
    proposal_nms_thresh: float = 0.85,
    proposal_nms_rotated: bool = True,
    train_proposal_post_maxsize: int = 512,
    train_proposal_nms_thresh: float = 0.8,
    roi_per_image: int = 128,
    fg_ratio: float = 0.5,
    reg_fg_thresh: float = 0.55,
    cls_fg_thresh: float = 0.6,
    cls_bg_thresh_lo: float = 0.1,
    hard_bg_ratio: float = 0.8,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: DetectionModel

PointRCNN two-stage point-based 3D object detector (packed point format).

Reference: Shi et al., 2019. Reference implementation: open-mmlab/OpenPCDet.

Stage 1 runs a multi-scale PointNet++ U-Net (PointNet2Encoder + PointNet2Decoder) over the raw point cloud, then a per-point head (PointHeadBox) predicts foreground scores and one box proposal per point. The top proposals (after class-agnostic NMS) become ROIs that stage 2 (PointRCNNRefinementHead) pools points around, canonically transforms, and refines into a confidence and a box correction.

Parameters:

  • in_channels (int, default: 4 ) –

    Raw point feature channels including xyz (e.g. \(4\) for \(x, y, z, \text{intensity}\)).

  • num_classes (int, default: 3 ) –

    Number of foreground classes.

  • mean_sizes (Union[Tensor, Sequence[Sequence[float]]]) –

    Per-class mean box size \((d_x, d_y, d_z)\), shape \((\text{num\_classes}, 3)\).

  • sa_channels (Sequence[Sequence[Sequence[int]]]) –

    Stage-1 per-SA-block, per-scale MLP channel lists.

  • sa_npoints (Sequence[int]) –

    Stage-1 per-SA-block sample counts.

  • sa_radii (Sequence[Sequence[float]]) –

    Stage-1 per-SA-block, per-scale ball-query radii.

  • sa_num_neighbors (Sequence[Sequence[int]]) –

    Stage-1 per-SA-block, per-scale neighbor caps.

  • fp_channels (Sequence[Sequence[int]]) –

    Stage-1 per-FP-block MLP channel lists, ordered from the coarsest skip level to the finest (PointNet2Decoder order).

  • point_cls_channels (Sequence[int], default: (256, 256) ) –

    Stage-1 classification MLP hidden channels.

  • point_reg_channels (Sequence[int], default: (256, 256) ) –

    Stage-1 box-regression MLP hidden channels.

  • roi_sa_channels (Sequence[Sequence[int]]) –

    Stage-2 per-SA-block MLP channel lists.

  • roi_sa_npoints (Sequence[int]) –

    Stage-2 per-SA-block sample counts (-1 groups all).

  • roi_sa_radii (Sequence[float]) –

    Stage-2 per-SA-block ball-query radii.

  • roi_sa_num_neighbors (Sequence[int]) –

    Stage-2 per-SA-block neighbor caps.

  • roi_xyz_up_channels (Sequence[int], default: (128, 128) ) –

    Stage-2 canonical-xyz lifting MLP channels.

  • roi_cls_channels (Sequence[int], default: (256, 256) ) –

    Stage-2 confidence MLP hidden channels.

  • roi_reg_channels (Sequence[int], default: (256, 256) ) –

    Stage-2 box-refinement MLP hidden channels.

  • num_sampled_points (int, default: 512 ) –

    Points pooled per ROI in stage 2.

  • pool_extra_width (Sequence[float], default: (0.0, 0.0, 0.0) ) –

    Per-axis enlargement of the stage-2 pooling box.

  • depth_normalizer (float, default: 70.0 ) –

    Divisor for the stage-2 point-depth feature.

  • proposal_pre_maxsize (int, default: 9000 ) –

    Proposals kept before stage-1 NMS.

  • proposal_post_maxsize (int, default: 100 ) –

    ROIs kept after stage-1 NMS at inference (the stage-2 batch size per scene).

  • proposal_nms_thresh (float, default: 0.85 ) –

    Stage-1 proposal NMS IoU threshold at inference.

  • proposal_nms_rotated (bool, default: True ) –

    Run the stage-1 NMS on the rotated BEV IoU (the reference nms_gpu) rather than the axis-aligned 3D IoU.

  • train_proposal_post_maxsize (int, default: 512 ) –

    Proposals kept after stage-1 NMS during training (before ROI sampling).

  • train_proposal_nms_thresh (float, default: 0.8 ) –

    Stage-1 proposal NMS IoU threshold during training.

  • roi_per_image (int, default: 128 ) –

    ROIs sampled per scene for stage-2 training.

  • fg_ratio (float, default: 0.5 ) –

    Target fraction of foreground ROIs in the sampled set.

  • reg_fg_thresh (float, default: 0.55 ) –

    ROI-to-GT IoU at or above which a sampled ROI is foreground (box regression valid).

  • cls_fg_thresh (float, default: 0.6 ) –

    ROI-to-GT IoU used with reg_fg_thresh to define the foreground sampling threshold.

  • cls_bg_thresh_lo (float, default: 0.1 ) –

    ROI-to-GT IoU below which a ROI is easy background (else hard background).

  • hard_bg_ratio (float, default: 0.8 ) –

    Fraction of the sampled background ROIs drawn from hard (higher-IoU) background.

  • act (Union[str, Callable, None], default: 'relu' ) –

    Activation type or callable.

  • act_kwargs (Optional[Dict[str, Any]], default: None ) –

    Extra activation arguments.

  • norm (Union[str, Callable, None], default: 'batch_norm' ) –

    Normalization type or callable.

  • norm_kwargs (Optional[Dict[str, Any]], default: None ) –

    Extra normalization arguments.

Methods:

  • configure_encoder –

    Build the stage-1 multi-scale PointNet++ encoder.

  • configure_decoder –

    Build the stage-1 PointNet++ feature-propagation decoder.

  • configure_point_head –

    Build the stage-1 per-point foreground and box-proposal head.

  • configure_roi_head –

    Build the stage-2 ROI refinement head.

  • forward –

    Run both stages; in train mode the ground truth drives the stage-2 ROI sampling.

  • decode –

    Decode a forward output into raw per-ROI detections (no score threshold or NMS).

Attributes:

  • num_features (int) –

    Channel count \(C\) of the per-point backbone features entering the heads.

num_features property 

num_features: int

Channel count \(C\) of the per-point backbone features entering the heads.

configure_encoder 

configure_encoder() -> PointNet2Encoder

Build the stage-1 multi-scale PointNet++ encoder.

configure_decoder 

configure_decoder() -> PointNet2Decoder

Build the stage-1 PointNet++ feature-propagation decoder.

configure_point_head 

configure_point_head() -> PointHeadBox

Build the stage-1 per-point foreground and box-proposal head.

configure_roi_head 

configure_roi_head() -> PointRCNNRefinementHead

Build the stage-2 ROI refinement head.

forward 

forward(
    x: OptTensor,
    pos: Tensor,
    batch: Tensor,
    gt_boxes: OptTensor = None,
    gt_labels: OptTensor = None,
    gt_batch: OptTensor = None,
) -> Union[PointRCNNTrainOutput, PointRCNNOutput]

Run both stages; in train mode the ground truth drives the stage-2 ROI sampling.

Stage-2 training samples its ROIs by matching stage-1 proposals to ground-truth boxes at forward time, so train mode requires the packed ground truth. The GT arguments default to None and are omitted at inference (a training pipeline passes box / label / batch_box after the point inputs, e.g. via input_keys).

Parameters:

  • x (OptTensor) –

    Per-point features including reflectance, shape \((N, \text{in\_channels} - 3)\).

  • pos (Tensor) –

    Per-point coordinates, shape \((N, 3)\).

  • batch (Tensor) –

    Per-point scene index, shape \((N,)\).

  • gt_boxes (OptTensor, default: None ) –

    Ground-truth boxes \((K, 7)\), required in train mode.

  • gt_labels (OptTensor, default: None ) –

    Ground-truth \(0\)-based classes, shape \((K,)\), required in train mode.

  • gt_batch (OptTensor, default: None ) –

    Per-box scene index, shape \((K,)\), required in train mode.

Returns:

Shape
  • x: \((N, \text{in\_channels} - 3)\), pos: \((N, 3)\), batch: \((N,)\)
  • gt_boxes: \((K, 7)\), gt_labels / gt_batch: \((K,)\)

decode 

decode(out: PointRCNNOutput) -> Detection3D

Decode a forward output into raw per-ROI detections (no score threshold or NMS).

Scores each refined box by its stage-2 confidence (sigmoid of rcnn_cls) and labels it by the stage-1 ROI label (shifted to 0-indexed). The full per-ROI set is returned; the evaluation pipeline applies class-agnostic 3D NMS then score thresholding via the torch_pointcloud.utils.box3d utilities (see the benchmark example).

Parameters:

  • out (PointRCNNOutput) –

    A forward output {"rcnn_cls", "boxes", "roi_labels", "roi_scores", "batch"}.

Returns:

  • Detection3D –

    Packed per-ROI detections {"boxes": (R, 7), "scores": (R,), "labels": (R,), "batch": (R,)}

  • Detection3D –

    (PyG layout).

rotate_points_along_z 

rotate_points_along_z(
    points: Tensor, angle: Tensor
) -> Tensor

Rotate point sets about the \(+z\) axis (angle increases \(x \to y\)).

Parameters:

  • points (Tensor) –

    Point sets, shape \((B, N, 3 + C)\); only the first three channels are rotated.

  • angle (Tensor) –

    Per-set yaw, shape \((B,)\).

Returns:

  • Tensor –

    The rotated point sets, shape \((B, N, 3 + C)\).

Shape
  • points: \((B, N, 3 + C)\)
  • angle: \((B,)\)
  • output: \((B, N, 3 + C)\)

decode_point_residuals 

decode_point_residuals(
    encodings: Tensor,
    points: Tensor,
    classes: Tensor,
    mean_sizes: Tensor,
) -> Tensor

Decode per-point box residuals with class mean-size anchors.

Decodes a stage-1 prediction \((x_t, y_t, z_t, d_{x,t}, d_{y,t}, d_{z,t}, \cos, \sin)\) at a foreground point into an oriented box \((c_x, c_y, c_z, d_x, d_y, d_z, \theta)\), using the predicted class mean size as the anchor for the size residuals.

Parameters:

  • encodings (Tensor) –

    Box residuals, shape \((N, 8)\).

  • points (Tensor) –

    Anchor point coordinates, shape \((N, 3)\).

  • classes (Tensor) –

    Predicted class index per point (\(1 \ldots \text{num\_classes}\)), shape \((N,)\).

  • mean_sizes (Tensor) –

    Per-class mean box size \((d_x, d_y, d_z)\), shape \((\text{num\_classes}, 3)\).

Returns:

  • Tensor –

    Decoded boxes \((c_x, c_y, c_z, d_x, d_y, d_z, \theta)\), shape \((N, 7)\).

Shape
  • encodings: \((N, 8)\)
  • points: \((N, 3)\)
  • output: \((N, 7)\)