pvcnn2

PVCNN++ classification and segmentation models.

First page of Point-Voxel CNN for Efficient 3D Deep Learning

1907.03739 · July 2019

Classes:

PVCNN2EncoderBlock –

One PVCNN++ encoder stage: optional set-abstraction downsampling followed by point-voxel convs.
PVCNN2Encoder –

Hierarchical PVCNN++ encoder.
PVCNN2DecoderBlock –

One PVCNN++ decoder stage: feature propagation followed by point-voxel conv refinement.
PVCNN2Decoder –

PVCNN++ decoder: a chain of feature-propagation blocks with point-voxel conv refinement.
PVCNN2Classification –

PVCNN++ classification model from
PVCNN2Segmentation –

PVCNN++ segmentation model from

PVCNN2EncoderBlock [source] ¶

PVCNN2EncoderBlock(
    in_channels: int,
    out_channels: int,
    depth: int,
    resolution: int,
    kernel_size: int,
    use_se: bool = False,
    normalize: bool = True,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    act_first: bool = False,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
    sa_module: Optional[PointNet2SetAbstraction] = None,
)

Bases: Module

One PVCNN++ encoder stage: optional set-abstraction downsampling followed by point-voxel convs.

When sa_module is given, the block first downsamples the cloud (farthest point sampling + ball grouping), then refines the abstracted features with depth point-voxel conv layers. A resolution of \(0\) or None swaps each point-voxel conv for a plain MLP layer.

Parameters:

in_channels (int) –

Number of input feature channels of the conv stack.
out_channels (int) –

Number of output feature channels of the conv stack.
depth (int) –

Number of point-voxel conv (or MLP) layers.
resolution (int) –

Voxel grid resolution of the point-voxel convs; \(0\) or None uses MLP layers.
kernel_size (int) –

Kernel size of the voxel branch convolutions.
use_se (bool, default: False ) –

Add a squeeze-and-excitation gate to every voxel branch.
normalize (bool, default: True ) –

Normalize coordinates into the unit voxel grid before voxelization.
act (Union[str, Callable, None], default: 'relu' ) –

Activation function.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation function.
act_first (bool, default: False ) –

Apply the activation before the normalization.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization function.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the normalization function.
sa_module (Optional[PointNet2SetAbstraction], default: None ) –

Optional set-abstraction module applied before the conv stack.

Shape

x: \((N, C_{in})\) point features.
pos: \((N, 3)\) point coordinates.
batch: \((N,)\) batch indices.
output: \((M, C_{out})\) features with matching pos / batch, where \(M \le N\) is the number of points kept by the set-abstraction module (\(M = N\) without one), and the \((M, C_{in})\) skip features the decoder consumes: the set-abstraction output before the conv stack (the block input without one).

PVCNN2Encoder [source] ¶

PVCNN2Encoder(
    *,
    channels: Sequence[int],
    depths: Sequence[int],
    resolutions: Sequence[Optional[int]],
    kernel_sizes: Sequence[int],
    use_se: bool = False,
    normalize: bool = True,
    sa_channels: Sequence[Sequence[Sequence[int]]],
    ratios: Sequence[float],
    radii: Sequence[Union[float, Sequence[float]]],
    num_neighbors: Sequence[Union[int, Sequence[int]]],
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    act_first: bool = False,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

Hierarchical PVCNN++ encoder.

Block \(0\) processes the full-resolution cloud with point-voxel conv layers; every following block \(i > 0\) first downsamples with a set-abstraction module configured by the \((i-1)\)-th entry of ratios / radii / num_neighbors / sa_channels, then refines with its own conv stack. Those four lists therefore pad to the block count and their final entry is unused.

Parameters:

channels (Sequence[int]) –

Per-block feature widths, one more entry than the number of blocks: channels[i + 1] is the output width of block \(i\).
depths (Sequence[int]) –

Number of point-voxel conv layers per block; \(0\) keeps the block set-abstraction-only.
resolutions (Sequence[Optional[int]]) –

Voxel grid resolution per block; \(0\) or None uses MLP layers instead of point-voxel convs.
kernel_sizes (Sequence[int]) –

Voxel conv kernel size per block.
use_se (bool, default: False ) –

Add a squeeze-and-excitation gate to every voxel branch.
normalize (bool, default: True ) –

Normalize coordinates into the unit voxel grid before voxelization.
sa_channels (Sequence[Sequence[Sequence[int]]]) –

MLP channels of each set-abstraction module. Nest twice for multi-scale grouping.
ratios (Sequence[float]) –

Farthest-point-sampling ratio per set-abstraction module.
radii (Sequence[Union[float, Sequence[float]]]) –

Ball-query radius per set-abstraction module. A nested sequence enables multi-scale grouping.
num_neighbors (Sequence[Union[int, Sequence[int]]]) –

Maximum number of neighbors per ball query.
act (Union[str, Callable, None], default: 'relu' ) –

Activation function.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation function.
act_first (bool, default: False ) –

Apply the activation before the normalization.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization function.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the normalization function.

Shape

x: \((N, C)\) point features with \(C = \text{channels}[0]\).
pos: \((N, 3)\) point coordinates.
batch: \((N,)\) batch indices.
output: \((M, \text{channels}[-1])\) features with matching pos / batch, where \(M\) is the number of points left after all set-abstraction stages. With return_intermediates, also the skips of every block but the last, fine to coarse: the encoder input, then each set-abstraction output before its block's conv stack.

PVCNN2DecoderBlock [source] ¶

PVCNN2DecoderBlock(
    in_channels: int,
    out_channels: int,
    depth: int,
    resolution: int,
    kernel_size: int,
    use_se: bool = False,
    normalize: bool = True,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    act_first: bool = False,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
    fp_module: Optional[PointNet2FeaturePropagation] = None,
)

Bases: Module

One PVCNN++ decoder stage: feature propagation followed by point-voxel conv refinement.

When fp_module is given, the block first upsamples the features to the skip resolution (k-NN interpolation + skip concatenation + MLP), then refines with depth point-voxel conv layers. A resolution of \(0\) or None swaps each point-voxel conv for a plain MLP layer.

Parameters:

in_channels (int) –

Number of input feature channels of the conv stack.
out_channels (int) –

Number of output feature channels of the conv stack.
depth (int) –

Number of point-voxel conv (or MLP) layers.
resolution (int) –

Voxel grid resolution of the point-voxel convs; \(0\) or None uses MLP layers.
kernel_size (int) –

Kernel size of the voxel branch convolutions.
use_se (bool, default: False ) –

Add a squeeze-and-excitation gate to every voxel branch.
normalize (bool, default: True ) –

Normalize coordinates into the unit voxel grid before voxelization.
act (Union[str, Callable, None], default: 'relu' ) –

Activation function.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation function.
act_first (bool, default: False ) –

Apply the activation before the normalization.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization function.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the normalization function.
fp_module (Optional[PointNet2FeaturePropagation], default: None ) –

Optional feature-propagation module applied before the conv stack.

Shape

x: \((M, C_{in})\) point features at the coarse resolution.
pos: \((M, 3)\) coarse point coordinates.
batch: \((M,)\) coarse batch indices.
x_skip / pos_skip / batch_skip: skip tensors at the target resolution \(N\).
output: \((N, C_{out})\) features with the skip pos / batch.

PVCNN2Decoder [source] ¶

PVCNN2Decoder(
    in_channels: int,
    depths: Sequence[int],
    channels: Sequence[int],
    skip_channels: Sequence[int],
    fp_channels: Sequence[Sequence[int]],
    resolutions: Sequence[Optional[int]],
    kernel_sizes: Sequence[int],
    use_se: bool = False,
    normalize: bool = True,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    act_first: bool = False,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

PVCNN++ decoder: a chain of feature-propagation blocks with point-voxel conv refinement.

Skips are consumed deepest-first from the encoder intermediates, and the last block always uses the earliest intermediate (the raw encoder input), so the chain ends at full resolution even when the encoder has more blocks than the decoder; the leftover shallow intermediates are skipped.

Parameters:

in_channels (int) –

Feature width entering the first block (the encoder output width).
depths (Sequence[int]) –

Number of point-voxel conv layers per block; \(0\) keeps the block feature-propagation-only.
channels (Sequence[int]) –

Per-block feature widths of the conv stacks.
skip_channels (Sequence[int]) –

Skip feature width consumed by each block, deepest first.
fp_channels (Sequence[Sequence[int]]) –

MLP channels of each feature-propagation module.
resolutions (Sequence[Optional[int]]) –

Voxel grid resolution per block; \(0\) or None uses MLP layers instead of point-voxel convs.
kernel_sizes (Sequence[int]) –

Voxel conv kernel size per block.
use_se (bool, default: False ) –

Add a squeeze-and-excitation gate to every voxel branch.
normalize (bool, default: True ) –

Normalize coordinates into the unit voxel grid before voxelization.
act (Union[str, Callable, None], default: 'relu' ) –

Activation function.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation function.
act_first (bool, default: False ) –

Apply the activation before the normalization.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization function.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the normalization function.

Shape

x: \((M, C_{in})\) encoder output features.
pos: \((M, 3)\) encoder output coordinates.
batch: \((M,)\) encoder output batch indices.
intermediates: per-encoder-block skip dicts with features / pos / batch keys.
output: \((N, C_{out})\) features at the resolution of the first intermediate.

PVCNN2Classification [source] ¶

PVCNN2Classification(
    in_channels: int,
    num_classes: int,
    *,
    ratios: Sequence[float],
    radii: Sequence[Union[float, Sequence[float]]],
    num_neighbors: Sequence[Union[int, Sequence[int]]],
    sa_channels: Sequence[
        Sequence[Union[int, Sequence[int]]]
    ],
    encoder_channels: Sequence[int],
    encoder_depths: Sequence[int],
    encoder_resolutions: Sequence[Optional[int]],
    encoder_kernel_sizes: Sequence[int],
    use_se: bool = False,
    normalize: bool = True,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    act_first: bool = False,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
    dropout: float = 0.0,
    global_pool: PoolLike = "max",
)

Bases: ClassificationModel

PVCNN++ classification model from Point-Voxel CNN for Efficient 3D Deep Learning by Zhijian Liu, Haotian Tang, Yujun Lin, Song Han.

PVCNN++ grafts point-voxel convolutions onto a PointNet++-style hierarchy: every encoder block downsamples the cloud with a set-abstraction module (farthest point sampling + ball grouping), then refines the abstracted features with PVConv layers that fuse a coarse voxel-grid convolution branch with a per-point MLP branch. The final features are pooled into a global embedding and classified with a linear head.

Parameters:

in_channels (int) –

Number of input feature channels. Takes precedence over encoder_channels[0] when the two disagree.
num_classes (int) –

Number of output classes. \(0\) replaces the head with nn.Identity.
ratios (Sequence[float]) –

Farthest-point-sampling ratio per encoder block. Block \(i\) uses ratios[i - 1] (block \(0\) has no set-abstraction module), so the list pads to the block count and its final entry is unused.
radii (Sequence[Union[float, Sequence[float]]]) –

Ball-query radius per encoder block, aligned like ratios. A nested sequence enables multi-scale grouping.
num_neighbors (Sequence[Union[int, Sequence[int]]]) –

Maximum number of neighbors per ball query, aligned like ratios.
sa_channels (Sequence[Sequence[Union[int, Sequence[int]]]]) –

MLP channels of each set-abstraction module, aligned like ratios. Nest twice for multi-scale grouping.
encoder_channels (Sequence[int]) –

Per-block feature widths, one more entry than the number of blocks: encoder_channels[i + 1] is the output width of block \(i\).
encoder_depths (Sequence[int]) –

Number of point-voxel conv layers per block; \(0\) keeps the block set-abstraction-only.
encoder_resolutions (Sequence[Optional[int]]) –

Voxel grid resolution per block; \(0\) or None uses MLP layers instead of point-voxel convs.
encoder_kernel_sizes (Sequence[int]) –

Voxel conv kernel size per block.
use_se (bool, default: False ) –

Add a squeeze-and-excitation gate to every voxel branch.
normalize (bool, default: True ) –

Normalize coordinates into the unit voxel grid before voxelization.
act (Union[str, Callable, None], default: 'relu' ) –

Activation function.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation function.
act_first (bool, default: False ) –

Apply the activation before the normalization.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization function.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the normalization function.
dropout (float, default: 0.0 ) –

Dropout probability applied to the pooled global embedding.
global_pool (PoolLike, default: 'max' ) –

Pooling used to aggregate per-point features into the global embedding.

Shape

x: \((N, C_{in})\) point features; when None, pos is used as features.
pos: \((N, 3)\) point coordinates.
batch: \((N,)\) batch indices.
output: \((B, \text{num\_classes})\) classification logits.

Methods:

configure_encoder –

Build the PVCNN2Encoder backbone.

Attributes:

num_features (int) –

Feature dimension \(C\) of the encoder output.

num_features `property` ¶

num_features: int

Feature dimension \(C\) of the encoder output.

configure_encoder [source] ¶

configure_encoder() -> PVCNN2Encoder

Build the PVCNN2Encoder backbone.

PVCNN2Segmentation [source] ¶

PVCNN2Segmentation(
    in_channels: int,
    num_classes: int,
    *,
    ratios: Sequence[float],
    radii: Sequence[Union[float, Sequence[float]]],
    num_neighbors: Sequence[Union[int, Sequence[int]]],
    sa_channels: Sequence[
        Sequence[Union[int, Sequence[int]]]
    ],
    encoder_channels: Sequence[int],
    encoder_depths: Sequence[int],
    encoder_resolutions: Sequence[Optional[int]],
    encoder_kernel_sizes: Sequence[int],
    fp_channels: Sequence[Sequence[int]],
    decoder_channels: Sequence[int],
    decoder_depths: Sequence[int],
    decoder_resolutions: Sequence[Optional[int]],
    decoder_kernel_sizes: Sequence[int],
    use_se: bool = False,
    normalize: bool = True,
    act: Union[str, Callable, None] = "relu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    act_first: bool = False,
    norm: Union[str, Callable, None] = "batch_norm",
    norm_kwargs: Optional[Dict[str, Any]] = None,
    dropout: float = 0.0,
    head_channels: Optional[Sequence[int]] = None,
    head_dropout: float = 0.0,
)

Bases: SemanticSegmentationModel

PVCNN++ segmentation model from Point-Voxel CNN for Efficient 3D Deep Learning by Zhijian Liu, Haotian Tang, Yujun Lin, Song Han.

A U-shaped PVCNN++: the encoder alternates set-abstraction downsampling with point-voxel conv refinement, and the decoder upsamples back with feature-propagation blocks followed by point-voxel convs. Decoder skips are consumed deepest-first, and the last decoder block uses the extra input features (the input without its 3 leading coordinate channels) as its skip, so the encoder may have more blocks than the decoder (the leftover shallow intermediates are skipped).

Parameters:

in_channels (int) –

Number of input feature channels. Takes precedence over encoder_channels[0] when the two disagree; \(0\) falls back to \(3\) (positions used as features).
num_classes (int) –

Number of output classes. \(0\) replaces the head with nn.Identity.
ratios (Sequence[float]) –

Farthest-point-sampling ratio per encoder block. Block \(i\) uses ratios[i - 1] (block \(0\) has no set-abstraction module), so the list pads to the block count and its final entry is unused.
radii (Sequence[Union[float, Sequence[float]]]) –

Ball-query radius per encoder block, aligned like ratios. A nested sequence enables multi-scale grouping.
num_neighbors (Sequence[Union[int, Sequence[int]]]) –

Maximum number of neighbors per ball query, aligned like ratios.
sa_channels (Sequence[Sequence[Union[int, Sequence[int]]]]) –

MLP channels of each set-abstraction module, aligned like ratios. Nest twice for multi-scale grouping.
encoder_channels (Sequence[int]) –

Per-block feature widths, one more entry than the number of encoder blocks: encoder_channels[i + 1] is the output width of block \(i\).
encoder_depths (Sequence[int]) –

Number of point-voxel conv layers per encoder block; \(0\) keeps the block set-abstraction-only.
encoder_resolutions (Sequence[Optional[int]]) –

Voxel grid resolution per encoder block; \(0\) or None uses MLP layers instead of point-voxel convs.
encoder_kernel_sizes (Sequence[int]) –

Voxel conv kernel size per encoder block.
fp_channels (Sequence[Sequence[int]]) –

MLP channels of each feature-propagation module, deepest stage first.
decoder_channels (Sequence[int]) –

Per-block feature widths of the decoder conv stacks.
decoder_depths (Sequence[int]) –

Number of point-voxel conv layers per decoder block; \(0\) keeps the block feature-propagation-only. At most one decoder block per set-abstraction level (encoder blocks minus one).
decoder_resolutions (Sequence[Optional[int]]) –

Voxel grid resolution per decoder block; \(0\) or None uses MLP layers instead of point-voxel convs.
decoder_kernel_sizes (Sequence[int]) –

Voxel conv kernel size per decoder block.
use_se (bool, default: False ) –

Add a squeeze-and-excitation gate to every voxel branch.
normalize (bool, default: True ) –

Normalize coordinates into the unit voxel grid before voxelization.
act (Union[str, Callable, None], default: 'relu' ) –

Activation function.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation function.
act_first (bool, default: False ) –

Apply the activation before the normalization.
norm (Union[str, Callable, None], default: 'batch_norm' ) –

Normalization function.
norm_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the normalization function.
dropout (float, default: 0.0 ) –

Dropout probability before the linear head. Ignored when head_channels is set: the MLP head applies its own head_dropout.
head_channels (Optional[Sequence[int]], default: None ) –

Hidden widths of the segmentation head MLP. None uses a single linear layer.
head_dropout (float, default: 0.0 ) –

Dropout probability inside the head MLP.

Shape

x: \((N, C_{in})\) point features; when None, pos is used as features.
pos: \((N, 3)\) point coordinates.
batch: \((N,)\) batch indices.
output: \((N, \text{num\_classes})\) per-point segmentation logits.

Methods:

configure_encoder –

Build the PVCNN2Encoder backbone.
configure_decoder –

Build the PVCNN2Decoder upsampling the coarsest features back through the encoder skips.

Attributes:

num_features (int) –

Feature dimension \(C\) of the decoder output.

num_features `property` ¶

num_features: int

Feature dimension \(C\) of the decoder output.

configure_encoder [source] ¶

configure_encoder() -> PVCNN2Encoder

Build the PVCNN2Encoder backbone.

configure_decoder [source] ¶

configure_decoder() -> PVCNN2Decoder

Build the PVCNN2Decoder upsampling the coarsest features back through the encoder skips.

pvcnn2

PVCNN2EncoderBlock [source] ¶

PVCNN2Encoder [source] ¶

PVCNN2DecoderBlock [source] ¶

PVCNN2Decoder [source] ¶

PVCNN2Classification [source] ¶

num_features property ¶

configure_encoder [source] ¶

PVCNN2Segmentation [source] ¶

num_features property ¶

configure_encoder [source] ¶

configure_decoder [source] ¶

num_features `property` ¶

num_features `property` ¶