PointGPT

PointGPT classification and generative pretraining models.

First page of PointGPT: Auto-regressively Generative Pre-training from Point Clouds

2305.11487 · May 2023

Classes:

PositionEmbeddingSine –

Parameter-free sinusoidal positional embedding of continuous coordinates.
PointGPTBlock –

GPT transformer block of PointGPT (masked multi-head attention then a residual MLP).
PointGPTExtractor –

Auto-regressive GPT extractor (encoder) of PointGPT.
PointGPTGenerator –

Auto-regressive GPT generator (decoder) of PointGPT.
PointGPTClassification –

PointGPT classification model.
PointGPTGenerativePretraining –

PointGPT auto-regressive generative pretraining model.

Functions:

morton_sort –

Greedy nearest-neighbor ("simplified Morton") ordering of patch centers.

PositionEmbeddingSine ¶

PositionEmbeddingSine(
    spatial_dim: int = 3,
    embed_dim: int = 384,
    temperature: float = 10000.0,
    scale: Optional[float] = None,
)

Bases: Module

Parameter-free sinusoidal positional embedding of continuous coordinates.

Implements the PositionEmbeddingCoordsSine of PointGPT: Auto-regressively Generative Pre-training from Point Clouds, adapted from CGuangyan-BIT/PointGPT. Each input dimension is scaled by \(2 \pi\) and expanded into interleaved sine / cosine features over a geometric range of frequencies; unused channels are zero-padded.

Parameters:

spatial_dim (int, default: 3 ) –

The number of input coordinate dimensions \(n\).
embed_dim (int, default: 384 ) –

The output embedding dimension \(d\).
temperature (float, default: 10000.0 ) –

The frequency base of the geometric progression.
scale (Optional[float], default: None ) –

The coordinate scale applied before \(2 \pi\) (defaults to \(1\)).

Shape

Input: \((*, n)\).
Output: \((*, d)\).

Example

import torch
from torch_pointcloud.models.pointgpt import PositionEmbeddingSine

pe = PositionEmbeddingSine(spatial_dim=3, embed_dim=384)
center = torch.randn(2, 64, 3)
emb = pe(center)
print(emb.shape)

PointGPTBlock ¶

PointGPTBlock(
    embed_dim: int,
    num_heads: int,
    mlp_ratio: float = 4.0,
    act: Union[str, Callable, None] = "gelu",
    act_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

GPT transformer block of PointGPT (masked multi-head attention then a residual MLP).

Implements the Block of PointGPT: Auto-regressively Generative Pre-training from Point Clouds, adapted from CGuangyan-BIT/PointGPT. Unlike the plain pre-norm ViT block, the attention residual adds the raw attention output to the normalized input (\(x \leftarrow \text{Norm}_1(x) + \text{Attn}(\text{Norm}_1(x)))\), then a standard residual MLP follows. Self-attention uses torch.nn.MultiheadAttention with an additive causal / masking attn_mask, so the weights are the fused in_proj / out_proj of that module.

Parameters:

embed_dim (int) –

The token dimension \(C\).
num_heads (int) –

The number of attention heads.
mlp_ratio (float, default: 4.0 ) –

The hidden-to-input ratio of the feed-forward MLP.
act (Union[str, Callable, None], default: 'gelu' ) –

The activation of the feed-forward MLP.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation.

Shape

Input: \((L, B, C)\) tokens and an additive / boolean attn_mask of shape \((L, L)\).
Output: \((L, B, C)\).

Example

import torch
from torch_pointcloud.models.pointgpt import PointGPTBlock

block = PointGPTBlock(embed_dim=384, num_heads=6)
x = torch.randn(66, 2, 384)
mask = torch.triu(torch.ones(66, 66, dtype=torch.bool), diagonal=1)
y = block(x, mask)
print(y.shape)

PointGPTExtractor ¶

PointGPTExtractor(
    embed_dim: int,
    num_heads: int,
    depth: int,
    act: Union[str, Callable, None] = "gelu",
    act_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

Auto-regressive GPT extractor (encoder) of PointGPT.

Implements the GPT_extractor of PointGPT: Auto-regressively Generative Pre-training from Point Clouds, adapted from CGuangyan-BIT/PointGPT. A start-of-sequence token is prepended to the patch-token sequence, a stack of causally-masked PointGPTBlock layers consumes the tokens plus their positional embedding, and a final layer normalization produces the encoded points. This is the backbone reused for downstream classification.

Parameters:

embed_dim (int) –

The token dimension \(C\).
num_heads (int) –

The number of attention heads.
depth (int) –

The number of transformer blocks.
act (Union[str, Callable, None], default: 'gelu' ) –

The activation of the feed-forward MLPs.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation.

Shape

Input: \((B, L, C)\) tokens, \((B, L + 1, C)\) positions, and an \((L + 1, L + 1)\) mask.
Output: \((B, L + 1, C)\) encoded points.

Example

import torch
from torch_pointcloud.models.pointgpt import PointGPTExtractor

extractor = PointGPTExtractor(embed_dim=384, num_heads=6, depth=12)
tokens = torch.randn(2, 65, 384)
pos = torch.randn(2, 66, 384)
mask = torch.triu(torch.ones(66, 66, dtype=torch.bool), diagonal=1)
out = extractor(tokens, pos, mask)
print(out.shape)

PointGPTGenerator ¶

PointGPTGenerator(
    embed_dim: int,
    num_heads: int,
    depth: int,
    group_size: int,
    act: Union[str, Callable, None] = "gelu",
    act_kwargs: Optional[Dict[str, Any]] = None,
)

Bases: Module

Auto-regressive GPT generator (decoder) of PointGPT.

Implements the GPT_generator of PointGPT: Auto-regressively Generative Pre-training from Point Clouds, adapted from CGuangyan-BIT/PointGPT. A stack of causally-masked PointGPTBlock layers consumes the encoded points plus a relative positional embedding and a per-token head reconstructs the next patch's \(M\) centered coordinates.

Parameters:

embed_dim (int) –

The token dimension \(C\).
num_heads (int) –

The number of attention heads.
depth (int) –

The number of transformer blocks.
group_size (int) –

The number of points \(M\) reconstructed per patch.
act (Union[str, Callable, None], default: 'gelu' ) –

The activation of the feed-forward MLPs.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation.

Shape

Input: \((B, L, C)\) encoded points, \((B, L, C)\) positions, and an \((L, L)\) mask.
Output: \((B \cdot L, M, 3)\) reconstructed patch coordinates.

Example

import torch
from torch_pointcloud.models.pointgpt import PointGPTGenerator

generator = PointGPTGenerator(embed_dim=384, num_heads=6, depth=4, group_size=32)
x = torch.randn(2, 64, 384)
pos = torch.randn(2, 64, 384)
mask = torch.triu(torch.ones(64, 64, dtype=torch.bool), diagonal=1)
out = generator(x, pos, mask)
print(out.shape)

PointGPTClassification ¶

PointGPTClassification(
    in_channels: int,
    num_classes: int,
    *,
    embed_dim: int = 384,
    depth: int = 12,
    num_heads: int = 6,
    num_group: int = 64,
    group_size: int = 32,
    decoder_depth: int = 4,
    encoder_local_channels: Sequence[int] = (128, 256),
    encoder_global_channels: Sequence[int] = (512,),
    dropout: float = 0.5,
    act: Union[str, Callable, None] = "gelu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    head_act: Union[str, Callable, None] = "relu",
    global_pool: AdaptivePoolLike = "max",
    spatial_dim: int = 3,
)

Bases: ClassificationModel

PointGPT classification model.

Implements the finetuning model (PointTransformer) of PointGPT: Auto-regressively Generative Pre-training from Point Clouds, adapted from CGuangyan-BIT/PointGPT. Patches are tokenized with a mini-PointNet, ordered by a greedy nearest-neighbor ("simplified Morton") traversal of the centers, and consumed by a causally-masked GPT extractor that prepends a start-of-sequence and a class token. The global feature concatenates the class-token output with the pooled patch outputs (global_pool, max-pool by default), so the head input dimension is \(2 \cdot \text{embed\_dim}\).

Parameters:

in_channels (int) –

The number of input channels (PointGPT uses coordinates only, so \(0\)).
num_classes (int) –

The number of output classes.
embed_dim (int, default: 384 ) –

The transformer / token-embedding dimension.
depth (int, default: 12 ) –

The number of extractor blocks.
num_heads (int, default: 6 ) –

The number of attention heads.
num_group (int, default: 64 ) –

The number of patches \(G\).
group_size (int, default: 32 ) –

The number of points \(M\) per patch.
decoder_depth (int, default: 4 ) –

The number of generator blocks (kept for weight compatibility).
encoder_local_channels (Sequence[int], default: (128, 256) ) –

Hidden widths of the patch embedder's per-point MLP.
encoder_global_channels (Sequence[int], default: (512,) ) –

Hidden widths of the patch embedder's per-patch MLP.
dropout (float, default: 0.5 ) –

The dropout rate of the classification head.
act (Union[str, Callable, None], default: 'gelu' ) –

The activation of the transformer MLPs.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation.
head_act (Union[str, Callable, None], default: 'relu' ) –

The activation of the classification head.
global_pool (AdaptivePoolLike, default: 'max' ) –

The pooling over the patch tokens for the global feature ("max" or "mean").
spatial_dim (int, default: 3 ) –

The spatial dimension of the input point cloud.

Shape

Input: \((N, 3)\) and \((N,)\).
Output: \((B, \text{num\_classes})\).

Example

import torch
from torch_pointcloud.models.pointgpt import PointGPTClassification

model = PointGPTClassification(in_channels=0, num_classes=40)
pos = torch.randn(2048, 3)
batch = torch.cat([torch.zeros(1024), torch.ones(1024)]).long()
logits = model(None, pos, batch)
print(logits.shape)

Methods:

configure_encoder –

Build the mini-PointNet patch embedder tokenizing each patch.
configure_pos_embed –

Build the sinusoidal positional embedding of the patch centers.
configure_blocks –

Build the causally-masked GPT extractor.
configure_generator_blocks –

Build the GPT generator predicting the next patch (unused at finetuning, kept for weight compatibility).

Attributes:

num_features (int) –

Channel count \(C\) of the pooled features entering the head.

num_features `property` ¶

num_features: int

Channel count \(C\) of the pooled features entering the head.

configure_encoder ¶

configure_encoder() -> PointPatchEmbed

Build the mini-PointNet patch embedder tokenizing each patch.

configure_pos_embed ¶

configure_pos_embed() -> PositionEmbeddingSine

Build the sinusoidal positional embedding of the patch centers.

configure_blocks ¶

configure_blocks() -> PointGPTExtractor

Build the causally-masked GPT extractor.

configure_generator_blocks ¶

configure_generator_blocks() -> PointGPTGenerator

Build the GPT generator predicting the next patch (unused at finetuning, kept for weight compatibility).

PointGPTGenerativePretraining ¶

PointGPTGenerativePretraining(
    in_channels: int,
    *,
    embed_dim: int = 384,
    depth: int = 12,
    decoder_depth: int = 4,
    num_heads: int = 6,
    num_group: int = 64,
    group_size: int = 32,
    encoder_local_channels: Sequence[int] = (128, 256),
    encoder_global_channels: Sequence[int] = (512,),
    mask_ratio: float = 0.7,
    keep_attend: int = 10,
    act: Union[str, Callable, None] = "gelu",
    act_kwargs: Optional[Dict[str, Any]] = None,
    spatial_dim: int = 3,
)

Bases: BaseModel

PointGPT auto-regressive generative pretraining model.

Implements the pretraining model (PointGPT / GPT_Transformer) of PointGPT: Auto-regressively Generative Pre-training from Point Clouds, adapted from CGuangyan-BIT/PointGPT.

Patches are tokenized, ordered by a greedy nearest-neighbor ("simplified Morton") traversal, and fed to a causally-masked GPT extractor with an additional column mask that randomly hides patches beyond the first keep_attend tokens (the dual-masking strategy). The generator then predicts the next patch from the extractor features and a relative positional embedding. forward returns the predicted and target patch coordinates for a set-to-set reconstruction objective such as chamfer_distance from torch_pointcloud.losses.

Parameters:

in_channels (int) –

The number of input channels (\(0\), coordinates only).
embed_dim (int, default: 384 ) –

The transformer / token-embedding dimension.
depth (int, default: 12 ) –

The number of extractor blocks.
decoder_depth (int, default: 4 ) –

The number of generator blocks.
num_heads (int, default: 6 ) –

The number of attention heads (shared by the extractor and the generator).
num_group (int, default: 64 ) –

The number of patches \(G\).
group_size (int, default: 32 ) –

The number of points \(M\) per patch.
encoder_local_channels (Sequence[int], default: (128, 256) ) –

Hidden widths of the patch embedder's per-point MLP.
encoder_global_channels (Sequence[int], default: (512,) ) –

Hidden widths of the patch embedder's per-patch MLP.
mask_ratio (float, default: 0.7 ) –

The fraction of maskable patches hidden by the column mask.
keep_attend (int, default: 10 ) –

The number of leading patches never hidden by the column mask.
act (Union[str, Callable, None], default: 'gelu' ) –

The activation of the transformer MLPs.
act_kwargs (Optional[Dict[str, Any]], default: None ) –

Keyword arguments for the activation.
spatial_dim (int, default: 3 ) –

The spatial dimension of the input point cloud.

Shape

Input: \((N, 3)\) and \((N,)\).
Output: predicted and target patches, each of shape \((B \cdot G, M, 3)\).

Example

import torch
from torch_pointcloud.models.pointgpt import PointGPTGenerativePretraining

model = PointGPTGenerativePretraining(in_channels=0)
pos = torch.randn(2048, 3)
batch = torch.cat([torch.zeros(1024), torch.ones(1024)]).long()
pred, target = model(None, pos, batch)
print(pred.shape, target.shape)

Methods:

configure_encoder –

Build the mini-PointNet patch embedder tokenizing each patch.
configure_pos_embed –

Build the sinusoidal positional embedding of the patch centers and relative directions.
configure_blocks –

Build the causally-masked GPT extractor.
configure_generator_blocks –

Build the GPT generator predicting the next patch from the extractor features.

configure_encoder ¶

configure_encoder() -> PointPatchEmbed

Build the mini-PointNet patch embedder tokenizing each patch.

configure_pos_embed ¶

configure_pos_embed() -> PositionEmbeddingSine

Build the sinusoidal positional embedding of the patch centers and relative directions.

configure_blocks ¶

configure_blocks() -> PointGPTExtractor

Build the causally-masked GPT extractor.

configure_generator_blocks ¶

configure_generator_blocks() -> PointGPTGenerator

Build the GPT generator predicting the next patch from the extractor features.

morton_sort ¶

morton_sort(center: Tensor) -> Tensor

Greedy nearest-neighbor ("simplified Morton") ordering of patch centers.

Reproduces the simplied_morton_sorting of PointGPT: Auto-regressively Generative Pre-training from Point Clouds, adapted from CGuangyan-BIT/PointGPT. Starting from the first center, the next center is repeatedly chosen as the nearest not-yet-visited center to the last one, giving a space-filling traversal that approximates a Z-order (Morton) curve.

Parameters:

center (Tensor) –

Patch centers of shape \((B, G, 3)\).

Returns:

Tensor –

A permutation of shape \((B, G)\) indexing the \(G\) centers of each sample.

Shape

Input: \((B, G, 3)\).
Output: \((B, G)\).

Example

import torch
from torch_pointcloud.models.pointgpt import morton_sort

center = torch.randn(2, 64, 3)
order = morton_sort(center)
print(order.shape)

PointGPT

PositionEmbeddingSine ¶

PointGPTBlock ¶

PointGPTExtractor ¶

PointGPTGenerator ¶

PointGPTClassification ¶

num_features property ¶

configure_encoder ¶

configure_pos_embed ¶

configure_blocks ¶

configure_generator_blocks ¶

PointGPTGenerativePretraining ¶

configure_encoder ¶

configure_pos_embed ¶

configure_blocks ¶

configure_generator_blocks ¶

morton_sort ¶

num_features `property` ¶