utils

Download, archive extraction, and hash verification helpers for disk-cached datasets.

Functions:

urltailname –

Get the name of the last segment of a URL.
urlsize –

Get the size of a URL.
download_url –

Download a file from a URL to a local path.
extract_zip –

Extract a zip file to a directory.
extract_tar –

Extract a tar file to a directory.
compute_hash –

Compute the hash of a file.
check_cache_meta –

Validate a processed cache against the parameters it was built with.
is_hash_valid –

Check if the hash of a file matches the expected hash.

urltailname ¶

urltailname(url: str) -> str

Get the name of the last segment of a URL.

Parameters:

url (str) –

The URL to get the basename from.

Returns:

str –

The (decoded) name of the last segment of the URL.

Examples:

>>> urltailname("https://example.com/file.zip")
'file.zip'
>>> urltailname("https://example.com/path/to/my%20file.zip")
'my file.zip'

urlsize ¶

urlsize(
    url: str,
    *,
    timeout: Optional[float] = None,
    cafile: Optional[str] = None,
    capath: Optional[str] = None,
    cadefault: bool = False,
    context: Optional[SSLContext] = None,
) -> Optional[int]

Get the size of a URL.

Parameters:

url (str) –

The URL to get the size of.
timeout (Optional[float], default: None ) –

Optional timeout in seconds for the request.
cafile (Optional[str], default: None ) –

Optional path to a CA file.
capath (Optional[str], default: None ) –

Optional path to a directory with CA certificates.
cadefault (bool, default: False ) –

Whether to use the default CA store.
context (Optional[SSLContext], default: None ) –

Optional SSLContext for the request. Built from cafile / capath / cadefault when not provided.

Returns:

Optional[int] –

The size of the URL in bytes, or None if the response has no content-length header.

Examples:

>>> urlsize("https://example.com/file.zip")  # doctest: +SKIP
1024

download_url ¶

download_url(
    url: str,
    file_path: PathLike = "",
    chunk_size: int = 1024 * 32,
    description: str = "Downloading",
    show_progress: bool = True,
    overwrite: Union[bool, Literal["incomplete"]] = False,
    timeout: float = 30.0,
) -> str

Download a file from a URL to a local path.

Parameters:

url (str) –

The URL to download the file from.
file_path (PathLike, default: '' ) –

The local path to save the file to (including the file name). If not provided, the file will be saved in the current working directory with the name taken from `Path
chunk_size (int, default: 1024 * 32 ) –

The size of the chunks to download in bytes.
description (str, default: 'Downloading' ) –

The description to display in the progress bar.
show_progress (bool, default: True ) –

Whether to display a progress bar.
overwrite (Union[bool, Literal['incomplete']], default: False ) –

Whether to overwrite the file if it already exists. If True, the local file will be overwritten even if it already exists. If 'incomplete', the local file will be overwritten if it already exists and its size does not match the expected size (when the remote size is unknown, the local file is kept). If False, the local file will not be overwritten if it already exists.
timeout (float, default: 30.0 ) –

Timeout in seconds passed to urlopen, covering the connection and each blocking read, so a stalled server raises instead of hanging forever.

Returns:

str –

The local path to the downloaded file.

Examples:

>>> download_url("https://example.com/file.zip")  # doctest: +SKIP
"file.zip"
>>> download_url("https://example.com/my%20file.zip", "my_file.zip", show_progress=False)  # doctest: +SKIP
"my_file.zip"

extract_zip ¶

extract_zip(
    zip_path: PathLike,
    out_dir: PathLike,
    relative_to: PathLike = "",
    show_progress: bool = True,
) -> str

Extract a zip file to a directory.

Parameters:

zip_path (PathLike) –

The path to the zip file to extract.
out_dir (PathLike) –

The directory to extract the zip file to.
relative_to (PathLike, default: '' ) –

If provided, extract the zip file relative to this directory. This is useful when the zip file contains nested directories but you want to extract the files on a specific level (e.g. for a nested zip A.zip containing files under directory A, then you will get A/A/*.png when extracting, but setting relative_to="A" will extract the files to A/*.png only).
show_progress (bool, default: True ) –

Whether to display a progress bar.

Returns:

str –

The path to the extracted directory.

Examples:

>>> extract_zip("A.zip", "A")  # doctest: +SKIP
"A"

extract_tar ¶

extract_tar(
    tar_path: PathLike,
    dst_dir: PathLike,
    /,
    relative_to: PathLike = "",
    show_progress: bool = True,
) -> str

Extract a tar file to a directory.

Parameters:

tar_path (PathLike) –

The path to the tar file to extract.
dst_dir (PathLike) –

The directory to extract the tar file to.
relative_to (PathLike, default: '' ) –

If provided, extract the tar file relative to this directory.
show_progress (bool, default: True ) –

Whether to display a progress bar.

compute_hash ¶

compute_hash(
    file_path: PathLike, hash_type: HashType = "md5"
) -> str

Compute the hash of a file.

Parameters:

file_path (PathLike) –

The path to the file to hash.
hash_type (HashType, default: 'md5' ) –

The type of hash to compute.

Returns:

str –

The hex digest of the file's hash.

Examples:

>>> compute_hash("file.zip")  # doctest: +SKIP
'9473fdd0d880a43c21b7778d34872157'

check_cache_meta ¶

check_cache_meta(
    meta_path: PathLike, meta: Dict[str, Any]
) -> None

Validate a processed cache against the parameters it was built with.

Compares the JSON metadata stored next to a processed cache with the metadata the current constructor parameters would produce, and raises a RuntimeError on mismatch so a stale cache is never silently served. A missing metadata file (legacy cache) is accepted as-is.

Parameters:

meta_path (PathLike) –

Path to the cache meta.json file.
meta (Dict[str, Any]) –

The metadata the requested parameters would produce.

Examples:

>>> check_cache_meta("data/ModelNet10/processed/train.meta.json", {"classes": ["chair"]})  # doctest: +SKIP

is_hash_valid ¶

is_hash_valid(
    file_path: PathLike,
    expected_hash: Optional[str] = None,
    hash_type: HashType = "md5",
) -> bool

Check if the hash of a file matches the expected hash.

Parameters:

file_path (PathLike) –

The path to the file to check the hash of.
expected_hash (Optional[str], default: None ) –

The expected hash of the file. When None, a UserWarning is emitted and the check passes without verifying the file.
hash_type (HashType, default: 'md5' ) –

The type of hash to use for the comparison. Prefer sha256 for new datasets.

Returns:

bool –

True if the hash of the file matches the expected hash, False otherwise.

Examples:

>>> is_hash_valid("file.zip", "f7f6b4e3a3e0f8e4e3e3e3e3e3e3", "md5")  # doctest: +SKIP
False
>>> is_hash_valid("file.zip", "f7f6b4e3a3e0f8e4e3e3e3e3e3e3", "sha256")  # doctest: +SKIP
True