hats.catalog.dataset

hats.catalog.dataset#

Submodules#

Classes#

Dataset

A base HATS dataset that contains a properties file

Package Contents#

class Dataset(catalog_info: hats.catalog.dataset.table_properties.TableProperties, catalog_path: str | pathlib.Path | upath.UPath | None = None, schema: pyarrow.Schema | None = None)[source]#

A base HATS dataset that contains a properties file and the data contained in parquet files

catalog_info#
catalog_name#
catalog_path = None#
on_disk#
catalog_base_dir = None#
schema = None#
aggregate_column_statistics(exclude_hats_columns: bool = True, exclude_columns: list[str] = None, include_columns: list[str] = None)[source]#

Read footer statistics in parquet metadata, and report on global min/max values.

Parameters:

  • exclude_hats_columns (bool) – exclude HATS spatial and partitioning fields from the statistics. Defaults to True.

  • exclude_columns (List[str]) – additional columns to exclude from the statistics.

  • include_columns (List[str]) – if specified, only return statistics for the column names provided. Defaults to None, and returns all non-hats columns.

per_pixel_statistics(exclude_hats_columns: bool = True, exclude_columns: list[str] = None, include_columns: list[str] = None, include_stats: list[str] = None, multi_index=False)[source]#

Read footer statistics in parquet metadata, and report on statistics about each pixel partition.

Parameters:

  • exclude_hats_columns (bool) – exclude HATS spatial and partitioning fields from the statistics. Defaults to True.

  • exclude_columns (List[str]) – additional columns to exclude from the statistics.

  • include_columns (List[str]) – if specified, only return statistics for the column names provided. Defaults to None, and returns all non-hats columns.

  • include_stats (List[str]) – if specified, only return the kinds of values from list (min_value, max_value, null_count, row_count). Defaults to None, and returns all values.

  • multi_index (bool) – should the returned frame be created with a multi-index, first on pixel, then on column name? Default is False, and instead indexes on pixel, with

  • combination. (separate columns per-data-column and stat value)

Contents