hats.io

hats.io#

Utilities for reading and writing catalog files

Submodules#

Functions#

write_parquet_metadata(catalog_path[, ...])

Generate parquet metadata, using the already-partitioned parquet files

get_common_metadata_pointer(→ upath.UPath)

Get file pointer to _common_metadata parquet metadata file

get_parquet_metadata_pointer(→ upath.UPath)

Get file pointer to _metadata parquet metadata file

get_partition_info_pointer(→ upath.UPath)

Get file pointer to partition_info.csv metadata file

get_point_map_file_pointer(→ upath.UPath)

Get file pointer to point_map.fits FITS image file.

pixel_catalog_file(→ upath.UPath)

Create path pointer for a pixel catalog file. This will not create the directory

pixel_directory(→ upath.UPath)

Create path pointer for a pixel directory. This will not create the directory.

Package Contents#

write_parquet_metadata(catalog_path: str | pathlib.Path | upath.UPath, order_by_healpix=True, output_path: str | pathlib.Path | upath.UPath | None = None)[source]#

Generate parquet metadata, using the already-partitioned parquet files for this catalog.

For more information on the general parquet metadata files, and why we write them, see https://arrow.apache.org/docs/python/parquet.html#writing-metadata-and-common-metadata-files

Parameters:

  • catalog_path (str) – base path for the catalog

  • order_by_healpix (bool) – use False if the dataset is not to be reordered by breadth-first healpix pixel (e.g. secondary indexes)

  • output_path (str) – base path for writing out metadata files defaults to catalog_path if unspecified

Returns:

sum of the number of rows in the dataset.

get_common_metadata_pointer(catalog_base_dir: str | pathlib.Path | upath.UPath) → upath.UPath[source]#

Get file pointer to _common_metadata parquet metadata file

Parameters:

catalog_base_dir – pointer to base catalog directory

Returns:

File Pointer to the catalog’s _common_metadata file

get_parquet_metadata_pointer(catalog_base_dir: str | pathlib.Path | upath.UPath) → upath.UPath[source]#

Get file pointer to _metadata parquet metadata file

Parameters:

catalog_base_dir – pointer to base catalog directory

Returns:

File Pointer to the catalog’s _metadata file

get_partition_info_pointer(catalog_base_dir: str | pathlib.Path | upath.UPath) → upath.UPath[source]#

Get file pointer to partition_info.csv metadata file

Parameters:

catalog_base_dir – pointer to base catalog directory

Returns:

File Pointer to the catalog’s partition_info.csv file

get_point_map_file_pointer(catalog_base_dir: str | pathlib.Path | upath.UPath) → upath.UPath[source]#

Get file pointer to point_map.fits FITS image file.

Parameters:

catalog_base_dir – pointer to base catalog directory

Returns:

File Pointer to the catalog’s point_map.fits FITS image file.

pixel_catalog_file(catalog_base_dir: str | pathlib.Path | upath.UPath | None, pixel: hats.pixel_math.healpix_pixel.HealpixPixel, query_params: dict | None = None, npix_suffix: str = '.parquet') → upath.UPath[source]#

Create path pointer for a pixel catalog file. This will not create the directory or file.

The catalog file name will take the HiPS standard form of:

<catalog_base_dir>/Norder=<pixel_order>/Dir=<directory number>/Npix=<pixel_number>.parquet

Where the directory number is calculated using integer division as:

(pixel_number/10000)*10000
Parameters:

  • catalog_base_dir (UPath) – base directory of the catalog (includes catalog name)

  • pixel (HealpixPixel) – the healpix pixel to create path to

  • query_params (dict) – Params to append to URL. Ex: {‘cols’: [‘ra’, ‘dec’], ‘fltrs’: [‘r>=10’, ‘g<18’]}

Returns:

string catalog file name

pixel_directory(catalog_base_dir: str | pathlib.Path | upath.UPath | None, pixel_order: int, pixel_number: int | None = None, directory_number: int | None = None) → upath.UPath[source]#

Create path pointer for a pixel directory. This will not create the directory.

One of pixel_number or directory_number is required. The directory name will take the HiPS standard form of:

<catalog_base_dir>/dataset/Norder=<pixel_order>/Dir=<directory number>

Where the directory number is calculated using integer division as:

(pixel_number/10000)*10000
Parameters:

  • catalog_base_dir (UPath) – base directory of the catalog (includes catalog name)

  • pixel_order (int) – the healpix order of the pixel

  • directory_number (int) – directory number

  • pixel_number (int) – the healpix pixel

Returns:

UPath directory name

Contents