hats.catalog.healpix_dataset.healpix_dataset ============================================ .. py:module:: hats.catalog.healpix_dataset.healpix_dataset Classes ------- .. autoapisummary:: hats.catalog.healpix_dataset.healpix_dataset.HealpixDataset Module Contents --------------- .. py:class:: HealpixDataset(catalog_info: hats.catalog.dataset.table_properties.TableProperties, pixels: hats.catalog.partition_info.PartitionInfo | hats.pixel_tree.pixel_tree.PixelTree | list[hats.pixel_math.HealpixPixel], catalog_path: str | pathlib.Path | upath.UPath | None = None, moc: mocpy.MOC | None = None, schema: pyarrow.Schema | None = None, snapshot: hats.catalog.catalog_snapshot.CatalogSnapshot | None = None, generate_snapshot: bool = False) Bases: :py:obj:`hats.catalog.dataset.Dataset` A HATS dataset partitioned with a HEALPix partitioning structure. Catalogs of this type are partitioned based on the ra and dec of the points with each partition containing points within a given HEALPix pixel. The files are in the form:: Norder=/Dir=/Npix=.parquet .. !! processed by numpydoc !! .. py:attribute:: partition_info .. py:attribute:: pixel_tree .. py:attribute:: moc :value: None .. py:method:: get_healpix_pixels() -> list[hats.pixel_math.HealpixPixel] Get healpix pixel objects for all pixels contained in the catalog. :Returns: list[HealpixPixel] List of HealpixPixel .. !! processed by numpydoc !! .. py:method:: __len__() The number of rows in the catalog. :Returns: int The number of rows in the catalog, as specified in its metadata. This value is undetermined when the catalog is modified, and therefore an error is raised. .. !! processed by numpydoc !! .. py:method:: get_max_coverage_order(default_order: int = 3) -> int Gets the maximum HEALPix order for which the coverage of the catalog is known from the pixel tree and moc if it exists :Parameters: **default_order** : int The order to return if the dataset has no pixels. (Default value = 3) :Returns: int maximum HEALPix order .. !! processed by numpydoc !! .. py:method:: filter_from_pixel_list(pixels: list[hats.pixel_math.HealpixPixel]) -> Self Filter the pixels in the catalog to only include any that overlap with the requested pixels. :Parameters: **pixels** : list[HealpixPixel] the pixels to include :Returns: HealpixDataset A new catalog with only the pixels that overlap with the given pixels. Note that we reset the total_rows to None, as updating would require a scan over the new pixel sizes. .. !! processed by numpydoc !! .. py:method:: filter_by_cone(ra: float, dec: float, radius_arcsec: float) -> Self Filter the pixels in the catalog to only include the pixels that overlap with a cone :Parameters: **ra** : float Right ascension of the center of the cone, in degrees **dec** : float Declination of the center of the cone, in degrees **radius_arcsec** : float Radius of the cone, in arcseconds :Returns: HealpixDataset A new catalog with only the pixels that overlap with the specified cone .. !! processed by numpydoc !! .. py:method:: filter_by_box(ra: tuple[float, float], dec: tuple[float, float]) -> Self Filter the pixels in the catalog to only include the pixels that overlap with a zone, defined by right ascension and declination ranges. The right ascension edges follow great arc circles and the declination edges follow small arc circles. :Parameters: **ra** : tuple[float, float] Right ascension range, in degrees **dec** : tuple[float, float] Declination range, in degrees :Returns: HealpixDataset A new catalog with only the pixels that overlap with the specified region .. !! processed by numpydoc !! .. py:method:: filter_by_polygon(vertices: list[tuple[float, float]]) -> Self Filter the pixels in the catalog to only include the pixels that overlap with a polygonal sky region. :Parameters: **vertices** : list[tuple[float, float]] The list of vertice coordinates for the polygon, (ra, dec), in degrees. :Returns: HealpixDataset A new catalog with only the pixels that overlap with the specified polygon. .. !! processed by numpydoc !! .. py:method:: filter_by_moc(moc: mocpy.MOC) -> Self Filter the pixels in the catalog to only include the pixels that overlap with the moc provided. :Parameters: **moc** : mocpy.MOC the moc to filter by :Returns: HealpixDataset A new catalog with only the pixels that overlap with the moc. Note that we reset the total_rows to 0, as updating would require a scan over the new pixel sizes. .. !! processed by numpydoc !! .. py:method:: align(other_cat: Self, alignment_type: hats.pixel_tree.PixelAlignmentType = PixelAlignmentType.INNER) -> hats.pixel_tree.PixelAlignment Performs an alignment to another catalog, using the pixel tree and mocs if available An alignment compares the pixel structures of the two catalogs, checking which pixels overlap. The alignment includes the mapping of all pairs of pixels in each tree that overlap with each other, and the aligned tree which consists of the overlapping pixels in the two input catalogs, using the higher order pixels where there is overlap with differing orders. For more information, see this document: https://docs.google.com/document/d/1gqb8qb3HiEhLGNav55LKKFlNjuusBIsDW7FdTkc5mJU/edit?usp=sharing :Parameters: **other_cat** : Catalog The catalog to align to **alignment_type** : PixelAlignmentType The type of alignment describing how to handle nodes which exist in one tree but not the other. Mirrors the 'how' argument of a pandas/sql join. Options are: - "inner" - only use pixels that appear in both catalogs - "left" - use all pixels that appear in the left catalog and any overlapping from the right - "right" - use all pixels that appear in the right catalog and any overlapping from the left - "outer" - use all pixels from both catalogs :Returns: PixelAlignment A `PixelAlignment` object with the alignment from the two catalogs .. !! processed by numpydoc !! .. py:method:: plot_pixels(**kwargs) Create a visual map of the pixel density of the catalog. :Parameters: **\*\*kwargs** Additional args to pass to `hats.inspection.visualize_catalog.plot_healpix_map` .. !! processed by numpydoc !! .. py:method:: plot_moc(**kwargs) Create a visual map of the coverage of the catalog. :Parameters: **\*\*kwargs** Additional args to pass to `hats.inspection.visualize_catalog.plot_moc` .. !! processed by numpydoc !! .. py:method:: aggregate_column_statistics(exclude_hats_columns: bool = True, exclude_columns: list[str] | None = None, include_columns: list[str] | None = None, only_numeric_columns: bool = False, include_pixels: list[hats.pixel_math.HealpixPixel] | None = None) Read footer statistics in parquet metadata, and report on global min/max values. :Parameters: **exclude_hats_columns** : bool exclude HATS spatial and partitioning fields from the statistics. Defaults to True. **exclude_columns** : list[str] | None additional columns to exclude from the statistics. **include_columns** : list[str] | None if specified, only return statistics for the column names provided. Defaults to None, and returns all non-hats columns. **only_numeric_columns** : bool only include columns that are numeric (integer or floating point) in the statistics. If True, the entire frame should be numeric. (Default value = False) **include_pixels: list[HealpixPixel] | None** (Default value = None) :Returns: Dataframe aggregated statistics .. !! processed by numpydoc !! .. py:method:: per_pixel_statistics(*, exclude_hats_columns: bool = True, exclude_columns: list[str] | None = None, include_columns: list[str] | None = None, only_numeric_columns: bool = False, include_stats: list[str] | None = None, multi_index=False, include_pixels: list[hats.pixel_math.HealpixPixel] | None = None, per_row_group: bool = False) Read footer statistics in parquet metadata, and report on statistics about each pixel partition. .. !! processed by numpydoc !! .. py:method:: per_partition_statistics(*, exclude_hats_columns: bool = True, exclude_columns: list[str] | None = None, include_columns: list[str] | None = None, only_numeric_columns: bool = False, include_stats: list[str] | None = None, multi_index=False, include_pixels: list[hats.pixel_math.HealpixPixel] | None = None, per_row_group: bool = False) Read footer statistics in parquet metadata, and report on statistics about each pixel partition. :Parameters: **exclude_hats_columns** : bool exclude HATS spatial and partitioning fields from the statistics. Defaults to True. **exclude_columns** : list[str] | None additional columns to exclude from the statistics. **include_columns** : list[str] | None if specified, only return statistics for the column names provided. Defaults to None, and returns all non-hats columns. **include_stats** : list[str] | None if specified, only return the kinds of values from list (min_value, max_value, null_count, row_count). Defaults to None, and returns all values. **multi_index** : bool should the returned frame be created with a multi-index, first on pixel, then on column name? (Default value = False) **include_pixels** : list[HealpixPixel] | None if specified, only return statistics for the pixels indicated. Defaults to none, and returns all pixels. :Returns: Dataframe granular statistics .. !! processed by numpydoc !! .. py:method:: has_healpix_column() Does this catalog's schema contain a healpix spatial index column? This is True if either: - there is a value for the ``hats_col_healpix`` property, and that string exists as a column name in the pyarrow schema - there is a ``_healpix_29`` column in the pyarrow schema :Returns: bool if the dataset has a healpix column in the properties .. !! processed by numpydoc !! .. py:method:: get_pixel_paths() Generate paths to all pixel files. Pixels will be traversed in "breadth-first" healpix order. If any spatial filters have been applied to this catalog, only those pixels that remain will be included. :Yields: UPath Universal Pathlib pointing to either an npix directory, or to a single pixel partition data file. .. !! processed by numpydoc !! .. py:method:: read_pixel_to_pandas(pixel: hats.pixel_math.HealpixPixel, **kwargs) -> nested_pandas.NestedFrame Read the parquet file(s) for this pixel into a pandas dataframe. :Parameters: **pixel** : HealpixPixel desired data partition, by healpix pixel **\*\*kwargs** Additional arguments to pass to pandas read_parquet method :Returns: NestedFrame Pandas DataFrame with the data from the parquet file(s) .. !! processed by numpydoc !!