piaso.pp — preprocessing

Reading data, cell metrics and filtering.

Function	What it does
align_spatial_coordinates	Put every sample’s spatial coordinates in a common frame.
alignSpatialCoordinates	Put every sample’s spatial coordinates in a common frame.
calculateCellMetrics	Calculate per-cell QC metrics and store them in adata.obs or cytome cells table.
calculateFeatureMetrics	Calculate the number of cells in which each feature is active (non-zero counts).
calculateGroupMetrics	Per-group (e.g. per cell type) summary metrics, streamed from a cytome.
calculatePeakMetrics	REMOVED — renamed to :func:calculateFeatureMetrics.
calculateTSSEnrichmentScore	Compute the TSS enrichment score for cells based on given parameters and files.
estimate_background	Estimate A/C/G/T composition from a set of sequences.
filter_cells	Filter cells. Polymorphic on AnnData / cytome inputs.
filter_features	Filter features (genes/peaks) based on cell or count thresholds.
getCrossCategories	Generates a new categorical column from the cross combinations of two specified columns.
importCellRanger	Build a Cytome dataset from Cell Ranger output(s), with Rust fragment import.
load_peaks	Load peaks from a BED / narrowPeak / broadPeak file.
normalize_log1p	Library-size normalization followed by log1p transform.
pvalue_to_threshold	Convert a right-tail p-value to a PSSM score cutoff via DP convolution.
read_10x	Read Cell Ranger output, dispatching on what path points at.
read_10x_h5	Read a Cell Ranger HDF5 matrix into an :class:~anndata.AnnData.
read_10x_mtx	Read a Cell Ranger MTX directory into an :class:~anndata.AnnData.
rotate_spatial_coordinates	Rotate spatial coordinates around their centroid.
rotateSpatialCoordinates	Rotate spatial coordinates around their centroid.
rust_ext_available	True if the compiled _piaso extension (Rust PWM scanner) is importable.
scan_motifs	Motif scanner: all PWMs × all sequences (Rust-accelerated when available).
scan_motifs_numpy	Reference (pure-numpy) motif scanner: all PWMs × all sequences.
scan_motifs_rust	Rust-backed equivalent of :func:piaso.pp.scan_motifs.
scanMotifs	Motif scanner: all PWMs × all sequences (Rust-accelerated when available).
scrublet	Detect doublets using the Scrublet algorithm (Wolock et al., 2019).
selectFeatures	Flag the features named in features on the cytome’s feature table.
subset_cells	Filter cells. Polymorphic on AnnData / cytome inputs.
sweep_intersect	Generator sweep-line intersection. Yields hits one at a time — O(1) memory for the hit stream.
table	Returns the counts of unique values in the given list or from a data source column.

align_spatial_coordinates

alignSpatialCoordinates(
    data,
    groupby: Optional[str] = None,
    spatial_key: str = 'spatial',
    key_added: str = 'spatial_aligned',
    backup_spatial_key: Optional[str] = None,
    with_std: bool = False,
    inplace: bool = True,
    copy: bool = False,
    batch_key: Optional[str] = None,
)

Signature defaults

data, groupby=None, spatial_key='spatial', key_added='spatial_aligned', backup_spatial_key=None, with_std=False, inplace=True, copy=False, batch_key=None

Put every sample’s spatial coordinates in a common frame.

Sections are placed on their chips independently, so two samples can sit hundreds of microns apart in raw coordinates. Nothing is wrong with the data, but a split plot then renders each panel at a different offset and the tissues look displaced relative to one another. Centring each group on its own centroid fixes the display without touching within-sample geometry.

The default is centring only (subtract the mean). with_std=True also divides by the per-group standard deviation, which equalises apparent section size — useful when samples differ a lot in extent, wrong when the relative sizes are part of what you are showing.

Parameters

data

AnnData, an open cytome dataset, or a path to a .cytome file.

groupby

Column naming the sample/section each cell belongs to (adata.obs or ds.cells).

spatial_key

Source coordinates. For AnnData an obsm key ('spatial' and 'X_spatial' both accepted); for a cytome a short embedding name resolved the same way basis= is in plotting.

key_added

Where the aligned coordinates are written.

backup_spatial_key

If given, the original coordinates are copied here first, so the un-aligned frame stays available.

with_std

Also scale each group to unit standard deviation.

inplace

AnnData only; False returns a modified copy. A cytome is always written in place.

copy

AnnData only; alias for not inplace.

Returns

The modified AnnData when inplace=False/copy=True, else None.

Examples

>>> piaso.pp.alignSpatialCoordinates(adata, groupby="Sample")
>>> piaso.pl.plot_embeddings_split(adata, color="cell_type",
...                                splitby="Sample",
...                                basis="spatial_aligned")

alignSpatialCoordinates

alignSpatialCoordinates(
    data,
    groupby: Optional[str] = None,
    spatial_key: str = 'spatial',
    key_added: str = 'spatial_aligned',
    backup_spatial_key: Optional[str] = None,
    with_std: bool = False,
    inplace: bool = True,
    copy: bool = False,
    batch_key: Optional[str] = None,
)

Signature defaults

data, groupby=None, spatial_key='spatial', key_added='spatial_aligned', backup_spatial_key=None, with_std=False, inplace=True, copy=False, batch_key=None

Put every sample’s spatial coordinates in a common frame.

Sections are placed on their chips independently, so two samples can sit hundreds of microns apart in raw coordinates. Nothing is wrong with the data, but a split plot then renders each panel at a different offset and the tissues look displaced relative to one another. Centring each group on its own centroid fixes the display without touching within-sample geometry.

The default is centring only (subtract the mean). with_std=True also divides by the per-group standard deviation, which equalises apparent section size — useful when samples differ a lot in extent, wrong when the relative sizes are part of what you are showing.

Parameters

data

AnnData, an open cytome dataset, or a path to a .cytome file.

groupby

Column naming the sample/section each cell belongs to (adata.obs or ds.cells).

spatial_key

Source coordinates. For AnnData an obsm key ('spatial' and 'X_spatial' both accepted); for a cytome a short embedding name resolved the same way basis= is in plotting.

key_added

Where the aligned coordinates are written.

backup_spatial_key

If given, the original coordinates are copied here first, so the un-aligned frame stays available.

with_std

Also scale each group to unit standard deviation.

inplace

AnnData only; False returns a modified copy. A cytome is always written in place.

copy

AnnData only; alias for not inplace.

Returns

The modified AnnData when inplace=False/copy=True, else None.

Examples

>>> piaso.pp.alignSpatialCoordinates(adata, groupby="Sample")
>>> piaso.pl.plot_embeddings_split(adata, color="cell_type",
...                                splitby="Sample",
...                                basis="spatial_aligned")

calculateCellMetrics

calculateCellMetrics(
    source,
    layer: Optional[str] = None,
    modality: str = 'RNA',
    measurement: str = 'counts',
    batch_size: int = 1024,
    verbose: bool = True,
    prefix_vars: Optional[dict] = None,
    feature_set_vars: Optional[dict] = None,
    feature_name_column: Optional[str] = None,
)

Signature defaults

source, layer=None, modality='RNA', measurement='counts', batch_size=1024, verbose=True, prefix_vars=None, feature_set_vars=None, feature_name_column=None

Calculate per-cell QC metrics and store them in adata.obs or cytome cells table.

Computes two values per cell from the chosen modality’s count matrix, plus (for fragment-backed modalities) the total fragment count:

• sum-per-cell — total counts/fragments (modality-specific column name)
• nnz-per-cell — number of active features (modality-specific column name)
• n_fragments — total fragments from the fragment_chunks table (ATAC / tiles only; cytome mode).
• frip — fraction of fragments in peaks = n_fragments_in_peak / n_fragments (ATAC only, cytome mode, written automatically when n_fragments is available). Filter on it directly, e.g. piaso.pp.filter_cells(ds, modality='ATAC', mask={'frip': (0.2, None)}).

The column names depend on modality (cytome mode):

======== ====================== ============ ===================== modality sum-per-cell nnz-per-cell fragment total ======== ====================== ============ ===================== ATAC n_fragments_in_peak n_peaks n_fragments RNA n_counts n_genes — (no fragments) GA GA_n_counts n_GA_genes — (no fragments) tiles n_fragments_in_tile n_tiles n_fragments ======== ====================== ============ =====================

Parameters

source — AnnData, cytome.Dataset, or str

Input data. For AnnData, uses the feature-cell matrix. For cytome Dataset or path to .cytome file, streams from the {modality}_{measurement} matrix on disk and (ATAC/tiles) reads total fragment counts.

layer — str, optional

AnnData layer to use. If None, uses .X. Ignored for cytome.

modality — str

Modality: 'RNA' (default), 'ATAC', 'GA', 'tiles'. Selects the cytome modality to stream AND the per-cell metric column names (see table above) — the AnnData path is now modality-aware too (e.g. an RNA AnnData gets n_counts/n_genes; pass modality='ATAC' for n_fragments_in_peak/n_peaks).

measurement — str

Cytome measurement/layer (default ‘counts’). Ignored for AnnData.

batch_size — int

Chunk size for streaming (cytome mode). Default 1024.

verbose — bool

Print progress messages.

prefix_vars — dict, optional

Map {key: prefix} or {key: [prefixes]} for per-cell percentage of counts in features whose name starts with any of the prefixes. Writes n_counts_{key} and pct_counts_{key} (= 100 * masked / total). Example: {'mt': 'MT-', 'ribo': ['RPS', 'RPL']} → pct_counts_mt, pct_counts_ribo.

feature_set_vars — dict, optional

Map {key: [exact feature names]} — same outputs as prefix_vars but matched by exact membership rather than prefix.

feature_name_column — str, optional

Which feature-name column to match against. AnnData: a var column (default var_names). Cytome: a column of the modality’s var table (default: the modality’s canonical name column, e.g. gene_id).

calculateFeatureMetrics

calculateFeatureMetrics(
    source,
    layer: str = None,
    modality: str = 'ATAC',
    measurement: str = 'counts',
    batch_size: int = 1024,
    verbose: bool = True,
)

Signature defaults

source, layer=None, modality='ATAC', measurement='counts', batch_size=1024, verbose=True

Calculate the number of cells in which each feature is active (non-zero counts).

For AnnData, stores result in adata.var['n_cells']. For cytome, stores n_cells in the modality’s var entity table (RNA → genes, GA → GA_genes, ATAC → peaks, tiles → tiles), resolved via the cytome modality registry.

Parameters

source — AnnData, cytome.Dataset, or str

Input data. For cytome, streams the {modality}_{measurement} matrix in chunks.

layer — str, optional

AnnData layer to use. Ignored for cytome.

modality — str

Cytome modality: 'ATAC' (default), 'RNA', 'GA', 'tiles'. Ignored for AnnData.

measurement — str

Cytome measurement (default ‘counts’). Ignored for AnnData.

batch_size — int

Chunk size for streaming. Default 1024.

verbose — bool

Print progress messages.

calculateGroupMetrics

calculateGroupMetrics(
    data,
    groupby: str,
    modalities=None,
    detection_pct=None,
    expression_cutoff: float = 0.0,
    measurement: str = 'counts',
    batch_size: int = 1024,
    key_added: Optional[str] = None,
    verbose: bool = True,
)

Signature defaults

data, groupby, modalities=None, detection_pct=None, expression_cutoff=0.0, measurement='counts', batch_size=1024, key_added=None, verbose=True

Per-group (e.g. per cell type) summary metrics, streamed from a cytome.

For every group in groupby and every modality present, computes how many features are detected — expressed in more than detection_pct of that group’s cells — plus per-cell count/feature summaries. Modalities are auto-detected (only those with a {mod}_counts matrix are reported).

Columns (per group; {m} = modality):

• n_cells
• {m}_n_features_detected — # features expressed in > detection_pct[m] of cells
• {m}_counts_mean / {m}_counts_median — per-cell total counts/fragments
• {m}_features_per_cell_median — median active features per cell
• ATAC/tiles: {m}_n_fragments_mean / {m}_n_fragments_median (from cells.n_fragments), {m}_frip_mean (if a frip column exists)

Parameters

data — cytome.Dataset or str, or AnnData

Cytome (streamed) or AnnData (single modality, in-memory).

groupby — str

Cell-grouping column (cells table / obs).

modalities — list of str, optional

Restrict to these modalities. Default: all present.

detection_pct — float or dict, optional

Detection fraction, global float or per-modality dict. Default {'RNA':0.10,'GA':0.10,'ATAC':0.05,'tiles':0.05}.

expression_cutoff — float, default 0.0

A cell “expresses” a feature when its count is > expression_cutoff.

measurement — str, default 'counts'

Source matrix layer.

batch_size — int, default 1024

Streaming chunk size (cytome).

key_added — str, optional

If given (cytome), also store the result under ds.metadata[key_added]. Default: return-only.

verbose — bool, default True

Returns

pandas.DataFrame

Rows = groups (ordered by the set_categories store when present), columns = metrics. df.attrs carries groupby, colors ({group: hex} if the store has them), and detection_pct so :func:piaso.pl.plotGroupMetrics can re-use the cell-type colors.

calculatePeakMetrics

calculatePeakMetrics(*args, **kwargs)

Signature defaults

*args, **kwargs

REMOVED — renamed to :func:calculateFeatureMetrics.

The function now supports all modalities (RNA / GA / ATAC / tiles), so the ATAC-specific “peak” name no longer fits. Update your call:

piaso.pp.calculatePeakMetrics(ds, ...)  →  piaso.pp.calculateFeatureMetrics(ds, ...)

calculateTSSEnrichmentScore

calculateTSSEnrichmentScore(
    fragment_file: str = None,
    adata=None,
    output_dir: str = None,
    barcodes: list = None,
    barcode_file: str = None,
    bedtools_path: str = None,
    tss_bed_file: str = None,
    genome_size_file: str = None,
    slop_l: int = 50,
    slop_r: int = 50,
    shift: int = 1550,
    prefix: str = 'tss',
    chromosome_selected: list = [],
    method: str = 'python',
)

Signature defaults

fragment_file=None, adata=None, output_dir=None, barcodes=None, barcode_file=None, bedtools_path=None, tss_bed_file=None, genome_size_file=None, slop_l=50, slop_r=50, shift=1550, prefix='tss', chromosome_selected=[], method='python'

Compute the TSS enrichment score for cells based on given parameters and files.

Parameters: :param fragment_file (str): Path to the fragment file. :param adata: AnnData object. If provided, TSS_score is written to adata.obs[‘TSS_score’]. :param output_dir (str): Directory to save intermediate and output files. :param barcodes: list, optional list of barcodes to include. If None and adata is provided, extracted from adata.obs_names. :param barcode_file: str, optional path to a file with one barcode per line. :param bedtools_path: str, the path to the bedtools binaries. Required if method=‘bedtools’. :param tss_bed_file: str, the path to the input TSS bed file. :param genome_size_file: str, the path to the genome size file. :param slop_l: int, the amount to slop (extend) the features to the left. :param slop_r: int, the amount to slop (extend) the features to the right. :param shift: int, the amount to shift the features to the left and right. :param prefix: str, a prefix for naming output files. Default is “tss”. :param chromosome_selected: list, list of chromosomes to keep. Default is empty, which keeps all. :param method: str, ‘python’ (default, no bedtools) or ‘bedtools’.

Returns:

• pandas.DataFrame: A dataframe with the ‘CellBarcode’ and ‘TSS_score’ columns.

estimate_background

estimate_background(sequences: 'list[str]', pseudocount: 'float' = 1.0)

Signature defaults

sequences, pseudocount=1.0

Estimate A/C/G/T composition from a set of sequences.

Ambiguous bases (N, etc.) are ignored. A pseudocount is added before normalisation so that even a monomer sequence does not produce zeros.

Parameters

sequences

List of DNA strings (any case; soft-masked allowed).

pseudocount

Added to each base count before normalisation (default 1.0).

Returns

np.ndarray

float64 array of length 4 (A, C, G, T) summing to 1.0. Returns uniform [0.25]*4 when no valid bases are found.

filter_cells

filter_cells(
    data,
    min_counts=None,
    max_counts=None,
    min_features=None,
    max_features=None,
    mask=None,
    inplace: 'bool' = True,
    output: 'str | Path | None' = None,
    overwrite: 'bool' = False,
    modality: 'str' = 'RNA',
    batch_size: 'int' = 2048,
    include_fragments: 'bool' = True,
    include_embeddings: 'bool' = True,
    verbose: 'int' = 1,
)

Signature defaults

data, min_counts=None, max_counts=None, min_features=None, max_features=None, mask=None, inplace=True, output=None, overwrite=False, modality='RNA', batch_size=2048, include_fragments=True, include_embeddings=True, verbose=1

Filter cells. Polymorphic on AnnData / cytome inputs.

The keep-mask is built from up to two independent sources, then intersected:

1. QC thresholds computed from the counts matrix (min_counts, max_counts, min_features, max_features). Backward-compatible with the prior AnnData-only signature.

2. General mask via the mask kwarg. Six shapes accepted:

   • boolean np.ndarray / pd.Series of length n_cells

   • integer np.ndarray of cell indices to keep

   • string pandas-query expression (e.g. "n_counts > 1000 and cluster.isin(['T','B'])")

   • dict mapping column → scalar / list / numeric 2-tuple range. Ranges may be open-ended via None: (lo, hi) → lo ≤ col ≤ hi, (lo, None) → col ≥ lo, (None, hi) → col ≤ hi. This is the clean way to express ATAC cell-QC after calculateCellMetrics / the importer have written the columns, e.g.::

     filter_cells(ds, modality="ATAC",
         mask={"n_fragments": (1000, None),
               "tss_score":   (3.0,  None),
               "frip":        (0.2,  None)})

   • callable fn(obs_df) -> bool series

Cytome path is RAM-efficient: dict masks and simple query strings push down to SQL WHERE and only matching cell_idx values are materialised. Complex queries / callables fall back to chunked streaming over the cells table.

Parameters

data

AnnData or cytome.Dataset. min_counts, max_counts, min_features, max_features Per-cell QC thresholds. Computed from the counts matrix.

mask

General-purpose mask (see above for accepted shapes).

inplace

If True, apply the filter in place. For cytome this calls ds.filter_cells(...) which atomically replaces the file. For AnnData this calls adata._inplace_subset_obs(...).

output

Cytome-only. When set with inplace=False, write a filtered copy to this path via ds.subset(mask, output=...) instead of modifying the original. AnnData ignores this with a warning.

overwrite

When output exists, replace it (default False raises).

modality

Cytome modality used to compute QC thresholds. Default ‘RNA’.

batch_size

Streaming batch size for QC threshold pass on cytome. include_fragments, include_embeddings Forwarded to ds.filter_cells / ds.subset.

verbose

0 silent, 1 (default) prints summary, 2 prints mask sources.

Returns

AnnData backend:

• inplace=True → None (modifies in place)
• inplace=False → bool mask of length n_cells

Cytome backend:

• inplace=True, output=None → int (n_kept)
• inplace=True, output=path → TypeError (contradictory)
• inplace=False, output=None → bool mask of length n_cells
• inplace=False, output=path → open cytome.Dataset at path

Raises

TypeError

inplace=True and output both given.

ValueError

No cells survive the filter, or a mask has the wrong length.

KeyError

mask dict references a missing column.

FileExistsError

output exists and overwrite=False.

filter_features

filter_features(
    adata,
    min_cells=None,
    max_cells=None,
    min_counts=None,
    max_counts=None,
    inplace=True,
)

Signature defaults

adata, min_cells=None, max_cells=None, min_counts=None, max_counts=None, inplace=True

Filter features (genes/peaks) based on cell or count thresholds.

AnnData only. Cytome equivalent lives in piaso.pp.selectPeaks / piaso.tl.infog (for highly-variable features).

Parameters

adata — AnnData

Annotated data matrix.

min_cells — int, optional

Minimum number of cells expressing the feature (non-zero).

max_cells — int, optional

Maximum number of cells expressing the feature.

min_counts — int, optional

Minimum total counts per feature.

max_counts — int, optional

Maximum total counts per feature.

inplace — bool

If True, subset adata in place. If False, return boolean mask.

Returns

If inplace=True: modifies adata in place and returns None. If inplace=False: returns boolean array of shape (n_vars,).

getCrossCategories

getCrossCategories(
    source,
    col1,
    col2,
    delimiter='@',
    iterate_by_second_column=True,
)

Signature defaults

source, col1, col2, delimiter='@', iterate_by_second_column=True

Generates a new categorical column from the cross combinations of two specified columns.

Accepts pd.DataFrame, AnnData (.obs), cytome Dataset (cells table), or str path to .cytome file.

Parameters

source — pd.DataFrame, AnnData, cytome.Dataset, or str

The data source containing the columns to be combined.

col1 — str

Name of the first column to combine.

col2 — str

Name of the second column to combine.

delimiter — str, optional

Delimiter used to join the column values. Defaults to ’@’.

iterate_by_second_column — bool, optional

If set to True, the function iterates by the values of the second column first when generating the combined categories. Defaults to True.

Returns

pd.Categorical

A Pandas Categorical series of the combined columns with a defined order.

importCellRanger

importCellRanger(
    path,
    output: 'str | Path',
    sample_name=None,
    modality: 'str' = 'both',
    genome: 'Optional[str]' = None,
    keep_chroms: 'str' = 'standard',
    min_fragments: 'int' = 0,
    threads: 'int' = 8,
    tss_bed: 'Optional[str]' = None,
    compression: 'str' = 'lz4',
    build_index: 'bool' = True,
    rust_binary: 'Optional[str]' = None,
    verbose: 'bool' = True,
    force: 'bool' = False,
)

Signature defaults

path, output, sample_name=None, modality='both', genome=None, keep_chroms='standard', min_fragments=0, threads=8, tss_bed=None, compression='lz4', build_index=True, rust_binary=None, verbose=True, force=False

Build a Cytome dataset from Cell Ranger output(s), with Rust fragment import.

cytome.from_cellranger writes the count matrices (RNA genes and/or ATAC peaks per modality); when ATAC is requested and atac_fragments.tsv.gz is present, :func:piaso.pp.importFragments imports the fragments with the Rust binary (inline tile quantification + optional TSS enrichment). For multiple folders the count matrices are merged first and then ALL fragment files are imported in a single Rust k-way merge (the merged barcodes are suffixed {barcode}-{i} per library so colliding Cell Ranger barcodes map to the right merged cells) — replacing the older per-library-then-merge fragment path.

Parameters

path — str | Path | list of (str | Path)

A single Cell Ranger output folder, or a list of folders (merged into one dataset).

output — str | Path

Output .cytome path.

sample_name — str | list of str, optional

Sample id(s) written to cells.sample_id (one per folder for a list).

modality — {“both”, “rna”, “atac”}, default “both”

"rna" → RNA genes + counts only (no ATAC, no fragments). "atac" → ATAC peaks + Rust fragments only (no RNA). "both" → RNA + ATAC peaks + Rust fragments.

genome — str, required for ATAC (no default)

Reference for the Rust importer’s tile quantification — 'hg38'/'hg19'/'mm10'/ 'mm39' or a .fai/.chrom.sizes path. Required whenever fragments are imported (modality="both"/"atac"); only modality="rna" may omit it. There is no default — naming the wrong genome (or relying on a silent default) builds the tile grid for the wrong assembly and corrupts the tiles, so the genome must be stated explicitly.

keep_chroms — {“standard”, “all”}, default “standard”

Forwarded to cytome.from_cellranger (drops non-standard scaffolds from ATAC peaks).

min_fragments — int, default 0

Per-barcode fragment floor for the Rust importer. 0 keeps every cell already present from the Cell Ranger filtered matrix (recommended; the matrix is already QC’d).

threads — int, default 8

Threads for the Rust importer.

tss_bed — str, optional

TSS BED for per-cell TSS enrichment during fragment import.

compression — str, default “lz4”

Fragment chunk compression ('lz4'/'zlib'/'zstd').

build_index — bool, default True

Build the peak / fragment spatial index.

rust_binary — str, optional

Explicit path to cytome-import-fragments (auto-discovered if None).

verbose — bool, default True

Returns

cytome.Dataset

Examples

>>> import piaso
>>> ds = piaso.pp.importCellRanger(
...     ["run/E15Satb2_ctrl", "run/E15Satb2_het", "run/E15Satb2_cko"],
...     output="E15Satb2.cytome",
...     sample_name=["ctrl", "het", "cko"],
...     genome="mm10",
... )
>>> # RNA-only
>>> ds = piaso.pp.importCellRanger("run/outs", "rna.cytome", modality="rna")

load_peaks

load_peaks(peak_file: str)

Signature defaults

peak_file

Load peaks from a BED / narrowPeak / broadPeak file.

Parameters

peak_file — str

Path to peak file.

Returns

peaks_by_chr — dict

{chrom: [(start, end, global_idx), ...]} sorted by start within each chromosome.

peak_names — list of str

"{chrom}:{start}-{end}" for each peak, in file order.

n_peaks — int

Total peak count.

normalize_log1p

normalize_log1p(
    data,
    target_sum=10000.0,
    key_added='log1p',
    save_layer=False,
    modality='RNA',
    layer='counts',
    batch_size=1024,
)

Signature defaults

data, target_sum=10000.0, key_added='log1p', save_layer=False, modality='RNA', layer='counts', batch_size=1024

Library-size normalization followed by log1p transform.

For each cell, divides by total counts, scales by target_sum, then applies log1p. Result is stored in data.layers[key_added].

Parameters

data — AnnData or cytome.Dataset

Input data. For AnnData, reads raw counts from .X. For cytome, streams from the specified modality/layer on disk.

target_sum — float

Target sum for per-cell normalization. Default: 1e4.

key_added — str

Layer name to store the result. Default: 'log1p'.

save_layer — bool

For cytome only. If True, writes the normalized layer to the cytome file. If False (default), only stores in-memory on the AnnData representation. Ignored for AnnData input.

modality — str

Cytome modality (default 'RNA'). Ignored for AnnData.

layer — str

Cytome layer within the modality (default 'counts'). Ignored for AnnData.

batch_size — int

Chunk size for streaming (cytome mode). Default: 1024.

Returns

None

Modifies data in place.

pvalue_to_threshold

pvalue_to_threshold(
    pssm: 'np.ndarray',
    background: 'np.ndarray',
    pvalue: 'float' = 0.0001,
    n_bins: 'int' = 1000,
)

Signature defaults

pssm, background, pvalue=0.0001, n_bins=1000

Convert a right-tail p-value to a PSSM score cutoff via DP convolution.

Implements the standard FIMO/MOODS algorithm:

1. Find the global min and max achievable score (sum of column mins/maxs).
2. Quantize the score range to n_bins integer bins.
3. For each PSSM column, build a probability distribution over quantized per-column scores; convolve (shift-add) across all columns.
4. Compute the right-tail CDF; return the smallest score whose right-tail probability ≤ pvalue.

Complexity: O(w × n_bins) time and space.

Parameters

pssm

(4, w) log2-odds matrix as returned by PWM.pssm().

background

Length-4 float64 background frequencies (sum to 1).

pvalue

Target right-tail probability (default 1e-4).

n_bins

Number of quantization bins (default 1000). Larger = more accurate threshold but linearly more compute. Approximation error is at most one bin width.

Returns

float

Score threshold t such that P(score ≥ t | null) ≤ pvalue. Approximate to within the bin resolution.

read_10x

read_10x(path, **kwargs)

Signature defaults

path, **kwargs

Read Cell Ranger output, dispatching on what path points at.

A .h5 file goes to :func:read_10x_h5; a directory goes to :func:read_10x_mtx. Keyword arguments are forwarded.

adata = piaso.pp.read_10x("filtered_feature_bc_matrix.h5")
adata = piaso.pp.read_10x("filtered_feature_bc_matrix/")

To read a Cell Ranger matrix into a cytome instead of an AnnData, use cytome.from_10x_h5(path, output) — writing a file is a different operation and keeps its own function.

read_10x_h5

read_10x_h5(
    path,
    modality: 'str' = 'rna',
    var_names: 'str' = 'gene_symbols',
    make_unique: 'bool' = True,
    dtype: 'str' = 'float32',
    genome: 'Optional[str]' = None,
)

Signature defaults

path, modality='rna', var_names='gene_symbols', make_unique=True, dtype='float32', genome=None

Read a Cell Ranger HDF5 matrix into an :class:~anndata.AnnData.

Parameters

path

Path to *_feature_bc_matrix.h5 (filtered or raw).

modality

Which features to keep: 'rna' (default, Gene Expression), 'atac' (Peaks), 'adt', 'crispr', a literal Cell Ranger feature_type string, or 'all' to keep everything. Multiome files contain more than one; with 'all' the type is kept in .var['feature_types'].

var_names

'gene_symbols' (default) or 'gene_ids'.

make_unique

Disambiguate repeated symbols with -1, -2, … Gene symbols are not unique in Cell Ranger references.

dtype

dtype of .X. The file stores integer counts; the default float32 keeps them exact well past any realistic UMI count while matching what downstream numerical code expects.

genome

Keep only features from this genome. Only meaningful for barnyard references; raises if the file has no such genome.

Returns

AnnData

n_cells x n_features, raw counts in .X, with gene_ids, feature_types, genome and (for peaks) interval in .var.

read_10x_mtx

read_10x_mtx(
    path,
    modality: 'str' = 'rna',
    var_names: 'str' = 'gene_symbols',
    make_unique: 'bool' = True,
    dtype: 'str' = 'float32',
    prefix: 'str' = '',
)

Signature defaults

path, modality='rna', var_names='gene_symbols', make_unique=True, dtype='float32', prefix=''

Read a Cell Ranger MTX directory into an :class:~anndata.AnnData.

Expects matrix.mtx[.gz] plus features.tsv[.gz] (or the v2 genes.tsv[.gz]) and barcodes.tsv[.gz]. Arguments match :func:read_10x_h5; prefix handles files named e.g. sample_matrix.mtx.gz.

rotate_spatial_coordinates

rotateSpatialCoordinates(
    adata,
    angle_degrees: float,
    spatial_key: str = 'X_spatial',
    clockwise: bool = False,
    inplace: bool = True,
    backup_spatial_key: Optional[str] = None,
)

Signature defaults

adata, angle_degrees, spatial_key='X_spatial', clockwise=False, inplace=True, backup_spatial_key=None

Rotate spatial coordinates around their centroid.

Works on an AnnData, an open cytome Dataset, or a path to a .cytome file. On AnnData the coordinates live in adata.obsm[spatial_key]; on a cytome, spatial_key resolves against the stored embeddings the same way plotting’s basis= does ('spatial', 'X_spatial' and the full stored name all work), the rotation is written back to the file, and — when the rotated embedding is the spatial one — the spatial_coords R*-tree index is rebuilt so cells_in_region stays consistent with the plot.

Parameters

adata — AnnData, cytome.Dataset, or str

The data whose coordinates to rotate.

angle_degrees — float

Rotation angle in degrees.

spatial_key — str, default ‘X_spatial’

obsm key (AnnData) or embedding name / short name (cytome). Tip: converted cytomes store obsm['spatial'] as {modality}_spatial — spatial_key='spatial' finds it.

clockwise — bool, default False

Rotate clockwise instead of the mathematical counter-clockwise.

inplace — bool, default True

AnnData only: modify in place (True) or return a rotated copy (False). A cytome is a file — the rotation is always written to it, and inplace=False raises rather than pretending to copy a dataset; use backup_spatial_key to keep the original.

backup_spatial_key — str, optional

Store the pre-rotation coordinates under this obsm key / embedding name first, so the original is one assignment away.

rotateSpatialCoordinates

rotateSpatialCoordinates(
    adata,
    angle_degrees: float,
    spatial_key: str = 'X_spatial',
    clockwise: bool = False,
    inplace: bool = True,
    backup_spatial_key: Optional[str] = None,
)

Signature defaults

adata, angle_degrees, spatial_key='X_spatial', clockwise=False, inplace=True, backup_spatial_key=None

Rotate spatial coordinates around their centroid.

Works on an AnnData, an open cytome Dataset, or a path to a .cytome file. On AnnData the coordinates live in adata.obsm[spatial_key]; on a cytome, spatial_key resolves against the stored embeddings the same way plotting’s basis= does ('spatial', 'X_spatial' and the full stored name all work), the rotation is written back to the file, and — when the rotated embedding is the spatial one — the spatial_coords R*-tree index is rebuilt so cells_in_region stays consistent with the plot.

Parameters

adata — AnnData, cytome.Dataset, or str

The data whose coordinates to rotate.

angle_degrees — float

Rotation angle in degrees.

spatial_key — str, default ‘X_spatial’

obsm key (AnnData) or embedding name / short name (cytome). Tip: converted cytomes store obsm['spatial'] as {modality}_spatial — spatial_key='spatial' finds it.

clockwise — bool, default False

Rotate clockwise instead of the mathematical counter-clockwise.

inplace — bool, default True

AnnData only: modify in place (True) or return a rotated copy (False). A cytome is a file — the rotation is always written to it, and inplace=False raises rather than pretending to copy a dataset; use backup_spatial_key to keep the original.

backup_spatial_key — str, optional

Store the pre-rotation coordinates under this obsm key / embedding name first, so the original is one assignment away.

rust_ext_available

_rust_ext_available()

True if the compiled _piaso extension (Rust PWM scanner) is importable.

scan_motifs

scan_motifs(
    pwms: 'list[PWM]',
    sequences: 'list[str]',
    background: 'Optional[np.ndarray]' = None,
    pvalue: 'float' = 0.0001,
    relative_frac: 'Optional[float]' = None,
    both_strands: 'bool' = True,
    pseudocount: 'float' = 0.01,
    backend: 'str' = 'auto',
)

Signature defaults

pwms, sequences, background=None, pvalue=0.0001, relative_frac=None, both_strands=True, pseudocount=0.01, backend='auto'

Motif scanner: all PWMs × all sequences (Rust-accelerated when available).

This is the public entry point (piaso.pp.scan_motifs). It dispatches between the Rust backend (_piaso.scan_motifs_fwd via :func:~piaso.preprocessing.grn._scan_rust.scan_motifs_rust) and the pure-numpy reference (:func:_scan_motifs_numpy); the two are numerically identical (same log-odds PSSM, N-augmentation, p-value/relative threshold and reverse-complement handling), so backend only trades speed for the no-compiler fallback.

Parameters

pwms, sequences, background, pvalue, relative_frac, both_strands, pseudocount See :func:_scan_motifs_numpy — forwarded unchanged to the selected backend.

backend

Which scanner to use:

"auto" (default) Use Rust if the _piaso extension is built, else fall back to numpy. "rust" Force the Rust backend; raise :class:ImportError if _piaso is not built. "numpy" Force the pure-numpy reference scanner.

Returns

dict

Same schema as :func:_scan_motifs_numpy (motif_ids, tf_names, best_score, hit_count).

scan_motifs_numpy

_scan_motifs_numpy(
    pwms: 'list[PWM]',
    sequences: 'list[str]',
    background: 'Optional[np.ndarray]' = None,
    pvalue: 'float' = 0.0001,
    relative_frac: 'Optional[float]' = None,
    both_strands: 'bool' = True,
    pseudocount: 'float' = 0.01,
)

Signature defaults

pwms, sequences, background=None, pvalue=0.0001, relative_frac=None, both_strands=True, pseudocount=0.01

Reference (pure-numpy) motif scanner: all PWMs × all sequences.

RAM behaviour

Peak resident memory is approximately::

n_motifs × (4 × w × 8 bytes) # all PSSMs (tiny)

• max_seq_len × 1 byte # encoded sequence
• n_windows × 8 bytes # sliding window scores (one seq)

PSSMs and thresholds are precomputed once per motif. Only one sequence’s window score array is live at a time (each is released before the next).

Parameters

pwms

List of PWM objects to scan.

sequences

List of DNA strings (promoter / peak sequences).

background

Length-4 float64 background frequencies. If None, estimated from sequences via estimate_background.

pvalue

Right-tail p-value threshold (used when relative_frac is None).

relative_frac

If given, use relative_threshold(pssm, relative_frac) instead of the DP p-value method.

both_strands

Scan both forward and reverse-complement strands.

pseudocount

Pseudocount passed to PWM.pssm() (default 0.01).

Returns

dict with keys:

"motif_ids" List[str] of length n_motifs. "tf_names" List[str] of length n_motifs. "best_score" float32 array of shape (n_motifs, n_seqs). NaN where no hit was found. "hit_count" int32 array of shape (n_motifs, n_seqs). 0 where no hit was found.

scan_motifs_rust

scan_motifs_rust(
    pwms: 'List',
    sequences: 'List[str]',
    background: 'Optional[np.ndarray]' = None,
    pvalue: 'float' = 0.0001,
    relative_frac: 'Optional[float]' = None,
    both_strands: 'bool' = True,
    pseudocount: 'float' = 0.01,
)

Signature defaults

pwms, sequences, background=None, pvalue=0.0001, relative_frac=None, both_strands=True, pseudocount=0.01

Rust-backed equivalent of :func:piaso.pp.scan_motifs.

scanMotifs

scan_motifs(
    pwms: 'list[PWM]',
    sequences: 'list[str]',
    background: 'Optional[np.ndarray]' = None,
    pvalue: 'float' = 0.0001,
    relative_frac: 'Optional[float]' = None,
    both_strands: 'bool' = True,
    pseudocount: 'float' = 0.01,
    backend: 'str' = 'auto',
)

Signature defaults

pwms, sequences, background=None, pvalue=0.0001, relative_frac=None, both_strands=True, pseudocount=0.01, backend='auto'

Motif scanner: all PWMs × all sequences (Rust-accelerated when available).

This is the public entry point (piaso.pp.scan_motifs). It dispatches between the Rust backend (_piaso.scan_motifs_fwd via :func:~piaso.preprocessing.grn._scan_rust.scan_motifs_rust) and the pure-numpy reference (:func:_scan_motifs_numpy); the two are numerically identical (same log-odds PSSM, N-augmentation, p-value/relative threshold and reverse-complement handling), so backend only trades speed for the no-compiler fallback.

Parameters

pwms, sequences, background, pvalue, relative_frac, both_strands, pseudocount See :func:_scan_motifs_numpy — forwarded unchanged to the selected backend.

backend

Which scanner to use:

"auto" (default) Use Rust if the _piaso extension is built, else fall back to numpy. "rust" Force the Rust backend; raise :class:ImportError if _piaso is not built. "numpy" Force the pure-numpy reference scanner.

Returns

dict

Same schema as :func:_scan_motifs_numpy (motif_ids, tf_names, best_score, hit_count).

scrublet

scrublet(
    data,
    library_key: 'Optional[str]' = None,
    n_components: 'int' = 30,
    sim_doublet_ratio: 'float' = 2.0,
    expected_doublet_rate: 'float' = 0.06,
    n_neighbors: 'int' = None,
    min_counts: 'int' = 3,
    min_cells: 'int' = 3,
    min_gene_variability_pctl: 'float' = 85,
    random_state: 'int' = 0,
    threshold: 'Optional[float]' = None,
    batch_size: 'int' = 1024,
    verbose: 'bool' = True,
)

Signature defaults

data, library_key=None, n_components=30, sim_doublet_ratio=2.0, expected_doublet_rate=0.06, n_neighbors=None, min_counts=3, min_cells=3, min_gene_variability_pctl=85, random_state=0, threshold=None, batch_size=1024, verbose=True

Detect doublets using the Scrublet algorithm (Wolock et al., 2019).

Processes each library independently. Supports both AnnData and cytome Dataset inputs. The cytome path is fully streaming with O(batch_size * n_genes) peak RAM per library.

Parameters

data — AnnData or cytome.Dataset or str

Input data. For AnnData, uses raw counts from X or raw. For cytome, streams from the RNA measurement layer.

library_key — str, optional

Column in obs/cells identifying libraries.

n_components — int

Number of PCA components for the manifold.

sim_doublet_ratio — float

Ratio of simulated doublets to observed cells.

expected_doublet_rate — float

Prior expected doublet rate (for Bayesian scoring).

n_neighbors — int, optional

Number of neighbors for KNN. Defaults to round(0.5 * sqrt(n_cells)).

min_counts — int

Min counts per gene for gene filtering.

min_cells — int

Min cells expressing gene for gene filtering.

min_gene_variability_pctl — float

V-score percentile threshold for gene filtering (default 85).

random_state — int

Random seed for reproducibility.

threshold — float, optional

Manual doublet score threshold. If None, auto-detected.

batch_size — int

Streaming batch size for cytome path (default 1024).

verbose — bool

Print progress messages.

Returns

None

Adds scrublet_score and is_doublet to obs/cells.

selectFeatures

selectFeatures(
    data,
    features: 'Iterable',
    key_added: 'str' = 'highly_variable',
    modality: 'str' = 'ATAC',
    match: 'str' = 'any',
    feature_col: 'Optional[str]' = None,
    overwrite: 'bool' = True,
    min_matched: 'Optional[float]' = None,
    return_mask: 'bool' = False,
    verbose: 'bool' = True,
)

Signature defaults

data, features, key_added='highly_variable', modality='ATAC', match='any', feature_col=None, overwrite=True, min_matched=None, return_mask=False, verbose=True

Flag the features named in features on the cytome’s feature table.

Parameters

data — cytome.Dataset or str

An open cytome, or a path to one.

features — iterable of str

The identifiers to mark. Peak ids ('chr1:100-200'), gene symbols, Ensembl ids — whatever the cytome stores, in any of its identifier columns unless feature_col pins one.

key_added — str, default ‘highly_variable’

The boolean column to write. This is the name runSVD and leiden_local read by default.

modality — str, default ‘ATAC’

Which feature table: 'ATAC' -> peaks, 'RNA' -> genes, 'GA' -> GA_genes, 'tiles' -> tiles. Resolved through the modality registry, never by guessing the table name.

match — {‘any’, ‘id’}, default ‘any’

'any' resolves an identifier against every candidate id/name column the modality registers, so a list in either vocabulary works. 'id' matches only the registry’s canonical id column, which is what you want when two columns could both match and you know which one you mean.

feature_col — str, optional

Match against exactly this column, overriding match.

overwrite — bool, default True

Refuse when key_added already exists and this is False. A name that differs from an existing column only in case is refused either way unless overwrite is set, because SQLite column names are case-insensitive and the two would silently become one.

min_matched — float, optional

Raise unless at least this fraction of features matched something. 0.9 turns “most of my list is missing” from a line of output into an error. The default reports and continues.

return_mask — bool, default False

Also return the boolean mask. Off by default: this writes the column in place like every other writer in the package, and returning an 860,000-entry array prints it into a notebook.

verbose — bool, default True

Report how many features were flagged and how many identifiers went unmatched.

Returns

None, or the boolean mask when return_mask=True.

Examples

>>> ids = marker_df[marker_df['rank'] < 2000]['feature'].values
>>> piaso.pp.selectFeatures(ds, ids, modality='ATAC')
  selectFeatures: flagged 45,398 of 834,686 peaks as 'highly_variable'
>>> piaso.tl.runSVD(ds, modality='ATAC', use_highly_variable=True)

subset_cells

filter_cells(
    data,
    min_counts=None,
    max_counts=None,
    min_features=None,
    max_features=None,
    mask=None,
    inplace: 'bool' = True,
    output: 'str | Path | None' = None,
    overwrite: 'bool' = False,
    modality: 'str' = 'RNA',
    batch_size: 'int' = 2048,
    include_fragments: 'bool' = True,
    include_embeddings: 'bool' = True,
    verbose: 'int' = 1,
)

Signature defaults

data, min_counts=None, max_counts=None, min_features=None, max_features=None, mask=None, inplace=True, output=None, overwrite=False, modality='RNA', batch_size=2048, include_fragments=True, include_embeddings=True, verbose=1

Filter cells. Polymorphic on AnnData / cytome inputs.

The keep-mask is built from up to two independent sources, then intersected:

1. QC thresholds computed from the counts matrix (min_counts, max_counts, min_features, max_features). Backward-compatible with the prior AnnData-only signature.

2. General mask via the mask kwarg. Six shapes accepted:

   • boolean np.ndarray / pd.Series of length n_cells

   • integer np.ndarray of cell indices to keep

   • string pandas-query expression (e.g. "n_counts > 1000 and cluster.isin(['T','B'])")

   • dict mapping column → scalar / list / numeric 2-tuple range. Ranges may be open-ended via None: (lo, hi) → lo ≤ col ≤ hi, (lo, None) → col ≥ lo, (None, hi) → col ≤ hi. This is the clean way to express ATAC cell-QC after calculateCellMetrics / the importer have written the columns, e.g.::

     filter_cells(ds, modality="ATAC",
         mask={"n_fragments": (1000, None),
               "tss_score":   (3.0,  None),
               "frip":        (0.2,  None)})

   • callable fn(obs_df) -> bool series

Cytome path is RAM-efficient: dict masks and simple query strings push down to SQL WHERE and only matching cell_idx values are materialised. Complex queries / callables fall back to chunked streaming over the cells table.

Parameters

data

AnnData or cytome.Dataset. min_counts, max_counts, min_features, max_features Per-cell QC thresholds. Computed from the counts matrix.

mask

General-purpose mask (see above for accepted shapes).

inplace

If True, apply the filter in place. For cytome this calls ds.filter_cells(...) which atomically replaces the file. For AnnData this calls adata._inplace_subset_obs(...).

output

Cytome-only. When set with inplace=False, write a filtered copy to this path via ds.subset(mask, output=...) instead of modifying the original. AnnData ignores this with a warning.

overwrite

When output exists, replace it (default False raises).

modality

Cytome modality used to compute QC thresholds. Default ‘RNA’.

batch_size

Streaming batch size for QC threshold pass on cytome. include_fragments, include_embeddings Forwarded to ds.filter_cells / ds.subset.

verbose

0 silent, 1 (default) prints summary, 2 prints mask sources.

Returns

AnnData backend:

• inplace=True → None (modifies in place)
• inplace=False → bool mask of length n_cells

Cytome backend:

• inplace=True, output=None → int (n_kept)
• inplace=True, output=path → TypeError (contradictory)
• inplace=False, output=None → bool mask of length n_cells
• inplace=False, output=path → open cytome.Dataset at path

Raises

TypeError

inplace=True and output both given.

ValueError

No cells survive the filter, or a mask has the wrong length.

KeyError

mask dict references a missing column.

FileExistsError

output exists and overwrite=False.

sweep_intersect

sweep_intersect(
    cell_indices,
    starts,
    ends,
    peaks_sorted: list,
    counting: str = 'overlaps',
)

Signature defaults

cell_indices, starts, ends, peaks_sorted, counting='overlaps'

Generator sweep-line intersection. Yields hits one at a time — O(1) memory for the hit stream.

Parameters

cell_indices — array-like of int

Cell index per fragment.

starts — array-like of int

Fragment start positions (sorted ascending within one chromosome).

ends — array-like of int

Fragment end positions.

peaks_sorted — list of (int, int, int)

(peak_start, peak_end, peak_global_idx) for one chromosome, sorted by peak_start.

Yields

(cell_idx, peak_global_idx) : tuple of int One pair per fragment-peak overlap.

Overlap condition (half-open intervals)::

peak_start < frag_end AND peak_end > frag_start

Adjacent intervals (peak_end == frag_start) are not overlaps.

Complexity — O(n + m + k) with n fragments, m peaks, k overlaps.

table

table(
    values,
    column: str = None,
    rank: bool = False,
    ascending: bool = False,
    as_dataframe: bool = False,
)

Signature defaults

values, column=None, rank=False, ascending=False, as_dataframe=False

Returns the counts of unique values in the given list or from a data source column.

Parameters

values — list, AnnData, cytome.Dataset, or str

A list of values, or a data source (AnnData, cytome Dataset, or path to .cytome file). When a data source is provided, column must also be specified.

column — str, optional

Column name to read from the data source. Required when values is an AnnData, cytome Dataset, or path.

rank — bool, optional

If True, the results are sorted by count. Default is False.

ascending — bool, optional

If True and rank is True, the results are sorted in ascending order. If False and rank is True, the results are sorted in descending order. Default is False.

as_dataframe — bool, optional

If True, the result is returned as a pandas DataFrame with columns ‘Value’ and ‘Count’. If False, the result is returned as a dictionary. Default is False.

Returns

dict or pandas.DataFrame A dictionary (or DataFrame, if as_dataframe is True) containing the counts of unique values. If rank is True, the dictionary is sorted by count.

Moved to cytorete

These names still work, but the method they call now lives in cytorete — pip install cytorete, then use it directly as cytorete.tl.<name>. Calling them through PIASO raises a pointer to that package if it is not installed.

build_cistrome, build_peak_cistrome, buildCistrome, buildPeakCistrome, bulk_base_cistrome, bulkBaseCistrome, extract_promoter_sequences, extractPromoterSequences