Release notes
v1.2.8
pip install -U piaso-toolsThis release fixes what a review of the whole package found in October 2026: three plots that showed wrong data, functions that gave another answer on a cytome than on an AnnData, and failures that passed without a word. A default call on an AnnData gives the same result as in 1.2.7 in most functions; every change in output is listed below, with what to do about it.
What to redo after upgrading:
- Redraw vertical
plotLigandReceptorLollipopplots (every row but the middle one named another interaction),plotLigandReceptorInteractionheatmaps drawn with one colormap (the ligand and receptor rows were swapped) andplot_dendrogramon a cytome (it clustered the groups on cell index and fragment counts). - Recompute ATAC FRiP from
piaso.pp.calculateCellMetrics: on a matrix that counts insertions, two per fragment (the default), it was twice the fraction of fragments in peaks (up to 2.0), so halve a threshold chosen on 1.2.7’s values. - Name or score a cytome again if you used
kamon(predictCellTypeByMarker) orcalculateScoreParallelon one with the default arguments: a store’s raw counts were scored where an AnnData’s INFOG layer is. - Batch-aware
runGDRon an AnnData (batch_key=...with the defaultlayer='infog') found each batch’s markers on the raw counts; it now uses INFOG, as a cytome and a single batch do, soX_gdrchanges. runGDRwithbatch_keyandgroupby, on an AnnData or a cytome: a batch that lacks one of the groups no longer adds a dimension of unrelated genes for it, soX_gdrhas one column fewer per such batch-group pair.predictCellTypeByGDRlabels: with COSG 1.2.0 or earlier, each reference type and each query cluster added a dimension of unrelated genes for the batch that lacks it, about half of the joint embedding. Labels transferred with it may change; re-run if affected.projectGDRnovelty for a reference built withbatch_keyreported a shift for every query, even for cells of the reference’s own distribution.- Scrublet scores all move a little, and a cell that cannot be scored now gets a missing call, not “not a doublet”.
- Motif scans: a motif whose best word cannot reach the scan’s p-value is no longer scanned; it was called at its best word’s score, and numpy and Rust disagreed there.
Motif occupancy from the scanner: scan_motifs(..., occupancy=True)
- New:
piaso.pp.scan_motifs(..., occupancy=True)also returns, per motif and sequence, the sum over the sequence’s windows (both strands) of each window’s likelihood ratio, motif against background. It is the evidence for one site at an unknown position integrated over the positions, so every good site counts, not only the best. - It is defined for every motif, including one too short to reach the p-value at any single site (a 6 bp core,
whose best word occurs by chance about once in 4 kb, has no hits at p 1e-4 but an occupancy that counts its copies).
Region-level calls built on it are in cytorete (
motif_calls="occupancy"). - The numpy and Rust backends agree to the bit. With occupancy the scan takes about twice as long.
Naming cells from marker genes is piaso.tl.kamon
The marker-based naming method has a name, KAMON, and its function is piaso.tl.kamon.
piaso.tl.predictCellTypeByMarker is the same function under its earlier name and keeps working unchanged, so
existing scripts and tutorials need no edit.
The API reference now documents each function once and lists its other names beside it (for example
align_spatial_coordinates beside alignSpatialCoordinates), instead of repeating the whole entry.
Naming and scoring a cytome score what an AnnData scores
- On a cytome,
kamonandcalculateScoreParallelscorescore_layer('infog'by default, normalized on the fly from the store’s INFOG parameters), as on an AnnData. They scored the store’s raw counts (cytome_layer='counts') whateverscore_layersaid.cytome_layernow defaults toNoneand still decides when given, andmodalitydefaults to'RNA'. A store without INFOG parameters raises an error that names the call computing them (piaso.tl.infog(ds, modality='RNA')). To reproduce a 1.2.7 call on a cytome, passcytome_layer='counts'. - On a cytome,
kamonwrites the columns an AnnData gets in.obsto the cells table: the call, its score or -log10 p-value, and with smoothing the call before smoothing and the confidence. - Ties between p-values go to the higher score (
use_score=False). With 100 control sets the -log10 p-value tops out at log10(101), so cell types that share markers tied and the order of the marker table decided: on a 10,003-nucleus human cortex dataset with its own markers, the p-value route went from 0.698 to 0.976 accuracy, the score route’s.n_ctrl_setis a parameter (100 by default), andscore_method='scanpy'withuse_score=Falseraises at once instead of failing later. predictCellTypeByGDRjoins the reference and the query as two batches of onerunGDR. With COSG 1.2.0 or earlier, each reference type and each query cluster added a dimension of unrelated genes for the batch that lacks it, about half of the joint embedding, which Harmony then aligned and the label transfer used. Labels transferred with it may change; re-run if affected. This needs COSG 1.2.1, which scores only the groups that have cells.predictCellTypeByGDRwith default arguments works: a missinglog1player is built from integer counts in.X, never on your object, and a query and a reference that share barcodes no longer fail.smoothCellTypePredictionon a graph pools the cell itself and itsk - 1strongest neighbors, as documented (it left the cell out); a missing label casts no vote instead of failing.- On an AnnData, the default call (
use_score=True) names cells as in 1.2.7.
score() and INFOG
cell_maskon an AnnData: the results land on your object, NaN outside the mask. They were written to a copy and lost. Integercell_maskindices select those cells, as documented (they were read as a boolean mask).- Inputs that failed: a dense layer scores (it raised), and a matrix of more than 2^31 non-zero values, about a million RNA cells, is scored in row blocks (the compiled kernel’s row pointers wrapped).
- Gene sets: a set with no gene in the data is named in one warning per call (it was dropped without a word, so its cell type could never be called), and a single set with no gene raises. A gene name that an AnnData holds twice is scored over both columns, as on a cytome, with the same warning. Gene weights of another length than their set raise instead of being cut short.
- INFOG refuses counts that are NaN or infinite (one such value made every value NaN), and the in-memory and streaming paths order genes of equal variance the same way.
ensure_infog_params(ds, modality)computes that modality’s parameters only; it computed RNA’s whatever was asked, and rewrote the store’shighly_variablecolumn on every call.
filter_cells: missing values count the same on a cytome and on an AnnData
- A condition about a label keeps the cells that have none, on both inputs (pandas’ rule). On a cytome,
cell_type != 'Doublet'dropped every unlabelled cell, and withinplace=Truethey were deleted for good, while an AnnData kept them.!=andnot inare true for a missing value, every other comparison false, andnot/~negate that. This also holds on pandas’ nullable columns, where AnnData queries failed with a TypeError. - In a dict mask,
Nonemeans “the value is missing” on both inputs (it matched nothing on a cytome). - Integer indices outside the cells raise (
-1selected the last cell), and a query whose value is a number ("flag + 1") is refused. layer=chooses the matrix the QC thresholds read (adata.layers[layer], or{modality}_{layer}on a cytome).
Scrublet
- Every score moves a little: each cell’s neighbor graph now leaves the cell itself out, as scrublet does (each simulated doublet counted itself as a doublet neighbor). Against scrublet 0.2.3 on 4,000 nuclei, 99.7 % of scores are now identical, against 54.4 % before.
- A cell that cannot be scored gets a missing value: a score of NaN and a missing doublet call, where it got 0
and “not a doublet” (a library of fewer than 50 cells, or with too few variable genes). On an AnnData the call is
then a nullable boolean, so
adata[~adata.obs['is_doublet']]raises until the missing calls are handled;piaso.pp.filter_cells(adata, mask="~is_doublet")keeps those cells. library_keynaming an empty column (no cell had a library) scored nothing and called every cell not a doublet; a key that is not a column pooled every cell. Both raise now.layer=names the raw counts; a matrix whose values are not integers is refused instead of being scored as counts.
runSVD: the same embedding on any number of threads
- The dense steps run on row blocks of a fixed height, so an embedding no longer depends on the number of cores (OpenBLAS split them by core count, which moved the last digits and could tip a kNN tie). Embeddings agree with 1.2.7’s to 1e-11, relative, and take about as long.
- Each component’s sign follows one rule: the cell coordinate of largest magnitude is positive. A stored embedding’s components may come out negated against 1.2.7’s once; distances, neighbors, Leiden and UMAP do not depend on signs.
- Fixed on a cytome:
- without
measurement, the number of features is the modality’s own (an ATAC-only store withuse_highly_variable=Falsewas refused as having no feature selected); - with
tfidf_paramsand a feature selection but nocol_mask, the TF-IDF took the IDF of the first features instead of the selected ones; - with
cell_mask, the TF-IDF computed in Python took the sequencing depths of the first cells instead of the masked ones; - cached TF-IDF statistics are checked against their matrix and recomputed when the matrix changed (after the matrix was written again they gave stale depths, an all-NaN IDF and “Eigenvalues did not converge”).
- without
- TF-IDF statistics that are NaN, infinite or negative are refused, and a fallback from the compiled engine to Python warns.
runGDR and projectGDR
- Batch-aware
runGDRon an AnnData found each batch’s markers on the raw counts whenlayer='infog'(the default), while a single batch and a cytome use the INFOG layer; the markers andX_gdrchange, and an AnnData and the same data in a cytome now agree. runGDRwithbatch_keyandgroupby: a batch that lacks one of the groups no longer adds a marker set of unrelated genes for it (one noise dimension ofX_gdrper missing batch-group pair). This needs COSG 1.2.1, which scores only the groups that have cells.- A marker set none of whose genes could be mapped shifted every block after it (AnnData) or failed (cytome); the
blocks now stay aligned. Repeated
obs_namesno longer fail. - On a cytome: a batch-aware run holds one batch’s scores at a time;
save_reference=Falseworks (NameError); a de novo run (groupby=None) clusters on INFOG’s 5,000 most variable genes, as on an AnnData (n_highly_variable_genesanduse_highly_variablewere ignored), keeps its labels asgdr_localand leaves no temporary embedding or graph. - On an AnnData, a de novo run no longer overwrites your INFOG layer and highly variable genes, and leaves no temporary graph.
projectGDR’s novelty for a reference built withbatch_keyis calibrated on the reference’s own cells projected as a query is; it was calibrated in another space, so held-out cells of the reference’s own distribution came out novel. For a single-batch reference nothing changes.projectGDR’s control genes: frozen control genes are matched to the query by gene name (a query with the same genes in another order drew other controls while reporting them frozen), and a cytome reference freezes them too, so a projection no longer depends on the size of the query. A read-only reference works on its second use.
stitchSpace on a cytome integrates
On a cytome, stitchSpace did no integration and reported success: it looked for an embedding name PIASO never
writes, ran COSG on a placeholder matrix, and so pruned every link between batches. It now reads the store’s
embedding (modality=), runs COSG per batch on the store’s expression, and writes the integrated embedding to the
store; the result equals the same data’s AnnData result. filter_bbknn_computation and filter_bbknn_metric reach
BBKNN (annoy was always used), and a failure raises instead of printing a message and returning as success does.
Clustering and UMAP
leidenandneighborstakecell_maskas indices, as documented (they read indices as a boolean mask), and refuse repeated or out-of-range ones.leiden_localputs each group’s labels back by position (cells sharing a barcode took the label of the last of them) and takessvd_drop_first_componentfor each group’s embedding.leidenandumapwarn when they fall back to igraph or umap-learn because the compiled extension lacks an entry point.- The compiled
leiden_csrdefaults to 10 iterations, astl.leidendoes (it was 2 when called directly);tl.leiden’s output does not change.
The motif scanner and the motif-to-TF tables
- A motif whose best word cannot reach the p-value is not scanned, and one warning names such motifs with
their best word’s p-value.
pvalue_to_thresholdreturns+inffor them; it returned the best word’s score, where numpy and Rust disagreed (one motif matched 0 % of regions on Rust and 28 % on numpy). At p = 1e-4 this is about a hundred of JASPAR 2024’s 879 vertebrate motifs, depending on the background; CIS-BP has few. - Short motifs’ thresholds are exact: the score distribution lost its top when rounded, so some thresholds let through up to 24 times the target rate (now under 2.6 times on 800 random motifs).
- numpy and Rust now score windows the same way and agree to the bit, and a minus-strand hit is reported at the forward start of its window.
build_tf_motif_mapgives a JASPAR heterodimer (FOS::JUN) to each partner, as cisTarget’s tables do; as one name it matched no gene, so 76 of JASPAR 2024’s 879 vertebrate motifs were lost.load_cistarget_motifs(max_per_tf=k)keeps each TF’s k motifs with the best similarity q-values (it kept the first k in the table’s order).
Specificity matrices and SCALAR
specificity_matrixon a cytome: a cached matrix is used only while the labels, the cells and the matrices it came from are unchanged. A relabelled or subset store returned the old matrix, and cytorete’sinferRegulonreads this cache by default. The cache holds no pickle; an older entry is recomputed once.specificity_matrixon an AnnData:cosg_layernames the matrix COSG scores (every layer gave.X’s result), anduns['cosg']is left alone.runSCALAR:call_breadthdivides by the groups where a pair was tested (by every group, so a pair the detection floor kept out of most cell types looked specific); a misspeltlayerraises (it fell back to.X); on a cytome,layer=Nonereads the matrix that was the AnnData’s.X(else the counts), so a store and its AnnData give the same p-values; a pair listed twice is tested once, with a warning;verbose=Falsesilences the messages and progress bars.
The PIASOmarkerDB client
- Requests are retried with a growing pause on a failed connection or a busy server (
retries=, 3 by default), and an answer that is not JSON raises an error naming the status and the start of the answer. cache_diris an opt-in response cache, kept forcache_ttlseconds (a day by default). It defaulted to~/.piaso/markers, created by every client although nothing was cached, so a read-only home failed every call.- Large requests are read in pages within the server’s limit (a larger page came back short and was read as the
last); a cell type whose name holds a comma is matched by its exact name;
offset=withoutlimit=works. as_dict=Truelists a gene once per cell type (it repeated it once per study or tissue, andscore()counted it each time) and warns when one cell type name gathers several contexts.loadCustomMarkersdrops repeated genes too,getRecommendedStudyreturns the studies’ current names, andanalyzeGeneswith several gene sets asks for all their genes in batches of 200 instead of once per set.
calculateCellMetrics: FRiP in one unit
ATAC FRiP divided the insertions in peaks (two per fragment) by the fragments, so a cell with every fragment in
peaks got 2.0. It is now the insertions in peaks over two cut sites per fragment, capped at 1. A matrix that
counts each fragment once is divided by the fragments (counting='overlaps'); a matrix that records no counting,
such as one converted from AnnData, is taken as insertions, with a warning; a presence matrix gets no FRiP. A
failure in this step raises instead of leaving the column out silently.
Downloads in piaso.data
- Every download goes to a temporary file, is checked (length, size or checksum) and only then renamed, so an interrupted download no longer counts as complete; connections time out after 60 s, and network errors are retried three times.
- The newest dataset registry wins: a registry cached by an older PIASO hid the datasets added since.
fetch_dataset(force=True)removes the old store’s-waland-shmfiles, which SQLite would replay onto the new one, and a downloaded file takes your usual file permissions.
Plotting: three plots that showed wrong data, and one rule for colours and figures
Redraw these. Three plots drew wrong data under their defaults:
plotLigandReceptorLollipop, vertical layout (the default): every row but the middle one was labelled with another interaction, since SCALAR was added. The rows were reversed for display and their labels reversed a second time; the label colours went with the labels, so text and colour agreed with each other but not with the row. Every vertical lollipop drawn with PIASO up to this release is affected, including the old SCALAR tutorial’s (vertical_layout=True); the horizontal layout was right.plotLigandReceptorInteraction, one colormap (the default), horizontal layout: the ligand specificities were drawn in the bottom row, under labels that read “Ligand” above “Receptor”. The two-colormap and vertical layouts were right. The labels now sit at each row’s centre.plot_dendrogramon a cytome:use_rep='auto'found no embedding (it looked for AnnData’sX_names only) and clustered the groups on the cells table’s numeric columns, cell index and fragment counts, without a warning. It now uses the store’s embedding (RNA_pca,ATAC_svd, …); a store with neither an embedding nor highly variable genes raises, naming what to pass.
Colours follow the category, by name, in every plot. One resolver now gives each category its colour over the
whole column, from the stored colours (adata.uns, a cytome’s set_categories, or the colours an AnnData carried
into a store) or PIASO’s palette in the column’s order. Colours used to be matched to the categories a plot happened
to draw, so these changed colour, and now keep it:
embedding(cell_mask=...), which drew a category in the colour of one the mask left out;- a cytome converted from AnnData, whose categorical order was ignored (now its categories follow the AnnData’s order and colours);
sankey(color_by='right')(offset by the number of left categories) andleft_order='auto';stacked_barplot(sort_groups=True); a violin with an empty group; the dendrogram’s leaves; the cell-level heatmap’s sidebar;plotGroupMetrics.
A store’s stored order and colours now reach the violin, the stacked bar, the Sankey, the dendrogram and the
heatmap, and labels without a stored order sort naturally (0, 1, 2, … 10) in the stacked bar, as in the
embedding. A qualitative palette named by string ('tab10', 'Set2') gives its colours in order instead of an
even sample across them, and a stored colour list shorter than its categories is ignored with a warning instead of
wrapped.
Missing values are drawn and named. A missing label (NaN, None, or the text 'nan' a cytome stores for one) is
drawn first, in na_color, and listed as “NA”, instead of becoming a 'nan' category (embedding,
split_embedding, scatter, which dropped those points). Continuous NaN values are drawn in na_color; they were
transparent.
Figures. show defaults to showing unless ax= is given; a function never closes a figure passed in through
ax= (six of them did, so a later plt.savefig wrote a blank page); a figure that is not shown is returned, and one a
function made is closed only once it is saved, so show=False followed by plt.savefig() keeps working. Layout
acts on the figure drawn, not the current one, and leaves a figure with constrained layout alone. PDFs and EPS files
saved with save= embed TrueType fonts, as journals ask. In a notebook, a show=False call left as a cell’s last
line is now displayed twice (once as the returned value); assign it (fig = ...) or end the line with ;.
Other changes in the plots’ output:
dotplot:expression_cutoffis applied (it was ignored, and a fraction was always of values above 0), andmean_only_expressedaverages the values above it; the dot-size key draws each fraction at its plotted size (it was capped, so the 80 % and 100 % dots were alike and smaller than a 100 % data dot).scatter: a cytome’s text column (leiden) and a boolean column are categories, as in the embedding (the first was drawn on a colorbar). Withon=, a text column’s missing values are drawn as “NA” (they were a category named “None”), andon=takes the store’s feature tables (genes,peaks,tiles) andsamples, read through cytome’s accessors; an internal table is no longer accepted.split_embedding: a boolean colour is categorical, as inembedding.plotLigandReceptorLollipop: the circle-size key draws the plotted sizes (its dots were 1.13x wider);score_range_minandscore_range_maxeach clip both ends of the stems (one alone clipped one end);circle_size_scalenever had an effect and now warns.plotLigandReceptorInteraction: withshared_legend=Trueandheatmap_vmaxthe colorbar is labelled (its ticks were at -inf), and withpreserve_input_order=Truethe shared scale covers the rows drawn.heatmap(cell_level=True)works (it raised NameError); withstandard_scalethe colorbar reads “z-score”.plotConfusionMatrix: the reference colour bar is on top, beside its labels.- New options:
sort_order=('size'draws rare categories on top) andvcenter=in the embedding plots; abasismatching bothRNA_umapandATAC_umapwarns, andmodality=chooses. import piasono longer reseeds numpy’s global random stream; the shuffled palettes are unchanged.
PIASOmarkerDB’s marker regions in PIASO: getMarkerRegions
- New:
piaso.tl.getMarkerRegionsqueries PIASOmarkerDB’s marker regions, the cell type-specific 500 bp tiles of single-nucleus chromatin atlases, up to 1,000 per cell type ranked by COSG.as_dict=Truereturns{cell_type: [tile IDs]}named as a PIASO store names its tiles ('chr1:3000500-3001000'), the formpredictCellTypeByMarkertakes, so nuclei can be named from their tile counts.top_nkeeps each cell type’s best regions, andtile_sizemaps them onto another grid. - The genome build is checked: each region study is on one build (hg38 for human, mm10 for mouse). Pass
genome=with your data’s build, and a study on the other build raises an error instead of matching nothing; regions are not lifted over between builds or species. getMarkerRegions(list_studies=True)lists the region studies; the client classPIASOmarkerDBhasgetMarkerRegionsandlistRegionStudies.
Installation and packaging
- PIASO needs Python 3.10 or later (
requires-python = ">=3.10"). Pre-compiled wheels are built for Linux on x86_64, CPython 3.10 to 3.12; elsewhere pip builds PIASO from source, which needs Rust and a C compiler. The README promised macOS and Windows wheels, which were never built. - Requires cytome 0.3.9 and COSG 1.2.1;
pip install -U piaso-toolsinstalls them. COSG 1.2.1 scores only the groups that have cells (therunGDRandpredictCellTypeByGDRfixes above), works under pandas 3 and reads a cytome’s column-major matrices. matplotlib’s floor is 3.6 (on 3.5 a colormap you named was silently ignored). - The source distribution ships
Cargo.lockand is built with it, so a build from source uses the Rust crates the tests ran with, and every wheel is installed and tested before it is published. - The tests carry markers (
slow,data,network), sopytest -m "not slow"is a fast subset.
Also in this release
- Cell Ranger feature names made unique never repeat a name (
['A', 'A', 'A-1']gave two'A-1'); a boolean mask with missing entries leaves them out; a mistyped cytome path raisesFileNotFoundErrorinstead of leaving an empty file behind.
v1.2.7
pip install -U piaso-toolsrunSVD stops when its singular vectors have converged, and ATAC embeddings come at unit length
Two defaults of piaso.tl.runSVD change, and with them the embedding a fixed
random_state gives:
- The stopping rule:
stop_on='residuals', the new default, stops when every component’s relative residual is belowtol, so the singular vectors themselves have converged. 1.2.6 stopped when the leading singular values settled, which can happen while the vectors still depend on the random start. A run may take a few more passes. - Unit length for ATAC: with
unit_lengthunset, the TF-IDF of ATAC peaks or tiles is embedded at unit length, soneighbors,leidenandumapwork with the angle between cells rather than a length that follows sequencing depth. Each cell’s length is kept (adata.obs['X_svd_length']; in a cytome, the cells table’s<embedding>_lengthcolumn). RNA embeddings are not scaled.
What to know when upgrading:
-
Embeddings change. To reproduce an embedding made with 1.2.6, pass both:
piaso.tl.runSVD(data, ..., stop_on="singular_values", unit_length=False) -
So does what embeds through
runSVD.leiden_localpasses both to each group’s SVD (svd_stop_on,svd_unit_length), andrunGDR’s own SVD stops on the residuals. -
An AnnData whose matrix already holds TF-IDF values carries nothing that says so: pass
modality='ATAC'orunit_length=True.
runSVD can leave out the first component, and runs the same engine on an AnnData as on a cytome
drop_first_component=Truecomputesn_components + 1components and keeps components 2 ton_components + 1, so the embedding still hasn_componentscolumns, andSandVthold the same components. For non-negative, uncentered inputs such as the TF-IDF of peak counts, the first singular vector is the axis of the mean profile and follows sequencing depth more than cell type. The choice and the left-out singular value are recorded with the embedding. It is not meant for RNA inputs.- An AnnData gets the solver and the Rust pass engine a cytome gets, with
the engine reading the matrix in memory:
method,tolandn_iterare honored, every allowed core is used, and an AnnData and the same matrix in a cytome give the same embedding,SandVtto the bit.auto_tfidf=True,cell_mask,scale_data,return_svd,tfidf_paramsandinfog_paramsnow work on an AnnData (the in-memory path ignored the last two). - What to know: an AnnData’s embedding changes once, since it came from
scikit-learn’s randomized SVD (seven fixed rounds on one core).
auto_tfidf=Trueon an AnnData whose values are not counts raises.
piaso.tl.umap is PIASO’s own: parallel, and the same picture on any number of threads
piaso.tl.umap now lays out the graph with PIASO’s extension by default
(backend='auto'), whenever a kNN is stored (after piaso.tl.neighbors) or
passed. It computes umap-learn’s algorithm: the same fuzzy graph, the same
curve, the same spectral start and the same schedule of attractions and
repulsions. What differs is the order of the work. umap-learn makes a seeded
run single-threaded, because its updates share one stream of random numbers.
PIASO’s layout cuts the nodes into 128 blocks by a hash of their names, and an
epoch visits every pair of blocks once; the pairs that run at the same time
share no node, so the threads together compute one sequential pass in a fixed
order. One graph and one random_state give the same coordinates on 1
thread or 20.
- Quality. Its pictures keep each cell’s graph neighbors as umap-learn’s do, within the variation between umap-learn’s own seeds.
- Speed. The layout uses every core the process may use, so a large dataset takes a fraction of umap-learn’s single-threaded time.
- The start does not depend on the seed, as umap-learn’s does not, and each axis’s sign is fixed, so two seeds no longer give mirrored pictures.
- Memory. The layout keeps 12 bytes per edge of the graph and about 40 per cell, the same on any number of threads.
What to know when upgrading:
-
The coordinates change. They differ from umap-learn’s as two umap-learn seeds differ. To reproduce an embedding made with 1.2.6:
piaso.tl.umap(data, ..., backend="umap-learn") -
Threads.
n_threads=0(the default) uses the cores the process may use; the picture does not depend on it. -
The whole stored kNN is used. umap-learn takes a stored kNN at its own
n_neighbors(15): it cuts a larger one to its first 15 columns and sets a smaller one aside for neighbors it computes itself. PIASO’s layout uses every column. Withneighbors’ default of 15 the two agree. -
umap-learn stays a dependency; without a stored kNN,
tl.umapstill calls umap-learn, which computes its own neighbors, with a warning.
score() leaves a gene set’s own genes out of its controls
Every path now draws control genes the same way: in numpy, a fresh random
state per gene set seeded by random_seed, gene by gene, with the genes’
means and variances summed in float64. The same seed gives the same control
sets and the same scores on an AnnData with or without the compiled
extension, in the single-set mode and on a cytome.
The new default, exclude_set_genes=True (in score,
calculateScoreParallel and predictCellTypeByMarker), draws each gene’s
controls from its nearest genes outside its own set. A member among a gene’s
nearest genes gives its place to the next nearest non-member, so only the
draws that would have been a member change.
What to know when upgrading:
- Scores change where a set’s genes are among each other’s nearest
genes, which is common for co-expressed marker sets.
exclude_set_genes=Falsekeeps the earlier draws of the numpy path; the compiled path, which drew its controls with another generator, now draws as numpy does, so its scores change once either way. - A panel with fewer features than the neighbors asked for (twice
n_nearest_neighborswith the exclusion) still scores, and one warning per call names the genes left with fewer candidate controls. - The nearest-gene table is int32, so the wider table the exclusion needs takes the memory the int64 table took before.
predictCellTypeByMarker(smooth_by='scores')
smooth_by='scores' averages each cell’s scores (or -log10 p-values) over its
smoothing neighbors before taking the best cell type, instead of voting the
neighbors’ own best types (smooth_by='labels', the default, unchanged). The
neighbors are the vote’s, the averaged matrix is kept in
obsm['{key_added}_score_smoothed'] (or '_nlog10pvals_smoothed'), and the
confidence is the share of the neighbors whose own best type is the one
chosen. On an AnnData and on a cytome.
Fixed: runSVD(ds, modality='ATAC') reads the peak counts
On a cytome, runSVD with no measurement read an INFOG layer whatever the
modality, so the plain call on a store holding ATAC peak or tile counts
stopped with KeyError: no ATAC_infog matrix. Left unnamed, the layer now
follows the modality: the counts for 'ATAC' and 'tiles', which the
default TF-IDF transforms, and INFOG for RNA. The call gives what
measurement='counts' gave, value for value.
Fixed: after runGDR on labels with ”/”, the AnnData can be written
anndata 0.13 refuses a ”/” in a key of an h5ad file by default
(anndata.settings.disallow_forward_slash_in_h5ad), and runGDR kept its
marker sets in adata.uns['gdr_reference'] as a table whose columns are the
cell types, so on labels such as “L2/3 IT” adata.write_h5ad failed with
“Forward slashes are not allowed in keys”. Such names are now stored with ”/”
written as “%2F”, and the exact names beside them (marker_gene_names);
projectGDR reads them back unchanged. Labels without ”/” are stored as
before, and a reference written by 1.2.6 still reads.
Also in this release
- Small inputs.
neighborscaps the number of neighbors at the number of cells, and replaces a graph in which the approximate search left a slot unfilled by the exact graph (a 10-cell batch failed with “negative axis 1 index: -1”).runGDR(batch_key=...)gives a batch of fewer than 16 cells no markers of its own, with a warning, instead of failing, and still embeds its cells. leiden_local(tmp_dir=)places the groups’ temporary stores, and when the disk is short writes them in batches that fit, one scan of the matrix each; the labels are those of a single split.filter_cells(check_integrity=True)on a cytome asks for SQLite’s integrity check of the filtered store; cytome 0.3.7 otherwise runs only its own consistency checks.makeGenomeFilesreads the GTF withcytome.read_gtf: one gene pergene_id, so genes that share a name are no longer merged into one span (in Cell Ranger ARC’s Mmul_10 reference, the copies of TRNAN-GUU spanned 118.6 Mb of chr1).specificity_matrix(cosg_remove_lowly_expressed=False)buildsrunSCALAR’s input: without COSG’s detection filter the matrix has no negative entries.- A renamed PIASOmarkerDB study’s former name still works in
analyzeGenesandanalyzeMarkers:studies=andexclude_studies=are resolved through the server’s map of former names (studyAliases()), in any case, with one warning per renamed name. A server without the map leaves the names as given. - Requires cytome 0.3.7 (for
cytome.read_gtf);pip install -U piaso-toolsinstalls it.
v1.2.6
pip install -U piaso-toolspiaso.tl.leiden is PIASO’s own: parallel, and the same clusters on any number of threads
Leiden now runs in PIASO’s Rust extension by default. It is the same algorithm (Traag, Waltman and van Eck 2019), with the same objective and the same randomised refinement as igraph’s, and it reaches the same modularity. What differs is how it is run:
| 10 iterations | cells | PIASO | igraph |
|---|---|---|---|
| mouse developing visual cortex | 200,061 | 3.4 s, 0.11 GB | 21.8 s, 0.32 GB |
| whole mouse brain | 2,341,350 | 26.3 s, 1.3 GB | 553.7 s, 3.7 GB |
Measured on a 10-core workstation (20 threads); memory is what the call adds to the process.
One graph and one random_state give one partition. The number of
threads, the cores of the machine and the layout of the graph in memory do
not enter the result: the labels are identical at 1, 2, 4, 10 and 20
threads. Parallel clustering usually gives that up, because threads see each
other’s moves in an order that changes from run to run. Here a round of
moves reads the state the round began with, every random number is a hash of
the seed and the cell, and edge weights are summed as integers, so nothing
depends on which thread did what first.
piaso.tl.leiden(adata, resolution=1.0) # all the cores this process may usepiaso.tl.leiden(adata, resolution=1.0, n_threads=4) # the same labelspiaso.tl.leiden(adata, resolution=1.0, backend="igraph") # the partitions of 1.2.5What to know when upgrading:
- Clusters change. The partitions differ from those of 1.2.5 by about as
much as another seed would change them (ARI 0.82 to 0.88 between the two on
the datasets above, where two seeds of igraph agree at 0.80 to 0.89). Pass
backend="igraph"to reproduce an earlier result. - So does everything that clusters.
leiden_localandrunGDRcallpiaso.tl.leiden, so their results follow.leiden_localtakesbackend=andn_threads=and passes them on. - The graph must be symmetric. A directed graph is refused, with the pair of nodes whose two directions differ. igraph’s path took the upper triangle and clustered it without a word.
n_iterationsstays at 10. Two runs give 96 to 99 % of the partition of ten in under half the time, which is worth knowing at tens of millions of cells; ten make two seeds agree a little more.- Ctrl-C stops a run. A graph of fewer than 32,768 nodes is clustered on one thread, which is faster there.
The Leiden at scale tutorial runs it on 200,000 cells and shows the clusters beside igraph’s.
Reading a store you cannot write
The plotting functions open a .cytome read-only, so plotting from a
downloaded or shared store leaves the file exactly as it was. Statistics
that are normally cached in the store (cell depth, INFOG and TF-IDF
parameters) are computed for the call instead when it cannot be written.
Functions that compute and then write (leiden, leiden_local,
neighbors, umap, runSVD, runGDR, infog, run_TFIDF) check first,
and a read-only store fails in a second with a message saying why, not at
the end of the run.
The motif scanner: twice as fast, and safe after fork
piaso.pp.scan_motifs(backend="rust") scores a window by walking it and the
motif’s columns together, with no index arithmetic in the inner loop: 300
motifs against 100,000 sequences of 300 bp take 5.7 s instead of 10.4 s, and
peak memory is 1.03 GB instead of 1.26 GB. The scores are identical to the
bit.
It also hung for ever in a process forked after it had run, which is
Python’s multiprocessing on Linux. It now runs on a thread pool of its
own, releases Python’s lock while it works (other Python threads keep
running) and stops on Ctrl-C. Every parallel function of the extension now
builds its pool the same way, and n_threads=0 means the same in each:
RAYON_NUM_THREADS if set, else the cores the process may use.
runSVD on a cytome no longer opens the file a second time
The SVD engine used to open the store with its own copy of SQLite while
Python’s connection held it. The two could not see each other’s locks, and
the engine could reset the shared index of the write-ahead log that Python
had mapped; the next large write then ended the process with a bus error
(SIGBUS). The engine now computes on the compressed chunks the open
dataset hands it (ds.chunk_source), so no second SQLite touches the file.
The result is identical to the bit, and cache_chunks=True now works with
the engine, keeping the compressed chunks within half the memory available.
Also in this release
leiden_localon a cytome makes every group’s store in one scan of the matrix (ds.split), instead of one subset per group. The labels are unchanged.piaso.data.chrom_sizes(genome)gives the chromosome sizes of hg38, mm10, Mmul_10, mCalJac1, hg19 and mm39 by name, or of anychrom.sizesor.faifile, reduced to the primary chromosomes.piaso.data.fetch_datasetchecks a download before it takes its final name, and never replaces a file that is already there: a downloaded.cytomeis a working file, and checking it on every call re-downloaded it over the results written into it.color=given a NumPy array of gene names draws one panel per name. It was read as one value per cell.- Fixed:
plot_embeddings_splitwithoutgroups=could give categories each other’s colours. Each panel’s palette was built in alphabetical order but applied in the column’s category order, so wherever the two differed (a categorical order, or one stored withds.set_categories) the colours were swapped, with no warning. Every panel now colours each category asplotEmbeddingdoes. Figures made with 1.2.5 or earlier withoutgroups=are worth a second look.
Requires cytome>=0.3.6.
v1.2.5
pip install -U piaso-toolsrunSVD on a cytome runs its passes in Rust, and converges by default
The streaming SVD’s cost is the pass over the store: one sparse product per
pass, twenty-odd passes per solve. Those passes now run in a Rust engine that
reads the chunks straight out of the file, in whichever layout the matrix is
stored (row-major or column-major, any integer width or float, and the zlib
blobs older files carry under the zstd label), blocks each chunk against the
dense operand so every random access stays in cache, and lets each thread own
its slice of the output. Every entry of every product is one sum in a fixed
order, so the result does not depend on the thread count and reruns are
bit-identical. Measured on a 37,609-cell, 317,726-peak matrix, one round of
the solver went from 27 s to 9.5 s; on a million-cell tile matrix, from
199 s to 135 s. n_threads=0 (the default) uses every core the process is
allowed; PIASO_SVD_PROFILE=1 prints, per pass, how long was spent reading,
bucketing and multiplying.
The solver changed with it. method="auto", now the default, runs block
Lanczos with a stopping rule (tol=1e-3) and reports what it did:
Solver: block Lanczos (at most 12 rounds = 26 passes)Solver done: block Lanczos, 9 rounds, 20 passes (converged)Solver time: 43.0 s, of which 41.6 s in 20 passes over the store and 1.4 s in numpy between themThe old power iteration at its default of seven rounds was not converged on
a peak matrix; it needs about twenty-two. method="power" with n_iter
still runs exactly that many rounds when you want a fixed budget, and
method="krylov" with n_iter does the same for Lanczos.
Two more defaults for peak and tile matrices on a cytome. TF-IDF is applied
on the fly to raw ATAC or tile counts unless auto_tfidf=False; the pass
engine weights each entry as it reads it, so no normalised layer is written.
The guard that refuses to normalise an already-normalised layer runs first:
inferred TF-IDF steps aside with a warning on a layer that does not look like
raw counts, and an explicit auto_tfidf=True raises. And the reads take no
file locks: once the write-ahead log has been checkpointed, the engine opens
the store as an immutable snapshot, which is what lets it run at full speed
from a network file system. On a cluster node reading from a network mount,
a twenty-pass solve went from 5 min 13 s to 44.9 s; from the node’s local
disk the same solve is 29 s. The installation page has a section on running
on a cluster.
leiden_local re-embeds every group with runSVD
dr_method="X_svd" is the new default: each group is embedded by runSVD
on its own selected features and clustered from there. On a cytome the
per-group work streams from disk for any modality, with the TF-IDF runSVD
applies to peaks or tiles and the store’s highly_variable selection, and
each group’s subset carries only the one matrix it needs. "X_svd_full" uses every feature
rather than the selected ones; "X_pca" keeps the INFOG route. min_cells,
svd_method, n_iter and svd_tol pass through, and the function no longer
touches scanpy. infog_svd likewise defaults to the infog layer, and where
there is no highly_variable column it uses every gene and says so.
Feature selection by name, and two primate genomes
piaso.pp.selectFeatures(ds, features) flags the named features (peak ids,
gene symbols, Ensembl ids, whatever the cytome stores) in one call, writing
the boolean column runSVD and leiden_local read by default. match="all"
and min_matched= turn a partial match into an error.
piaso.data.fetch_genome knows rhesus macaque (Mmul_10) and common
marmoset (mCalJac1), with chromosome names as the Cell Ranger ARC
references carry them; piaso.data.makeGenomeFiles builds the same files for
any assembly from a GTF and a chromosome-sizes file. A genome without a GTF
source no longer announces one.
Every reader takes a column-major matrix
Peak and tile matrices stored column-major (the layout the SVD engine
prefers; cytome 0.3.3 reads and subsets it as such) are read that way
throughout:
score, calculateCellMetrics, calculateFeatureMetrics, normalize_log1p,
the ligand-receptor readers, the TF-IDF raw-count guard and the feature reads
behind plotting all take the columns when that is what the store holds,
instead of asking for row chunks a column-major matrix does not have.
A breaking change: getMarkers(as_dict=True) returns the dictionary
It used to return (DataFrame, dict). Almost every use of it wanted the dict —
it is the form predictCellTypeByMarker takes — and paid for that by unpacking
a pair and discarding the first element. Asking for a dict now gets a dict:
marker_db = piaso.tl.getMarkers(study="AllenWholeMouseBrain_isocortex", as_dict=True)piaso.tl.predictCellTypeByMarker(adata, marker_gene_set=marker_db)as_dict='both' returns the old pair. It is deprecated and will be removed in
the next release. as_dict=False, the default, is unchanged.
The change announces itself once per session on the first as_dict=True,
because a caller that keeps unpacking does not always fail loudly: with exactly
two cell types, df, d = getMarkers(as_dict=True) succeeds and binds two
cell-type names. Search your code for , ... = piaso.tl.getMarkers( — the
tutorials are updated.
Dot plots: the same numbers, two orders of magnitude faster
piaso.pl.dotplot computed each group’s mean and expressed fraction with one
Python comparison per (feature, group) pair, each scanning every cell. It now
builds a one-hot group indicator once and takes two sparse products. The
numbers are identical to 5e-16. A 100,000-cell, 60-feature, 30-group plot goes
from 68 s to under half a second; at 20,000 cells it is 9.1 s to 0.05 s.
A missing groupby column is an error, not a wrong answer
SQLite has a compatibility misfeature: a double-quoted token that does not
resolve to a column is treated as a string literal rather than raising. So
SELECT "Leiden" FROM cells on a cytome with no Leiden column returns the
string 'Leiden' for every row — and the caller sees one group, named after
the typo, with a marker table that looks entirely plausible. That is how a COSG
run reported 1 groups for a column that was never there.
Every place a user-supplied column name reaches such a query now looks it up
first: cosg.run_cosg_cytome, piaso.tl.runSCALAR, and the new fragment-length
functions. The error lists the columns that do exist. A case variant is still
accepted, because SQLite column names are case-insensitive and leiden really
does name Leiden. piaso.utils.require_cells_column is the shared check.
groups= draws a legend of what is on the plot
plotEmbedding(color='CellTypes', groups=['A','B']) draws A and B in their
palette colours and everything else as one grey cloud. The legend was built
from the full category list, so on a forty-cluster column it listed forty
entries in forty colours, thirty-eight of which appeared nowhere on the figure
— while the grey covering most of the points had no entry at all.
It now lists the highlighted groups and one final entry for the grey, labelled
by the new na_label (default 'other') and shown only when something was
actually left out. Highlighted groups keep the colour they have on an
unfiltered plot.
color= can be the values, not only a column name
Anything computed on the fly — a regulon activity vector, a per-cell score
never written back, a mask — had to be assigned into obs first. Passing the
array reached color in adata.obs.columns and came back as
TypeError: unhashable type: 'numpy.ndarray'.
plotEmbedding and plot_embeddings_split now accept a numpy array, a pandas
Series or a Categorical as one value per cell. Numeric values get a colorbar,
everything else a category legend; a Categorical keeps its own order, and a
named Series titles the panel. A wrong length names both numbers. A plain
list/tuple still means one panel per colour.
A Series is aligned by name when its index resolves to the cell names —
anything that came out of a groupby, a sort or a .loc colours the right
cells instead of being silently permuted. A RangeIndex, or an index sharing
no labels with the cells, is used positionally; an index that resolves some
of its labels and not others is refused, naming the count, because neither
reading is safe. Duplicate cell names make alignment ambiguous, so those fall
back to position with a warning.
predictCellTypeByMarker can weight its markers
A marker list treats a textbook marker and a weak, broadly expressed one
identically. marker_gene_weights= passes per-gene weights through to
piaso.tl.score(gene_weights=): a dict keyed by cell type, a DataFrame read
column by column, or a sequence in set order.
The weights are used as given, with no normalisation per cell type. score()
divides by median(weights) * n_genes and builds its control sets from the
same weights, so rescaling one cell type’s weights cannot change which cell
type wins — measured at 9e-16 across a 1000× rescale. Normalising would be a
step that provably changes nothing.
What a rescale cannot survive is a median of zero: the divisor becomes zero and
every score becomes NaN without a word. score() now rejects that, and
non-finite weights, naming the gene set. Mismatched lengths, a missing cell
type and an unknown one are each refused with both numbers.
Smaller things
-
predictCellTypeByGDRfits on the shared label column. With a query that lacks the reference’s label column, the classifier was fitted on a column that only the reference rows carried and failed with aKeyError; it now fits on the combined column both sides share. -
Twenty-two public functions have complete docstrings, checked by a test against their signatures: every parameter present, every default the one the code uses, the cytome-only parameters and the deprecated aliases documented where they live. The remaining functions are on the same test’s backlog.
-
leiden_localstops warning about its own work. It calledneighborswithout passing the modality on, so an ATAC run warned on every call that it had found the embedding PIASO itself had just written under another modality. The argument is forwarded; graphs and labels are unchanged, and a genuine cross-modality guess still warns. -
A per-cell column will not land on someone else’s. SQLite column names are case-insensitive, and cytome resolves a new spelling to an existing column with only a warning — so writing
tss_scoreinto a cytome already carrying aTSS_scorecolumn from another tool replaced it.piaso.utils.write_cells_columnis the shared guard: the same spelling overwrites (re-running a QC step replaces its own column), a spelling that differs only in case is refused unlessoverwrite=True. -
The TF-IDF cache no longer grows with the dataset.
compute_tfidf_statskept its per-cell depth and per-feature IDF vectors inds.metadata, which is a JSON document rewritten whole on every save — large enough on a real ATAC cytome for cytome to warn about its size. The depth is now acellscolumn, the IDF stays the feature column it was already written to, and metadata keeps the scale factor and the two column names. At 20K cells by 30K peaks that is 98 bytes instead of 1.0 MB. Cytomes written the old way are read unchanged.
v1.2.4
pip install -U piaso-toolsSCALAR: a detection floor, applied in the test
piaso.tl.runSCALAR gains expressed_pct (default 0.1) and groupby. An
interaction is tested only if the ligand is detected in at least that fraction
of the sender’s cells and the receptor in the receiver’s, and each gene’s
matched control genes are drawn from genes that clear the same floor in the
same cell type. The null therefore never contains a control that is not
expressed where it is used, and the k × k support stays full. Rows that fail
the floor are returned with testable = False, p_value = 1 and are excluded
from FDR; two new columns, ligand_expressed_pct and receptor_expressed_pct,
carry the fractions the floor was applied to. groupby names the cell-type
column and is required when the floor is on; expressed_pct=None reproduces
the previous behaviour exactly.
Why in the test and not in the matrix: zeroing lowly-detected genes in the specificity matrix leaves the observed pair conditioned on passing the floor while its controls are zeroed wherever they fall below it, so most of the null becomes zeros and the p-value drifts toward “detected in more cells than its peers”. Measured on the SCALAR tutorial data, a quarter of the calls that approach produces are significant only because of the zero padding.
runSCALAR now validates its input. NaN specificity scores raise (they are
what cosg.indexByGene leaves for genes outside a top-N list unless
set_nan_to_zero=True, and a NaN observation would otherwise score at the
p-value floor). Negative scores raise (COSG marks genes failing its own
expressed_pct filter with −1, and two such sentinels multiply to +1, the
largest product in the table). Existing callers that pass a clean matrix and
no groupby will now see an error asking for one; add groupby= or
expressed_pct=None.
The SCALAR tutorial’s recipe changes with it: COSG at mu=10 over all genes,
raw scores, no COSG-side filter, and the floor applied by runSCALAR. Any
per-cell-type linear rescaling of the specificity matrix (for instance
dividing by an IQR) leaves every p-value unchanged, because the observed
product and its null scale together; cosg.iqrLogNormalize applies a log1p
on top and is not recommended as SCALAR input.
SCALAR: a readable score, and a magnitude that compares across pairs
The interaction score is a product of two specificities, each below 1, so a pair of two 0.02 specificities scored 0.0004 — a number no one can hold, and one that is not comparable across sender–receiver pairs because specificity is calibrated within a cell type. Four columns fix both without touching a single p-value:
ligand_specificityandreceptor_specificity, the two inputs.specificity_geomean, their geometric mean: a monotone function of the product, so it ranks exactly as the score does, on the specificity scale. The tutorial’s plots are sized by it.null_meanandlog2_enrichment: the mean of the interaction’s own matched null and the log2 ratio of the observation to it (a 10⁻¹² guard against division by zero, nothing more), so every pair is referenced to what expressed, expression-matched gene pairs achieve between the same two cell types. This is the number to compare across pairs. On the tutorial data the largest raw score in the table (PTPRC→CD22) is ten times its null; a somatostatin pair scoring 0.0008 is six thousand times its null.
statistic="min" is a second test statistic: the smaller of the two
specificities, so both sides must beat the controls’ weaker sides — the
“are both specific” question. Its exact null factorises into two one-sided
counts. The default remains the product.
piaso.tl.umap and piaso.tl.leiden no longer return a value
Breaking. Both wrote their result and returned it on AnnData, while
returning None on a cytome — one function with two contracts, and a
161,027-row array echoed into every notebook cell that called it. They now
return None on both backends; read the result from adata.obsm[key_added]
or adata.obs[key_added], as with every other piaso.tl function.
The two calls that have nowhere to write still return their result: data=None
with a knn_result (in-memory), and leiden(..., cell_mask=...), whose labels
cover the masked cells only and would mis-align if written full-length.
If you have emb = piaso.tl.umap(adata) in a script, it now binds None.
TF lists: one source that works, and a file that is checked
piaso.data.fetch_tf_list(species, source=...) replaces
fetch_animaltfdb_tf_list as the entry point, defaulting to the
cisTarget/SCENIC+ TF universe (~1,890 human, ~1,860 mouse).
AnimalTFDB was removed as a source. Its only host (guolab.wchscu.cn)
sits behind a WAF that answers every non-browser client — urllib, requests
and curl alike, under any User-Agent — with a captcha page or 405, and the
old bioinfo.life.hust.edu.cn mirror no longer resolves. source="animaltfdb"
now raises and says so; fetch_animaltfdb_tf_list, which shipped in 1.2.3,
raises without making a request and will be deleted in 1.3. An AnimalTFDB
table saved from a browser still works: pass it to load_tf_list(path=...),
whose TSV parser reads its Symbol column.
Any TF-list file is now validated. The captcha page arrived as HTTP 200 and used to parse into a dozen plausible-looking “symbols” that silently shrank the TF universe. A file that starts with markup, or that yields tokens no gene symbol could be, is refused; a downloaded catalogue must also yield at least 50 symbols, and one that does not is deleted rather than cached. No minimum applies to a file you pass yourself — a ten-TF list is a legitimate thing to hand in.
For regulon inference you need neither: cytorete.tl.inferRegulon with
tf_list=None already restricts to TFs that have a motif and are expressed
in the data, which is the only set that can produce an edge.
Fixed: the source distribution was too large for PyPI
maturin sdist builds from the cargo package, which swept in docs/ — the
rendered tutorials and their images. The archive came to 407 MB against
2.1 MB of actual source, and PyPI refuses any file over 100 MiB.
The three wheels upload first and succeed, so the effect was a red publish
step at the end of every release since 1.2.0 and no sdist on PyPI for any
version. Cargo.toml now excludes the documentation and the other
non-source trees; the archive is 590 kB, installs from source, and passes
twine check.
Fixed: a dict palette set the colours and lost the order
palette={"Neuron": ..., "Glia": ...} coloured the categories correctly and
then re-sorted them alphabetically, so the sequence written at the call site
was dropped without a word. A dict states an order — Python has preserved
insertion order since 3.7 — and it now becomes the category order, ahead of a
stored categorical order, because the call site is the more specific of the
two. A partial dict leads and the rest follow alphabetically; a list palette
is unaffected, since a list cannot state an order.
plot_embeddings_split had a second half of the same bug: it resolved one
order across all panels and then did not pass it to the renderer, which
re-sorted each panel on its own.
Fixed: legend_loc="none" left the colour bar behind
For a continuous colour the colour bar is the legend, and legend_loc never
reached it — plotEmbedding drew one anyway, and plot_embeddings_split drew
one per panel. Both now honour "none". Categorical legends were already
suppressed correctly.
umap-learn must be 0.5.8 or newer
Earlier versions call scikit-learn’s check_array(force_all_finite=...), which
scikit-learn renamed in 1.6 and removed in 1.8. The combination raises a
TypeError from three frames inside scikit-learn with no mention of umap,
which reads like a PIASO bug; it is an upstream one, fixed upstream in
umap-learn 0.5.8. The requirement is now umap-learn>=0.5.8, and
piaso.tl.umap catches that specific failure and names the package to
upgrade, for environments that already have the bad pair.
pandas 3: supported, not excluded
pandas 3 turns on future.infer_string, so a string column is StringDtype
rather than object. Two checks had been asking numpy about that dtype:
np.issubdtype(StringDtype, np.number)raises rather than returning False, so the numeric-versus-categorical dispatch crashed and took every categorical embedding with it. It now asks pandas, which understands both its own extension dtypes and numpy’s. Booleans stay categorical — the one place the two libraries disagree, pandas callingboolnumeric.- A string cluster label could return from a
.cytomeas an integer. That was in cytome’s entity writer, which reachedTEXTonly throughis_object_dtypeand otherwise defaulted toINTEGER; fixed in cytome 0.3.2, which this release now requires. Nothing raised at the time, because SQLite stores what it is handed and only the read is typed — the damage appeared later, wherever a label was compared to a string or looked up in a stored category order.
There is no upper bound on pandas. An earlier build of this release capped
it at <3; that was wrong. The problem was ours, it is fixed, and the suite
passes on 3.x. A cap would have made PIASO uninstallable beside anything that
requires pandas 3, which in this ecosystem is increasingly everything.
Both behaviours are covered by tests that run on pandas 2 as well, through the
future.infer_string option that has existed since 2.1 — so this stays tested
wherever it is installed rather than only where pandas 3 happens to be.
Fixed: piaso.tl.umap(data=None, knn_result=...) crashed
The in-memory path — documented, and advertised in this release as one of the
two calls that still return their result — asked None for .obsm and died
with AttributeError. It had been invisible because the test covering it skips
wherever the installed umap-learn and scikit-learn disagree, which is most
machines. Calling it with no knn_result now explains itself instead of
raising from three frames down.
Fixed: importCellRanger reported a broken install
It imported the ATAC fragment leg unconditionally, so in the released package
every call failed with ModuleNotFoundError, including modality='rna',
which needs nothing from it. RNA-only now works; asking for fragments says
which method is unavailable and why.
Fixed: embedding dots were drawn as rings, at 3x the size requested
plotEmbedding, plot_embeddings_split and scatter never passed
linewidths to ax.scatter, so matplotlib stroked every marker at
rcParams['patch.linewidth'] — 1.0 pt, in edgecolors='face'.
On a cell-sized point that stroke is most of the dot. point_size=0.2 is a
disc 0.45 pt across, so it drew 1.45 pt wide: 3.2x the size asked for. With
alpha < 1 the stroke and the fill composite twice around the rim, so each
point rendered as an annulus with a washed-out core whose colour was neither
the palette colour nor the background. Measured on a 600 dpi render, ink peaked
two pixels off-centre.
All three scatter calls now pass linewidths=0, edgecolors='none', and both
are exposed as parameters for anyone who wants a deliberate outline. The same
fix is applied in piaso.pl.scatter and the plotByCluster jitter points. No
visible change on large markers; the workaround
plt.rc_context({"patch.linewidth": 0}) is no longer needed.
Fixed: plot_embeddings_split silently ignored extra keywords
It accepted **kwargs and never forwarded them, so styling a split panel had
no effect and no error. Unrecognised keywords now raise TypeError, as
plotEmbedding does. show is accepted as an alias for show_figure — it
was among the dropped keywords, which meant show=False displayed the figure
anyway.
Codebook motifs: about twice the TF universe, opt-in
piaso.data.fetch_codebook() and load_codebook() add the representative PWM
set of Jolma, Laverty, Fathi et al., Nature 657, 275-283 (2026) —
1,421 human TFs against JASPAR 2024 CORE vertebrates’ 754, covering all but
eleven of them. On SEA-AD middle temporal gyrus that lifts the count of TFs
with a motif and expression from 483 to 1,023. The archive is ~1 MB,
downloaded from the publisher under the article’s CC-BY-4.0 licence on first
use and cached; nothing is redistributed with PIASO.
cytorete.tl.inferRegulon(..., motif_db="codebook") uses it. JASPAR remains
the default, and that is now measured rather than assumed. Both were run
end-to-end on the same 20,000 SEA-AD nuclei:
| JASPAR 2024 | Codebook | |
|---|---|---|
| TFs with motifs in the data | 740 | 1,408 |
| regulons | 415 | 982 (412 of JASPAR’s 415 among them) |
| zinc-finger regulons | 65 (16%) | 385 (39%) |
| agreement with the TF’s own specificity | 26.7% | 12.7% |
| runtime | 13 min | 56 min |
The result is mixed rather than bad. Codebook finds more agreeing regulons in absolute terms (125 against 111) and fixes cases JASPAR gets wrong — OLIG2 now tops OPC rather than oligodendrocytes, and SOX10, absent from JASPAR, tops oligodendrocytes. But it dilutes: agreement halves, the zinc-finger share more than doubles, and neuronal factors such as RORB and LHX6 fall in rank. Reach for it when the question is glial, zinc-finger, or coverage-limited; keep JASPAR when it is a survey.
The loader repairs what the archive carries: construct names (CASZ1.FL,
ZNF729.DBD3) are filed under the gene symbol, three clone identifiers are
dropped, and five CIS-BP conversions with all-zero positions are cleaned and
renormalised.
Removed: cytorete.tl.markerMotifEnrichment
Never released, and removed after measurement rather than for tidiness. Two cutoff-free designs were tried on SEA-AD: a Fisher test on top-N markers against a GC- and length-matched background returned 0, 1 and 5 significant pairs at 50, 100 and 200 markers, with Jaccard 0.00 between the first two; a PIASOscore of each TF’s motif gene set against matched controls agreed with the TF’s own expression in 3-4% of cases.
The same agreement measure on inferRegulon’s 415 regulons, same cells and
same motifs, is 26.7%. A PWM is a family address, not a TF address, so a
motif-only statistic cannot say which cell type a TF acts in — the TF’s
expression profile is what separates SPI1 from ELF1. §10 of the motif analysis
tutorial now reports both measurements and hands
over to inferRegulon, which adds exactly that step.
cytorete.pp.build_cistrome still returns the TF × gene matrix for anyone who
wants to run their own test.
Sankey: node heights, and one size key instead of two
piaso.pl.sankey gains min_node_size (height of the shortest category block,
in points) and node_scale ('linear', 'sqrt', 'log'), the node-level
parallel of min_flow_width and flow_scale. Both are solved the same affine
way, so blocks stay monotone in group size and nothing is tied.
This fixes a distortion the default layout has always had: a node is exactly as
tall as the ribbons landing on it, so under min_flow_width its height is
m * size + b * fan_out, and two equally large categories are drawn at
different heights when one splits more ways than the other. Measured on a
two-category example at a 20 pt baseline, the four-way split draws twice as
tall as the equal one-way split.
What it costs is stated in the docstring and enforced in the key: each node’s
ribbons are renormalised to fill its height, so the count-to-thickness map
becomes per-node and min_flow_width stops being exact.
show_width_legend / width_legend_loc are now show_size_legend /
size_legend_loc, and draw one key rather than two. Only one map can be
global at a time, so only one key can be true: the key describes flows when the
flow map is non-proportional, and switches to describing nodes — suppressing
the flow key — as soon as the nodes are sized.
Confusion matrices: hierarchical ordering
order_query / order_reference accept 'hclust' (average linkage on that
axis’s own profiles) and 'hclust_symmetric' (one clustered order on both
axes, for a square table over one label set). 'auto' now prefers the
symmetric order when the two axes carry the same labels — two independent
orders moved a real diagonal off the diagonal — and is unchanged otherwise.
Fixed
-
The GDR
SettingWithCopyWarning. Each batch was scored by slicing an anndata view, whose constructor rewrites categoricalobscolumns through_remove_unused_categories— an anndata-internal pandas idiom, and pure overhead here since the scorer reads only the matrix andvar_names. The batch object is now built directly. This also removes twowarnings.catch_warningsblocks, one of them inside aThreadPoolExecutorworker, where mutating the process-global filter list races between threads. -
plotLigandReceptorLollipoptick labels were formatted to one decimal, so an axis spanning a few 10⁻⁴ — what neuronal ligand–receptor scores look like — read0.0at every tick. The precision now follows the axis range.
v1.2.3
pip install -U piaso-toolsFixed
-
piaso.tl.predictCellTypeByGDRdiscarded its own result. The function copies the query internally to avoid mutating.X, and the copy rebound the local name, so the prediction was written to the copy. It printed “All finished. The predicted cell types are saved asCellTypes_gdrin adata.obs” and the caller’s AnnData came back without that column. The cytome path was unaffected — it writes through a separate handle, which is why this survived. If you called it on an AnnData and found no prediction, this was why. -
plot_embeddings_split(vmin_pct=…, vmax_pct=…)was accepted and ignored. The parameters arrived through**kwargsand were dropped, so a request for 10/90 silently produced the full range. Nothing raised and the figure still rendered — only the contrast was wrong. Now implemented to matchplotEmbedding, with explicitvmin/vmaxstill taking precedence.
Added
-
piaso.pp.alignSpatialCoordinates. Sections are placed on their chips independently, so a split plot renders each panel at its own offset; this centres each sample on its own centroid, andwith_std=Truealso equalises apparent section size. Works on AnnData, an open cytome, or a path, and takesgroupby=orbatch_key=for the same column.This was described in the v1.2.2 notes but did not make the 1.2.2 wheel. It ships here.
-
palette=accepts a colormap name for a categorical column, sampled across the categories in their resolved order. On ordered categories — ages, timepoints, stages — that turns the legend into a ramp instead of a set of unrelated hues.
v1.2.2
pip install -U piaso-toolsSpatial
- Tissue images and regions of interest.
piaso.pl.plotEmbeddingandplot_embeddings_splittakeimage=Trueto draw a registered morphology image under a spatial embedding, with orientation and units handled: the image is placed in coordinate space, so nothing needs flipping or scaling. Needs cytome ≥ 0.2.6, which stores the image in the file. cell_mask=plots a subset — a boolean mask or integer indices — without writing a subset file. It pairs with cytome’scells_in_region, so one rectangle selects its cells and, viaspatial_images.crop, its pixels.piaso.pp.rotateSpatialCoordinatesworks directly on a cytome, not only on an AnnData, and rebuilds the coordinate index in the same call so region queries keep agreeing with what you plot.
Plotting
- Continuous colour now depends on where the value came from. A feature
keeps PIASO’s sequential
color_1; a numeric cell column — regulon activity, a QC metric, pseudotime — getsSpectral_r. An explicitcmap=still wins. Both are continuous, but drawing metadata on the expression ramp made a metadata panel read as expression. - Fixed:
plot_embeddings_split(fix_coordinate_ratio=True)is documented as the default but only theFalsebranch acted, so panels were stretched to their subplot box and spatial tissue rendered squashed. - Fixed:
basis='spatial'resolved to the last matching embedding, so writing an aligned or backup copy beside it silently redirected existing calls. Candidates are ranked now — the name that is the key beats one that merely contains it.
v1.2.1
pip install -U piaso-toolspiaso.tl.cospecificity_transis public; it is the trans co-specificity used by cytorete to decide regulon edges.- The GRN entry points that used to live in
piaso.tlremain as lazy forwarders to cytorete, raising an actionableImportErrorwhen it is not installed. - Motif scanning (
piaso.pp.scan_motifs), the PWM loaders and the SCREEN cCRE registry are part of the public API.
v1.2.0
pip install -U piaso-toolscytome datasets and streaming
PIASO works directly on cytome files.
Anywhere a function took an AnnData, it also takes a path to a .cytome file
or an open dataset:
import piaso, cytome
ds = cytome.open("atlas.cytome")piaso.tl.runGDR(ds, groupby="cell_type", layer="infog")The matrix is read in chunks rather than loaded, so peak memory is set by the batch size instead of by the number of cells. cytome is installed automatically — there is no extra to remember.
A self-contained analysis workflow
From raw UMI counts to clusters, an embedding and marker genes:
import piaso, cosg
# Input must be RAW UMI counts. This reads adata.X by default; if your raw# counts live in a layer, pass it explicitly:# piaso.tl.infog(adata, layer="counts", n_top_genes=3000)piaso.tl.infog(adata, n_top_genes=3000)
# Pass layer="infog" — runSVD defaults to adata.X, which would silently# ignore the normalization above.piaso.tl.runSVD(adata, layer="infog", n_components=50, key_added="X_svd")
piaso.tl.neighbors(adata, use_rep="X_svd", n_neighbors=15)piaso.tl.leiden(adata, resolution=1.0, key_added="leiden")piaso.tl.umap(adata, use_rep="X_svd")
cosg.cosg(adata, groupby="leiden", key_added="cosg")piaso.pl.embedding(adata, basis="X_umap", color="leiden")Every step runs on a plain pip install piaso-tools — no scanpy required.
scanpy remains an optional extra (pip install 'piaso-tools[scanpy]') for
interoperability.
One key to watch when moving older code over: piaso.tl.leiden writes its
labels to adata.obs['leiden'] by default, whereas the scanpy-based tutorials
that preceded this release passed key_added='Leiden' explicitly. Downstream
calls that name the column need the same spelling.
Motif scanning
piaso.pp.scan_motifs is a pure-numpy PWM scanner; scan_motifs_rust is a
rayon-parallel implementation with the same contract. The motif-database loaders
and .2bit sequence access live alongside it in piaso.data, so the whole
workflow — fetch a genome, extract sequences, load PWMs, scan — is available
from one install.
Reference data
piaso.data fetches and caches what analyses need: genome sequence and
annotation, example datasets, motif databases (JASPAR, CIS-BP, cisTarget), and
the SCREEN cCRE registry.
Requirements
Requires cosg>=1.1.0. pip install -U piaso-tools pulls it in.
Security
pyo3 upgraded to 0.29.2, closing GHSA-36hh-v3qg-5jq4 and GHSA-chgr-c6px-7xpp. PIASO called neither affected API, so exposure was nil, but the crate linked the code.
v1.1.0
Marker-gene-guided dimensionality reduction (GDR), INFOG normalization, gene-set scoring, cell-type annotation and label transfer, and the plotting suite. See the GitHub releases for the full history.