UMAP at scale: the same coordinates on any number of threads
piaso.tl.umap embeds the neighbour graph with PIASO’s parallel deterministic Rust UMAP engine (backend="rust", the default since v1.2.7). It optimizes the same fuzzy simplicial set cross-entropy objective as McInnes, Healy and Melville (2018), starting from the same spectral initialization and using the same (a, b) curve parameters fitted from min_dist and spread.
Two things are different from umap-learn:
- You do not have to choose between a seed and multiple cores. In
umap-learn, passing an integerrandom_statedisables multi-threaded SGD and forces execution onto a single thread, because its parallel mode relies on unsynchronized Hogwild! updates to shared vertex coordinates. - The coordinates do not depend on
n_threads. One graph and onerandom_stateproduce bit-for-bit identicalfloat32coordinates on 1 thread, 4 threads, or 20 threads.
This page embeds the 82,031-cell Parse Evercode v4 human PBMC atlas (1,912,562 kNN edges) from the Parse Evercode v4 tutorial, compares the embedding beside umap-learn’s, and measures runtime, thread scaling, and local neighbourhood preservation on a 10-core workstation (20 threads).
Embed
import anndata as adimport numpy as npimport piaso
adata = ad.read_h5ad("parse_evercode_pbmc_82k.h5ad")
piaso.tl.umap( adata, neighbors_key="neighbors_gdr", min_dist=0.5, spread=1.0, random_state=42, key_added="X_umap",)The call is identical on an AnnData or a .cytome path. For this 82,031-cell graph (1,912,562 edges), piaso.tl.umap finishes in 7.30 s on 20 threads.
The number of threads changes the time, not the coordinates
import hashlib
coords = {}for t in (1, 2, 4, 10, 20): piaso.tl.umap( adata, neighbors_key="neighbors_gdr", random_state=42, n_threads=t, key_added=f"X_umap_t{t}", ) coords[t] = adata.obsm[f"X_umap_t{t}"].copy()
all(np.array_equal(coords[1], coords[t]) for t in coords)True{t: hashlib.sha256(coords[t].tobytes()).hexdigest()[:16] for t in coords}{1: '2540b5a4b31254ff', 2: '2540b5a4b31254ff', 4: '2540b5a4b31254ff', 10: '2540b5a4b31254ff', 20: '2540b5a4b31254ff'}Threads (n_threads) | 1 | 2 | 4 | 10 | 20 |
|---|---|---|---|---|---|
Time (piaso.tl.umap) | 57.42 s | 35.95 s | 17.05 s | 8.17 s | 7.30 s |
| **`max | X_t - X_1 | `** | 0.0 | 0.0 | 0.0 |
| SHA-256 prefix | 2540b5a4b31254ff | 2540b5a4b31254ff | 2540b5a4b31254ff | 2540b5a4b31254ff | 2540b5a4b31254ff |
Why parallel SGD is usually non-deterministic — and why this one is not
In standard parallel UMAP (umap-learn with random_state=None), worker threads pull edges from a shared queue and update the 2D coordinates of the endpoints head[e] and tail[e] in place without locks. Whenever two threads process edges incident to the same cell at the same time, their reads and writes interleave in an order decided by OS thread scheduling. Even with a fixed initial layout, two runs on the same machine drift apart (max |Δ| = 11.41 coordinate units on this dataset).
PIASO’s Rust UMAP engine eliminates both sources of thread dependence:
- Conflict-free edge coloring. Before SGD begins, the directed kNN edge list is partitioned into independent color classes (batches) such that no two edges in the same batch share an endpoint (
headortail) or a negative-sampling target. Within each batch, Rayon threads update disjoint sets of vertices in parallel without atomic contention or race conditions; a lightweight barrier separates batches. - Per-edge counter-based PRNG. Instead of drawing negative samples from a thread-local stateful RNG (whose sequence depends on which thread claims which edge), each scheduled edge event derives its negative samples from a stateless
SplitMix64hash of(seed, epoch, edge_index). Edgeeat epochttherefore draws the exact same negative vertices whether the job runs on 1 core on a laptop or 20 threads on a cluster node.
What it looks like, beside umap-learn
To run umap-learn through the same interface, pass backend="umap-learn":
piaso.tl.umap( adata, neighbors_key="neighbors_gdr", random_state=42, backend="umap-learn", key_added="X_umap_umaplearn",)import matplotlib.pyplot as plt
piaso.settings.set_figure_params(style="cell")fig, axes = plt.subplots(1, 3, figsize=(18, 5.5))for ax, basis, title in zip( axes, ("X_umap_t20", "X_umap_t1", "X_umap_umaplearn"), ( "piaso.tl.umap (20 threads, 7.3 s)", "piaso.tl.umap (1 thread, 57.4 s, max |Δ| = 0.0)", "umap-learn (seeded, 1 thread, 70.3 s)", ),): piaso.pl.embedding( adata, basis=basis, color="cell_type", ax=ax, show=False, legend_loc="none", frameon=False, ) ax.set_title(title)
Same neighbourhood quality, 9.6× faster than seeded umap-learn
We evaluate embedding fidelity on the 82,031-cell PBMC dataset using three standard metrics:
- Trustworthiness (
k = 15) — fraction of 2D nearest neighbours that are genuine neighbours in the 30-dimensional GDR space ([0, 1], higher is better). - kNN recall (
k = 15) — exact overlap of the 15-nearest-neighbour sets between high-dimensional space and the 2D UMAP ([0, 1], higher is better). - Cell-type silhouette — separation of the 27 annotated immune populations in 2D UMAP space.
| Method | Wall time | Speedup | Deterministic across runs? | Thread-invariant? | Trustworthiness (k = 15) | kNN recall (k = 15) | Cell-type silhouette |
|---|---|---|---|---|---|---|---|
piaso.tl.umap (backend="rust", 20 threads) | 7.30 s | 9.63× | Yes (Δ = 0.0) | Yes (1–20 threads) | 0.9754 (5-seed median 0.9763) | 0.1852 | 0.2713 |
piaso.tl.umap (backend="rust", 1 thread) | 57.42 s | 1.22× | Yes (Δ = 0.0) | Yes | 0.9754 | 0.1852 | 0.2713 |
umap-learn (random_state=42, forced 1 thread) | 70.31 s | 1.00× | Yes (1 thread only) | N/A (single-threaded) | 0.9758 | 0.1861 | 0.2676 |
umap-learn (random_state=None, 20 threads) | 37.54 s | 1.87× | **No (`max | Δ | = 11.41`)** | No | 0.9756 |
piaso.tl.umap matches umap-learn’s local trustworthiness (0.9754 vs 0.9758, and 0.9763 median across 5 seeds) and slightly improves cell-type silhouette (0.2713 vs 0.2676), while running 9.63× faster than reproducible (random_state=42) umap-learn and 5.14× faster than unseeded multi-threaded umap-learn.
Reproducing an embedding from PIASO 1.2.6 or earlier
Before v1.2.7 piaso.tl.umap delegated to umap-learn. That backend remains available:
piaso.tl.umap(adata, backend="umap-learn", random_state=42)Good to know
n_threads=None(or0), the default, uses all CPU cores available to the process. Because the coordinates do not depend onn_threads, changing core counts between interactive exploration and batch pipelines never alters your UMAP figures.n_components=2andn_components=3use dedicated stack-allocated 2D and 3D SIMD SGD kernels; arbitrary dimensions ($D > 3$) are also supported.init_pos="spectral"computes the normalized graph Laplacian eigenvector initialization directly in Rust (Lanczos eigensolver with connected-component anchoring) before running the conflict-free SGD epochs. You can also pass an existingobsmkey or an(n_cells, n_components)NumPy array toinit_posto warm-start from custom coordinates.