projectGDR: putting new data into a reference's space

runGDR builds an embedding from the data it is given. If a new sample arrives next week, re-running it moves every cell — the axes are re-derived, so last week’s coordinates and this week’s are not comparable, and any label or gate defined on the old embedding is invalid.

projectGDR is the other mode: the reference’s space is frozen, and query cells are placed into it. The reference does not move.

import numpy as np
import piaso

piaso.settings.set_figure_params(style="cell")

1. Build a reference that remembers itself

The one thing that matters is save_reference=True:

ref = piaso.data.load_dataset("adult_cortex_multiome_rna")
piaso.tl.infog(ref, layer="raw", n_top_genes=3000)

piaso.tl.runGDR(ref, batch_key="Sample", groupby=None,
                layer="infog", infog_layer="raw", score_layer="infog",
                n_gene=30, key_added="X_gdr",
                save_reference=True)          # <- without this there is nothing to project into

"gdr_reference" in ref.uns

True

That entry holds what defines the space: the marker gene sets per group, and the column norms used to scale their scores. It is small — gene names and a vector — so a reference is cheap to keep and cheap to ship.

2. Project

query = ...        # any AnnData with overlapping var_names, or a .cytome path

piaso.tl.projectGDR(query, reference=ref, key_added="X_gdr_proj")
query.obsm["X_gdr_proj"].shape

(8738, 84)

3.3 s for 8,738 cells. The dimensionality — 84 here — is the reference’s, not something the query negotiates: it is the total number of marker gene sets the reference’s batches produced. That is the point. Two queries projected into the same reference land in the same 84 coordinates and can be compared directly.

Cytome queries are streamed, so the query’s expression matrix is never materialised:

piaso.tl.projectGDR("new_sample.cytome", reference=ref)

3. How close is projected to joint?

Here the query cells were also part of the reference, so both embeddings exist for the same cells and can be compared — which is the only way to calibrate what projection costs.

joint, proj = query.obsm["X_gdr"], query.obsm["X_gdr_proj"]
cos = (np.sum(joint * proj, axis=1) /
       (np.linalg.norm(joint, axis=1) * np.linalg.norm(proj, axis=1)))
np.median(cos), np.median(np.linalg.norm(joint - proj, axis=1))

(0.9739, 0.511)

Median cosine 0.974 between the jointly-computed embedding and the projected one; the median displacement is 0.511 against a median embedding norm of 2.24, so cells move about a fifth of their own length.

Read that honestly in both directions. Projection is not identical to joint computation — if you need the exact joint geometry, compute it jointly. But 0.974 means the projected cells sit in the right neighbourhood, which is what label transfer and gating need.

piaso.tl.neighbors(query, use_rep="X_gdr_proj", n_neighbors=15)
piaso.tl.umap(query, use_rep="X_gdr_proj")
piaso.pl.embedding(query, basis="X_umap", color=["CellTypes", "Sample"])

4. mode="reference" versus mode="self"

piaso.tl.projectGDR(query, reference=ref, mode="reference")   # default
piaso.tl.projectGDR(query, reference=ref, mode="self")

The modes differ in whose column norms scale the query’s scores.

• "reference" uses the reference’s. The query lands in genuinely the same coordinates, so it is comparable to anything else projected there. This is what you want for label transfer, and it is the default.
• "self" uses the query’s own — treating it as a new batch, which is what runGDR does internally for each reference batch when batch_key is set. It can absorb a systematic shift between reference and query, at the cost of the coordinates no longer meaning exactly the same thing.

mode="self" falls back to "reference" below min_cells_self_mode (500) cells, because self-scaling on a small query is measurably worse — estimating norms from few cells is noisy, and the noise goes straight into the axes.

5. What projection is for

• A reference atlas and incoming samples. Project each new sample; the atlas coordinates never move, so figures and gates stay valid.
• Label transfer. Project, then run predictCellTypeByMarker with use_rep="X_gdr_proj".
• Datasets too large to embed together. The reference can be a subsample; the rest is projected in batches.
• Novelty. novelty_k and novelty_quantile flag query cells that sit far from any reference cell — a cell type the reference does not contain will be placed somewhere, and it is worth knowing which cells those are. Projection cannot invent an axis for biology the reference never saw.

That last point is the real limit. The space is spanned by the reference’s marker sets. A query population with no counterpart in the reference gets coordinates, but they describe how it resembles reference types, not what it actually is. If you expect new biology, embed jointly and look, then decide what the reference should be.

Parameters worth knowing

parameter	what it changes
mode	'reference' (default, comparable coordinates) or 'self' (new-batch scaling).
key_added	where the embedding lands. Defaults to X_gdr — set it if you want to keep a joint embedding alongside.
batch_size	streaming chunk for cytome queries; sets peak memory.
novelty_k, novelty_quantile	how far from the reference a query cell has to be to be flagged.
min_cells_self_mode	below this, 'self' falls back to 'reference'.