Multiple samples in one cytome

The multi-sample tutorial concatenates AnnData objects in memory. This does the same analysis with the samples merged into one cytome on disk, which is the version that keeps working when the samples stop fitting in RAM.

Nothing about the method changes. What changes is where the matrix lives.

1. One cytome per sample

load_dataset writes a cytome directly — the registry entries are 10x .h5 files and cytome reads that format, so there is no AnnData in this step:

import cytome, piaso

parts = []
for name in ("pbmc_multiome_san1", "pbmc_multiome_san2"):
    piaso.data.load_dataset(name, return_type="cytome",
                            cytome_path=f"{name}.cytome").close()
    parts.append(f"{name}.cytome")

Converting per sample rather than concatenating first is the point: each file is read, written and closed one at a time, so peak memory is one sample rather than all of them.

One call per file, for now. cytome.from_10x_h5 takes a single path and cytome.merge takes cytomes, so importing several .h5 files in one streaming pass is not available yet. It would be a convenience rather than a memory win — the two-step route above is already bounded, because neither step holds more than one sample.

2. Merge

ds = cytome.merge(parts, output="pbmc_merged.cytome",
                  batch_key="sample", batch_labels=["SAN1", "SAN2"],
                  gene_strategy="inner")

ds.n_cells, ds.n_genes          # 7905, 36601

Close what merge returns. It hands back an open CytomeDataset, and a second connection to the same file then hits sqlite3.OperationalError: database is locked on the first write — which surfaces several steps later, inside infog, pointing at nothing useful.

batch_labels writes the sample name into ds.cells["sample"], so provenance travels with the cells instead of living in a variable you have to remember. gene_strategy="inner" keeps genes present in both; "union" keeps everything and zero-fills, which is right when the panels genuinely differ and wrong when one sample simply failed to detect a gene.

Barcodes collide across libraries — the same sequence means different cells — and merge disambiguates them, which is the same problem index_unique solves on the AnnData side.

3. Normalise, embed, cluster

Everything below takes the open dataset. There is no reopening and no flushing by hand: infog persists its own writes, and runGDR accepts a cytome object as readily as a path.

piaso.tl.infog(ds, modality="RNA", layer="counts",
               key_added="infog", save_layer=True, inplace=True)

piaso.tl.runGDR(ds, batch_key="sample", groupby=None,
                layer="infog", score_layer="infog", max_workers=8)

piaso.tl.neighbors(ds, use_rep="X_gdr", n_neighbors=15)
piaso.tl.umap(ds, key_added="X_umap")
piaso.tl.leiden(ds, resolution=1.0, key_added="leiden")

batch_key="sample" is the whole difference from a single-sample run: GDR finds marker genes within each library and scores every cell against all of them, so an axis means the same thing in both samples. groupby=None because these libraries have no annotation yet — GDR clusters within each batch and uses what it finds.

step	time
convert both samples	22 s
merge	16 s
INFOG	4 s
`runGDR` (`batch_key="sample"`)	32 s
neighbours + UMAP + Leiden	14 s

4. Read it the same way

piaso.pl.plotEmbedding(ds, color="sample", basis="X_umap")
piaso.pl.plotEmbedding(ds, color="leiden", basis="X_umap", legend_loc=None)

Two libraries merged, coloured by sample and by cluster

Both panels, again, because either alone is unreadable as evidence. By sample, SAN1 and SAN2 interleave through every population rather than forming two islands. By cluster, 16 clusters resolve on the same points. Batch mixed, biology separated — the same check as every other integration on this site.

Which route to use

	AnnData	cytome (this page)
samples in memory	all, concatenated	one at a time, then none
provenance	`obs["sample"]` you set	`batch_labels` written at merge
barcode collisions	`index_unique="-"`	handled by `merge`
scales to	what fits in RAM	what fits on disk

At two PBMC libraries the AnnData route is simpler and there is no reason to avoid it. The cytome route earns its keep at the point where concatenating would not finish — and the analysis code above is unchanged from that point on, which is the argument for learning it before you need it.