bkbit.data_translators.ait_taxonomy_translator module

AIT .h5ad -> Cell Taxonomy JSON-LD, driven by the model’s own annotations.

The point of this script is that it does not hardcode where fields live. Every slot in the cell_taxonomy model carries an ait_location annotation saying where its value comes from in an AIT .h5ad. linkml’s pydantic generator keeps those annotations, so they are read straight off bkbit.models.cell_taxonomy – no LinkML YAML needs to ship alongside the generated model.

Where a declared location cannot be honoured, the script may fall back to another source – but every such deviation is recorded in the report, never applied silently. That is the whole contract:

read from where the model says; if you cannot, say so.

Location grammar (as written in the sheet’s AIT Location column):

obs.<field> a column of obs obs.index the obs index obs.<annotation_level> template: one column per uns/hierarchy level uns.<key> a uns scalar uns.hierarchy (key|value) hierarchy keys (level names) or values (order) uns.cluster_info the whole cluster_info table uns.cluster_info.<pre>_<level> per-level wide column, needs un-pivoting obsm.X_<embedding> (key|value) an obsm embedding X / raw.X expression matrix groups (derived: <rule>) not stored; computed per <rule> (synthesized) minted identifier, no file source <path> (data only) present in files but NOT in the AIT schema

Reads only metadata groups over HTTP range requests; never materialises X.

curated_markers has range GeneAnnotation, so its values must be links. The file stores symbols; –gene_annotation (one gff2jsonld output, Turtle or JSON-LD, of the taxonomy’s species – checked) supplies the GeneAnnotation IRIs. Symbols that do not resolve are reported and left out, never written as strings – a bare string in an @id slot becomes a relative IRI against the output file’s path.

Usage:

python ait2jsonld.py URL_OR_PATH [–out out.jsonld] [–report rep.json]

[–species NAME] [–no-counts] [–gene_annotation ANNOTATION.ttl|.jsonld]

bkbit.data_translators.ait_taxonomy_translator.GENE_ANNOTATION_CLASS = 'https://identifiers.org/brain-bican/vocab/GeneAnnotation'

CellTypeTaxon.curated_markers has range GeneAnnotation, so the generated context types it "@type": "@id". The AIT file stores gene symbols (“PDGFRA, DCN”). Written as they are, a JSON-LD processor reads each symbol as a relative IRI and resolves it against the document’s base – the output file’s own path – so the graph ends up with file:///.../ACTA2: a node nothing else refers to. The value the model asks for is the IRI of a GeneAnnotation, which gff2jsonld mints. So symbols are looked up in the ONE gff2jsonld output the caller names for this h5ad, and a symbol that cannot be resolved is reported, never written as a string.

class bkbit.data_translators.ait_taxonomy_translator.GeneIndex(taxon: str, source: str = '', label: str = '', by_symbol: dict = <factory>, by_folded: dict = <factory>, by_synonym: dict = <factory>)

Bases: object

Symbol lookup over the GeneAnnotations of one genome annotation file.

by_folded: dict

by_symbol: dict

by_synonym: dict

label: str = ''

resolve(symbol: str) → tuple[str | None, str]

(GeneAnnotation IRI, tier) or (None, why).

Tiers, strongest first: symbol (exact), symbol_case (the same symbol in another case, e.g. mouse-style “Pou6f2” for POU6F2), synonym (exact). Each tier must name exactly one gene; an ambiguous tier is reported, never broken by picking one.

source: str = ''

taxon: str

bkbit.data_translators.ait_taxonomy_translator.KNOWN_DATABASE_PREFIXES = frozenset({'ENSEMBL', 'NCBIGene'})

Prefixes known to be database identifiers, so correctly left in xref. Anything in neither set is reported rather than silently assumed to be one.

bkbit.data_translators.ait_taxonomy_translator.ONTOLOGY_PREDICATE = 'broad_match'

Which SKOS predicate the ontology terms are asserted with.

broad_match, not exact_match: the mapping is many-to-one. Measured across the HMBA files, up to three distinct CellTypeTaxon entries share one CL term (CL:0000128, CL:4042025, CL:0000192), and abbreviations reuse terms too. skos:exactMatch is symmetric and transitive, so asserting it three times against one term would entail that those three taxa are equivalent to each other, which is false – they are distinct taxa sharing a coarser class. skos:broadMatch says the true thing: the external concept is broader than ours.

bkbit.data_translators.ait_taxonomy_translator.ONTOLOGY_PREFIXES = frozenset({'CL', 'DHBA', 'UBERON'})

Prefixes whose terms are concepts in a published ontology, as opposed to records in a database.

The distinction decides which slot a value lands in. Biolink defines xref as a cross reference to a database identifier, and it serialises as a literal, so a CL or UBERON term sitting there joins with nothing. The SKOS mapping slots have a class range, so the generated context gives them "@type": "@id" and the CURIE becomes a real node that resolves against the published ontology. Database identifiers stay in xref, which is what it is for.

Keyed on the prefix rather than the table’s abbreviation_type column, because the two can disagree – the prefix is what decides whether a value is resolvable, and a row typed gene could still carry an ontology term. An unrecognised prefix is left in xref and reported, so a new ontology shows up as a deviation rather than silently becoming an unjoinable literal.

class bkbit.data_translators.ait_taxonomy_translator.Report(source_file: str = '', species: str = '', resolved: list = <factory>, deviations: list = <factory>, unpopulated: list = <factory>, counts: dict = <factory>)

Bases: object

Records, per slot, whether the model’s declared location was honoured.

counts: dict

deviate(slot, declared, actual, why)

deviations: list

empty(slot, declared, why)

ok(slot, loc, note='')

resolved: list

source_file: str = ''

species: str = ''

to_dict()

unpopulated: list

bkbit.data_translators.ait_taxonomy_translator.accepts(cls_name, slot, value, rep, loc, seen) → bool

True if the model accepts this value for this slot.

A rejected value is dropped and reported once per distinct value, never raised: a tracking placeholder in one column should not fail a whole taxonomy. Deliberate – see the ASK NELSON note on CL:ID_<level>.

bkbit.data_translators.ait_taxonomy_translator.blankish(v)

None for a cell that is empty or a stringified null, else its text.

bkbit.data_translators.ait_taxonomy_translator.build(path: str, species: str = '', want_counts: bool = False, cache_dir: str | None = None, abbreviations: str | None = None, gene_annotation: str | None = None)

bkbit.data_translators.ait_taxonomy_translator.dec(v)

bkbit.data_translators.ait_taxonomy_translator.df_columns(group) → list[str]

bkbit.data_translators.ait_taxonomy_translator.gene_annotation_format(path) → str

‘json-ld’ or ‘turtle’, from the content, checked against the extension.

The content decides (a JSON document starts with ‘{’ or ‘[‘); a known extension that disagrees with it is an error rather than a guess, since a mislabelled file is usually the wrong file.

bkbit.data_translators.ait_taxonomy_translator.guard_links(ents, rep, context_path=None) → int

Drop values that would become relative IRIs, and report each one.

The model is the authority on which slots hold links (the generated context marks them "@type": "@id"); this applies it to every class and slot, so a new link slot fed a plain string fails loudly here rather than quietly becoming a file:/// node. Returns the number dropped.

bkbit.data_translators.ait_taxonomy_translator.is_link(value, prefixes) → bool

True for an absolute IRI or a CURIE whose prefix the context defines.

Anything else in an @id-typed slot is a relative IRI reference, which a JSON-LD processor silently resolves against the document base.

bkbit.data_translators.ait_taxonomy_translator.load_abbreviations(spec: str) → tuple[dict, list]

{term: {meaning, entity_type, xref}} from an abbreviation table.

spec is a path, or the name of a table bundled with bkbit (see BUNDLED_ABBREVIATIONS). The table is taxonomy-specific – the HMBA basal-ganglia one does not describe the tokens in other taxonomies – so there is no default: nothing is enriched unless a table is named.

bkbit.data_translators.ait_taxonomy_translator.load_gene_annotation(path) → GeneIndex

GeneIndex over one gff2jsonld output, in Turtle or JSON-LD.

The file’s species is the NCBITaxon:<n> xref of the OrganismTaxon its genes are in_taxon of – the same number the AIT file stores in organism_ontology_term_id, which build() checks it against. A file whose genes span several species is rejected: one h5ad, one annotation.

bkbit.data_translators.ait_taxonomy_translator.load_locations() → dict

{(class, slot): {‘loc’:…, ‘sep’:…}} from the model’s annotations.

These come from the ait_location and ait_separator annotations on the cell_taxonomy LinkML schema. linkml’s pydantic generator preserves them verbatim under each field’s json_schema_extra["linkml_meta"], so the generated model is a complete source and the LinkML YAML does not have to ship alongside it. Verified equal to reading the schema with SchemaView: the same (class, slot) pairs with identical values.

The sheet also has Notes and Source columns, but both are deliberately mapped to ignore and so never reach the model; this used to read them and no longer does.

bkbit.data_translators.ait_taxonomy_translator.multivalued_map() → dict

{(class, slot): bool} – cardinality per class, not per slot name.

Keying by name alone is wrong: Cluster.has_parent is multivalued while CellTypeTaxon.has_parent is not, so a global set wraps the latter in a list and the model rejects it. Read straight off the generated field types, which encode exactly this.

bkbit.data_translators.ait_taxonomy_translator.ncbi_taxon_number(value) → str | None

‘9606’ from NCBITaxon:9606, NCBITaxon_9606 (IRI) or NCBI:txid9606.

bkbit.data_translators.ait_taxonomy_translator.obs_counts(h5, column)

Per-category row counts for an obs column, without loading all of obs.

bkbit.data_translators.ait_taxonomy_translator.oid(cls_name: str, attrs: dict) → str

Deterministic id, identical to every other bkbit translator.

Delegates to bkbit.utils.generate_bkbit_id.generate_object_id(), so the same entity minted here and by taxonomy2jsonld gets the same IRI. cls_name is accepted for call-site readability but is deliberately NOT hashed: including it would fork this translator’s ids away from the rest of bkbit. Checked against all 13 released AIT files – hashing attributes alone produces no cross-class collisions.

bkbit.data_translators.ait_taxonomy_translator.open_h5(path: str, cache_dir: str | None = None)

Open local or remote. Remote uses a local block cache so that repeat runs do not re-fetch the same HDF5 metadata blocks (the dominant cost: listing uns alone is ~26s cold).

bkbit.data_translators.ait_taxonomy_translator.parse_loc(loc: str) → dict

Parse a declared location into something resolvable.

bkbit.data_translators.ait_taxonomy_translator.read_col(group, name)

Decode ONE dataframe column (categorical or plain). Reading only the columns the model asks for avoids pulling all ~206 datasets of a 103-column table, which is what made the first version unusably slow.

bkbit.data_translators.ait_taxonomy_translator.read_uns_scalar(h5, key)

bkbit.data_translators.ait_taxonomy_translator.resolve_markers(symbols, gene_index, tally) → list[str]

GeneAnnotation IRIs for the symbols that resolve; the rest go to tally.

With no gene index nothing is emitted: a symbol is not a valid value of a GeneAnnotation-ranged slot. A cell that is not a symbol list at all (e.g. “Six3 and Foxp1”) is not re-split – the declared separator is the only one – so it is reported unresolved for the data to be fixed upstream.

bkbit.data_translators.ait_taxonomy_translator.source_as_uri(path: str) → str | None

The location the file was actually read from, as a URI.

An https URL pointing at S3 is rewritten to its s3:// form, so a taxonomy read over range requests records the same identifier the bucket owner would quote. Returns None for a local path – a filesystem location is not a permanent URL and must not be published as one.

bkbit.data_translators.ait_taxonomy_translator.split_values(value, sep)

Split one stored cell into the values of a multivalued slot.

sep comes from the model’s ait_separator annotation. With no separator declared the cell is a single value – guessing a delimiter would silently invent or merge entities.

bkbit.data_translators.ait_taxonomy_translator.to_jsonld(ents, context_path=None)

bkbit.data_translators.ait_taxonomy_translator.value_validators(cls_name: str) → dict

{slot: callable} for slots the model constrains, e.g. with a pattern.

linkml’s pydantic generator compiles a pattern into a @field_validator rather than leaving it on the field, so it cannot be read off FieldInfo or the JSON schema. Pulling the validators out of __pydantic_decorators__ and calling them keeps the model the single authority – no regex is restated here.