annotations.precomputed#

Export annotations in neuroglancer’s precomputed annotations format. The single entry point is write_precomputed_annotations(), which supports five annotation types: 'point', 'line', 'axis_aligned_bounding_box', 'ellipsoid', and 'polyline'.

Geometry columns#

Annotations can live in a coordinate space of any dimensionality (not just 3D); the column names are derived from coord_space.names. For example, with coord_space.names == ['x', 'y', 'z'], the input table must have the following geometry columns (plus any property / relationship columns):

'point': x, y, z
'line' and 'axis_aligned_bounding_box': xa, ya, za, xb, yb, zb
'ellipsoid': x, y, z, rx, ry, rz
'polyline': no geometry columns; vertices are supplied separately via polyline_points

For a pandas DataFrame input the index supplies the annotation ID; for a Feather file input the ID is resolved from (in order) an explicit annotation_id column, a pandas-index column carried in the file’s schema metadata, or finally synthesized as 0, 1, 2, ... if neither is available – so a file written via df.to_feather(path) just works. See Streaming from a Feather file below for the full resolution order. In most use-cases the annotation ID is not user-visible, so the values need not be carefully chosen; any unique uint64-compatible values will do (e.g. range(len(df))).

Examples#

Point annotations#

import pandas as pd
from ngsidekick.annotations.precomputed import write_precomputed_annotations

df = pd.DataFrame({
    'x': [10.0, 20.0, 30.0],
    'y': [10.0, 20.0, 30.0],
    'z': [10.0, 20.0, 30.0],
})

# df:
#       x     y     z
#    0  10.0  10.0  10.0
#    1  20.0  20.0  20.0
#    2  30.0  30.0  30.0

write_precomputed_annotations(df, 'xyz', 'point', output_dir='out/points')

Line annotations#

df = pd.DataFrame({
    'xa': [0.0, 10.0], 'ya': [0.0, 0.0], 'za': [0.0, 0.0],
    'xb': [5.0, 15.0], 'yb': [5.0, 5.0], 'zb': [0.0, 0.0],
})

# df:
#         xa   ya   za    xb   yb   zb
#    0   0.0  0.0  0.0   5.0  5.0  0.0
#    1  10.0  0.0  0.0  15.0  5.0  0.0

write_precomputed_annotations(df, 'xyz', 'line', output_dir='out/lines')

Bounding boxes use the same column convention as lines, with annotation_type='axis_aligned_bounding_box'.

Ellipsoid annotations#

df = pd.DataFrame({
    'x':  [10.0, 20.0], 'y':  [10.0, 20.0], 'z':  [10.0, 20.0],
    'rx': [ 2.0,  3.0], 'ry': [ 2.0,  3.0], 'rz': [ 2.0,  3.0],
})

# df:
#          x     y     z   rx   ry   rz
#    0  10.0  10.0  10.0  2.0  2.0  2.0
#    1  20.0  20.0  20.0  3.0  3.0  3.0

write_precomputed_annotations(df, 'xyz', 'ellipsoid', output_dir='out/ellipsoids')

Polyline annotations#

Polylines have a variable number of vertices, so vertex coordinates are passed in a separate auxiliary DataFrame supplied via the polyline_points argument: one row per vertex, with coordinate columns plus an 'annotation_id' column linking each vertex back to its polyline. Vertex order within an annotation defines the polyline’s traversal order.

The main DataFrame carries any per-annotation properties or relationships; its index supplies the annotation IDs referenced by polyline_points['annotation_id']. In the example below, two polylines each get a single mycolor rgb color property; the main DataFrame’s default RangeIndex [0, 1] matches the annotation_id values in the points table.

main_df = pd.DataFrame({
    'mycolor_r': [255,   0],
    'mycolor_g': [128, 200],
    'mycolor_b': [  0, 255],
})

# main_df:
#       mycolor_r  mycolor_g  mycolor_b
#    0        255        128          0
#    1          0        200        255

polyline_points = pd.DataFrame({
    'x':             [0.0, 1.0, 2.0,    5.0, 5.0],
    'y':             [0.0, 0.5, 1.0,    5.0, 6.0],
    'z':             [0.0, 0.0, 0.0,    0.0, 0.0],
    'annotation_id': [   0,   0,   0,     1,   1],
})

# polyline_points:
#         x    y    z  annotation_id
#    0  0.0  0.0  0.0              0
#    1  1.0  0.5  0.0              0
#    2  2.0  1.0  0.0              0
#    3  5.0  5.0  0.0              1
#    4  5.0  6.0  0.0              1

write_precomputed_annotations(
    main_df, 'xyz', 'polyline',
    properties=['mycolor'],
    polyline_points=polyline_points,
    output_dir='out/polylines',
)

See Properties and relationships below for the full set of supported property and relationship column conventions, which apply identically to polyline annotations.

If your polylines have no properties or relationships, you can omit the main DataFrame entirely by passing None as the first argument; the main table is synthesized from the unique annotation IDs in polyline_points:

write_precomputed_annotations(
    None, 'xyz', 'polyline',
    polyline_points=polyline_points,
    output_dir='out/polylines',
)

Properties and relationships#

In addition to geometry columns, the main DataFrame can carry annotation properties (per-annotation attributes like color or a confidence score) and relationships (per-annotation lists of related segment IDs that neuroglancer can use to filter annotations by segment).

Numeric properties are plain numeric columns. The column dtype determines the encoded type (uint8, int8, …, float32).
Enum properties are pandas categorical columns. Each category becomes a discrete enum value with the category label shown in the neuroglancer UI.
Color properties (rgb or rgba) are split across one column per channel: <name>_r, <name>_g, <name>_b (and optionally <name>_a). List the base name in properties; the suffixed columns are picked up automatically.
Relationships are columns whose values are lists of related segment IDs (uint64). As a shortcut, if every annotation has exactly one related segment, the column may have dtype=np.uint64 (a scalar per row) instead of containing lists.

The example below demonstrates all four on 'line' annotations. The two single-segment relationships (body_pre / body_post) use scalar uint64 columns; the multi-segment relationship (nearby_mito) uses lists.

import numpy as np
import pandas as pd
from ngsidekick.annotations.precomputed import write_precomputed_annotations

df = pd.DataFrame({
    # line geometry columns
    'xa': [0.0, 10.0], 'ya': [0.0, 0.0], 'za': [0.0, 0.0],
    'xb': [5.0, 15.0], 'yb': [5.0, 5.0], 'zb': [0.0, 0.0],

    # numeric property
    'confidence': [0.92, 0.71],

    # enum property (pandas categorical)
    'kind': pd.Categorical(['excitatory', 'inhibitory']),

    # color property: one column per channel, rgb(a)
    'mycolor_r': [255,   0], 'mycolor_g': [128, 200], 'mycolor_b': [  0, 255],
    'mycolor_a': [255, 255],  # (alpha is optional)

    # single-segment relationships: scalar uint64 per row
    'body_pre':  np.array([100, 200], dtype=np.uint64),
    'body_post': np.array([300, 400], dtype=np.uint64),

    # multi-segment relationship: list of uint64 per row
    'nearby_mito': [[10, 11], [20, 21, 22]],
})

# df:
#         xa   ya   za    xb   yb   zb  confidence        kind  mycolor_r  mycolor_g  mycolor_b  mycolor_a  body_pre  body_post   nearby_mito
#    0   0.0  0.0  0.0   5.0  5.0  0.0        0.92  excitatory        255        128          0        255       100        300      [10, 11]
#    1  10.0  0.0  0.0  15.0  5.0  0.0        0.71  inhibitory          0        200        255        255       200        400  [20, 21, 22]

write_precomputed_annotations(
    df, 'xyz', 'line',
    # 'mycolor' is the base name; the _r/_g/_b/_a columns are picked up automatically.
    properties=['confidence', 'kind', 'mycolor'],
    relationships=['body_pre', 'body_post', 'nearby_mito'],
    output_dir='out/lines',
)

Streaming from a Feather file#

For datasets that don’t fit comfortably in pandas memory, df can be a path (str or os.PathLike) to a Feather/Arrow IPC file. The file is memory-mapped via PyArrow and registered into DuckDB; the writers stream through it in shard-aligned batches so the full input never materializes in the Python heap.

import pyarrow.feather as feather

# Pre-built annotation table on disk -- one row per annotation, with
# an explicit ``annotation_id`` column (Feather files don't carry a
# pandas index).
feather.write_feather(df, 'annotations.feather')

write_precomputed_annotations(
    'annotations.feather', 'xyz', 'line',
    properties=['confidence'],
    relationships=['body_pre'],
    output_dir='out/lines',
)

Annotation IDs in Feather files are resolved in priority order:

If the file has a column named annotation_id, it is used directly.
If the file was written by pandas with a real index column (named or anonymous), that column is reused as annotation_id via a zero-copy PyArrow rename. So if you wrote your file with something like df.to_feather(path), the index comes through automatically no matter what it was called.
If neither applies (e.g. the file was written from a pandas RangeIndex, or by a non-pandas tool with no id column), an annotation_id is synthesized on the fly via DuckDB’s ROW_NUMBER(). A pandas RangeIndex(start, stop, step) descriptor in the file’s schema metadata, if present, is honored; otherwise IDs are 0, 1, 2, ....

Other notes specific to the Feather path:

Polyline auxiliary data (polyline_points) is still required to be an in-memory pandas DataFrame. The main table may be Feather even for polyline annotations; only the aux table is restricted.
DuckDB’s documented order-preservation guarantee ensures that the streamed batches deliver rows in the file’s storage order, which matters when shuffle_spatial_ordering=False.

Tuning tensorstore writes#

The sharded write path uses tensorstore, which can be tuned via three related arguments. The defaults are tuned for high-throughput multi-core machines and should be fine for most cases.

max_threads (default: LSB_DJOB_NUMPROC on LSF, otherwise the local CPU count) sets the limit on tensorstore’s data_copy_concurrency and file_io_concurrency pools — i.e. how many threads tensorstore is allowed to use for shard encoding/compression and file I/O.

max_shards_per_transaction (default: equal to max_threads) controls how many shards are committed in a single tensorstore transaction. A transaction holds all of its shards’ staged data in memory until commit, so this is the main knob for trading RAM for throughput during writes: more shards per transaction → more parallelism at commit time but a higher peak RAM during sharded writes; fewer shards per transaction → less RAM, slower commits.

# Lower memory pressure at the cost of less commit parallelism.
write_precomputed_annotations(
    df, 'xyz', 'line', output_dir='out/lines',
    max_threads=64,
    max_shards_per_transaction=16,
)

tensorstore_context accepts a JSON-shaped dict matching tensorstore’s Context spec, which is useful when you want finer control over tensorstore’s resource pools than max_threads alone provides. The most useful key in practice is cache_pool.total_bytes_limit, which caps the in-memory shard staging that tensorstore retains across transactions (and tends to dominate sustained RAM use on very large runs):

write_precomputed_annotations(
    df, 'xyz', 'line', output_dir='out/lines',
    tensorstore_context={
        # Cap tensorstore's internal cache + write-staging pool at 4 GB.
        'cache_pool': {'total_bytes_limit': 4_000_000_000},
    },
)

Any keys you provide are passed through verbatim; the data_copy_concurrency and file_io_concurrency keys are filled in from max_threads only when your dict doesn’t already specify them, so you can override one without touching the other.

Bounding DuckDB’s working set#

The shard-streamed write pipeline materializes a per-level working table inside DuckDB (joining the spatial assignments with the input geometry+properties, sorted by shard_id). For large inputs this working set can become several GB. If you want to cap how much of it DuckDB keeps resident, pass duckdb_memory_limit:

write_precomputed_annotations(
    'annotations.feather', 'xyz', 'line', output_dir='out/lines',
    duckdb_memory_limit='40GB',
)

DuckDB spills excess to its temp directory once the limit is reached, trading some I/O for a tighter resident-RAM ceiling. Leaving the argument as None (the default) lets DuckDB choose – typically ~80% of system RAM, which is fine for most jobs.

DuckDB’s spill directory defaults to .tmp/ under the process’s current working directory. On a cluster node whose CWD lives on a slow shared filesystem (GPFS, NFS, etc.), it’s worth pointing DuckDB at fast local scratch instead via duckdb_temp_directory:

write_precomputed_annotations(
    'annotations.feather', 'xyz', 'line', output_dir='out/lines',
    duckdb_memory_limit='40GB',
    duckdb_temp_directory='/scratch/myuser/duckdb-spill',
)

API reference#

ngsidekick.annotations.precomputed.write_precomputed_annotations(df, coord_space, annotation_type, properties=(), relationships=(), output_dir='annotations', write_sharded=True, *, polyline_points=None, write_by_id=True, write_by_relationship=True, write_by_spatial_chunk=True, num_spatial_levels=64, target_chunk_limit=10000, shuffle_spatial_ordering=True, max_threads=None, max_shards_per_transaction=None, duckdb_memory_limit=None, duckdb_temp_directory=None, tensorstore_context=None, description='')[source]#

Export the data from a pandas DataFrame into neuroglancer’s precomputed annotations format as described in the neuroglancer spec.

A progress bar is shown when writing each portion of the export (annotation ID index, related ID indexes), but there may be a significant amount of preprocessing time that occurs before the actual writing begins.

Note

Internally the writers stream through the input in shard-aligned batches via DuckDB, so peak RAM during the encode+write phase is roughly one batch’s worth of encoded bytes rather than the full dataset. To take maximum advantage of this for large inputs, pass the path to a Feather/Arrow IPC file in df rather than a fully-materialized pandas DataFrame.

Parameters:

df (Union[DataFrame, str, PathLike, None]) –
The annotations table. Accepted forms:
- pandas DataFrame: the DataFrame’s index is used as the annotation ID and must be unique. Columns supply geometry, properties, and relationships per the rules below.
- Path-like (str or os.PathLike): a Feather/Arrow IPC file carrying the same columns as the DataFrame form. The annotation ID is taken from an explicit annotation_id column if present; otherwise from a pandas-index column recorded in the file’s schema metadata (so a file written via df.to_feather(path) just works); otherwise synthesized as 0, 1, 2, .... The data is streamed from the file via DuckDB and never fully materialized in pandas memory.
- None: only valid for annotation_type='polyline' when you have no properties or relationships – the main table is synthesized from the unique annotation IDs in polyline_points.
Required geometry columns depend on the annotation_type and the coordinate space. For example, assuming coord_space.names == ['x', 'y', 'z']:
- For point annotations, provide [‘x’, ‘y’, ‘z’]
- For line annotations or axis_aligned_bounding_box annotations, provide [‘xa’, ‘ya’, ‘za’, ‘xb’, ‘yb’, ‘zb’]
- For ellipsoid annotations, provide [‘x’, ‘y’, ‘z’, ‘rx’, ‘ry’, ‘rz’] for the center point and radii.
- For polyline annotations, do not provide x/y/z columns here. Instead, provide them in the polyline_points argument.
You may also provide additional columns to use as annotation properties, in which case their column names should be listed in the ‘properties’ argument. (See below.)
coord_space (CoordinateSpace | str | list[str] | dict[str, list]) –
neuroglancer.coordinate_space.CoordinateSpace or equivalent. The coordinate space of the annotations. Among other things, this determines which input columns represent the annotation geometry. For convenience, we accept a couple different formats for the coordinate space, assuming a default scale of 1 nm if no scale/units are provided.

Examples (all equivalent):
```
>>> coord_space = "xyz"
>>> coord_space = ['x', 'y', 'z']
>>> coord_space = {"names": ['x', 'y', 'z']}
>>> coord_space = {
    "names": ['x', 'y', 'z'],
    "units": ['nm', 'nm', 'nm'],
    "scales": [1, 1, 1]
}
>>> coord_space = CoordinateSpace(
...     names=['x', 'y', 'z'],
...     scales=[1.0, 1.0, 1.0],
...     units=['nm', 'nm', 'nm']
... )
```
annotation_type (Literal['point', 'line', 'ellipsoid', 'axis_aligned_bounding_box', 'polyline']) – Literal[‘point’, ‘line’, ‘ellipsoid’, ‘axis_aligned_bounding_box’, ‘polyline’] The type of annotation to export. Note that the columns you provide in the DataFrame depend on the annotation type.
properties (list[str] | list[AnnotationPropertySpec] | dict[str, AnnotationPropertySpec] | list[dict]) –
If your dataframe contains columns for annotation properties, list the names of those columns here.

Categorical columns will be automatically converted to integers with associated enum labels.

To provide an rgb or rgba property such as ‘mycolor’, provide separate columns in your dataframe named ‘mycolor_r’, ‘mycolor_g’, ‘mycolor_b’ (and ‘mycolor_a’), and then include ‘mycolor’ in the properties list here.

The full property spec for each property will be inferred from the column dtype, but if you want to explicitly override any property specs yourself, you can pass a list of AnnotationPropertySpec objects here instead of just listing column names.

Property names must start with a lowercase letter and may contain only letters, numbers, and underscores.
relationships (list[str]) – list[str] If your annotations have related segment IDs, such relationships can be provided in the columns of your DataFrame. Each relationship should be listed in a single column, whose values are lists of segment IDs. In the special case where each annotation has exactly one related segment, the column may have dtype=np.uint64 instead of containing lists.
output_dir (str) – str The directory into which the exported annotations will be written. Subdirectories will be created for the “annotation ID index” and each “related object id index” as needed.
write_sharded (bool) – bool Whether to write the output as sharded files. The sharded format is preferable for most use cases. Without sharding, every annotation results in a separate file in the annotation ID index. Similarly, every related ID results in a separate file in the related ID index.
polyline_points (DataFrame | None) –
pandas DataFrame. Required when annotation_type='polyline'; must be None otherwise. (Feather input is not supported here – the polyline aux table must be an in-memory pandas DataFrame.)

One row per polyline vertex, with one column per coordinate axis plus an 'annotation_id' column indicating which polyline each vertex belongs to. For example, assuming coord_space.names == ['x', 'y', 'z'], then provide the following columns: [‘annotation_id’, ‘x’, ‘y’, ‘z’]. (For a polyline with N vertices, its annotation_id should appear N times.)

For each polyline, the point order in the emitted annotation will match the order in which they appear in this dataframe.
write_by_id (bool) – bool Whether to write the annotations to the “Annotation ID Index”. If False, skip writing.
write_relationships – bool Whether to write the relationships to the “Related Object ID Index”. If False, skip writing.
write_by_spatial_chunk (bool) – bool Whether to write the spatial index.
num_spatial_levels (int) – int The maximum number of spatial index levels to write. If not all levels are needed (because all annotations fit within the first N levels), then the actual number of levels written will be less than this value. The default allows up to 64 levels (at least 9e18 spatial subdivisions at the finest level), which far exceeds the max that any real dataset would need.
target_chunk_limit (int) –
int For the spatial index, this is how many annotations we aim to place in each chunk (regardless of the level). If there are more annotations than fit within the specified num_spacial_levels while (approximately) adhering to the target_chunk_limit at each level, then the extra annotations will be assigned to the last level.

Note

Instead of specifying a valid limit here, you can disable subsampling in neuroglancer by setting this to the special value of 0. In our implementation, this is only valid when num_spatial_levels=1.
shuffle_spatial_ordering (bool) –
bool Whether to randomize the spatial assignment. When True (the default), two things happen randomly:
1. which level each annotation lands at is uniformly random, so coarse levels carry a uniform random sample of all annotations – the neuroglancer spec recommendation.
2. the within-chunk order is also random, so that neuroglancer’s prefix-based subsampling (it draws the first N annotations from a chunk’s stored list) produces an unbiased sample at any zoom level.
When False, both orderings use the input row order: earlier input rows go to coarser levels, and within each chunk annotations are stored in input row order. Set this False when you have deliberately ordered your input (e.g. by importance) and want neuroglancer to render the most important annotations first.
max_threads (int | None) – int or None Default cap on tensorstore’s data-copy and file-I/O thread pools. Used to populate the data_copy_concurrency and file_io_concurrency keys of the tensorstore Context when tensorstore_context doesn’t already specify them. Defaults to LSB_DJOB_NUMPROC on LSF clusters, otherwise multiprocessing.cpu_count().
max_shards_per_transaction (int | None) –
int or None (Sharded mode only.) Caps the number of shards committed in a single tensorstore transaction. Tensorstore parallelizes the per-shard work (encode, compress, write) inside a transaction across its internal thread pool, so this knob trades RAM (more shards staged in memory at once) for throughput and effective CPU utilization (more parallel work available at commit).

Defaults to max_threads so each transaction can saturate the available threads. Set higher for better throughput at extra RAM cost, or lower to reduce peak RAM.
duckdb_memory_limit (str | None) – str or None Forwarded to DuckDB’s memory_limit setting (e.g. '40GB') when opening the connection used for all shard-streamed writes. DuckDB will spill to its temp directory once its working set exceeds this value, which caps DuckDB’s contribution to peak RAM at the cost of some extra I/O. Defaults to None, which lets DuckDB pick its own limit (~80% of system RAM).
duckdb_temp_directory (str | None) – str or None Forwarded to DuckDB’s temp_directory setting – the location used for spill files when DuckDB’s working set exceeds duckdb_memory_limit. Defaults to None, which leaves DuckDB on its own default of .tmp/ under the process’s current working directory. Set this to a fast local-scratch path (e.g. /scratch/...) when running on a cluster node whose CWD is on a slow shared filesystem.
tensorstore_context (dict | None) –
dict or None Optional JSON spec for the tensorstore Context used to open every kvstore in this run. Useful for tuning resource pool sizes (e.g. cache_pool.total_bytes_limit) to cap peak RAM during sharded writes.

Any key you supply is passed through verbatim; data_copy_concurrency and file_io_concurrency are filled in from max_threads only when you haven’t already specified them.
description (str) – str A description of the annotation collection.
write_by_relationship (bool)

annotations.precomputed

Contents

annotations.precomputed#

Geometry columns#

Examples#

Point annotations#

Line annotations#

Ellipsoid annotations#

Polyline annotations#

Properties and relationships#

Streaming from a Feather file#

Tuning tensorstore writes#

Bounding DuckDB’s working set#

API reference#