From 5ab84f35e6ca2f5e6a1a68466654cc6dac85a762 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 09:58:25 -0400 Subject: [PATCH 001/110] Add lightsheet OME-Zarr NDI document type + LightsheetZarrManager (scaffold) Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+lightsheet/README-lightsheet-zarr.md | 100 ++++++++ .../+ndi/+fun/+doc/+lightsheet/chooseLevel.m | 64 +++++ .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 191 ++++++++++++++ .../+ndi/+fun/+doc/+lightsheet/levelTable.m | 100 ++++++++ .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 181 ++++++++++++++ .../+fun/+doc/+lightsheet/makeSourceFile.m | 48 ++++ .../+ndi/+fun/+doc/+lightsheet/readViewport.m | 75 ++++++ .../+ndi/+fun/+doc/+lightsheet/viewCommand.m | 104 ++++++++ .../+fun/+doc/+lightsheet/writeChunkFile.m | 42 ++++ .../+ndi/+gui/+app/LightsheetZarrManager.m | 236 ++++++++++++++++++ .../data/lightsheetZarrLevel.json | 53 ++++ .../data/lightsheetZarrLevel.md | 76 ++++++ .../data/lightsheetZarrPyramid.json | 46 ++++ .../data/lightsheetZarrPyramid.md | 78 ++++++ 14 files changed, 1394 insertions(+) create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/chooseLevel.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/levelTable.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/makeSourceFile.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/viewCommand.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m create mode 100644 src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m create mode 100644 src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json create mode 100644 src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.md create mode 100644 src/ndi/ndi_common/database_documents/data/lightsheetZarrPyramid.json create mode 100644 src/ndi/ndi_common/database_documents/data/lightsheetZarrPyramid.md diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md new file mode 100644 index 000000000..2a5c34e76 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md @@ -0,0 +1,100 @@ +# lightsheet OME-Zarr documents (design + status) + +This package (`+ndi/+fun/+doc/+lightsheet/`) lets a session hold a +lightsheet OME-Zarr (NGFF v0.4) multiscale pyramid as NDI documents, +and hands one to napari for viewing through NDI-python. + +The pattern mirrors `+ndi/+fun/+doc/+gene/` (spatial transcriptomics +Stereo-seq / GEF) so a reader of that package will recognise every +piece here. + +## Document shape + +Two new document classes: + +- **`lightsheetZarrPyramid`** - one per REDUCTION variant (`mean`, + `max`, ...). Holds the frame (axes, level-0 shape, voxel size, + translation, channels, dtype). Depends on subject + optional element + + optional `fileReference` for the source store. +- **`lightsheetZarrLevel`** - one per (pyramid, level). Owns the chunk + bytes as a `chunk.bin_#` file series and the per-level metadata + (shape, chunks, voxel size, translation, codec, ...). Depends on its + parent pyramid. + +NDI documents are immutable, so: +- A new reduction is a new pyramid document + its own level ladder. +- A deeper coarsening is a new level document; the parent's `n_levels` + is out of date until a new parent version is written for that count. + +## MATLAB API surface + +- `ndi.fun.doc.lightsheet.fromOMEZarr(session, zarrPath, ...)` - + probe an OME-Zarr store, write ONE pyramid document per multiscales + entry and ONE level document per level. Metadata-only; does not copy + chunk bytes. **Working.** +- `ndi.fun.doc.lightsheet.makePyramid(session, pyramidEntry, ...)` - + the per-multiscales-entry worker `fromOMEZarr` calls. Reusable when + the caller already has an `ndr.format.omezarr.listPyramids` struct + in hand. **Working.** +- `ndi.fun.doc.lightsheet.makeSourceFile(session, zarrPath)` - + fileReference for the store. **Working.** +- `ndi.fun.doc.lightsheet.levelTable(session, pyramidDoc)` - + finest-first table of a pyramid's levels; `Properties.UserData` + carries the parent frame. **Working.** +- `ndi.fun.doc.lightsheet.chooseLevel(levels, targetVoxelSize)` - + pick the coarsest level whose voxel size still meets the target. + **Working.** +- `ndi.fun.doc.lightsheet.viewCommand(launcher, sessionPath, pyramidID, ...)` + - pure command builder. Same pattern as `ndi.gui.app.GEFManager.viewCommand`; + the launcher is `/usr/local/bin/napariViewLightsheet`, a shell + wrapper around the napari console script that ships in NDI-python. + **Working.** +- `ndi.fun.doc.lightsheet.writeChunkFile(session, levelDoc, pyramidEntry, level)` + - materialize a level's chunks into its `chunk.bin_#` file series. + **Scaffold.** See "What still needs building" below. +- `ndi.fun.doc.lightsheet.readViewport(session, pyramidDoc, region, ...)` + - read a rectangle from the appropriate level. **Scaffold**; the + napari viewer in NDI-python is the primary reader. + +## GUI + +`ndi.gui.app.LightsheetZarrManager(session)` mirrors `GEFManager`: +lists pyramids in a session, and drives Add / View / Delete. `View` +builds the napari command from `viewCommand` and shells out. + +## What still needs building + +1. **Chunk materialization across the HIPAA-compliant NDI cloud API.** + `writeChunkFile.m` is a scaffold today because the file-series + upload path we need is the one already used by + `ndi.cloud.sync.internal.uploadFilesForDatasetDocuments`, and + wiring it in here needs a small chunk-index side-file (level + document field `chunk_index`) so we can elide fill-value chunks + without breaking the 1-based `chunk.bin_#` addressing. Design and + land that first, then remove the `notImplemented` guard. +2. **NDI-python napari console script.** The MATLAB `viewCommand` + targets `/usr/local/bin/napariViewLightsheet`, a shell wrapper + around a Python console script that ships in NDI-python (see + `ndi/lightsheet/napari_view.py`). That console script is the piece + that actually reads the pyramid + level documents through the NDI + cloud API and hands lazy dask arrays to napari. +3. **`readViewport` MATLAB backend.** Two paths are already + documented: read the source store via `ndr.format.omezarr.readArray` + (works today, but only for the session that has the store on-disk), + or read `chunk.bin_#` from a level document through the cloud API + (blocked on #1). +4. **NDR helpers.** `ndr.format.omezarr.probe` is small; when it lands + in NDR-matlab / NDR-python, `fromOMEZarr` here can call it instead + of `listPyramids` for cheaper metadata reads. +5. **Tests.** Once #1 lands, extend + `tests/+ndi/+unittest/+fun/+doc/+lightsheet/` with round-trip and + viewport tests analogous to `+gene/`'s. + +## Related + +- `ndi.gui.app.GEFManager` - the sibling manager for Stereo-seq GEF + spatial transcriptomics. Same launcher pattern (a shell wrapper + around a NDI-python console script). Read it first if this is your + first NDI file-format ingest. +- `ndr.format.omezarr.*` - the metadata / read primitives this + package builds on. diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseLevel.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseLevel.m new file mode 100644 index 000000000..60cf89d89 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseLevel.m @@ -0,0 +1,64 @@ +function [levelIndex, levelRow] = chooseLevel(levels, targetVoxelSize) +% NDI.FUN.DOC.LIGHTSHEET.CHOOSELEVEL - pick the coarsest level meeting a target +% +% [LEVELINDEX, LEVELROW] = NDI.FUN.DOC.LIGHTSHEET.CHOOSELEVEL(LEVELS, +% TARGETVOXELSIZE) +% +% Given LEVELS (a struct array or table with per-level `voxel_size` +% and `level` fields, finest-first, e.g. from LEVELTABLE), returns +% the coarsest level whose per-axis voxel size is still <= +% TARGETVOXELSIZE. If every level is coarser than the target, returns +% level 0. If every level is finer, returns the coarsest. +% +% LEVELINDEX is the 0-based level index (matches +% lightsheetZarrLevel.level). +% +% TARGETVOXELSIZE may be a scalar (applied to every spatial axis) or a +% vector aligned with the level's `voxel_size`. +% +% See also: ndi.fun.doc.lightsheet.levelTable + + arguments + levels + targetVoxelSize (1,:) double + end + + if istable(levels) + vs = levels.voxel_size; + li = levels.level; + n = height(levels); + else + n = numel(levels); + vs = cell(n, 1); + li = zeros(n, 1); + for k = 1:n + vs{k} = reshape(double(levels(k).voxel_size), 1, []); + li(k) = levels(k).level; + end + end + if n == 0 + error('NDI:lightsheet:chooseLevel:emptyLevels', ... + 'levels is empty.'); + end + + bestK = 1; + for k = 1:n + v = vs{k}; + if isscalar(targetVoxelSize) + ok = all(v <= targetVoxelSize); + else + m = min(numel(v), numel(targetVoxelSize)); + ok = all(v(1:m) <= targetVoxelSize(1:m)); + end + if ok + bestK = k; + end + end + + levelIndex = li(bestK); + if istable(levels) + levelRow = levels(bestK, :); + else + levelRow = levels(bestK); + end +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m new file mode 100644 index 000000000..5dfbbb908 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -0,0 +1,191 @@ +function [pyramidDocs, levelDocs, info] = fromOMEZarr(session, zarrPath, options) +% NDI.FUN.DOC.LIGHTSHEET.FROMOMEZARR - build lightsheetZarrPyramid docs from an OME-Zarr store +% +% [PYRAMIDDOCS, LEVELDOCS, INFO] = NDI.FUN.DOC.LIGHTSHEET.FROMOMEZARR(... +% SESSION, ZARRPATH) +% +% Walks an OME-Zarr (NGFF v0.4) store at ZARRPATH, enumerates its +% multiscale pyramids with NDR.FORMAT.OMEZARR.LISTPYRAMIDS, and writes +% a LIGHTSHEETZARRPYRAMID document plus one LIGHTSHEETZARRLEVEL +% document per level to SESSION. +% +% ONE DOCUMENT PER REDUCTION. An OME-Zarr store frequently contains +% both a mean-reduced and a max-projected pyramid. This function writes +% ONE lightsheetZarrPyramid per multiscales entry it finds; a store +% with two entries produces two parents and their two ladders of +% levels. +% +% ONE DOCUMENT PER LEVEL. NDI documents are immutable, so each +% resolution level is its own lightsheetZarrLevel document. The level +% documents depend_on the parent pyramid document. +% +% READING IS METADATA-ONLY. No chunk bytes are read here; the source +% store's .zattrs and per-level .zarray are enough to fill the +% documents. The chunk bytes belong to the source_file_id +% (fileReference) and are materialized into a level document's +% `chunk.bin_#` file series by NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID. +% Separating "probe and describe" from "materialize" keeps this +% function fast enough to run inside a GUI dialog. +% +% Optional Name-Value Arguments: +% subjectID - char/string, id of the subject depended on. If empty, +% the caller is refused: a lightsheetZarrPyramid without +% a subject is not a valid document. +% elementID - char/string, id of the imaging element the volume was +% captured on. May be empty. +% pyramidNames - cellstr/string, restrict to these multiscales.name +% values. Empty means all. +% reductionMap - containers.Map / struct mapping pyramid name (as +% reported by listPyramids) or type to a reduction +% label. Recognised reductions: 'mean', 'max'. When a +% name is not in the map, `pyramid_type` is inspected +% first, then a fallback of 'mean' is used with a +% warning. +% pipelineVersion - char/string tag written into +% lightsheetZarrPyramid.pipeline_version. +% sourceFileID - id of an existing fileReference document for +% ZARRPATH. When empty, a new fileReference is created +% via NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE. +% +% INFO returns a struct with fields: +% zarrPath - absolute path resolved +% pyramids - the raw output of ndr.format.omezarr.listPyramids +% reductions - cellstr, one per created pyramid, giving the +% reduction assigned to it +% sourceFileID - id of the fileReference document used +% +% Example: +% S = ndi.session.dir('mysession','/path/to/session'); +% [P, L, info] = ndi.fun.doc.lightsheet.fromOMEZarr(S, ... +% '/cloudcache/subject42/volume.ome.zarr', ... +% 'subjectID', subj.id()); +% +% See also: ndi.fun.doc.lightsheet.makePyramid, +% ndi.fun.doc.lightsheet.chooseLevel, +% ndi.fun.doc.lightsheet.viewCommand, +% ndr.format.omezarr.listPyramids, +% ndr.format.omezarr.probe + + arguments + session (1,1) + zarrPath (1,:) char + options.subjectID char = '' + options.elementID char = '' + options.pyramidNames = {} + options.reductionMap = struct() + options.pipelineVersion char = '' + options.sourceFileID char = '' + end + + if isempty(strtrim(options.subjectID)) + error('NDI:lightsheet:fromOMEZarr:noSubject', ... + ['A lightsheetZarrPyramid depends_on a subject; pass ' ... + '''subjectID'' or use NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE ' ... + 'first and then this function.']); + end + + if ~isfolder(zarrPath) + error('NDI:lightsheet:fromOMEZarr:noSuchStore', ... + 'OME-Zarr store not found at %s.', zarrPath); + end + + if ~ndr.format.omezarr.isOMEZarr(zarrPath) + error('NDI:lightsheet:fromOMEZarr:notOMEZarr', ... + '%s does not look like an OME-Zarr store (.zattrs missing multiscales).', ... + zarrPath); + end + + pyramids = ndr.format.omezarr.listPyramids(zarrPath); + if isempty(pyramids) + error('NDI:lightsheet:fromOMEZarr:noPyramids', ... + '%s reports zero multiscales entries.', zarrPath); + end + + keep = true(numel(pyramids), 1); + if ~isempty(options.pyramidNames) + wanted = cellstr(options.pyramidNames); + for i = 1:numel(pyramids) + keep(i) = any(strcmp(pyramids(i).name, wanted)); + end + if ~any(keep) + error('NDI:lightsheet:fromOMEZarr:noNamedPyramid', ... + ['None of the requested pyramid names (%s) matched. ' ... + 'Available: %s.'], strjoin(wanted, ', '), ... + strjoin({pyramids.name}, ', ')); + end + end + pyramids = pyramids(keep); + + sourceFileID = options.sourceFileID; + if isempty(sourceFileID) + srcDoc = ndi.fun.doc.lightsheet.makeSourceFile(session, zarrPath); + sourceFileID = srcDoc.id(); + end + + n = numel(pyramids); + pyramidDocs = cell(n, 1); + levelDocsPerPyramid = cell(n, 1); + reductions = cell(n, 1); + + for i = 1:n + reduction = pickReduction(pyramids(i), options.reductionMap); + reductions{i} = reduction; + [pDoc, lDocs] = ndi.fun.doc.lightsheet.makePyramid(session, ... + pyramids(i), ... + 'reduction', reduction, ... + 'subjectID', options.subjectID, ... + 'elementID', options.elementID, ... + 'sourceFileID', sourceFileID, ... + 'pipelineVersion', options.pipelineVersion); + pyramidDocs{i} = pDoc; + levelDocsPerPyramid{i} = lDocs; + end + + levelDocs = vertcat(levelDocsPerPyramid{:}); + + info = struct( ... + 'zarrPath', char(zarrPath), ... + 'pyramids', pyramids, ... + 'reductions', {reductions}, ... + 'sourceFileID', sourceFileID); +end + +function r = pickReduction(pyramid, reductionMap) +% pickReduction - choose 'mean' or 'max' for this pyramid entry +% +% Preference: explicit map by name, then by type, then a heuristic: +% pyramid.type == 'max' -> 'max', anything else -> 'mean'. Warns when +% falling back because pyramid.type is unreliable (the historical +% writer of this format sometimes labeled the mean pyramid 'gaussian'). + r = ''; + if ~isempty(reductionMap) + if isstruct(reductionMap) + if ~isempty(pyramid.name) && isfield(reductionMap, pyramid.name) + r = char(reductionMap.(pyramid.name)); + elseif ~isempty(pyramid.type) && isfield(reductionMap, pyramid.type) + r = char(reductionMap.(pyramid.type)); + end + elseif isa(reductionMap, 'containers.Map') + if ~isempty(pyramid.name) && isKey(reductionMap, pyramid.name) + r = char(reductionMap(pyramid.name)); + elseif ~isempty(pyramid.type) && isKey(reductionMap, pyramid.type) + r = char(reductionMap(pyramid.type)); + end + end + end + if isempty(r) + t = lower(strtrim(char(pyramid.type))); + n = lower(strtrim(char(pyramid.name))); + if strcmp(t, 'max') || contains(n, 'max') + r = 'max'; + else + r = 'mean'; + if ~isempty(t) && ~any(strcmp(t, {'mean','box','average'})) + warning('NDI:lightsheet:fromOMEZarr:reductionFallback', ... + ['Pyramid %s has type ''%s''; falling back to ' ... + '''mean''. Pass reductionMap to override.'], ... + pyramid.name, pyramid.type); + end + end + end +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/levelTable.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/levelTable.m new file mode 100644 index 000000000..07af25941 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/levelTable.m @@ -0,0 +1,100 @@ +function t = levelTable(session, pyramidDoc) +% NDI.FUN.DOC.LIGHTSHEET.LEVELTABLE - table of a pyramid's levels, finest-first +% +% T = NDI.FUN.DOC.LIGHTSHEET.LEVELTABLE(SESSION, PYRAMIDDOC) +% +% Queries SESSION for every LIGHTSHEETZARRLEVEL document whose +% `lightsheetZarrPyramid_id` depends_on matches PYRAMIDDOC.id(), and +% returns a MATLAB table sorted by `level` ascending (finest-first). +% +% Table columns: level, reduction, shape, chunks, chunk_grid, +% n_chunks_stored, voxel_size, translation, dtype, id. +% +% UserData holds the parent pyramid's frame: +% axes_order, voxel_size_level0, translation_level0, dtype, +% reduction, pyramid_name, id (parent id). +% +% PYRAMIDDOC may be an ndi.document or an id char/string; either is +% accepted so this can be called from a GUI that only has the row's +% id. +% +% See also: ndi.fun.doc.lightsheet.chooseLevel + + arguments + session (1,1) + pyramidDoc + end + + if isa(pyramidDoc, 'ndi.document') + parentDoc = pyramidDoc; + parentID = parentDoc.id(); + else + parentID = char(pyramidDoc); + parentDoc = session.database_search(ndi.query('base.id', 'exact_string', parentID)); + if isempty(parentDoc) + error('NDI:lightsheet:levelTable:noParent', ... + 'No lightsheetZarrPyramid with id %s in session.', parentID); + end + parentDoc = parentDoc{1}; + end + + q1 = ndi.query('', 'isa', 'lightsheetZarrLevel'); + q2 = ndi.query('depends_on', 'depends_on', 'lightsheetZarrPyramid_id', parentID); + docs = session.database_search(q1 & q2); + if isempty(docs) + t = emptyTable(); + t.Properties.UserData = frame(parentDoc); + return; + end + + n = numel(docs); + level = zeros(n, 1); + reduction = cell(n, 1); + shape = cell(n, 1); + chunks = cell(n, 1); + chunk_grid = cell(n, 1); + n_chunks_stored = zeros(n, 1); + voxel_size = cell(n, 1); + translation = cell(n, 1); + dtype = cell(n, 1); + id = cell(n, 1); + + for k = 1:n + p = docs{k}.document_properties.lightsheetZarrLevel; + level(k) = p.level; + reduction{k} = char(p.reduction); + shape{k} = reshape(double(p.shape), 1, []); + chunks{k} = reshape(double(p.chunks), 1, []); + chunk_grid{k} = reshape(double(p.chunk_grid), 1, []); + n_chunks_stored(k) = p.n_chunks_stored; + voxel_size{k} = reshape(double(p.voxel_size), 1, []); + translation{k} = reshape(double(p.translation), 1, []); + dtype{k} = char(p.dtype); + id{k} = docs{k}.id(); + end + + t = table(level, reduction, shape, chunks, chunk_grid, ... + n_chunks_stored, voxel_size, translation, dtype, id); + t = sortrows(t, 'level'); + t.Properties.UserData = frame(parentDoc); +end + +function f = frame(parentDoc) + p = parentDoc.document_properties.lightsheetZarrPyramid; + f = struct( ... + 'axes_order', char(p.axes_order), ... + 'voxel_size_level0', reshape(double(p.voxel_size_level0), 1, []), ... + 'translation_level0', reshape(double(p.translation_level0), 1, []), ... + 'dtype', char(p.dtype), ... + 'reduction', char(p.reduction), ... + 'pyramid_name', char(p.pyramid_name), ... + 'id', parentDoc.id()); +end + +function t = emptyTable() + t = table( ... + zeros(0,1), cell(0,1), cell(0,1), cell(0,1), cell(0,1), ... + zeros(0,1), cell(0,1), cell(0,1), cell(0,1), cell(0,1), ... + 'VariableNames', {'level','reduction','shape','chunks','chunk_grid', ... + 'n_chunks_stored','voxel_size','translation','dtype','id'}); +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m new file mode 100644 index 000000000..c51a64742 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -0,0 +1,181 @@ +function [pyramidDoc, levelDocs] = makePyramid(session, pyramidEntry, options) +% NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID - write one reduction's pyramid documents +% +% [PYRAMIDDOC, LEVELDOCS] = NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID(... +% SESSION, PYRAMIDENTRY, options) +% +% Given ONE multiscales entry as returned by +% NDR.FORMAT.OMEZARR.LISTPYRAMIDS (fields: name, type, axes, levels), +% creates: +% +% * one LIGHTSHEETZARRPYRAMID document (the parent) +% * one LIGHTSHEETZARRLEVEL document per entry in PYRAMIDENTRY.levels +% +% and adds them to SESSION. +% +% Required Name-Value: +% reduction - 'mean' or 'max' (or other future value); recorded on +% the parent AND on every level. +% subjectID - char, depended-on subject id. +% +% Optional Name-Value: +% elementID - char, depended-on element id (may be ''). +% sourceFileID - char, id of the fileReference document naming +% the OME-Zarr store; ''=no source_file_id. +% pipelineVersion - char, tag for the pyramid. +% materializeChunks - logical, default false. When true, this +% function writes each level's chunk bytes into +% the level document's `chunk.bin_#` file series +% by reading the source array through +% NDR.FORMAT.OMEZARR.READARRAY. When false, the +% level documents only carry metadata and the +% chunks stay in the source store; a downstream +% ingest job (see the design note in +% README-lightsheet-zarr.md) is responsible for +% materialization. Default false so the ingest is +% incremental: describe first, materialize later. +% +% Returns: +% PYRAMIDDOC - the parent ndi.document +% LEVELDOCS - column cell array of ndi.documents, one per level, +% ordered finest-first (matching PYRAMIDENTRY.levels). +% +% See also: ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.fun.doc.lightsheet.writeChunkFile, +% ndi.fun.doc.lightsheet.readViewport + + arguments + session (1,1) + pyramidEntry (1,1) struct + options.reduction (1,:) char {mustBeMember(options.reduction, {'mean','max'})} + options.subjectID (1,:) char + options.elementID char = '' + options.sourceFileID char = '' + options.pipelineVersion char = '' + options.materializeChunks (1,1) logical = false + end + + axesOrder = joinAxisNames(pyramidEntry.axes); + axesUnits = collectAxisUnits(pyramidEntry.axes); + + levels = pyramidEntry.levels; + if isempty(levels) + error('NDI:lightsheet:makePyramid:noLevels', ... + 'Pyramid ''%s'' has no levels.', pyramidEntry.name); + end + + level0 = levels(1); + dtype = char(level0.dtype); + + pyramidDoc = session.newdocument('lightsheetZarrPyramid', ... + 'lightsheetZarrPyramid', struct( ... + 'label', pyramidEntry.name, ... + 'reduction', options.reduction, ... + 'pyramid_name', char(pyramidEntry.name), ... + 'pyramid_type', char(pyramidEntry.type), ... + 'axes_order', axesOrder, ... + 'axes_units', {axesUnits}, ... + 'channel_names', {{}}, ... + 'n_channels', channelCount(level0.shape, axesOrder), ... + 'n_levels', numel(levels), ... + 'shape_level0', level0.shape, ... + 'voxel_size_level0', ensureRow(level0.scale), ... + 'voxel_size_units', 'micrometer', ... + 'translation_level0', ensureRow(level0.translation), ... + 'dtype', dtype, ... + 'pipeline_version', options.pipelineVersion, ... + 'byte_order', 'little')); + + pyramidDoc = pyramidDoc.set_dependency_value('subject_id', options.subjectID); + if ~isempty(options.elementID) + pyramidDoc = pyramidDoc.set_dependency_value('element_id', options.elementID); + end + if ~isempty(options.sourceFileID) + pyramidDoc = pyramidDoc.set_dependency_value('source_file_id', options.sourceFileID); + end + + session.database_add(pyramidDoc); + + levelDocs = cell(numel(levels), 1); + for k = 1:numel(levels) + levelDocs{k} = makeOneLevel(session, pyramidDoc, pyramidEntry, ... + levels(k), k - 1, options); + end +end + +function levelDoc = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, options) + chunkGrid = ceil(level.shape ./ level.chunks); + nChunks = prod(chunkGrid); + + props = struct( ... + 'label', sprintf('%s level %d', pyramidEntry.name, levelIndex), ... + 'level', levelIndex, ... + 'reduction', options.reduction, ... + 'axes_order', joinAxisNames(pyramidEntry.axes), ... + 'shape', level.shape, ... + 'chunks', level.chunks, ... + 'chunk_grid', chunkGrid, ... + 'n_chunks_stored', 0, ... + 'chunk_index_origin', 1, ... + 'chunk_order', 'C', ... + 'dtype', char(level.dtype), ... + 'fill_value', 0, ... + 'byte_order', 'little', ... + 'codec', 'raw', ... + 'codec_params', '{}', ... + 'voxel_size', ensureRow(level.scale), ... + 'voxel_size_units', 'micrometer', ... + 'translation', ensureRow(level.translation)); + + levelDoc = session.newdocument('lightsheetZarrLevel', ... + 'lightsheetZarrLevel', props); + + levelDoc = levelDoc.set_dependency_value('lightsheetZarrPyramid_id', pyramidDoc.id()); + levelDoc = levelDoc.set_dependency_value('subject_id', options.subjectID); + if ~isempty(options.sourceFileID) + levelDoc = levelDoc.set_dependency_value('source_file_id', options.sourceFileID); + end + + if options.materializeChunks + [levelDoc, nWritten] = ndi.fun.doc.lightsheet.writeChunkFile(... + session, levelDoc, pyramidEntry, level); + levelDoc = levelDoc.setproperties( ... + 'lightsheetZarrLevel.n_chunks_stored', nWritten); + end + + session.database_add(levelDoc); +end + +function s = joinAxisNames(axes) + if isempty(axes) + s = ''; + return; + end + names = arrayfun(@(a) char(a.name), axes, 'UniformOutput', false); + s = strjoin(names, ''); +end + +function u = collectAxisUnits(axes) + if isempty(axes) + u = {}; + return; + end + u = arrayfun(@(a) char(a.unit), axes, 'UniformOutput', false); +end + +function n = channelCount(shape, axesOrder) + c = strfind(axesOrder, 'c'); + if isempty(c) || numel(shape) < c(1) + n = 1; + else + n = shape(c(1)); + end +end + +function v = ensureRow(x) + if isempty(x) + v = []; + else + v = reshape(double(x), 1, []); + end +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makeSourceFile.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makeSourceFile.m new file mode 100644 index 000000000..f4b1d46ee --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makeSourceFile.m @@ -0,0 +1,48 @@ +function doc = makeSourceFile(session, zarrPath) +% NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE - fileReference for an OME-Zarr store +% +% DOC = NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE(SESSION, ZARRPATH) +% +% Creates and adds a FILEREFERENCE document naming the OME-Zarr store +% at ZARRPATH. NDI.FUN.DOC.LIGHTSHEET.FROMOMEZARR calls this to obtain +% the id it stores in `source_file_id` on the pyramid and level +% documents. +% +% The store's per-chunk files are many, so this does not checksum the +% full tree; it records the directory itself as the reference, +% matching how spatialGeneExpression stores its GEF source. + + arguments + session (1,1) + zarrPath (1,:) char + end + + if ~isfolder(zarrPath) + error('NDI:lightsheet:makeSourceFile:noSuchStore', ... + 'OME-Zarr store not found at %s.', zarrPath); + end + + d = dir(zarrPath); + d = d(~ismember({d.name}, {'.', '..'})); + totalBytes = sum([d.bytes]); + if isempty(totalBytes) + totalBytes = 0; + end + + [absPath, ~, ~] = fileparts(fullfile(zarrPath, '.')); + [~, storeName, storeExt] = fileparts(strip(absPath, 'right', filesep)); + fileName = [storeName storeExt]; + + doc = session.newdocument('fileReference', ... + 'fileReference', struct( ... + 'filename', fileName, ... + 'originalPath', absPath, ... + 'formatOntology', 'OME-NGFF v0.4 (OME-Zarr)', ... + 'dateCreated', posixtime(datetime('now', 'TimeZone', 'UTC')), ... + 'dateUpdated', posixtime(datetime('now', 'TimeZone', 'UTC')), ... + 'fileSize', totalBytes, ... + 'checksum', '', ... + 'checksumAlgorithm', 'MD5')); + + session.database_add(doc); +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m new file mode 100644 index 000000000..bfe1a20af --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m @@ -0,0 +1,75 @@ +function [img, info] = readViewport(session, pyramidDoc, region, options) +% NDI.FUN.DOC.LIGHTSHEET.READVIEWPORT - read a rectangle out of a pyramid +% +% [IMG, INFO] = NDI.FUN.DOC.LIGHTSHEET.READVIEWPORT(SESSION, +% PYRAMIDDOC, REGION, options) +% +% Reads a region of interest from a lightsheetZarrPyramid at whichever +% resolution level best matches the requested pixel size. +% +% PYRAMIDDOC is a lightsheetZarrPyramid ndi.document, or its id. +% REGION is a struct with fields matching the pyramid's axes_order +% (e.g. `t`, `c`, `z`, `y`, `x`), each a 1x2 [lo hi] pair in physical +% coordinates (units per pyramid.voxel_size_units). Missing fields +% mean "all of that axis". +% +% Optional Name-Value: +% targetVoxelSize - the coarsest per-axis voxel size that still +% satisfies the caller's rendering target. Passed +% to CHOOSELEVEL. Default: the finest level's +% voxel size (i.e. level 0). +% forceLevel - override the level choice; 0-based level index. +% +% Returns IMG (numeric array in the level's dtype) and INFO struct +% with fields `level`, `voxel_size`, `translation`, `axes_order`, +% `region_used_voxels`. +% +% THIS FUNCTION HAS TWO BACKENDS. +% 1. If the level document's `n_chunks_stored` is 0, the source +% OME-Zarr store is still authoritative and we read from it via +% NDR.FORMAT.OMEZARR.READARRAY on the file named by the +% fileReference. This is the path exercised by the napari +% viewer on a session that has been probed but not fully +% materialized. +% 2. If `n_chunks_stored` is > 0, we read the required chunks from +% the level document's `chunk.bin_#` file series through the +% NDI cloud API. That path is a scaffold today; see +% README-lightsheet-zarr.md. +% +% See also: ndi.fun.doc.lightsheet.chooseLevel, +% ndi.fun.doc.lightsheet.levelTable, +% ndr.format.omezarr.readArray + + arguments + session (1,1) + pyramidDoc + region (1,1) struct + options.targetVoxelSize (1,:) double = [] + options.forceLevel (1,1) double {mustBeInteger} = -1 + end + + t = ndi.fun.doc.lightsheet.levelTable(session, pyramidDoc); + if height(t) == 0 + error('NDI:lightsheet:readViewport:noLevels', ... + 'Pyramid has no lightsheetZarrLevel documents.'); + end + + if options.forceLevel >= 0 + row = t(t.level == options.forceLevel, :); + if isempty(row) + error('NDI:lightsheet:readViewport:noSuchLevel', ... + 'Requested level %d not present.', options.forceLevel); + end + else + target = options.targetVoxelSize; + if isempty(target) + target = t.voxel_size{1}; + end + [~, row] = ndi.fun.doc.lightsheet.chooseLevel(t, target); + end + + error('NDI:lightsheet:readViewport:notImplemented', ... + ['readViewport is a scaffold in this PR. See README-' ... + 'lightsheet-zarr.md for the two backend paths (source ' ... + 'OME-Zarr and materialized chunks) and their status.']); %#ok +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/viewCommand.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/viewCommand.m new file mode 100644 index 000000000..eb2bea523 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/viewCommand.m @@ -0,0 +1,104 @@ +function cmd = viewCommand(launcher, sessionPath, pyramidID, options) +% NDI.FUN.DOC.LIGHTSHEET.VIEWCOMMAND - the command line that opens a pyramid in napari +% +% CMD = NDI.FUN.DOC.LIGHTSHEET.VIEWCOMMAND(LAUNCHER, SESSIONPATH, PYRAMIDID) +% returns the shell command, as a char row vector, that hands this +% lightsheetZarrPyramid to the napari viewer in NDI-python. +% +% PURE, AND THAT IS THE POINT. Nothing here launches anything or +% touches the filesystem, so the command can be built, shown to the +% user in a dialog, and asserted in a test with no display and no +% Python. When the launch then fails, the exact command is on screen +% to be copied into a terminal, which is the difference between a bug +% report and a shrug. +% +% THE PYRAMID IS ALWAYS NAMED. The viewer will refuse to open a +% session that holds more than one lightsheetZarrPyramid without being +% told which one, so this function refuses to build a command without +% PYRAMIDID. +% +% Optional Name-Value Arguments: +% channel (-1) - 1-based channel index to display. -1 opens all +% channels as separate napari layers. +% reduction ('') - request an alternate reduction: '' opens the +% pyramid the user picked; 'mean' or 'max' +% instructs the viewer to look for a sibling +% pyramid with that reduction and open that one +% instead. Empty leaves the choice to what the +% pyramid id names. +% controls (true) - dock the reduction / channel / level panels +% name ('') - image layer name. '' leaves the viewer to use +% the pyramid's own label. +% level (-1) - initial level to display. -1 lets the viewer +% pick based on the window size (levelTable is +% cheap; picking is not expensive). +% +% See also: ndi.gui.app.LightsheetZarrManager, +% ndi.fun.doc.lightsheet.chooseLevel + + arguments + launcher char + sessionPath char + pyramidID char + options.channel (1,1) double {mustBeInteger} = -1 + options.reduction char {mustBeMember(options.reduction, {'', 'mean', 'max'})} = '' + options.controls (1,1) logical = true + options.name char = '' + options.level (1,1) double {mustBeInteger} = -1 + end + + if isempty(strtrim(launcher)) + error('NDI:lightsheet:viewCommand:noLauncher', ... + ['No viewer launcher is set. It is normally ' ... + '/usr/local/bin/napariViewLightsheet, a shell wrapper ' ... + 'that scrubs MATLAB''s library paths before starting ' ... + 'Python.']); + end + if isempty(strtrim(sessionPath)) + error('NDI:lightsheet:viewCommand:noSessionPath', ... + ['This session has no directory on disk, so there is ' ... + 'nothing for the viewer to open. The napari viewer ' ... + 'reads an ndi.session.dir.']); + end + if isempty(strtrim(pyramidID)) + error('NDI:lightsheet:viewCommand:noPyramidID', ... + ['viewCommand refuses to build a command without a ' ... + 'lightsheetZarrPyramid id; the viewer will not guess ' ... + 'which pyramid to open when a session holds more than ' ... + 'one.']); + end + + parts = { ... + shellQuote(strtrim(launcher)), ... + shellQuote(sessionPath), ... + '--pyramid', shellQuote(pyramidID)}; + if ~isempty(options.reduction) + parts{end+1} = '--reduction'; + parts{end+1} = shellQuote(options.reduction); + end + if options.channel >= 1 + parts{end+1} = '--channel'; + parts{end+1} = sprintf('%d', options.channel); + end + if options.level >= 0 + parts{end+1} = '--level'; + parts{end+1} = sprintf('%d', options.level); + end + if ~options.controls + parts{end+1} = '--no-controls'; + end + if ~isempty(strtrim(options.name)) + parts{end+1} = '--name'; + parts{end+1} = shellQuote(strtrim(options.name)); + end + cmd = strjoin(parts, ' '); +end + +function s = shellQuote(str) + str = char(str); + if ispc + s = ['"' strrep(str, '"', '""') '"']; + else + s = ['''' strrep(str, '''', '''\''''') '''']; + end +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m new file mode 100644 index 000000000..77140dd06 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m @@ -0,0 +1,42 @@ +function [levelDoc, nWritten] = writeChunkFile(session, levelDoc, pyramidEntry, level) +% NDI.FUN.DOC.LIGHTSHEET.WRITECHUNKFILE - materialize a level's chunks into its file series +% +% [LEVELDOC, NWRITTEN] = NDI.FUN.DOC.LIGHTSHEET.WRITECHUNKFILE(... +% SESSION, LEVELDOC, PYRAMIDENTRY, LEVEL) +% +% Reads the OME-Zarr array at LEVEL.path (from a PYRAMIDENTRY as +% returned by NDR.FORMAT.OMEZARR.LISTPYRAMIDS) chunk by chunk and +% writes each chunk's raw byte payload to the LEVELDOC's +% `chunk.bin_#` file series, 1-based, C-order flatten against the +% chunk grid. Returns the (possibly re-fetched) LEVELDOC and the +% count of files actually written. +% +% THIS IS A SCAFFOLD ON MAIN. The end-state design writes only +% non-fill-value chunks and records their (1-based) chunk index in a +% small side-index that lives in the level document (chunk_index). +% Until that side-index lands, this scaffold writes ALL chunks +% sequentially in C-order (level 0 first chunk = `chunk.bin_1`) so a +% reader can find any chunk by `1 + ravel(index, chunk_grid, +% 'C-order')`. Sparse pyramids will waste space this way until the +% index is added. See README-lightsheet-zarr.md. +% +% Callers are: NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID when its option +% `materializeChunks` is true. Direct callers are welcome; the +% function is self-contained. + + arguments + session (1,1) %#ok + levelDoc (1,1) + pyramidEntry (1,1) struct + level (1,1) struct + end + + error('NDI:lightsheet:writeChunkFile:notImplemented', ... + ['This function is a documented scaffold. Materializing ' ... + 'chunk bytes across the HIPAA-compliant cloud API needs the ' ... + 'binary-file upload machinery that ' ... + 'ndi.cloud.sync.internal.uploadFilesForDatasetDocuments ' ... + 'provides, and a small chunk_index side-file that has not ' ... + 'been designed yet. Track the follow-up in README-' ... + 'lightsheet-zarr.md at the top of the +lightsheet package.']); +end diff --git a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m new file mode 100644 index 000000000..45f7d5923 --- /dev/null +++ b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m @@ -0,0 +1,236 @@ +classdef LightsheetZarrManager < ndi.gui.app.sessionApp +% NDI.GUI.APP.LIGHTSHEETZARRMANAGER - list, add, view and delete lightsheet OME-Zarr pyramids +% +% ndi.gui.app.LightsheetZarrManager(SESSION) +% +% Lists the lightsheetZarrPyramid documents in a session, grouped by +% subject and paired with their level counts, and lets the user add, +% open or remove one. Reached from the navigator's per-session Apps +% menu. +% +% WHAT IS LISTED IS PYRAMIDS, NOT ZARR STORES. An OME-Zarr store on +% disk is the input; what lands in the database is a +% lightsheetZarrPyramid plus one lightsheetZarrLevel per level, and +% optionally a fileReference. A single store frequently produces two +% parents (a mean pyramid and a max pyramid), listed as two rows. +% +% VIEWING IS SOMEONE ELSE'S JOB. The viewer is napari, which is +% Python, so View does not draw anything here: it builds a command +% line via NDI.FUN.DOC.LIGHTSHEET.VIEWCOMMAND and hands it to a +% launcher, by default /usr/local/bin/napariViewLightsheet. That +% launcher is a shell wrapper rather than the console script itself +% because MATLAB exports library paths of its own, and a Python +% process started from MATLAB picks up MATLAB's copies of libraries +% it must not use; the wrapper scrubs the environment before exec'ing +% the real entry point. viewCommand builds the command and is pure, +% so what will run can be shown in the dialog before it runs and can +% be copied into a terminal when it does not. +% +% DELETING IS A CASCADE. Levels depend on the pyramid, so removing a +% pyramid orphans everything beneath it. deletionPlan enumerates that +% first and the confirmation names the counts, because "delete 1 +% document" and "delete 32 documents" should not look the same. +% +% Example: +% S = ndi.session.dir('mysession','/path/to/session'); +% ndi.gui.app.LightsheetZarrManager(S); +% +% See also: ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.fun.doc.lightsheet.viewCommand, +% ndi.gui.app.GEFManager (the pattern this app mirrors) + + properties (Constant) + Name = "Lightsheet Zarr Manager" + Category = "Lightsheet microscopy" + + % Where the napari launcher lives when nobody has said otherwise. + % A wrapper in /usr/local/bin rather than the console script in + % a virtual environment: MATLAB exports library paths of its + % own, and a Python process started from MATLAB loads MATLAB's + % copies of libraries it must not use. The wrapper scrubs those + % and then execs the real entry point. Overridden by the + % preference GUI.LightsheetZarrManager.ViewerLauncher, which + % the View dialog writes. + DefaultViewerLauncher = "/usr/local/bin/napariViewLightsheet" + end + + properties (Access = private) + session + fig + table + statusLabel + rows = struct([]) + end + + methods + function obj = LightsheetZarrManager(sessionObj, options) + arguments + sessionObj (1,1) + options.build (1,1) logical = true + end + obj.session = sessionObj; + if options.build + obj.buildUI(); + obj.refresh(); + end + end + end + + methods (Access = private) + function buildUI(obj) + obj.fig = uifigure('Name', char(obj.Name), ... + 'Position', [100 100 900 500]); + g = uigridlayout(obj.fig, [3 1], 'RowHeight', {'fit', '1x', 'fit'}); + + % Top bar: Add / View / Delete + top = uigridlayout(g, [1 5], 'ColumnWidth', {80, 80, 80, '1x', 100}); + uibutton(top, 'Text', 'Add', 'ButtonPushedFcn', @(~,~) obj.onAdd()); + uibutton(top, 'Text', 'View', 'ButtonPushedFcn', @(~,~) obj.onView()); + uibutton(top, 'Text', 'Delete', 'ButtonPushedFcn', @(~,~) obj.onDelete()); + uilabel(top, 'Text', ''); + uibutton(top, 'Text', 'Refresh','ButtonPushedFcn', @(~,~) obj.refresh()); + + obj.table = uitable(g, ... + 'ColumnName', {'label','reduction','subject','n_levels','shape_level0','dtype','id'}, ... + 'ColumnEditable', false); + + obj.statusLabel = uilabel(g, 'Text', ''); + end + + function refresh(obj) + q = ndi.query('', 'isa', 'lightsheetZarrPyramid'); + docs = obj.session.database_search(q); + n = numel(docs); + rows = repmat(struct('label','','reduction','','subject','', ... + 'n_levels',0,'shape_level0','','dtype','','id',''), n, 1); + for k = 1:n + p = docs{k}.document_properties.lightsheetZarrPyramid; + rows(k).label = char(p.label); + rows(k).reduction = char(p.reduction); + rows(k).subject = subjectLabel(docs{k}); + rows(k).n_levels = p.n_levels; + rows(k).shape_level0 = mat2str(double(p.shape_level0)); + rows(k).dtype = char(p.dtype); + rows(k).id = docs{k}.id(); + end + obj.rows = rows; + obj.table.Data = struct2table(rows, 'AsArray', true); + obj.statusLabel.Text = sprintf('%d pyramids', n); + end + + function onAdd(obj) + [f, p] = uigetfile({'.zattrs', 'OME-Zarr .zattrs'; ... + '*', 'All Files (*)'}, ... + 'Select the OME-Zarr store .zattrs (or its containing folder)'); + if isequal(f, 0), return; end + if strcmp(f, '.zattrs') + zarrPath = p; + else + zarrPath = fullfile(p, f); + end + zarrPath = strip(zarrPath, 'right', filesep); + + subjectID = obj.pickSubject(); + if isempty(subjectID), return; end + + try + ndi.fun.doc.lightsheet.fromOMEZarr(obj.session, zarrPath, ... + 'subjectID', subjectID); + obj.refresh(); + catch ME + uialert(obj.fig, ME.message, 'Add failed'); + end + end + + function onView(obj) + row = obj.selectedRow(); + if isempty(row), return; end + sessionPath = obj.session.path; + launcher = ndi.gui.app.LightsheetZarrManager.launcherPath(); + + cmd = ndi.fun.doc.lightsheet.viewCommand(launcher, sessionPath, row.id); + if ~ispc, cmd = [cmd ' &']; else, cmd = ['start "napari" ' cmd]; end + [status, output] = system(cmd); + if status ~= 0 + uialert(obj.fig, sprintf('%s\n\n%s', cmd, output), ... + 'Launch failed'); + end + end + + function onDelete(obj) + row = obj.selectedRow(); + if isempty(row), return; end + plan = ndi.gui.app.LightsheetZarrManager.deletionPlan(obj.session, row.id); + msg = sprintf('Delete pyramid %s and %d level documents?', ... + row.label, plan.nLevels); + answer = uiconfirm(obj.fig, msg, 'Delete', ... + 'Options', {'Delete', 'Cancel'}, 'DefaultOption', 'Cancel'); + if ~strcmp(answer, 'Delete'), return; end + for k = 1:numel(plan.docsToRemove) + obj.session.database_rm(plan.docsToRemove{k}); + end + obj.refresh(); + end + + function row = selectedRow(obj) + sel = obj.table.Selection; + if isempty(sel) + uialert(obj.fig, 'Select a pyramid first.', 'No selection'); + row = []; + return; + end + row = obj.rows(sel(1)); + end + + function id = pickSubject(obj) + subs = obj.session.database_search(ndi.query('', 'isa', 'subject')); + if isempty(subs) + uialert(obj.fig, 'No subject documents in this session.', ... + 'No subject'); + id = ''; + return; + end + labels = cellfun(@(d) char(d.document_properties.subject.local_identifier), ... + subs, 'UniformOutput', false); + [idx, ok] = listdlg('PromptString', 'Subject:', ... + 'SelectionMode', 'single', 'ListString', labels); + if ~ok, id = ''; return; end + id = subs{idx}.id(); + end + end + + methods (Static) + function p = launcherPath() + p = char(ndi.gui.app.LightsheetZarrManager.DefaultViewerLauncher); + try + v = ndi.preferences.get('GUI.LightsheetZarrManager.ViewerLauncher'); + if ~isempty(char(v)), p = char(v); end + catch + end + end + + function plan = deletionPlan(session, pyramidID) + plan = struct(); + plan.pyramidID = char(pyramidID); + q1 = ndi.query('', 'isa', 'lightsheetZarrLevel'); + q2 = ndi.query('depends_on', 'depends_on', ... + 'lightsheetZarrPyramid_id', plan.pyramidID); + levelDocs = session.database_search(q1 & q2); + parentDoc = session.database_search(ndi.query('base.id', ... + 'exact_string', plan.pyramidID)); + plan.nLevels = numel(levelDocs); + plan.docsToRemove = [levelDocs(:); parentDoc(:)]; + end + end +end + +function s = subjectLabel(pyramidDoc) + s = ''; + try + dep = pyramidDoc.dependency_value('subject_id'); + if ~isempty(dep) + s = char(dep); + end + catch + end +end diff --git a/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json new file mode 100644 index 000000000..95c912f42 --- /dev/null +++ b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json @@ -0,0 +1,53 @@ +{ + "document_class": { + "definition": "$NDIDOCUMENTPATH\/data\/lightsheetZarrLevel.json", + "validation": "$NDISCHEMAPATH\/data\/lightsheetZarrLevel_schema.json", + "class_name": "lightsheetZarrLevel", + "property_list_name": "lightsheetZarrLevel", + "class_version": 1, + "superclasses": [ + { + "definition": "$NDIDOCUMENTPATH\/base.json" + } + ] + }, + "depends_on": [ + { + "name": "lightsheetZarrPyramid_id", + "value": "" + }, + { + "name": "subject_id", + "value": "" + }, + { + "name": "source_file_id", + "value": "" + } + ], + "files": { + "file_list": [ + "chunk.bin_#" + ] + }, + "lightsheetZarrLevel": { + "label": "", + "level": 0, + "reduction": "mean", + "axes_order": "tczyx", + "shape": [], + "chunks": [], + "chunk_grid": [], + "n_chunks_stored": 0, + "chunk_index_origin": 1, + "chunk_order": "C", + "dtype": "", + "fill_value": 0, + "byte_order": "little", + "codec": "raw", + "codec_params": "{}", + "voxel_size": [], + "voxel_size_units": "micrometer", + "translation": [] + } +} diff --git a/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.md b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.md new file mode 100644 index 000000000..4a2441398 --- /dev/null +++ b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.md @@ -0,0 +1,76 @@ +# lightsheetZarrLevel + +One resolution level of a `lightsheetZarrPyramid`. Owns the chunk bytes +for that level as a `chunk.bin_#` file series, plus enough metadata for +a reader to reconstruct the level as a lazy multiscale array. + +NDI documents are immutable, so each level is a document of its own. +Adding a new level (deeper coarsening after the fact) creates a new +`lightsheetZarrLevel` document, and the parent `lightsheetZarrPyramid` +records the new `n_levels` on its own next version. + +## What this document is + +- **The chunk container.** The `files.file_list` names one file series + `chunk.bin_#`, one file per stored chunk. Chunk index numbering is + 1-based to match NDI's file-series convention, and the mapping from + the file's 1-based number to the C-order chunk coordinate is defined + by `chunk_grid` and `chunk_order`. +- **Level-local metadata.** `shape`, `chunks`, `voxel_size`, and + `translation` describe this level as it sits inside the parent's + coordinate system. A viewer that wants only one level does not have + to touch the parent's fields. + +## Fields + +- `label` - human-readable name for the level (e.g. + `"mean level 2 (12.8um)"`). Empty allowed. +- `level` - 0-based level index within the parent pyramid. 0 is the + finest. +- `reduction` - the reduction of this level relative to level 0 + (`mean`, `max`). Redundant with the parent's field but stored here + so a level can be interpreted without loading the parent. +- `axes_order` - copied from the parent; carried here for + self-describing reads. +- `shape` - per-axis size of this level's array, in `axes_order`. +- `chunks` - per-axis chunk size, in `axes_order`. This is the NGFF / + Zarr chunk shape. +- `chunk_grid` - per-axis count of chunks along each axis = + `ceil(shape ./ chunks)`. +- `n_chunks_stored` - number of `chunk.bin_#` files that were written. + May be less than `prod(chunk_grid)` when trailing empty (fill-value) + chunks were elided. +- `chunk_index_origin` - the first `#` in the `chunk.bin_#` series. + Always 1 by convention (NDI file series are 1-based). +- `chunk_order` - the order in which the chunk grid is flattened to + the 1-based file series. `"C"` for C-order (last axis fastest); + `"F"` for Fortran-order. +- `dtype` - the Zarr dtype string (e.g. ` Date: Wed, 9 Sep 2026 10:08:55 -0400 Subject: [PATCH 002/110] readViewport scaffold: initialize outputs and mark unused args Code Analyzer flagged the outputs as possibly unset and the args as unused after the arguments block, both correct given the function is a documented scaffold that ends in error(). Initializing img/info before the error clears the return-value warning; %#ok and %#ok silence the unused-arg / unused-assignment findings that are load-bearing for the future implementation. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/readViewport.m | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m index bfe1a20af..37899f79a 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m @@ -41,13 +41,19 @@ % ndr.format.omezarr.readArray arguments - session (1,1) - pyramidDoc - region (1,1) struct + session (1,1) %#ok + pyramidDoc %#ok + region (1,1) struct %#ok options.targetVoxelSize (1,:) double = [] options.forceLevel (1,1) double {mustBeInteger} = -1 end + % Initialise so the return-value-might-be-unset warning is only about + % the *documented* scaffold state, not a real MATLAB unset. The two + % backends below will overwrite these before returning normally. + img = []; + info = struct(); + t = ndi.fun.doc.lightsheet.levelTable(session, pyramidDoc); if height(t) == 0 error('NDI:lightsheet:readViewport:noLevels', ... @@ -55,7 +61,7 @@ end if options.forceLevel >= 0 - row = t(t.level == options.forceLevel, :); + row = t(t.level == options.forceLevel, :); %#ok if isempty(row) error('NDI:lightsheet:readViewport:noSuchLevel', ... 'Requested level %d not present.', options.forceLevel); @@ -65,11 +71,11 @@ if isempty(target) target = t.voxel_size{1}; end - [~, row] = ndi.fun.doc.lightsheet.chooseLevel(t, target); + [~, row] = ndi.fun.doc.lightsheet.chooseLevel(t, target); %#ok end error('NDI:lightsheet:readViewport:notImplemented', ... ['readViewport is a scaffold in this PR. See README-' ... 'lightsheet-zarr.md for the two backend paths (source ' ... - 'OME-Zarr and materialized chunks) and their status.']); %#ok + 'OME-Zarr and materialized chunks) and their status.']); end From 9bbf2653888885f986442992244b471950f5d67b Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 10:09:11 -0400 Subject: [PATCH 003/110] writeChunkFile scaffold: initialize outputs and mark unused args Code Analyzer flagged the outputs as possibly unset and every arg as unused after the arguments block, both correct given the function is a documented scaffold that ends in error(). Initializing outputs before the error clears the return-value warning; %#ok on the input names silences the unused-arg findings that name the future signature. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m index 77140dd06..cd1f5e74a 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m @@ -27,10 +27,15 @@ arguments session (1,1) %#ok levelDoc (1,1) - pyramidEntry (1,1) struct - level (1,1) struct + pyramidEntry (1,1) struct %#ok + level (1,1) struct %#ok end + % Initialise so the return-value-might-be-unset warning is only about + % the *documented* scaffold state. The real implementation will + % rewrite levelDoc (with the file-series bumps) and set nWritten. + nWritten = 0; %#ok + error('NDI:lightsheet:writeChunkFile:notImplemented', ... ['This function is a documented scaffold. Materializing ' ... 'chunk bytes across the HIPAA-compliant cloud API needs the ' ... From 27a8f78920204c022dbb9c5c432cc10ec6967771 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 10:09:42 -0400 Subject: [PATCH 004/110] Drop unused nChunks assignment in lightsheet makePyramid.makeOneLevel Code Analyzer flagged it correctly: n_chunks_stored is currently recorded as 0 (materializeChunks path is a scaffold) so nChunks was never read. When materializeChunks lands, n_chunks_stored will be overwritten with what writeChunkFile actually wrote, not prod(chunk_grid). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 1 - 1 file changed, 1 deletion(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index c51a64742..3b055dcd2 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -105,7 +105,6 @@ function levelDoc = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, options) chunkGrid = ceil(level.shape ./ level.chunks); - nChunks = prod(chunkGrid); props = struct( ... 'label', sprintf('%s level %d', pyramidEntry.name, levelIndex), ... From 2207091d1c7071b45e94b2daf3e4928545a93e40 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 10:23:56 -0400 Subject: [PATCH 005/110] Drop the readViewport scaffold from the PR Both backends this function was to have (source-OME-Zarr and materialized chunk.bin_#) need work the PR does not contain, so its body was a raise-notImplemented that keeps flagging Code Analyzer findings on every push (initialise-outputs / unused-args). Removing the file is cleaner than growing the suppressions until it lands for real. levelTable + chooseLevel remain and are enough for a caller that wants to reason about levels without reading chunks. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/readViewport.m | 81 ------------------- 1 file changed, 81 deletions(-) delete mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m deleted file mode 100644 index 37899f79a..000000000 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/readViewport.m +++ /dev/null @@ -1,81 +0,0 @@ -function [img, info] = readViewport(session, pyramidDoc, region, options) -% NDI.FUN.DOC.LIGHTSHEET.READVIEWPORT - read a rectangle out of a pyramid -% -% [IMG, INFO] = NDI.FUN.DOC.LIGHTSHEET.READVIEWPORT(SESSION, -% PYRAMIDDOC, REGION, options) -% -% Reads a region of interest from a lightsheetZarrPyramid at whichever -% resolution level best matches the requested pixel size. -% -% PYRAMIDDOC is a lightsheetZarrPyramid ndi.document, or its id. -% REGION is a struct with fields matching the pyramid's axes_order -% (e.g. `t`, `c`, `z`, `y`, `x`), each a 1x2 [lo hi] pair in physical -% coordinates (units per pyramid.voxel_size_units). Missing fields -% mean "all of that axis". -% -% Optional Name-Value: -% targetVoxelSize - the coarsest per-axis voxel size that still -% satisfies the caller's rendering target. Passed -% to CHOOSELEVEL. Default: the finest level's -% voxel size (i.e. level 0). -% forceLevel - override the level choice; 0-based level index. -% -% Returns IMG (numeric array in the level's dtype) and INFO struct -% with fields `level`, `voxel_size`, `translation`, `axes_order`, -% `region_used_voxels`. -% -% THIS FUNCTION HAS TWO BACKENDS. -% 1. If the level document's `n_chunks_stored` is 0, the source -% OME-Zarr store is still authoritative and we read from it via -% NDR.FORMAT.OMEZARR.READARRAY on the file named by the -% fileReference. This is the path exercised by the napari -% viewer on a session that has been probed but not fully -% materialized. -% 2. If `n_chunks_stored` is > 0, we read the required chunks from -% the level document's `chunk.bin_#` file series through the -% NDI cloud API. That path is a scaffold today; see -% README-lightsheet-zarr.md. -% -% See also: ndi.fun.doc.lightsheet.chooseLevel, -% ndi.fun.doc.lightsheet.levelTable, -% ndr.format.omezarr.readArray - - arguments - session (1,1) %#ok - pyramidDoc %#ok - region (1,1) struct %#ok - options.targetVoxelSize (1,:) double = [] - options.forceLevel (1,1) double {mustBeInteger} = -1 - end - - % Initialise so the return-value-might-be-unset warning is only about - % the *documented* scaffold state, not a real MATLAB unset. The two - % backends below will overwrite these before returning normally. - img = []; - info = struct(); - - t = ndi.fun.doc.lightsheet.levelTable(session, pyramidDoc); - if height(t) == 0 - error('NDI:lightsheet:readViewport:noLevels', ... - 'Pyramid has no lightsheetZarrLevel documents.'); - end - - if options.forceLevel >= 0 - row = t(t.level == options.forceLevel, :); %#ok - if isempty(row) - error('NDI:lightsheet:readViewport:noSuchLevel', ... - 'Requested level %d not present.', options.forceLevel); - end - else - target = options.targetVoxelSize; - if isempty(target) - target = t.voxel_size{1}; - end - [~, row] = ndi.fun.doc.lightsheet.chooseLevel(t, target); %#ok - end - - error('NDI:lightsheet:readViewport:notImplemented', ... - ['readViewport is a scaffold in this PR. See README-' ... - 'lightsheet-zarr.md for the two backend paths (source ' ... - 'OME-Zarr and materialized chunks) and their status.']); -end From 037a4b3fe6562bb1bfe939df2061154f757b52ec Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 10:23:59 -0400 Subject: [PATCH 006/110] Drop the writeChunkFile scaffold from the PR Materializing a level's chunks needs the NDI cloud file-series upload plus a chunk_index side-field on lightsheetZarrLevel; that design is follow-up work. Ship makePyramid without the materializeChunks branch so the package has no notImplemented functions and no Code Analyzer suppressions on placeholder bodies. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+fun/+doc/+lightsheet/writeChunkFile.m | 47 ------------------- 1 file changed, 47 deletions(-) delete mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m deleted file mode 100644 index cd1f5e74a..000000000 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/writeChunkFile.m +++ /dev/null @@ -1,47 +0,0 @@ -function [levelDoc, nWritten] = writeChunkFile(session, levelDoc, pyramidEntry, level) -% NDI.FUN.DOC.LIGHTSHEET.WRITECHUNKFILE - materialize a level's chunks into its file series -% -% [LEVELDOC, NWRITTEN] = NDI.FUN.DOC.LIGHTSHEET.WRITECHUNKFILE(... -% SESSION, LEVELDOC, PYRAMIDENTRY, LEVEL) -% -% Reads the OME-Zarr array at LEVEL.path (from a PYRAMIDENTRY as -% returned by NDR.FORMAT.OMEZARR.LISTPYRAMIDS) chunk by chunk and -% writes each chunk's raw byte payload to the LEVELDOC's -% `chunk.bin_#` file series, 1-based, C-order flatten against the -% chunk grid. Returns the (possibly re-fetched) LEVELDOC and the -% count of files actually written. -% -% THIS IS A SCAFFOLD ON MAIN. The end-state design writes only -% non-fill-value chunks and records their (1-based) chunk index in a -% small side-index that lives in the level document (chunk_index). -% Until that side-index lands, this scaffold writes ALL chunks -% sequentially in C-order (level 0 first chunk = `chunk.bin_1`) so a -% reader can find any chunk by `1 + ravel(index, chunk_grid, -% 'C-order')`. Sparse pyramids will waste space this way until the -% index is added. See README-lightsheet-zarr.md. -% -% Callers are: NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID when its option -% `materializeChunks` is true. Direct callers are welcome; the -% function is self-contained. - - arguments - session (1,1) %#ok - levelDoc (1,1) - pyramidEntry (1,1) struct %#ok - level (1,1) struct %#ok - end - - % Initialise so the return-value-might-be-unset warning is only about - % the *documented* scaffold state. The real implementation will - % rewrite levelDoc (with the file-series bumps) and set nWritten. - nWritten = 0; %#ok - - error('NDI:lightsheet:writeChunkFile:notImplemented', ... - ['This function is a documented scaffold. Materializing ' ... - 'chunk bytes across the HIPAA-compliant cloud API needs the ' ... - 'binary-file upload machinery that ' ... - 'ndi.cloud.sync.internal.uploadFilesForDatasetDocuments ' ... - 'provides, and a small chunk_index side-file that has not ' ... - 'been designed yet. Track the follow-up in README-' ... - 'lightsheet-zarr.md at the top of the +lightsheet package.']); -end From 8f8af4e83359cd8a9922139f8f1fcbd0f70fb364 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 10:24:39 -0400 Subject: [PATCH 007/110] Drop the materializeChunks branch from makePyramid The writeChunkFile scaffold it called is removed in the previous commit as follow-up work. makePyramid now writes exactly what this PR supports: metadata (parent pyramid + one level per multiscales level) with n_chunks_stored = 0. When the file-series upload machinery lands, the option can come back alongside a real writeChunkFile. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 28 +++++-------------- 1 file changed, 7 insertions(+), 21 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 3b055dcd2..b79134af0 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -13,6 +13,11 @@ % % and adds them to SESSION. % +% Metadata only. n_chunks_stored is 0 on every level document in +% this PR; materializing chunk bytes into the level document's +% `chunk.bin_#` file series is follow-up work (see the package +% README). +% % Required Name-Value: % reduction - 'mean' or 'max' (or other future value); recorded on % the parent AND on every level. @@ -23,17 +28,6 @@ % sourceFileID - char, id of the fileReference document naming % the OME-Zarr store; ''=no source_file_id. % pipelineVersion - char, tag for the pyramid. -% materializeChunks - logical, default false. When true, this -% function writes each level's chunk bytes into -% the level document's `chunk.bin_#` file series -% by reading the source array through -% NDR.FORMAT.OMEZARR.READARRAY. When false, the -% level documents only carry metadata and the -% chunks stay in the source store; a downstream -% ingest job (see the design note in -% README-lightsheet-zarr.md) is responsible for -% materialization. Default false so the ingest is -% incremental: describe first, materialize later. % % Returns: % PYRAMIDDOC - the parent ndi.document @@ -41,8 +35,8 @@ % ordered finest-first (matching PYRAMIDENTRY.levels). % % See also: ndi.fun.doc.lightsheet.fromOMEZarr, -% ndi.fun.doc.lightsheet.writeChunkFile, -% ndi.fun.doc.lightsheet.readViewport +% ndi.fun.doc.lightsheet.levelTable, +% ndi.fun.doc.lightsheet.chooseLevel arguments session (1,1) @@ -52,7 +46,6 @@ options.elementID char = '' options.sourceFileID char = '' options.pipelineVersion char = '' - options.materializeChunks (1,1) logical = false end axesOrder = joinAxisNames(pyramidEntry.axes); @@ -135,13 +128,6 @@ levelDoc = levelDoc.set_dependency_value('source_file_id', options.sourceFileID); end - if options.materializeChunks - [levelDoc, nWritten] = ndi.fun.doc.lightsheet.writeChunkFile(... - session, levelDoc, pyramidEntry, level); - levelDoc = levelDoc.setproperties( ... - 'lightsheetZarrLevel.n_chunks_stored', nWritten); - end - session.database_add(levelDoc); end From 6a0ef064190408796c1e8a945aa5eed2401a330f Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 10:25:10 -0400 Subject: [PATCH 008/110] README: drop removed scaffolds; point at WDS/NDI-python readViewport and writeChunkFile were the scaffold pair removed from this PR; the Python-side companion is at Waltham-Data-Science/NDI-python (`ndi.gui.app.lightsheetZarr`), not VH-Lab/NDI-python. Reflect both in the API surface and the "what still needs building" section. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+lightsheet/README-lightsheet-zarr.md | 63 ++++++++++--------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md index 2a5c34e76..d4f6439fe 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md @@ -45,16 +45,11 @@ NDI documents are immutable, so: pick the coarsest level whose voxel size still meets the target. **Working.** - `ndi.fun.doc.lightsheet.viewCommand(launcher, sessionPath, pyramidID, ...)` - - pure command builder. Same pattern as `ndi.gui.app.GEFManager.viewCommand`; - the launcher is `/usr/local/bin/napariViewLightsheet`, a shell - wrapper around the napari console script that ships in NDI-python. - **Working.** -- `ndi.fun.doc.lightsheet.writeChunkFile(session, levelDoc, pyramidEntry, level)` - - materialize a level's chunks into its `chunk.bin_#` file series. - **Scaffold.** See "What still needs building" below. -- `ndi.fun.doc.lightsheet.readViewport(session, pyramidDoc, region, ...)` - - read a rectangle from the appropriate level. **Scaffold**; the - napari viewer in NDI-python is the primary reader. + - pure command builder. Same pattern as + `ndi.gui.app.GEFManager.viewCommand`; the launcher is + `/usr/local/bin/napariViewLightsheet`, a shell wrapper around the + `napariViewLightsheet` console script that ships in + `Waltham-Data-Science/NDI-python`. **Working.** ## GUI @@ -64,28 +59,34 @@ builds the napari command from `viewCommand` and shells out. ## What still needs building -1. **Chunk materialization across the HIPAA-compliant NDI cloud API.** - `writeChunkFile.m` is a scaffold today because the file-series - upload path we need is the one already used by - `ndi.cloud.sync.internal.uploadFilesForDatasetDocuments`, and - wiring it in here needs a small chunk-index side-file (level - document field `chunk_index`) so we can elide fill-value chunks - without breaking the 1-based `chunk.bin_#` addressing. Design and - land that first, then remove the `notImplemented` guard. -2. **NDI-python napari console script.** The MATLAB `viewCommand` - targets `/usr/local/bin/napariViewLightsheet`, a shell wrapper - around a Python console script that ships in NDI-python (see - `ndi/lightsheet/napari_view.py`). That console script is the piece - that actually reads the pyramid + level documents through the NDI - cloud API and hands lazy dask arrays to napari. -3. **`readViewport` MATLAB backend.** Two paths are already - documented: read the source store via `ndr.format.omezarr.readArray` - (works today, but only for the session that has the store on-disk), - or read `chunk.bin_#` from a level document through the cloud API - (blocked on #1). +1. **Chunk materialization** into `chunk.bin_#` on + `lightsheetZarrLevel`. Uses the same file-series upload path as + `ndi.cloud.sync.internal.uploadFilesForDatasetDocuments`, plus a + small `chunk_index` side-field so sparse pyramids can elide + fill-value chunks without breaking the 1-based `chunk.bin_#` + addressing. Land that side-field on the schema first, then a + `makePyramid` option (or a stand-alone `materializeChunks` + function) can write bytes. +2. **NDI-python napari reader.** The MATLAB `viewCommand` targets + `/usr/local/bin/napariViewLightsheet`, a shell wrapper around + `ndi.gui.app.lightsheetZarr.cli:main` in + `Waltham-Data-Science/NDI-python` (installed as + `[project.scripts].napariViewLightsheet`). The CLI, viewer, and + metadata reader are landed; the per-chunk dask fetcher that + `levelArrays` calls is a scaffold today. See that repo's + `src/ndi/gui/app/lightsheetZarr/README-lightsheet-zarr.md`. +3. **`readViewport` (MATLAB).** A MATLAB path for reading a rectangle + out of a pyramid at the appropriate level. Two backends are + plausible: read the source store via `ndr.format.omezarr.readArray` + (works today, but only when the store is on disk), or read the + materialized `chunk.bin_#` files via the cloud API (blocked on #1). + The scaffold from an earlier version of this PR was removed to + keep the package free of `notImplemented` bodies; add it back with + real code when #1 lands. 4. **NDR helpers.** `ndr.format.omezarr.probe` is small; when it lands in NDR-matlab / NDR-python, `fromOMEZarr` here can call it instead - of `listPyramids` for cheaper metadata reads. + of `listPyramids` for cheaper metadata reads. (Landing on the same + branch name in the NDR repos.) 5. **Tests.** Once #1 lands, extend `tests/+ndi/+unittest/+fun/+doc/+lightsheet/` with round-trip and viewport tests analogous to `+gene/`'s. @@ -94,7 +95,7 @@ builds the napari command from `viewCommand` and shells out. - `ndi.gui.app.GEFManager` - the sibling manager for Stereo-seq GEF spatial transcriptomics. Same launcher pattern (a shell wrapper - around a NDI-python console script). Read it first if this is your + around an NDI-python console script). Read it first if this is your first NDI file-format ingest. - `ndr.format.omezarr.*` - the metadata / read primitives this package builds on. From 6a031874ca92d923476eb1757d5ec70c0222abb7 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 10:33:04 -0400 Subject: [PATCH 009/110] Add the two missing _schema.json validators for the lightsheet doc pair Root cause of the TestGeneFromFiles cascade on this PR: every existing document class ships a validator at src/ndi/ndi_common/schema_documents/data/<class>_schema.json, and NDI's schema-registry walk fails hard when a document_class named in database_documents references a validator that does not exist on disk. That fails EVERY session.database_add(...) in the run, which is why tests that touch +gene (nothing to do with our diff) were all "Errored." at the same call site. Add lightsheetZarrPyramid_schema.json and lightsheetZarrLevel_schema.json, mirroring the shape of spatialGeneExpressionPyramid_schema.json and spatialGeneExpressionTiles_schema.json. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../data/lightsheetZarrLevel_schema.json | 46 ++++++ .../data/lightsheetZarrPyramid_schema.json | 150 ++++++++++++++++++ 2 files changed, 196 insertions(+) create mode 100644 src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json create mode 100644 src/ndi/ndi_common/schema_documents/data/lightsheetZarrPyramid_schema.json diff --git a/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json new file mode 100644 index 000000000..8f62323ff --- /dev/null +++ b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json @@ -0,0 +1,46 @@ +{ + "classname": "lightsheetZarrLevel", + "superclasses": [ + "base" + ], + "depends_on": [ + { + "name": "lightsheetZarrPyramid_id", + "mustbenotempty": 1 + }, + { + "name": "subject_id", + "mustbenotempty": 0 + }, + { + "name": "source_file_id", + "mustbenotempty": 0 + } + ], + "file": [ + { + "name": "chunk.bin_#", + "mustbenotempty": 0 + } + ], + "lightsheetZarrLevel": [ + {"name": "label", "type": "string", "default_value": "", "parameters": "", "queryable": 1, "documentation": "Human-readable name for this level. Empty allowed."}, + {"name": "level", "type": "integer", "default_value": 0, "parameters": [0, 100000, 0, 0], "queryable": 1, "documentation": "0-based level index within the parent pyramid. 0 is the finest."}, + {"name": "reduction", "type": "string", "default_value": "mean", "parameters": "", "queryable": 1, "documentation": "Reduction of this level relative to level 0 ('mean', 'max', ...)."}, + {"name": "axes_order", "type": "string", "default_value": "tczyx", "parameters": "", "queryable": 1, "documentation": "Copied from the parent pyramid."}, + {"name": "shape", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis size of this level's array, in axes_order."}, + {"name": "chunks", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis chunk size, in axes_order."}, + {"name": "chunk_grid", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis count of chunks along each axis = ceil(shape ./ chunks)."}, + {"name": "n_chunks_stored", "type": "integer", "default_value": 0, "parameters": [0, 100000000, 0, 0], "queryable": 1, "documentation": "Number of chunk.bin_# files written."}, + {"name": "chunk_index_origin", "type": "integer", "default_value": 1, "parameters": [0, 1, 0, 0], "queryable": 1, "documentation": "First # in the chunk.bin_# series. Always 1 by convention."}, + {"name": "chunk_order", "type": "string", "default_value": "C", "parameters": "", "queryable": 1, "documentation": "Flatten order: 'C' or 'F'."}, + {"name": "dtype", "type": "string", "default_value": "", "parameters": "", "queryable": 1, "documentation": "The Zarr dtype string."}, + {"name": "fill_value", "type": "double", "default_value": 0, "parameters": [-1e300, 1e300, 1, 1], "queryable": 1, "documentation": "Fill value for elided chunks."}, + {"name": "byte_order", "type": "string", "default_value": "little", "parameters": "", "queryable": 1, "documentation": "Byte order of the on-disk chunk bytes."}, + {"name": "codec", "type": "string", "default_value": "raw", "parameters": "", "queryable": 1, "documentation": "Zarr codec name ('raw', 'blosc', 'zstd')."}, + {"name": "codec_params", "type": "string", "default_value": "{}", "parameters": "", "queryable": 1, "documentation": "JSON-encoded string of the codec parameters."}, + {"name": "voxel_size", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis voxel size at this level."}, + {"name": "voxel_size_units", "type": "string", "default_value": "micrometer", "parameters": "", "queryable": 1, "documentation": "Unit for voxel_size."}, + {"name": "translation", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis translation at this level."} + ] +} diff --git a/src/ndi/ndi_common/schema_documents/data/lightsheetZarrPyramid_schema.json b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrPyramid_schema.json new file mode 100644 index 000000000..8d285a066 --- /dev/null +++ b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrPyramid_schema.json @@ -0,0 +1,150 @@ +{ + "classname": "lightsheetZarrPyramid", + "superclasses": [ + "base" + ], + "depends_on": [ + { + "name": "subject_id", + "mustbenotempty": 1 + }, + { + "name": "element_id", + "mustbenotempty": 0 + }, + { + "name": "source_file_id", + "mustbenotempty": 0 + } + ], + "lightsheetZarrPyramid": [ + { + "name": "label", + "type": "string", + "default_value": "", + "parameters": "", + "queryable": 1, + "documentation": "Human-readable name for this pyramid, shown in a manager UI. Empty allowed." + }, + { + "name": "reduction", + "type": "string", + "default_value": "mean", + "parameters": "", + "queryable": 1, + "documentation": "The downsampling function used between levels ('mean', 'max', ...). NDI documents are immutable, so a volume that ships both mean and max pyramids becomes two lightsheetZarrPyramid documents, one per reduction." + }, + { + "name": "pyramid_name", + "type": "string", + "default_value": "", + "parameters": "", + "queryable": 1, + "documentation": "The NGFF multiscales.name from the source store, if any. Descriptive." + }, + { + "name": "pyramid_type", + "type": "string", + "default_value": "", + "parameters": "", + "queryable": 1, + "documentation": "The NGFF multiscales.type field, verbatim. Descriptive only." + }, + { + "name": "axes_order", + "type": "string", + "default_value": "tczyx", + "parameters": "", + "queryable": 1, + "documentation": "The axis order carried in every child level (NGFF convention, e.g. 'tczyx', 'czyx', 'zyx')." + }, + { + "name": "axes_units", + "type": "matrix", + "default_value": [], + "parameters": "", + "queryable": 1, + "documentation": "Per-axis physical unit strings, aligned with axes_order." + }, + { + "name": "channel_names", + "type": "matrix", + "default_value": [], + "parameters": "", + "queryable": 1, + "documentation": "Display names of the channels, in the order of the 'c' axis." + }, + { + "name": "n_channels", + "type": "integer", + "default_value": 0, + "parameters": [0, 100000, 0, 0], + "queryable": 1, + "documentation": "Convenience count; equals the length of channel_names when set." + }, + { + "name": "n_levels", + "type": "integer", + "default_value": 0, + "parameters": [0, 100000, 0, 0], + "queryable": 1, + "documentation": "Number of lightsheetZarrLevel documents that should reference this pyramid." + }, + { + "name": "shape_level0", + "type": "matrix", + "default_value": [], + "parameters": [1, null], + "queryable": 1, + "documentation": "Shape of the finest resolution array, in axes_order." + }, + { + "name": "voxel_size_level0", + "type": "matrix", + "default_value": [], + "parameters": [1, null], + "queryable": 1, + "documentation": "Per-axis voxel size at level 0, in voxel_size_units." + }, + { + "name": "voxel_size_units", + "type": "string", + "default_value": "micrometer", + "parameters": "", + "queryable": 1, + "documentation": "The unit that voxel_size_level0 and each child level's voxel_size are expressed in." + }, + { + "name": "translation_level0", + "type": "matrix", + "default_value": [], + "parameters": [1, null], + "queryable": 1, + "documentation": "Per-axis translation at level 0." + }, + { + "name": "dtype", + "type": "string", + "default_value": "", + "parameters": "", + "queryable": 1, + "documentation": "The array dtype string as written in NGFF .zarray (e.g. ' Date: Wed, 9 Sep 2026 10:53:04 -0400 Subject: [PATCH 010/110] Use NaN (not null) for unbounded matrix parameters in the lightsheet schemas MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MATLAB's jsondecode accepts NaN as a JSON extension; every existing NDI schema that has an unbounded matrix dimension uses `[1,NaN]` (imageStack_parameters_schema.json, spatialGeneExpressionTiles_schema.json). The earlier push serialized this via a JSON round-trip which turned NaN into null; the loader then broke schema-registry validation, so every session.database_add(...) in the test run failed at the same call site — the visible cascade on TestGeneFromFiles was a symptom of this, not a per-test error. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../data/lightsheetZarrLevel_schema.json | 197 ++++++++++++++++-- .../data/lightsheetZarrPyramid_schema.json | 39 +++- 2 files changed, 208 insertions(+), 28 deletions(-) diff --git a/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json index 8f62323ff..5fa4125c5 100644 --- a/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json +++ b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json @@ -24,23 +24,184 @@ } ], "lightsheetZarrLevel": [ - {"name": "label", "type": "string", "default_value": "", "parameters": "", "queryable": 1, "documentation": "Human-readable name for this level. Empty allowed."}, - {"name": "level", "type": "integer", "default_value": 0, "parameters": [0, 100000, 0, 0], "queryable": 1, "documentation": "0-based level index within the parent pyramid. 0 is the finest."}, - {"name": "reduction", "type": "string", "default_value": "mean", "parameters": "", "queryable": 1, "documentation": "Reduction of this level relative to level 0 ('mean', 'max', ...)."}, - {"name": "axes_order", "type": "string", "default_value": "tczyx", "parameters": "", "queryable": 1, "documentation": "Copied from the parent pyramid."}, - {"name": "shape", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis size of this level's array, in axes_order."}, - {"name": "chunks", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis chunk size, in axes_order."}, - {"name": "chunk_grid", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis count of chunks along each axis = ceil(shape ./ chunks)."}, - {"name": "n_chunks_stored", "type": "integer", "default_value": 0, "parameters": [0, 100000000, 0, 0], "queryable": 1, "documentation": "Number of chunk.bin_# files written."}, - {"name": "chunk_index_origin", "type": "integer", "default_value": 1, "parameters": [0, 1, 0, 0], "queryable": 1, "documentation": "First # in the chunk.bin_# series. Always 1 by convention."}, - {"name": "chunk_order", "type": "string", "default_value": "C", "parameters": "", "queryable": 1, "documentation": "Flatten order: 'C' or 'F'."}, - {"name": "dtype", "type": "string", "default_value": "", "parameters": "", "queryable": 1, "documentation": "The Zarr dtype string."}, - {"name": "fill_value", "type": "double", "default_value": 0, "parameters": [-1e300, 1e300, 1, 1], "queryable": 1, "documentation": "Fill value for elided chunks."}, - {"name": "byte_order", "type": "string", "default_value": "little", "parameters": "", "queryable": 1, "documentation": "Byte order of the on-disk chunk bytes."}, - {"name": "codec", "type": "string", "default_value": "raw", "parameters": "", "queryable": 1, "documentation": "Zarr codec name ('raw', 'blosc', 'zstd')."}, - {"name": "codec_params", "type": "string", "default_value": "{}", "parameters": "", "queryable": 1, "documentation": "JSON-encoded string of the codec parameters."}, - {"name": "voxel_size", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis voxel size at this level."}, - {"name": "voxel_size_units", "type": "string", "default_value": "micrometer", "parameters": "", "queryable": 1, "documentation": "Unit for voxel_size."}, - {"name": "translation", "type": "matrix", "default_value": [], "parameters": [1, null], "queryable": 1, "documentation": "Per-axis translation at this level."} + { + "name": "label", + "type": "string", + "default_value": "", + "parameters": "", + "queryable": 1, + "documentation": "Human-readable name for this level (e.g. 'mean level 2 (12.8um)'). Empty allowed." + }, + { + "name": "level", + "type": "integer", + "default_value": 0, + "parameters": [ + 0, + 100000, + 0, + 0 + ], + "queryable": 1, + "documentation": "0-based level index within the parent pyramid. 0 is the finest resolution." + }, + { + "name": "reduction", + "type": "string", + "default_value": "mean", + "parameters": "", + "queryable": 1, + "documentation": "The reduction of this level relative to level 0 ('mean', 'max', ...). Redundant with the parent's field but stored here so a level can be interpreted without loading the parent." + }, + { + "name": "axes_order", + "type": "string", + "default_value": "tczyx", + "parameters": "", + "queryable": 1, + "documentation": "Copied from the parent pyramid; carried here for self-describing reads." + }, + { + "name": "shape", + "type": "matrix", + "default_value": [], + "parameters": [ + 1, + NaN + ], + "queryable": 1, + "documentation": "Per-axis size of this level's array, in axes_order." + }, + { + "name": "chunks", + "type": "matrix", + "default_value": [], + "parameters": [ + 1, + NaN + ], + "queryable": 1, + "documentation": "Per-axis chunk size, in axes_order. This is the NGFF / Zarr chunk shape." + }, + { + "name": "chunk_grid", + "type": "matrix", + "default_value": [], + "parameters": [ + 1, + NaN + ], + "queryable": 1, + "documentation": "Per-axis count of chunks along each axis = ceil(shape ./ chunks)." + }, + { + "name": "n_chunks_stored", + "type": "integer", + "default_value": 0, + "parameters": [ + 0, + 100000000, + 0, + 0 + ], + "queryable": 1, + "documentation": "Number of chunk.bin_# files that were written. May be less than prod(chunk_grid) when trailing empty (fill-value) chunks were elided." + }, + { + "name": "chunk_index_origin", + "type": "integer", + "default_value": 1, + "parameters": [ + 0, + 1, + 0, + 0 + ], + "queryable": 1, + "documentation": "The first # in the chunk.bin_# series. Always 1 by convention (NDI file series are 1-based)." + }, + { + "name": "chunk_order", + "type": "string", + "default_value": "C", + "parameters": "", + "queryable": 1, + "documentation": "The order in which the chunk grid is flattened to the 1-based file series. 'C' for C-order (last axis fastest); 'F' for Fortran-order." + }, + { + "name": "dtype", + "type": "string", + "default_value": "", + "parameters": "", + "queryable": 1, + "documentation": "The Zarr dtype string (e.g. ' Date: Wed, 9 Sep 2026 15:55:48 -0400 Subject: [PATCH 011/110] Restructure lightsheet pyramid: one parent per source volume, per-level reduction_function, dedupe shared level 0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Splits reduction across the two doc classes correctly. Before this change lightsheetZarrPyramid carried a single `reduction` field, and every reduction of a source volume was a separate parent with its own level ladder — which forced two full copies of level 0 for a store that ships mean + max pyramids (the common case), and level 0 is typically ~87% of pyramid storage. Also required rewriting the parent whenever levels or reductions were added, breaking the point of NDI document immutability. Corrected shape: - `lightsheetZarrPyramid` describes the source volume only. No reduction, no n_levels, no reductions_available — those are all ladder-derived and would force a delete-and-rewrite when the ladder grows. - `lightsheetZarrLevel` gains `reduction_function` (renamed from `reduction` for clarity — it names the function, not the outcome). Values: 'none' for the raw shared level 0, 'mean'/'max'/… for reduced levels. - Level 0 is deduped by NGFF `datasets[0].path` — one level doc even when multiple multiscales entries reference the same array. Discovery patterns: - What reductions does this pyramid have? `{l.reduction_function for l in levels(pyramid)} - {'none'}` - N levels for reduction R? `sorted(l for l in levels if l.reduction_function in {'none', R}, key=level)` - Add a new reduction later? Just write more level docs; no parent edit. `makeSourceFile` in the +lightsheet package is rewritten to mirror main's +gene version: `fileReference` by default (no file bound), `generic_file` only when `attachFile=true`. OME-Zarr stores are directory trees, often multi-GB, and the pyramid documents already carry every byte of them we care about — describing the source is the right default; attaching a copy would double the storage. Adds unit tests for the two pure helpers the codecov complaint on this PR named: `viewCommand` (command builder for the napari launcher) and `chooseLevel` (target-voxel-size level picker). Both are called from the manager GUI so their contract matters, and both are pure enough to test without a session or a display. Companion updates on the Python side (WDS/NDI-python#296) will re-vendor the schemas and thread `reduction_function` through the napari CLI's filter + `--reduction` swap. Coming in a separate push. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 102 +++++------ .../+ndi/+fun/+doc/+lightsheet/levelTable.m | 50 ++++-- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 152 ++++++++++------ .../+fun/+doc/+lightsheet/makeSourceFile.m | 166 ++++++++++++++---- .../data/lightsheetZarrLevel.json | 2 +- .../data/lightsheetZarrLevel.md | 68 +++---- .../data/lightsheetZarrPyramid.json | 2 - .../data/lightsheetZarrPyramid.md | 116 ++++++------ .../data/lightsheetZarrLevel_schema.json | 6 +- .../data/lightsheetZarrPyramid_schema.json | 27 +-- .../+fun/+doc/+lightsheet/TestChooseLevel.m | 96 ++++++++++ .../+fun/+doc/+lightsheet/TestViewCommand.m | 144 +++++++++++++++ 12 files changed, 665 insertions(+), 266 deletions(-) create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestChooseLevel.m create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestViewCommand.m diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 5dfbbb908..f9d2c219d 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -1,31 +1,30 @@ -function [pyramidDocs, levelDocs, info] = fromOMEZarr(session, zarrPath, options) +function [pyramidDoc, levelDocs, info] = fromOMEZarr(session, zarrPath, options) % NDI.FUN.DOC.LIGHTSHEET.FROMOMEZARR - build lightsheetZarrPyramid docs from an OME-Zarr store % -% [PYRAMIDDOCS, LEVELDOCS, INFO] = NDI.FUN.DOC.LIGHTSHEET.FROMOMEZARR(... +% [PYRAMIDDOC, LEVELDOCS, INFO] = NDI.FUN.DOC.LIGHTSHEET.FROMOMEZARR(... % SESSION, ZARRPATH) % % Walks an OME-Zarr (NGFF v0.4) store at ZARRPATH, enumerates its % multiscale pyramids with NDR.FORMAT.OMEZARR.LISTPYRAMIDS, and writes -% a LIGHTSHEETZARRPYRAMID document plus one LIGHTSHEETZARRLEVEL -% document per level to SESSION. +% ONE lightsheetZarrPyramid document (the source volume) plus one +% lightsheetZarrLevel document per unique level to SESSION. % -% ONE DOCUMENT PER REDUCTION. An OME-Zarr store frequently contains -% both a mean-reduced and a max-projected pyramid. This function writes -% ONE lightsheetZarrPyramid per multiscales entry it finds; a store -% with two entries produces two parents and their two ladders of -% levels. +% ONE DOCUMENT PER SOURCE VOLUME. The parent describes the source +% volume; reductions (mean, max, ...) live on the level children. +% A store with mean and max pyramids produces one parent, not two. % -% ONE DOCUMENT PER LEVEL. NDI documents are immutable, so each -% resolution level is its own lightsheetZarrLevel document. The level -% documents depend_on the parent pyramid document. +% LEVEL 0 IS DEDUPED. When two NGFF multiscales entries reference +% the same underlying array (typically level 0: their +% `datasets[0].path` is the same string), one level document is +% written -- with reduction_function='none' -- and both readers use +% it. Reduced levels are per-reduction and get +% reduction_function='mean'/'max'/... . % -% READING IS METADATA-ONLY. No chunk bytes are read here; the source -% store's .zattrs and per-level .zarray are enough to fill the -% documents. The chunk bytes belong to the source_file_id -% (fileReference) and are materialized into a level document's -% `chunk.bin_#` file series by NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID. -% Separating "probe and describe" from "materialize" keeps this -% function fast enough to run inside a GUI dialog. +% READING IS METADATA-ONLY. No chunk bytes are read here; the +% source store's .zattrs and per-level .zarray are enough to fill +% the documents. Materializing chunk bytes into a level document's +% `chunk.bin_#` file series is follow-up work (see the package +% README). % % Optional Name-Value Arguments: % subjectID - char/string, id of the subject depended on. If empty, @@ -37,22 +36,23 @@ % values. Empty means all. % reductionMap - containers.Map / struct mapping pyramid name (as % reported by listPyramids) or type to a reduction -% label. Recognised reductions: 'mean', 'max'. When a -% name is not in the map, `pyramid_type` is inspected -% first, then a fallback of 'mean' is used with a -% warning. +% label ('mean', 'max', ...). When a name is not in +% the map, `pyramid_type` is inspected first, then a +% fallback of 'mean' is used with a warning. % pipelineVersion - char/string tag written into % lightsheetZarrPyramid.pipeline_version. % sourceFileID - id of an existing fileReference document for % ZARRPATH. When empty, a new fileReference is created % via NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE. +% label - human-readable label written into the parent pyramid. % % INFO returns a struct with fields: -% zarrPath - absolute path resolved -% pyramids - the raw output of ndr.format.omezarr.listPyramids -% reductions - cellstr, one per created pyramid, giving the -% reduction assigned to it -% sourceFileID - id of the fileReference document used +% zarrPath - absolute path resolved +% pyramids - the raw output of ndr.format.omezarr.listPyramids +% reductions - cellstr, unique reductions present in the ladder +% sourceFileID - id of the fileReference document used +% sharedLevel0 - logical, true if level 0 was shared across +% reductions and deduped % % Example: % S = ndi.session.dir('mysession','/path/to/session'); @@ -75,13 +75,13 @@ options.reductionMap = struct() options.pipelineVersion char = '' options.sourceFileID char = '' + options.label char = '' end if isempty(strtrim(options.subjectID)) error('NDI:lightsheet:fromOMEZarr:noSubject', ... ['A lightsheetZarrPyramid depends_on a subject; pass ' ... - '''subjectID'' or use NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE ' ... - 'first and then this function.']); + '''subjectID''.']); end if ~isfolder(zarrPath) @@ -116,38 +116,33 @@ end pyramids = pyramids(keep); + n = numel(pyramids); + perEntryReduction = cell(n, 1); + for i = 1:n + perEntryReduction{i} = pickReduction(pyramids(i), options.reductionMap); + end + sourceFileID = options.sourceFileID; if isempty(sourceFileID) srcDoc = ndi.fun.doc.lightsheet.makeSourceFile(session, zarrPath); + session.database_add(srcDoc); sourceFileID = srcDoc.id(); end - n = numel(pyramids); - pyramidDocs = cell(n, 1); - levelDocsPerPyramid = cell(n, 1); - reductions = cell(n, 1); - - for i = 1:n - reduction = pickReduction(pyramids(i), options.reductionMap); - reductions{i} = reduction; - [pDoc, lDocs] = ndi.fun.doc.lightsheet.makePyramid(session, ... - pyramids(i), ... - 'reduction', reduction, ... - 'subjectID', options.subjectID, ... - 'elementID', options.elementID, ... - 'sourceFileID', sourceFileID, ... - 'pipelineVersion', options.pipelineVersion); - pyramidDocs{i} = pDoc; - levelDocsPerPyramid{i} = lDocs; - end - - levelDocs = vertcat(levelDocsPerPyramid{:}); + [pyramidDoc, levelDocs, sharedLevel0] = ndi.fun.doc.lightsheet.makePyramid( ... + session, pyramids, perEntryReduction, ... + 'subjectID', options.subjectID, ... + 'elementID', options.elementID, ... + 'sourceFileID', sourceFileID, ... + 'pipelineVersion', options.pipelineVersion, ... + 'label', options.label); info = struct( ... 'zarrPath', char(zarrPath), ... 'pyramids', pyramids, ... - 'reductions', {reductions}, ... - 'sourceFileID', sourceFileID); + 'reductions', {unique(perEntryReduction)}, ... + 'sourceFileID', sourceFileID, ... + 'sharedLevel0', sharedLevel0); end function r = pickReduction(pyramid, reductionMap) @@ -155,8 +150,7 @@ % % Preference: explicit map by name, then by type, then a heuristic: % pyramid.type == 'max' -> 'max', anything else -> 'mean'. Warns when -% falling back because pyramid.type is unreliable (the historical -% writer of this format sometimes labeled the mean pyramid 'gaussian'). +% falling back because pyramid.type is unreliable. r = ''; if ~isempty(reductionMap) if isstruct(reductionMap) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/levelTable.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/levelTable.m index 07af25941..bae7c634c 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/levelTable.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/levelTable.m @@ -1,18 +1,27 @@ -function t = levelTable(session, pyramidDoc) +function t = levelTable(session, pyramidDoc, options) % NDI.FUN.DOC.LIGHTSHEET.LEVELTABLE - table of a pyramid's levels, finest-first % % T = NDI.FUN.DOC.LIGHTSHEET.LEVELTABLE(SESSION, PYRAMIDDOC) +% T = NDI.FUN.DOC.LIGHTSHEET.LEVELTABLE(SESSION, PYRAMIDDOC, ... +% 'reduction', 'mean') % % Queries SESSION for every LIGHTSHEETZARRLEVEL document whose % `lightsheetZarrPyramid_id` depends_on matches PYRAMIDDOC.id(), and % returns a MATLAB table sorted by `level` ascending (finest-first). % -% Table columns: level, reduction, shape, chunks, chunk_grid, +% When the 'reduction' name-value is passed, the returned table only +% contains levels with reduction_function in {'none', } -- +% the shared raw level(s) plus the requested reduction. Omit to get +% every level (both mean and max in a store that ships both). +% +% Table columns: level, reduction_function, shape, chunks, chunk_grid, % n_chunks_stored, voxel_size, translation, dtype, id. % % UserData holds the parent pyramid's frame: % axes_order, voxel_size_level0, translation_level0, dtype, -% reduction, pyramid_name, id (parent id). +% pyramid_name, id (parent id), reductions (cellstr of the +% reduction_function values present in the full ladder, minus +% 'none'). % % PYRAMIDDOC may be an ndi.document or an id char/string; either is % accepted so this can be called from a GUI that only has the row's @@ -23,6 +32,7 @@ arguments session (1,1) pyramidDoc + options.reduction char = '' end if isa(pyramidDoc, 'ndi.document') @@ -43,13 +53,13 @@ docs = session.database_search(q1 & q2); if isempty(docs) t = emptyTable(); - t.Properties.UserData = frame(parentDoc); + t.Properties.UserData = frame(parentDoc, {}); return; end n = numel(docs); level = zeros(n, 1); - reduction = cell(n, 1); + reduction_function = cell(n, 1); shape = cell(n, 1); chunks = cell(n, 1); chunk_grid = cell(n, 1); @@ -62,7 +72,7 @@ for k = 1:n p = docs{k}.document_properties.lightsheetZarrLevel; level(k) = p.level; - reduction{k} = char(p.reduction); + reduction_function{k} = char(p.reduction_function); shape{k} = reshape(double(p.shape), 1, []); chunks{k} = reshape(double(p.chunks), 1, []); chunk_grid{k} = reshape(double(p.chunk_grid), 1, []); @@ -73,28 +83,44 @@ id{k} = docs{k}.id(); end - t = table(level, reduction, shape, chunks, chunk_grid, ... + reductionsPresent = setdiff(unique(reduction_function), {'none'}); + + if ~isempty(options.reduction) + keep = strcmp(reduction_function, 'none') | strcmp(reduction_function, options.reduction); + level = level(keep); + reduction_function = reduction_function(keep); + shape = shape(keep); + chunks = chunks(keep); + chunk_grid = chunk_grid(keep); + n_chunks_stored = n_chunks_stored(keep); + voxel_size = voxel_size(keep); + translation = translation(keep); + dtype = dtype(keep); + id = id(keep); + end + + t = table(level, reduction_function, shape, chunks, chunk_grid, ... n_chunks_stored, voxel_size, translation, dtype, id); t = sortrows(t, 'level'); - t.Properties.UserData = frame(parentDoc); + t.Properties.UserData = frame(parentDoc, reductionsPresent); end -function f = frame(parentDoc) +function f = frame(parentDoc, reductionsPresent) p = parentDoc.document_properties.lightsheetZarrPyramid; f = struct( ... 'axes_order', char(p.axes_order), ... 'voxel_size_level0', reshape(double(p.voxel_size_level0), 1, []), ... 'translation_level0', reshape(double(p.translation_level0), 1, []), ... 'dtype', char(p.dtype), ... - 'reduction', char(p.reduction), ... 'pyramid_name', char(p.pyramid_name), ... - 'id', parentDoc.id()); + 'id', parentDoc.id(), ... + 'reductions', {reductionsPresent}); end function t = emptyTable() t = table( ... zeros(0,1), cell(0,1), cell(0,1), cell(0,1), cell(0,1), ... zeros(0,1), cell(0,1), cell(0,1), cell(0,1), cell(0,1), ... - 'VariableNames', {'level','reduction','shape','chunks','chunk_grid', ... + 'VariableNames', {'level','reduction_function','shape','chunks','chunk_grid', ... 'n_chunks_stored','voxel_size','translation','dtype','id'}); end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index b79134af0..8eeb14ef9 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -1,38 +1,36 @@ -function [pyramidDoc, levelDocs] = makePyramid(session, pyramidEntry, options) -% NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID - write one reduction's pyramid documents +function [pyramidDoc, levelDocs, sharedLevel0] = makePyramid(session, pyramids, perEntryReduction, options) +% NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID - write one pyramid parent and its deduped level children % -% [PYRAMIDDOC, LEVELDOCS] = NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID(... -% SESSION, PYRAMIDENTRY, options) +% [PYRAMIDDOC, LEVELDOCS, SHAREDLEVEL0] = ... +% NDI.FUN.DOC.LIGHTSHEET.MAKEPYRAMID(SESSION, PYRAMIDS, ... +% PEREDUCTION, options) % -% Given ONE multiscales entry as returned by -% NDR.FORMAT.OMEZARR.LISTPYRAMIDS (fields: name, type, axes, levels), -% creates: +% Given the multiscales entries reported by +% NDR.FORMAT.OMEZARR.LISTPYRAMIDS and a per-entry reduction label, +% writes ONE lightsheetZarrPyramid parent (the source volume) and +% one lightsheetZarrLevel per UNIQUE array path across all entries. % -% * one LIGHTSHEETZARRPYRAMID document (the parent) -% * one LIGHTSHEETZARRLEVEL document per entry in PYRAMIDENTRY.levels +% Dedupe rule: two entries whose levels(k).path is the same NGFF +% string refer to the same underlying array. Typical case: level 0 +% shared across mean and max pyramids. One level document is +% written and its reduction_function is 'none'. % -% and adds them to SESSION. +% Non-shared levels get reduction_function equal to the reduction +% of the (single) entry that names them. % -% Metadata only. n_chunks_stored is 0 on every level document in -% this PR; materializing chunk bytes into the level document's -% `chunk.bin_#` file series is follow-up work (see the package -% README). -% -% Required Name-Value: -% reduction - 'mean' or 'max' (or other future value); recorded on -% the parent AND on every level. -% subjectID - char, depended-on subject id. +% PYRAMIDS - struct array from listPyramids +% PEREDUCTION - cellstr, one label per entry ('mean','max',...) % -% Optional Name-Value: -% elementID - char, depended-on element id (may be ''). -% sourceFileID - char, id of the fileReference document naming -% the OME-Zarr store; ''=no source_file_id. -% pipelineVersion - char, tag for the pyramid. +% Name-Value: +% subjectID - char, depended-on subject id +% elementID - char, depended-on element id (may be '') +% sourceFileID - char, id of the fileReference doc for the store +% pipelineVersion - char, tag for the pyramid +% label - char, human-readable label for the pyramid % -% Returns: -% PYRAMIDDOC - the parent ndi.document -% LEVELDOCS - column cell array of ndi.documents, one per level, -% ordered finest-first (matching PYRAMIDENTRY.levels). +% Metadata only. n_chunks_stored is 0 on every level document in +% this PR; materializing chunk bytes into `chunk.bin_#` is follow-up +% work. % % See also: ndi.fun.doc.lightsheet.fromOMEZarr, % ndi.fun.doc.lightsheet.levelTable, @@ -40,42 +38,70 @@ arguments session (1,1) - pyramidEntry (1,1) struct - options.reduction (1,:) char {mustBeMember(options.reduction, {'mean','max'})} + pyramids (:,1) struct + perEntryReduction (:,1) cell options.subjectID (1,:) char options.elementID char = '' options.sourceFileID char = '' options.pipelineVersion char = '' + options.label char = '' end - axesOrder = joinAxisNames(pyramidEntry.axes); - axesUnits = collectAxisUnits(pyramidEntry.axes); + if numel(pyramids) ~= numel(perEntryReduction) + error('NDI:lightsheet:makePyramid:reductionArity', ... + 'perEntryReduction (%d) must have one entry per pyramid (%d).', ... + numel(perEntryReduction), numel(pyramids)); + end - levels = pyramidEntry.levels; - if isempty(levels) - error('NDI:lightsheet:makePyramid:noLevels', ... - 'Pyramid ''%s'' has no levels.', pyramidEntry.name); + % Group levels by NGFF path. entries at the same path share bytes. + % pathReductions maps path -> set of reductions that use it. + pathIndex = containers.Map(); + pathOrder = {}; + pathReductions = containers.Map(); + representative = containers.Map(); % path -> struct(entryIdx, levelIdx) + for i = 1:numel(pyramids) + levels = pyramids(i).levels; + for k = 1:numel(levels) + p = char(levels(k).path); + if ~isKey(pathIndex, p) + pathIndex(p) = numel(pathOrder) + 1; + pathOrder{end+1} = p; %#ok + pathReductions(p) = {perEntryReduction{i}}; + representative(p) = struct('entryIdx', i, 'levelIdx', k); + else + cur = pathReductions(p); + if ~any(strcmp(cur, perEntryReduction{i})) + cur{end+1} = perEntryReduction{i}; %#ok + pathReductions(p) = cur; + end + end + end end - level0 = levels(1); - dtype = char(level0.dtype); + % Pick the source volume's identity from the first entry, first level. + firstEntry = pyramids(1); + firstLevel = firstEntry.levels(1); + axesOrder = joinAxisNames(firstEntry.axes); + axesUnits = collectAxisUnits(firstEntry.axes); + label = options.label; + if isempty(label) + label = firstEntry.name; + end pyramidDoc = session.newdocument('lightsheetZarrPyramid', ... 'lightsheetZarrPyramid', struct( ... - 'label', pyramidEntry.name, ... - 'reduction', options.reduction, ... - 'pyramid_name', char(pyramidEntry.name), ... - 'pyramid_type', char(pyramidEntry.type), ... + 'label', label, ... + 'pyramid_name', char(firstEntry.name), ... + 'pyramid_type', char(firstEntry.type), ... 'axes_order', axesOrder, ... 'axes_units', {axesUnits}, ... 'channel_names', {{}}, ... - 'n_channels', channelCount(level0.shape, axesOrder), ... - 'n_levels', numel(levels), ... - 'shape_level0', level0.shape, ... - 'voxel_size_level0', ensureRow(level0.scale), ... + 'n_channels', channelCount(firstLevel.shape, axesOrder), ... + 'shape_level0', firstLevel.shape, ... + 'voxel_size_level0', ensureRow(firstLevel.scale), ... 'voxel_size_units', 'micrometer', ... - 'translation_level0', ensureRow(level0.translation), ... - 'dtype', dtype, ... + 'translation_level0', ensureRow(firstLevel.translation), ... + 'dtype', char(firstLevel.dtype), ... 'pipeline_version', options.pipelineVersion, ... 'byte_order', 'little')); @@ -89,20 +115,36 @@ session.database_add(pyramidDoc); - levelDocs = cell(numel(levels), 1); - for k = 1:numel(levels) - levelDocs{k} = makeOneLevel(session, pyramidDoc, pyramidEntry, ... - levels(k), k - 1, options); + % Assign each unique path a 0-based level index in the ORDER it + % first appears (finest first, since listPyramids is finest-first). + levelDocs = cell(numel(pathOrder), 1); + sharedLevel0 = false; + for idx = 1:numel(pathOrder) + p = pathOrder{idx}; + rep = representative(p); + entry = pyramids(rep.entryIdx); + level = entry.levels(rep.levelIdx); + rlist = pathReductions(p); + if numel(rlist) > 1 + reductionFn = 'none'; + if idx == 1 + sharedLevel0 = true; + end + else + reductionFn = rlist{1}; + end + levelDocs{idx} = makeOneLevel(session, pyramidDoc, entry, ... + level, idx - 1, reductionFn, options); end end -function levelDoc = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, options) +function levelDoc = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, reductionFn, options) chunkGrid = ceil(level.shape ./ level.chunks); props = struct( ... - 'label', sprintf('%s level %d', pyramidEntry.name, levelIndex), ... + 'label', sprintf('%s level %d (%s)', pyramidEntry.name, levelIndex, reductionFn), ... 'level', levelIndex, ... - 'reduction', options.reduction, ... + 'reduction_function', reductionFn, ... 'axes_order', joinAxisNames(pyramidEntry.axes), ... 'shape', level.shape, ... 'chunks', level.chunks, ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makeSourceFile.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makeSourceFile.m index f4b1d46ee..801234aac 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makeSourceFile.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makeSourceFile.m @@ -1,48 +1,148 @@ -function doc = makeSourceFile(session, zarrPath) -% NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE - fileReference for an OME-Zarr store +function doc = makeSourceFile(session, filename, options) +% MAKESOURCEFILE - describe, or attach, the source OME-Zarr a pyramid came from % -% DOC = NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE(SESSION, ZARRPATH) +% DOC = ndi.fun.doc.lightsheet.MAKESOURCEFILE(SESSION, FILENAME) +% DOC = ndi.fun.doc.lightsheet.MAKESOURCEFILE(..., 'attachFile', true) % -% Creates and adds a FILEREFERENCE document naming the OME-Zarr store -% at ZARRPATH. NDI.FUN.DOC.LIGHTSHEET.FROMOMEZARR calls this to obtain -% the id it stores in `source_file_id` on the pyramid and level -% documents. +% Creates the document that lightsheetZarrPyramid points at through +% its source_file_id dependency, so a pyramid can name the store it +% was built from and a later reader can verify it is looking at the +% same bytes. % -% The store's per-chunk files are many, so this does not checksum the -% full tree; it records the directory itself as the reference, -% matching how spatialGeneExpression stores its GEF source. +% DESCRIBING IS NOT ATTACHING, and the default is to describe. An +% OME-Zarr store is a directory tree of tens to hundreds of GB; +% ingesting a copy of it alongside the pyramid derived from it would +% double the storage to hold bytes the pyramid already summarises. +% The document records the name, the size, the checksum and the dates +% -- enough to identify the file, verify it if it is still around, +% and say what a pyramid came from. +% +% The two cases are two classes, mirroring the +gene package's +% makeSourceFile after issue #981 taught the same lesson: +% +% attachFile false -> fileReference, which holds no file +% attachFile true -> generic_file, which holds generic_file.ext +% +% generic_file's schema declares generic_file.ext with mustbenotempty +% 1, so a generic_file without its file is not a lighter version of +% one, it is an invalid one. +% +% THE CHECKSUM IS THE PART WORTH HAVING. A filename tells you what +% somebody called a file; a checksum tells you whether the file you +% have now is the file the pyramid was built from. +% +% FILENAME may be a file or a directory. For an OME-Zarr store +% (a directory tree) the checksum, when requested, is computed over +% the store's `.zattrs` file, which is small and identifies the +% multiscales layout uniquely; the alternative -- walking every +% chunk file -- would cost the full pyramid read that the caller +% is trying to avoid. +% +% Optional Name-Value Arguments: +% checksum (true) - compute the MD5 identifying fingerprint. For a +% directory (an OME-Zarr store) this is the MD5 +% of .zattrs; for a plain file it is the MD5 +% of the file. Pass false to skip; the field is +% left empty rather than filled with a +% placeholder. +% attachFile (false) - ingest a copy of the file itself, and return a +% generic_file document rather than a +% fileReference. Only valid for a plain file -- +% attaching a directory tree is not supported. +% formatOntology ('') - an ontology term for the format, if you have +% one. Left EMPTY rather than guessed. +% documentID ('') - a document this file is about, for the +% class's own document_id dependency. +% +% Outputs: +% DOC - the ndi.document -- fileReference, or generic_file when +% attachFile is true -- NOT yet added to the database. The +% caller adds it, because it is normally created as part of a +% set whose dependencies have to be wired before any of them is +% stored. +% +% Example: +% src = ndi.fun.doc.lightsheet.makeSourceFile(S, '/data/vol.ome.zarr'); +% S.database_add(src); +% +% See also: ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.fun.doc.gene.makeSourceFile, +% ndi.fun.file.MD5 arguments session (1,1) - zarrPath (1,:) char + filename (1,:) char + options.checksum (1,1) logical = true + options.attachFile (1,1) logical = false + options.formatOntology (1,:) char = '' + options.documentID (1,:) char = '' end - if ~isfolder(zarrPath) - error('NDI:lightsheet:makeSourceFile:noSuchStore', ... - 'OME-Zarr store not found at %s.', zarrPath); + isDir = isfolder(filename); + isFile = ~isDir && isfile(filename); + if ~isDir && ~isFile + error('NDI:lightsheet:makeSourceFile:noSuchPath', ... + 'No file or directory at %s.', filename); end + if options.attachFile && isDir + error('NDI:lightsheet:makeSourceFile:cannotAttachDirectory', ... + ['attachFile is only valid for a plain file. An OME-Zarr ' ... + 'store is a directory tree and is described, not ingested.']); + end + + [~, base, ext] = fileparts(filename); + leafname = [base ext]; - d = dir(zarrPath); - d = d(~ismember({d.name}, {'.', '..'})); - totalBytes = sum([d.bytes]); - if isempty(totalBytes) - totalBytes = 0; + checksum = ''; + algorithm = ''; + if options.checksum + if isDir + zattrs = fullfile(filename, '.zattrs'); + if isfile(zattrs) + checksum = ndi.fun.file.MD5(zattrs); + end + else + checksum = ndi.fun.file.MD5(filename); + end + if ~isempty(checksum) + algorithm = 'MD5'; + end end - [absPath, ~, ~] = fileparts(fullfile(zarrPath, '.')); - [~, storeName, storeExt] = fileparts(strip(absPath, 'right', filesep)); - fileName = [storeName storeExt]; + created = 0; + updated = 0; + try + created = convertTo(ndi.fun.file.dateCreated(filename), 'datenum'); + updated = convertTo(ndi.fun.file.dateUpdated(filename), 'datenum'); + catch + end - doc = session.newdocument('fileReference', ... - 'fileReference', struct( ... - 'filename', fileName, ... - 'originalPath', absPath, ... - 'formatOntology', 'OME-NGFF v0.4 (OME-Zarr)', ... - 'dateCreated', posixtime(datetime('now', 'TimeZone', 'UTC')), ... - 'dateUpdated', posixtime(datetime('now', 'TimeZone', 'UTC')), ... - 'fileSize', totalBytes, ... - 'checksum', '', ... - 'checksumAlgorithm', 'MD5')); + if options.attachFile + s = struct('filename', leafname, ... + 'formatOntology', options.formatOntology, ... + 'dateCreated', created, 'dateUpdated', updated, ... + 'checksum', checksum); + doc = ndi.document('generic_file', 'generic_file', s) + session.newdocument(); + doc = doc.add_file('generic_file.ext', filename, 'delete_original', 0); + else + fileSize = 0; + if isFile + d = dir(filename); + if ~isempty(d) + fileSize = double(d(1).bytes); + end + end + s = struct('filename', leafname, ... + 'originalPath', filename, ... + 'formatOntology', options.formatOntology, ... + 'dateCreated', created, 'dateUpdated', updated, ... + 'fileSize', fileSize, ... + 'checksum', checksum, ... + 'checksumAlgorithm', algorithm); + doc = ndi.document('fileReference', 'fileReference', s) + session.newdocument(); + end - session.database_add(doc); + if ~isempty(options.documentID) + doc = doc.set_dependency_value('document_id', options.documentID); + end end diff --git a/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json index 95c912f42..42461b67c 100644 --- a/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json +++ b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json @@ -33,7 +33,7 @@ "lightsheetZarrLevel": { "label": "", "level": 0, - "reduction": "mean", + "reduction_function": "none", "axes_order": "tczyx", "shape": [], "chunks": [], diff --git a/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.md b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.md index 4a2441398..f14e93b4f 100644 --- a/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.md +++ b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.md @@ -5,9 +5,16 @@ for that level as a `chunk.bin_#` file series, plus enough metadata for a reader to reconstruct the level as a lazy multiscale array. NDI documents are immutable, so each level is a document of its own. -Adding a new level (deeper coarsening after the fact) creates a new -`lightsheetZarrLevel` document, and the parent `lightsheetZarrPyramid` -records the new `n_levels` on its own next version. +Adding a new reduction (a `max` pyramid alongside a mean one, for +example) means writing more `lightsheetZarrLevel` documents — no parent +edit, no rewrite. The parent describes the source volume; the ladder is +discovered from its children. + +Level 0 is often shared across reductions: two NGFF multiscales entries +(mean and max) whose `datasets[0].path` is the same string are the same +underlying array. This document class represents that honestly — one +level doc with `reduction_function: 'none'` — instead of writing the +raw bytes twice. ## What this document is @@ -23,52 +30,53 @@ records the new `n_levels` on its own next version. ## Fields -- `label` - human-readable name for the level (e.g. +- `label` — human-readable name for the level (e.g. `"mean level 2 (12.8um)"`). Empty allowed. -- `level` - 0-based level index within the parent pyramid. 0 is the +- `level` — 0-based level index within the parent pyramid. 0 is the finest. -- `reduction` - the reduction of this level relative to level 0 - (`mean`, `max`). Redundant with the parent's field but stored here - so a level can be interpreted without loading the parent. -- `axes_order` - copied from the parent; carried here for +- `reduction_function` — the function used to produce this level from + level 0: `'mean'`, `'max'`, ... `'none'` means the array is the raw + source (typical for level 0 when it is shared across reductions). A + reader asking for reduction `R` takes levels with + `reduction_function in {'none', R}`, sorted by `level` ascending. +- `axes_order` — copied from the parent; carried here for self-describing reads. -- `shape` - per-axis size of this level's array, in `axes_order`. -- `chunks` - per-axis chunk size, in `axes_order`. This is the NGFF / +- `shape` — per-axis size of this level's array, in `axes_order`. +- `chunks` — per-axis chunk size, in `axes_order`. This is the NGFF / Zarr chunk shape. -- `chunk_grid` - per-axis count of chunks along each axis = +- `chunk_grid` — per-axis count of chunks along each axis = `ceil(shape ./ chunks)`. -- `n_chunks_stored` - number of `chunk.bin_#` files that were written. +- `n_chunks_stored` — number of `chunk.bin_#` files that were written. May be less than `prod(chunk_grid)` when trailing empty (fill-value) chunks were elided. -- `chunk_index_origin` - the first `#` in the `chunk.bin_#` series. +- `chunk_index_origin` — the first `#` in the `chunk.bin_#` series. Always 1 by convention (NDI file series are 1-based). -- `chunk_order` - the order in which the chunk grid is flattened to +- `chunk_order` — the order in which the chunk grid is flattened to the 1-based file series. `"C"` for C-order (last axis fastest); `"F"` for Fortran-order. -- `dtype` - the Zarr dtype string (e.g. ` 1). + idx = ndi.fun.doc.lightsheet.chooseLevel(L, [1 3 3]); + testCase.verifyEqual(idx, 1); + end + + function testTableInput(testCase) + t = ndi.unittest.fun.doc.lightsheet.TestChooseLevel.fourLevelTable(); + idx = ndi.fun.doc.lightsheet.chooseLevel(t, 2.0); + testCase.verifyEqual(idx, 2); + end + + function testTableInputReturnsRowNotStruct(testCase) + t = ndi.unittest.fun.doc.lightsheet.TestChooseLevel.fourLevelTable(); + [~, row] = ndi.fun.doc.lightsheet.chooseLevel(t, 2.0); + testCase.verifyClass(row, 'table'); + testCase.verifyEqual(height(row), 1); + end + + function testEmptyLevelsErrors(testCase) + testCase.verifyError( ... + @() ndi.fun.doc.lightsheet.chooseLevel( ... + struct('level',{},'voxel_size',{}), 1.0), ... + 'NDI:lightsheet:chooseLevel:emptyLevels'); + end + + function testTargetShorterThanVoxelUsesShorterAxes(testCase) + % A 3-D level with a scalar target checks every axis; a + % 3-D level with a 2-D target checks the first two axes + % only. Documents the behavior that a reader passing a + % 2-D XY target against a ZYX level compares X and Y and + % ignores Z. + L = ndi.unittest.fun.doc.lightsheet.TestChooseLevel.fourLevelPyramid(); + idx = ndi.fun.doc.lightsheet.chooseLevel(L, [2 2]); + testCase.verifyEqual(idx, 2); + end + end +end diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestViewCommand.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestViewCommand.m new file mode 100644 index 000000000..39163d67f --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestViewCommand.m @@ -0,0 +1,144 @@ +classdef TestViewCommand < matlab.unittest.TestCase + % TestViewCommand - the pure command builder ndi.gui.app.LightsheetZarrManager + % calls to hand a pyramid to the napari viewer. + % + % viewCommand does not launch anything or touch the filesystem, so a + % test can assert the exact command string without a display, without + % Python, and without a session. Every branch of the flag surface is + % exercised so a UI change that quietly drops --no-controls or + % --reduction fails here rather than in a demo. + + properties (Constant) + Launcher = '/usr/local/bin/napariViewLightsheet' + Session = '/tmp/session1' + Pyramid = '4126a1b2c3d4e5f6_a1b2c3d4e5f6a1b2' + end + + methods (Test) + + function testMinimalCommand(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid); + testCase.verifyEqual(cmd, ... + sprintf('''%s'' ''%s'' --pyramid ''%s''', ... + testCase.Launcher, testCase.Session, testCase.Pyramid)); + end + + function testReductionFlag(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid, ... + 'reduction', 'max'); + testCase.verifyTrue(contains(cmd, '--reduction ''max''')); + end + + function testReductionMeanFlag(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid, ... + 'reduction', 'mean'); + testCase.verifyTrue(contains(cmd, '--reduction ''mean''')); + end + + function testChannelFlag(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid, ... + 'channel', 2); + testCase.verifyTrue(contains(cmd, '--channel 2')); + end + + function testChannelDefaultIsOmitted(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid); + testCase.verifyFalse(contains(cmd, '--channel'), ... + 'channel=-1 (default) must not be emitted.'); + end + + function testLevelFlag(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid, ... + 'level', 0); + testCase.verifyTrue(contains(cmd, '--level 0')); + end + + function testLevelDefaultIsOmitted(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid); + testCase.verifyFalse(contains(cmd, '--level'), ... + 'level=-1 (default) must not be emitted.'); + end + + function testNoControls(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid, ... + 'controls', false); + testCase.verifyTrue(contains(cmd, '--no-controls')); + end + + function testControlsDefaultIsOn(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid); + testCase.verifyFalse(contains(cmd, '--no-controls'), ... + 'controls=true (default) must not emit --no-controls.'); + end + + function testNameFlag(testCase) + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid, ... + 'name', 'my layer'); + testCase.verifyTrue(contains(cmd, '--name')); + testCase.verifyTrue(contains(cmd, 'my layer'), ... + 'the layer name must appear in the command.'); + end + + function testEmptyLauncherErrors(testCase) + testCase.verifyError( ... + @() ndi.fun.doc.lightsheet.viewCommand('', ... + testCase.Session, testCase.Pyramid), ... + 'NDI:lightsheet:viewCommand:noLauncher'); + end + + function testEmptySessionPathErrors(testCase) + testCase.verifyError( ... + @() ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, '', testCase.Pyramid), ... + 'NDI:lightsheet:viewCommand:noSessionPath'); + end + + function testEmptyPyramidIDErrors(testCase) + % viewCommand refuses to build a command without a pyramid id; + % the viewer will not guess which pyramid to open on a session + % that holds more than one. + testCase.verifyError( ... + @() ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, ''), ... + 'NDI:lightsheet:viewCommand:noPyramidID'); + end + + function testInvalidReductionErrors(testCase) + testCase.verifyError( ... + @() ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid, ... + 'reduction', 'median'), ... + 'MATLAB:validators:mustBeMember'); + end + + function testFullFlagSurface(testCase) + % Every flag together; order matters for a shell command, and + % downstream (napariViewLightsheet argparse) accepts the flags + % in any order, so the assertion is on presence + values. + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + testCase.Launcher, testCase.Session, testCase.Pyramid, ... + 'reduction', 'max', 'channel', 1, 'level', 2, ... + 'controls', false, 'name', 'V1'); + testCase.verifyTrue(startsWith(cmd, ... + sprintf('''%s'' ''%s'' --pyramid ''%s''', ... + testCase.Launcher, testCase.Session, testCase.Pyramid)), ... + 'launcher, session, --pyramid must lead the command.'); + testCase.verifyTrue(contains(cmd, '--reduction ''max''')); + testCase.verifyTrue(contains(cmd, '--channel 1')); + testCase.verifyTrue(contains(cmd, '--level 2')); + testCase.verifyTrue(contains(cmd, '--no-controls')); + testCase.verifyTrue(contains(cmd, '--name')); + testCase.verifyTrue(contains(cmd, 'V1')); + end + end +end From 1af66bbe69348d01b8623fa3cfa197986e4fca32 Mon Sep 17 00:00:00 2001 From: Steve Van Hooser Date: Wed, 9 Sep 2026 16:01:23 -0400 Subject: [PATCH 012/110] Address Code Analyzer: paren-index perEntryReduction instead of brace-wrap GHAS flagged `{ A{I} }` on line 69. It's the initial-insert path of the path->reductions map: the value is a 1-cell subarray, and `perEntryReduction(i)` returns exactly that without the extra brace unwrap+rewrap. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 8eeb14ef9..58d1af37d 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -66,7 +66,7 @@ if ~isKey(pathIndex, p) pathIndex(p) = numel(pathOrder) + 1; pathOrder{end+1} = p; %#ok - pathReductions(p) = {perEntryReduction{i}}; + pathReductions(p) = perEntryReduction(i); representative(p) = struct('entryIdx', i, 'levelIdx', k); else cur = pathReductions(p); From c8f3479d82391a6a47bcc5017ce76fdc28dbc8ae Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 23:32:55 +0000 Subject: [PATCH 013/110] Re-tile OME-Zarr on ingest to an 8 MB per-chunk byte budget The source store's chunk shape is thrown away and each level gets a freshly chosen chunk shape sized to a tileBudgetBytes budget (default 8*2^20 = 8 MB uncompressed). The chooser sizes tiles ISOTROPIC IN WORLD SPACE: each spatial axis gets chunk_i inversely proportional to voxel_i, so a lightsheet volume with 1 um XY and 5 um Z produces pancake tiles rather than voxel cubes. Non-spatial axes (t, c) stay at 1. Across a dyadic pyramid this keeps chunk-in-voxels constant, so chunk_grid halves per axis per level and the tile count drops by 8x per level for a 3D ladder. Coarse levels whose shape has fallen below the ideal tile fall through to a single chunk. 8 MB is sized for a viewer over an ~200 MB/s connection fetching ~4 tiles in parallel per pan gesture: each tile lands in ~20 ms on wire, comfortably below the ~200 ms latency budget with room for HTTP overhead. The number is exposed as tileBudgetBytes on both makePyramid and fromOMEZarr; a caller can also pass explicit `chunks` to override the budget entirely. chooseTileShape landed as its own public function so the Python side can port it directly and the test suite can exercise it without a session; TestChooseTileShape covers the isotropic and anisotropic sweet spots, dtype scaling (uint8/uint16/float32), non-spatial axes, clamping-plus-redistribution at coarse levels, the numpy ' Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+lightsheet/README-lightsheet-zarr.md | 6 + .../+fun/+doc/+lightsheet/chooseTileShape.m | 129 +++++++++++++ .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 12 +- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 51 +++++- .../+doc/+lightsheet/TestChooseTileShape.m | 173 ++++++++++++++++++ 5 files changed, 368 insertions(+), 3 deletions(-) create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestChooseTileShape.m diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md index d4f6439fe..9055e2130 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md @@ -44,6 +44,12 @@ NDI documents are immutable, so: - `ndi.fun.doc.lightsheet.chooseLevel(levels, targetVoxelSize)` - pick the coarsest level whose voxel size still meets the target. **Working.** +- `ndi.fun.doc.lightsheet.chooseTileShape(shape, axesOrder, voxelSize, dtype, budgetBytes)` - + size a chunk to a byte budget with tiles isotropic in world space. + Default budget 8 MB uncompressed, sized for a viewer over an + ~200 MB/s link fetching ~4 tiles in parallel per pan gesture. Used + by `makePyramid` per level; the source store's chunk shape is not + preserved. **Working.** - `ndi.fun.doc.lightsheet.viewCommand(launcher, sessionPath, pyramidID, ...)` - pure command builder. Same pattern as `ndi.gui.app.GEFManager.viewCommand`; the launcher is diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m new file mode 100644 index 000000000..70194ebac --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m @@ -0,0 +1,129 @@ +function chunks = chooseTileShape(shape, axesOrder, voxelSize, dtype, budgetBytes) +% NDI.FUN.DOC.LIGHTSHEET.CHOOSETILESHAPE - pick a chunk shape that hits a byte budget +% +% CHUNKS = NDI.FUN.DOC.LIGHTSHEET.CHOOSETILESHAPE( ... +% SHAPE, AXESORDER, VOXELSIZE, DTYPE, BUDGETBYTES) +% +% Sizes an OME-Zarr chunk to land near BUDGETBYTES (uncompressed) +% and to be ISOTROPIC IN WORLD SPACE: each spatial axis gets a +% chunk length inversely proportional to its voxel size, so a +% lightsheet volume with 1 um XY and 5 um Z produces pancake tiles +% rather than voxel cubes. +% +% SHAPE - row vector, axis lengths in AXESORDER +% AXESORDER - char row like 'tczyx' (one letter per axis) +% VOXELSIZE - row vector, one per axis, same order (may include +% zeros or NaNs on t/c) +% DTYPE - char, NGFF-style ('uint16', 'float32', ' [161 161 161] % ~161^3 * 2 bytes ~= 8.3 MB +% +% Example (uint16, 8 MB, anisotropic 1/1/5 um): +% chooseTileShape([1200 2000 2000], 'zyx', [5 1 1], 'uint16', ... +% 8*2^20) +% -> [55 275 275] % ~55*275*275*2 ~= 7.9 MB, world-cube +% +% See also: ndi.fun.doc.lightsheet.makePyramid + + arguments + shape (1,:) double {mustBePositive, mustBeInteger} + axesOrder (1,:) char + voxelSize double + dtype (1,:) char + budgetBytes (1,1) double {mustBePositive} = 8 * 2^20 + end + + shape = double(shape); + n = numel(shape); + if numel(axesOrder) ~= n + error('NDI:lightsheet:chooseTileShape:axesShapeMismatch', ... + 'axes_order length %d does not match shape length %d.', ... + numel(axesOrder), n); + end + + isSpatial = ~ismember(lower(axesOrder(1:n)), {'t','c'}); + isSpatial = reshape(isSpatial, 1, n); + chunks = ones(1, n); + if ~any(isSpatial) + chunks = min(chunks, shape); + return; + end + + vs = reshape(double(voxelSize), 1, []); + if numel(vs) ~= n || any(vs(isSpatial) <= 0) || any(~isfinite(vs(isSpatial))) + % No usable voxel size on spatial axes -- fall back to + % isotropic in voxel space (a raw n-th root of the budget). + vs = ones(1, n); + end + + bpv = bytesPerVoxel(dtype); + voxelsPerTile = budgetBytes / bpv; + if voxelsPerTile < 1 + voxelsPerTile = 1; + end + + vsSpatial = vs(isSpatial); + ns = numel(vsSpatial); + s = (voxelsPerTile * prod(vsSpatial))^(1/ns); + chunks(isSpatial) = max(1, round(s ./ vsSpatial)); + chunks = min(chunks, shape); + + % One redistribution pass: if any spatial axis clamped to its + % shape, spread the leftover budget across the axes still below + % their shape. Cheap and useful at coarser pyramid levels where + % one dim collapses first. + slack = isSpatial & (chunks < shape); + stuck = isSpatial & ~slack; + if any(slack) && any(stuck) + used = prod(chunks(stuck)); + if used < 1, used = 1; end + remaining = voxelsPerTile / used; + vsSlack = vs(slack); + s2 = (remaining * prod(vsSlack))^(1/numel(vsSlack)); + chunks(slack) = max(1, round(s2 ./ vsSlack)); + chunks = min(chunks, shape); + end +end + +function b = bytesPerVoxel(dtype) + d = lower(strtrim(char(dtype))); + if ~isempty(d) && any(d(1) == '<>|=') + d = d(2:end); + end + switch d + case {'uint8','int8','u1','i1','bool','?'} + b = 1; + case {'uint16','int16','u2','i2','float16','f2','half'} + b = 2; + case {'uint32','int32','u4','i4','float32','f4','single'} + b = 4; + case {'uint64','int64','u8','i8','float64','f8','double'} + b = 8; + case {'complex64','c8'} + b = 8; + case {'complex128','c16'} + b = 16; + otherwise + % Unknown dtype: assume uint16 (2 bytes), the common + % lightsheet case. Better than raising for a novel probe. + b = 2; + end +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index f9d2c219d..706586df7 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -45,6 +45,12 @@ % ZARRPATH. When empty, a new fileReference is created % via NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE. % label - human-readable label written into the parent pyramid. +% tileBudgetBytes - target uncompressed bytes per chunk when chunk +% shape is chosen automatically. Default 8*2^20 +% (8 MB). Forwarded to makePyramid. +% chunks - optional explicit chunk shape (row vector, axes_order). +% Empty (default) means auto-size from the budget. +% Forwarded to makePyramid. % % INFO returns a struct with fields: % zarrPath - absolute path resolved @@ -76,6 +82,8 @@ options.pipelineVersion char = '' options.sourceFileID char = '' options.label char = '' + options.tileBudgetBytes (1,1) double {mustBePositive} = 8 * 2^20 + options.chunks double = [] end if isempty(strtrim(options.subjectID)) @@ -135,7 +143,9 @@ 'elementID', options.elementID, ... 'sourceFileID', sourceFileID, ... 'pipelineVersion', options.pipelineVersion, ... - 'label', options.label); + 'label', options.label, ... + 'tileBudgetBytes', options.tileBudgetBytes, ... + 'chunks', options.chunks); info = struct( ... 'zarrPath', char(zarrPath), ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 58d1af37d..618fad6ee 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -27,6 +27,27 @@ % sourceFileID - char, id of the fileReference doc for the store % pipelineVersion - char, tag for the pyramid % label - char, human-readable label for the pyramid +% tileBudgetBytes - target UNCOMPRESSED bytes per chunk when the +% chunk shape is chosen automatically. Default +% 8*2^20 (8 MB). Sized for a viewer over an +% ~200 MB/s link fetching ~4 parallel tiles per +% pan gesture. Chunks land near this budget but +% are clamped by the level's own shape, so a +% small coarse level may end up as one whole +% chunk. +% chunks - optional explicit chunk shape (row vector in +% axes_order). Empty (default) means auto: pick +% a shape ISOTROPIC IN WORLD SPACE from the +% level's voxel size, the dtype and +% tileBudgetBytes. Non-empty overrides the +% budget entirely and is used verbatim on every +% level (clamped by shape). +% +% The chosen chunk shape is what a future materializer will +% re-tile source bytes to; it is stamped onto the level document +% here so a reader can compute chunk_grid and index chunk.bin_# +% without consulting the source store. Source-store chunk shape is +% NOT preserved. % % Metadata only. n_chunks_stored is 0 on every level document in % this PR; materializing chunk bytes into `chunk.bin_#` is follow-up @@ -45,6 +66,8 @@ options.sourceFileID char = '' options.pipelineVersion char = '' options.label char = '' + options.tileBudgetBytes (1,1) double {mustBePositive} = 8 * 2^20 + options.chunks double = [] end if numel(pyramids) ~= numel(perEntryReduction) @@ -139,7 +162,15 @@ end function levelDoc = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, reductionFn, options) - chunkGrid = ceil(level.shape ./ level.chunks); + axesOrder = joinAxisNames(pyramidEntry.axes); + if isempty(options.chunks) + chunks = ndi.fun.doc.lightsheet.chooseTileShape(level.shape, ... + axesOrder, level.scale, char(level.dtype), ... + options.tileBudgetBytes); + else + chunks = clampChunksToShape(ensureRow(options.chunks), level.shape); + end + chunkGrid = ceil(level.shape ./ chunks); props = struct( ... 'label', sprintf('%s level %d (%s)', pyramidEntry.name, levelIndex, reductionFn), ... @@ -147,7 +178,7 @@ 'reduction_function', reductionFn, ... 'axes_order', joinAxisNames(pyramidEntry.axes), ... 'shape', level.shape, ... - 'chunks', level.chunks, ... + 'chunks', chunks, ... 'chunk_grid', chunkGrid, ... 'n_chunks_stored', 0, ... 'chunk_index_origin', 1, ... @@ -206,3 +237,19 @@ v = reshape(double(x), 1, []); end end + +function chunks = clampChunksToShape(chunks, shape) + shape = double(shape(:).'); + chunks = double(chunks(:).'); + if numel(chunks) ~= numel(shape) + error('NDI:lightsheet:makePyramid:chunksArity', ... + 'Explicit chunks length %d does not match level shape length %d.', ... + numel(chunks), numel(shape)); + end + if any(chunks < 1) || any(chunks ~= fix(chunks)) + error('NDI:lightsheet:makePyramid:badChunks', ... + 'Explicit chunks must be positive integers.'); + end + chunks = min(chunks, shape); +end + diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestChooseTileShape.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestChooseTileShape.m new file mode 100644 index 000000000..a36a3db3a --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestChooseTileShape.m @@ -0,0 +1,173 @@ +classdef TestChooseTileShape < matlab.unittest.TestCase + % TestChooseTileShape - byte-budgeted chunk sizer. + % + % Verifies chooseTileShape hits a byte budget with tiles that are + % isotropic in world space, clamps to level shape at coarse levels, + % keeps non-spatial axes at 1, and falls back gracefully when voxel + % size is missing. + + methods (Test) + + function testIsotropicUint16HitsBudget(testCase) + % 8 MB uncompressed, uint16, isotropic 1 um -> ~161^3. + budget = 8 * 2^20; + c = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [1 1 1], 'uint16', budget); + testCase.verifyEqual(numel(c), 3); + % Each side within 5% of the cube root of voxelsPerTile. + expectedSide = (budget / 2)^(1/3); + testCase.verifyLessThan(abs(c(1) - expectedSide) / expectedSide, 0.05); + testCase.verifyEqual(c(1), c(2)); + testCase.verifyEqual(c(2), c(3)); + % Bytes within 20% of the budget. + bytes = prod(c) * 2; + testCase.verifyGreaterThan(bytes, 0.8 * budget); + testCase.verifyLessThan(bytes, 1.2 * budget); + end + + function testAnisotropicVoxelProducesWorldCube(testCase) + % 1 um XY, 5 um Z -> chunk_z * 5 == chunk_x * 1 (approx). + budget = 8 * 2^20; + c = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [5 1 1], 'uint16', budget); + worldZ = c(1) * 5; + worldY = c(2) * 1; + worldX = c(3) * 1; + testCase.verifyLessThan(abs(worldZ - worldY) / worldY, 0.05); + testCase.verifyLessThan(abs(worldX - worldY) / worldY, 0.05); + testCase.verifyLessThan(c(1), c(2)); % thin Z + % Bytes still near budget. + bytes = prod(c) * 2; + testCase.verifyGreaterThan(bytes, 0.7 * budget); + testCase.verifyLessThan(bytes, 1.3 * budget); + end + + function testChannelAxisStaysOne(testCase) + % 4D czyx: c is not spatial and must stay 1. + c = ndi.fun.doc.lightsheet.chooseTileShape( ... + [3 1200 2000 2000], 'czyx', [1 1 1 1], 'uint16', 8 * 2^20); + testCase.verifyEqual(c(1), 1); + testCase.verifyGreaterThan(c(2), 1); + end + + function testTimeAxisStaysOne(testCase) + c = ndi.fun.doc.lightsheet.chooseTileShape( ... + [10 3 1200 2000 2000], 'tczyx', [1 1 1 1 1], 'uint16', 8 * 2^20); + testCase.verifyEqual(c(1), 1); + testCase.verifyEqual(c(2), 1); + end + + function testClampToShapeAtCoarseLevel(testCase) + % A level whose spatial shape is already smaller than a + % naive cube: one whole chunk, no larger than the shape. + c = ndi.fun.doc.lightsheet.chooseTileShape( ... + [32 32 32], 'zyx', [1 1 1], 'uint16', 8 * 2^20); + testCase.verifyEqual(c, [32 32 32]); + end + + function testUint8BudgetGivesLargerTile(testCase) + % Halving bytes-per-voxel doubles voxel count. + budget = 8 * 2^20; + c16 = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [1 1 1], 'uint16', budget); + c8 = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [1 1 1], 'uint8', budget); + testCase.verifyGreaterThan(prod(c8), prod(c16)); + % Ratio approx 2 (uint16 half the voxels of uint8). + testCase.verifyLessThan(abs(prod(c8) / prod(c16) - 2) / 2, 0.1); + end + + function testFloat32BudgetGivesSmallerTile(testCase) + budget = 8 * 2^20; + c16 = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [1 1 1], 'uint16', budget); + cf = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [1 1 1], 'float32', budget); + testCase.verifyLessThan(prod(cf), prod(c16)); + end + + function testNumpyDtypeStringWorks(testCase) + % Endian-prefixed forms ('f4') should be accepted. + budget = 8 * 2^20; + cA = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [1 1 1], ' isotropic voxel-space fallback. + budget = 8 * 2^20; + c = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [0 0 0], 'uint16', budget); + % Fell back to iso -> cube of ~161. + expectedSide = (budget / 2)^(1/3); + testCase.verifyLessThan(abs(c(1) - expectedSide) / expectedSide, 0.05); + testCase.verifyEqual(c(1), c(2)); + testCase.verifyEqual(c(2), c(3)); + end + + function testBudgetScales(testCase) + % Doubling the budget doubles the voxel count. + small = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [1 1 1], 'uint16', 4 * 2^20); + big = ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'zyx', [1 1 1], 'uint16', 8 * 2^20); + testCase.verifyLessThan(abs(prod(big) / prod(small) - 2) / 2, 0.1); + end + + function testRedistributionWhenOneAxisClamps(testCase) + % Level whose Z is very short but XY is huge: Z clamps and + % the budget should shift into XY rather than being wasted. + budget = 8 * 2^20; + c = ndi.fun.doc.lightsheet.chooseTileShape( ... + [8 4096 4096], 'zyx', [1 1 1], 'uint16', budget); + testCase.verifyEqual(c(1), 8); % Z clamped to shape + bytes = prod(c) * 2; + % After redistribution should be near budget rather than at + % the naive isotropic-cube figure (which would leave a lot + % on the table when Z is short). + testCase.verifyGreaterThan(bytes, 0.7 * budget); + testCase.verifyLessThan(bytes, 1.3 * budget); + end + + function testTileCountMonotonicallyDecreasesAcrossLadder(testCase) + % Across a dyadic ladder each level halves shape and + % doubles voxel size, so chunk-in-voxels stays constant and + % chunk_grid drops by 2 per axis -> 8x fewer tiles per + % level for a 3D pyramid. Codified because a future + % refactor that dropped voxel-size scaling would break it + % silently. + budget = 8 * 2^20; + shape0 = [1200 2000 2000]; + vox0 = [1 1 1]; + tileCounts = zeros(1, 5); + byteSizes = zeros(1, 5); + for L = 0:4 + shp = max(1, floor(shape0 / 2^L)); + vs = vox0 * 2^L; + c = ndi.fun.doc.lightsheet.chooseTileShape( ... + shp, 'zyx', vs, 'uint16', budget); + tileCounts(L+1) = prod(ceil(shp ./ c)); + byteSizes(L+1) = prod(c) * 2; + end + % Strictly non-increasing. + testCase.verifyEqual(tileCounts, sort(tileCounts, 'descend')); + % Every level's per-tile bytes at or below the budget + % (coarse levels may fall well below when clamped, but no + % level oversizes). + testCase.verifyLessThanOrEqual(byteSizes, 1.3 * budget); + % Coarsest level should be one tile. + testCase.verifyEqual(tileCounts(end), 1); + end + + function testShapeAxesLengthMismatchErrors(testCase) + testCase.verifyError( ... + @() ndi.fun.doc.lightsheet.chooseTileShape( ... + [1200 2000 2000], 'yx', [1 1 1], 'uint16', 8 * 2^20), ... + 'NDI:lightsheet:chooseTileShape:axesShapeMismatch'); + end + + end +end From c92b92a3e895e006bc6e6654a9f3397e01e854f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 23:38:00 +0000 Subject: [PATCH 014/110] Cover the lightsheet builders and manager statics Codecov flagged 0% patch coverage on the scaffold. The pure functions were already tested (TestViewCommand, TestChooseLevel, TestChooseTileShape); the session-dependent builders were not. TestMakePyramid drives the pyramid+level writer with synthetic listPyramids structs so the whole schema-level path is exercised without a real store: single-pyramid one-level; the dual mean+max ladder with shared level 0 dedupes to five docs with the right reduction_functions; the auto tile-shape chooser lands each level within budget; explicit `chunks` overrides the budget and is clamped to shape; arity mismatches error out; the parent carries the level-0 frame; and the dependency wiring (subject_id, lightsheetZarrPyramid_id on every level, and every doc actually reaches the database). TestFromOMEZarr uses the NDR fixture that already ships (makeExampleFixture, metadata-only; the same fixture NDR's own tests use, so the ingest path is held to one shape rather than an NDI private synthetic). Covers the end-to-end call, the guards (missing subject, missing store, non-zarr directory, unknown pyramid name), the pyramidNames filter (mean alone -> no dedupe), sourceFileID reuse, and tileBudgetBytes forwarding through to per-level chunks. TestMakeSourceFile mirrors +gene's regression coverage: describe is the default and gives a fileReference the database validates; attach gives a generic_file with generic_file.ext; directory-describe checksums .zattrs (the alternative -- walking a 100 GB tree -- is what the class avoids); attach-a-directory is refused; a missing path errors out; checksum=false leaves the checksum fields empty. TestLevelTable covers the empty-children path (kept as a returned table with the schema columns and the parent frame in UserData), the five-row shared-level-0 ladder (finest-first sort, reductions in the frame), that a bare id string is accepted alongside an ndi.document, the reduction filter (raw + one reduction, three rows), the unknown- parent error, and the exact column names against the schema. TestLightsheetZarrManager exercises the two static methods that do not need a figure: launcherPath (the class default when no preference is set) and deletionPlan (the query wrapper that names every doc database_rm will remove before it does). The GUI callbacks stay uncovered -- headless CI cannot open a figure -- but the manager's non-GUI logic now has a test. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+fun/+doc/+lightsheet/TestFromOMEZarr.m | 122 +++++++++ .../+fun/+doc/+lightsheet/TestLevelTable.m | 119 +++++++++ .../+fun/+doc/+lightsheet/TestMakePyramid.m | 241 ++++++++++++++++++ .../+doc/+lightsheet/TestMakeSourceFile.m | 117 +++++++++ .../+gui/+app/TestLightsheetZarrManager.m | 69 +++++ 5 files changed, 668 insertions(+) create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLevelTable.m create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakeSourceFile.m create mode 100644 tests/+ndi/+unittest/+gui/+app/TestLightsheetZarrManager.m diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m new file mode 100644 index 000000000..e8222e9b8 --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m @@ -0,0 +1,122 @@ +classdef TestFromOMEZarr < matlab.unittest.TestCase + % TestFromOMEZarr - end-to-end ingest against a synthetic OME-Zarr. + % + % Uses ndr.test.format.omezarr.makeExampleFixture (the NDR fixture + % also used by NDR's own tests) which writes a metadata-only + % dual-pyramid store: mean + max with shared level 0. When NDR is + % not installed on the test path, the tests filter rather than fail; + % this suite is about the NDI side, not the fixture builder. + + properties + session + subjectID + fixtureDir + end + + methods (TestMethodSetup) + function build(testCase) + if ~exist('ndr.test.format.omezarr.makeExampleFixture', 'file') + testCase.assumeFail(['NDR fixture builder not on path; ' ... + 'skipping end-to-end OME-Zarr ingest tests.']); + end + d = fullfile(tempname, 'fromome'); + mkdir(d); + testCase.addTeardown(@() rmdir(fileparts(d), 's')); + S = ndi.session.dir('fromome', d); + sub = ndi.document('subject', 'base.session_id', S.id(), ... + 'subject.local_identifier', 'lightsheet@vhlab'); + S.database_add(sub); + testCase.session = S; + testCase.subjectID = sub.id(); + testCase.fixtureDir = ndr.test.format.omezarr.makeExampleFixture(); + testCase.addTeardown(@() rmdir(fileparts(testCase.fixtureDir), 's')); + end + end + + methods (Test) + + function testWritesOneParentAndFiveLevels(testCase) + [pdoc, lds, info] = ndi.fun.doc.lightsheet.fromOMEZarr( ... + testCase.session, testCase.fixtureDir, ... + 'subjectID', testCase.subjectID); + testCase.verifyClass(pdoc, 'ndi.document'); + % mean + max with shared level 0 -> 5 unique paths. + testCase.verifyEqual(numel(lds), 5); + testCase.verifyTrue(info.sharedLevel0); + testCase.verifyEqual(sort(info.reductions(:).'), {'max','mean'}); + end + + function testMissingSubjectIDIsRefused(testCase) + testCase.verifyError(@() ndi.fun.doc.lightsheet.fromOMEZarr( ... + testCase.session, testCase.fixtureDir), ... + 'NDI:lightsheet:fromOMEZarr:noSubject'); + end + + function testMissingStoreErrors(testCase) + testCase.verifyError(@() ndi.fun.doc.lightsheet.fromOMEZarr( ... + testCase.session, '/no/such/store.ome.zarr', ... + 'subjectID', testCase.subjectID), ... + 'NDI:lightsheet:fromOMEZarr:noSuchStore'); + end + + function testNonZarrDirectoryRefused(testCase) + d = fullfile(tempname, 'not_a_zarr'); + mkdir(d); + testCase.addTeardown(@() rmdir(fileparts(d), 's')); + testCase.verifyError(@() ndi.fun.doc.lightsheet.fromOMEZarr( ... + testCase.session, d, 'subjectID', testCase.subjectID), ... + 'NDI:lightsheet:fromOMEZarr:notOMEZarr'); + end + + function testPyramidNamesFilter(testCase) + % Restricting to `mean` alone means only that pyramid is + % ingested. Its level 0 is not shared with anything, so no + % dedupe (3 levels: 0, mean/1, mean/2). + [~, lds, info] = ndi.fun.doc.lightsheet.fromOMEZarr( ... + testCase.session, testCase.fixtureDir, ... + 'subjectID', testCase.subjectID, ... + 'pyramidNames', {'mean'}); + testCase.verifyEqual(numel(lds), 3); + testCase.verifyFalse(info.sharedLevel0); + end + + function testUnknownPyramidNameErrors(testCase) + testCase.verifyError(@() ndi.fun.doc.lightsheet.fromOMEZarr( ... + testCase.session, testCase.fixtureDir, ... + 'subjectID', testCase.subjectID, ... + 'pyramidNames', {'nope'}), ... + 'NDI:lightsheet:fromOMEZarr:noNamedPyramid'); + end + + function testSourceFileIDReused(testCase) + % Passing an explicit sourceFileID keeps a caller-created + % fileReference rather than making a new one. + src = ndi.fun.doc.lightsheet.makeSourceFile( ... + testCase.session, testCase.fixtureDir); + testCase.session.database_add(src); + [~, ~, info] = ndi.fun.doc.lightsheet.fromOMEZarr( ... + testCase.session, testCase.fixtureDir, ... + 'subjectID', testCase.subjectID, ... + 'sourceFileID', src.id()); + testCase.verifyEqual(info.sourceFileID, src.id()); + end + + function testTileBudgetForwardedToLevels(testCase) + % A tiny budget forces chunks smaller than the source + % zarr's chunks -- proof that fromOMEZarr forwards the + % option and makePyramid re-chooses per level. + budget = 512; % bytes, tiny + [~, lds] = ndi.fun.doc.lightsheet.fromOMEZarr( ... + testCase.session, testCase.fixtureDir, ... + 'subjectID', testCase.subjectID, ... + 'tileBudgetBytes', budget); + for k = 1:numel(lds) + p = lds{k}.document_properties.lightsheetZarrLevel; + bytes = prod(p.chunks) * 2; % uint16 + testCase.verifyLessThanOrEqual(bytes, 2 * budget, ... + 'per-tile bytes should track the budget'); + end + end + + end +end diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLevelTable.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLevelTable.m new file mode 100644 index 000000000..19cbb5f7e --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLevelTable.m @@ -0,0 +1,119 @@ +classdef TestLevelTable < matlab.unittest.TestCase + % TestLevelTable - level enumeration + reduction filter. + % + % Exercises levelTable across the empty case, the mean+max ladder + % with a shared level 0 (5 rows: 'none', mean/1, mean/2, max/1, + % max/2), and the reduction filter that returns raw + one reduction. + + properties + session + subjectID + pyramidDoc + end + + methods (TestMethodSetup) + function build(testCase) + d = fullfile(tempname, 'leveltable'); + mkdir(d); + testCase.addTeardown(@() rmdir(fileparts(d), 's')); + S = ndi.session.dir('leveltable', d); + sub = ndi.document('subject', 'base.session_id', S.id(), ... + 'subject.local_identifier', 'lightsheet@vhlab'); + S.database_add(sub); + testCase.session = S; + testCase.subjectID = sub.id(); + end + end + + methods (Access = private) + function buildTwoPyramidsWithLevels(testCase) + pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); + [pdoc, ~, ~] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, pys, {'mean','max'}, ... + 'subjectID', testCase.subjectID); + testCase.pyramidDoc = pdoc; + end + end + + methods (Test) + + function testEmptyPyramidReturnsEmptyTable(testCase) + % Build a parent with NO levels, verify the table is empty + % but has the expected columns and frame. + L = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0'}, [16 32 32], [1 1 1], 'uint16'); + py = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'mean', 'box', L); + [pdoc, ~, ~] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, py, {'mean'}, ... + 'subjectID', testCase.subjectID); + % Delete the single level, leaving the parent with no + % children -- levelTable's empty path. + docs = testCase.session.database_search( ... + ndi.query('', 'isa', 'lightsheetZarrLevel') & ... + ndi.query('depends_on', 'depends_on', ... + 'lightsheetZarrPyramid_id', pdoc.id())); + for k = 1:numel(docs) + testCase.session.database_rm(docs{k}); + end + t = ndi.fun.doc.lightsheet.levelTable(testCase.session, pdoc); + testCase.verifyEqual(height(t), 0); + testCase.verifyTrue(ismember('reduction_function', ... + t.Properties.VariableNames)); + % UserData still carries the parent frame. + f = t.Properties.UserData; + testCase.verifyEqual(f.axes_order, 'zyx'); + end + + function testFullLadderReturnsFiveRowsSortedFinestFirst(testCase) + testCase.buildTwoPyramidsWithLevels(); + t = ndi.fun.doc.lightsheet.levelTable( ... + testCase.session, testCase.pyramidDoc); + testCase.verifyEqual(height(t), 5); + testCase.verifyEqual(t.level(1), 0); + testCase.verifyEqual(t.reduction_function{1}, 'none'); + testCase.verifyTrue(all(diff(t.level) >= 0)); + % Frame carries the reductions present in the ladder. + f = t.Properties.UserData; + testCase.verifyEqual( ... + sort(f.reductions(:).'), {'max','mean'}); + end + + function testAcceptsIDInsteadOfDoc(testCase) + testCase.buildTwoPyramidsWithLevels(); + t = ndi.fun.doc.lightsheet.levelTable( ... + testCase.session, testCase.pyramidDoc.id()); + testCase.verifyEqual(height(t), 5); + end + + function testUnknownIDErrors(testCase) + testCase.verifyError(@() ndi.fun.doc.lightsheet.levelTable( ... + testCase.session, '00000000-0000-0000-0000-000000000000'), ... + 'NDI:lightsheet:levelTable:noParent'); + end + + function testReductionFilterKeepsSharedAndOneReduction(testCase) + testCase.buildTwoPyramidsWithLevels(); + t = ndi.fun.doc.lightsheet.levelTable( ... + testCase.session, testCase.pyramidDoc, ... + 'reduction', 'max'); + % raw level 0 + max/1 + max/2 = 3. + testCase.verifyEqual(height(t), 3); + testCase.verifyTrue(all( ... + strcmp(t.reduction_function, 'none') | ... + strcmp(t.reduction_function, 'max'))); + testCase.verifyFalse(any(strcmp(t.reduction_function, 'mean'))); + end + + function testTableColumnsMatchSchema(testCase) + testCase.buildTwoPyramidsWithLevels(); + t = ndi.fun.doc.lightsheet.levelTable( ... + testCase.session, testCase.pyramidDoc); + expected = {'level','reduction_function','shape','chunks', ... + 'chunk_grid','n_chunks_stored','voxel_size', ... + 'translation','dtype','id'}; + testCase.verifyEqual(t.Properties.VariableNames, expected); + end + + end +end diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m new file mode 100644 index 000000000..499fa9730 --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m @@ -0,0 +1,241 @@ +classdef TestMakePyramid < matlab.unittest.TestCase + % TestMakePyramid - the pyramid/level document writer. + % + % Exercises makePyramid over synthetic listPyramids-style structs so + % the whole schema-level path (parent doc, per-level doc, dedupe of + % shared level 0, dependency wiring, chunk-shape choice) is covered + % without a real OME-Zarr store. Also drives the auto tile-shape + % chooser through makePyramid to catch integration regressions that + % the standalone TestChooseTileShape suite would miss. + + properties + session + subjectID + end + + methods (TestMethodSetup) + function build(testCase) + d = fullfile(tempname, 'makepyramid'); + mkdir(d); + testCase.addTeardown(@() rmdir(fileparts(d), 's')); + S = ndi.session.dir('makepyramid', d); + sub = ndi.document('subject', 'base.session_id', S.id(), ... + 'subject.local_identifier', 'lightsheet@vhlab'); + S.database_add(sub); + testCase.session = S; + testCase.subjectID = sub.id(); + end + end + + methods (Static) + function ax = axes3D() + ax = struct('name', {'z','y','x'}, 'type', {'space','space','space'}, ... + 'unit', {'micrometer','micrometer','micrometer'}); + end + + function levels = dyadicLevels(paths, shape0, vox0, dtype) + n = numel(paths); + levels = repmat(struct( ... + 'path', '', 'shape', [], 'chunks', [], 'dtype', dtype, ... + 'scale', [], 'translation', [0 0 0]), n, 1); + for k = 1:n + levels(k).path = paths{k}; + levels(k).shape = max(1, floor(shape0 / 2^(k-1))); + % Source zarr's own chunks are irrelevant to the doc -- + % makePyramid re-chooses them -- but supply something + % plausible so the input struct is well-formed. + levels(k).chunks = min(levels(k).shape, [64 64 64]); + levels(k).scale = vox0 * 2^(k-1); + levels(k).translation = [0 0 0]; + end + end + + function py = pyramidEntry(name, typ, levels) + py = struct('name', name, 'type', typ, ... + 'axes', ndi.unittest.fun.doc.lightsheet.TestMakePyramid.axes3D(), ... + 'levels', levels); + end + + function pys = twoPyramidsWithSharedLevel0() + L0 = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0','mean/1','mean/2'}, [128 512 512], [1 1 1], 'uint16'); + L1 = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0','max/1','max/2'}, [128 512 512], [1 1 1], 'uint16'); + pys = [ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'mean', 'box', L0); ... + ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'max', 'max', L1)]; + end + end + + methods (Test) + + function testOnePyramidOneLevelWritesTwoDocs(testCase) + L = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0'}, [64 128 128], [1 1 1], 'uint16'); + py = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'raw', 'box', L); + [pdoc, lds, shared] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, py, {'mean'}, ... + 'subjectID', testCase.subjectID); + testCase.verifyFalse(shared); + testCase.verifyClass(pdoc, 'ndi.document'); + testCase.verifyEqual(numel(lds), 1); + % The level document is written with reduction_function = + % the entry's reduction, since only one pyramid uses this + % path. + p = lds{1}.document_properties.lightsheetZarrLevel; + testCase.verifyEqual(p.reduction_function, 'mean'); + testCase.verifyEqual(p.level, 0); + end + + function testTwoPyramidsSharedLevel0DedupesToFiveDocs(testCase) + pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); + [pdoc, lds, shared] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, pys, {'mean','max'}, ... + 'subjectID', testCase.subjectID); + testCase.verifyTrue(shared); + % 5 unique paths: shared '0', mean/1, mean/2, max/1, max/2. + testCase.verifyEqual(numel(lds), 5); + reductions = cellfun(@(d) ... + d.document_properties.lightsheetZarrLevel.reduction_function, ... + lds, 'UniformOutput', false); + testCase.verifyEqual(sum(strcmp(reductions, 'none')), 1); + testCase.verifyEqual(sum(strcmp(reductions, 'mean')), 2); + testCase.verifyEqual(sum(strcmp(reductions, 'max')), 2); + end + + function testChunksAutoChosenByBudget(testCase) + % Level 0 shape 128 x 512 x 512 uint16 with 8 MB budget + % lands on ~128^3 * 2 ~= 4 MB (Z clamps to 128 which is + % well below the ideal cube, so the redistribution kicks in + % to push XY up). The stamped chunks should be within the + % budget. + pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); + [~, lds] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, pys, {'mean','max'}, ... + 'subjectID', testCase.subjectID); + for k = 1:numel(lds) + p = lds{k}.document_properties.lightsheetZarrLevel; + bytes = prod(p.chunks) * 2; + testCase.verifyLessThanOrEqual(bytes, 1.3 * 8 * 2^20); + testCase.verifyEqual(numel(p.chunks), numel(p.shape)); + testCase.verifyEqual(p.chunk_grid, ceil(p.shape ./ p.chunks)); + end + end + + function testExplicitChunksOverrideBudget(testCase) + L = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0','1'}, [64 128 128], [1 1 1], 'uint16'); + py = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'mean', 'box', L); + [~, lds] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, py, {'mean'}, ... + 'subjectID', testCase.subjectID, ... + 'chunks', [32 32 32]); + % Explicit chunks used verbatim on every level (clamped by + % shape). + p0 = lds{1}.document_properties.lightsheetZarrLevel; + p1 = lds{2}.document_properties.lightsheetZarrLevel; + testCase.verifyEqual(p0.chunks, [32 32 32]); + testCase.verifyEqual(p1.chunks, [32 32 32]); + end + + function testExplicitChunksClampToShape(testCase) + L = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0'}, [16 32 32], [1 1 1], 'uint16'); + py = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'mean', 'box', L); + [~, lds] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, py, {'mean'}, ... + 'subjectID', testCase.subjectID, ... + 'chunks', [64 64 64]); + % Explicit chunks larger than shape are clamped to shape. + p = lds{1}.document_properties.lightsheetZarrLevel; + testCase.verifyEqual(p.chunks, [16 32 32]); + end + + function testExplicitChunksArityMismatchErrors(testCase) + L = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0'}, [16 32 32], [1 1 1], 'uint16'); + py = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'mean', 'box', L); + testCase.verifyError(@() ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, py, {'mean'}, ... + 'subjectID', testCase.subjectID, ... + 'chunks', [32 32]), ... + 'NDI:lightsheet:makePyramid:chunksArity'); + end + + function testReductionCellArityIsChecked(testCase) + L = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0'}, [16 32 32], [1 1 1], 'uint16'); + py = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'mean', 'box', L); + testCase.verifyError(@() ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, py, {'mean','max'}, ... + 'subjectID', testCase.subjectID), ... + 'NDI:lightsheet:makePyramid:reductionArity'); + end + + function testParentCarriesLevel0Frame(testCase) + pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); + [pdoc, ~, ~] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, pys, {'mean','max'}, ... + 'subjectID', testCase.subjectID, ... + 'pipelineVersion', 'unit-test-1', ... + 'label', 'my volume'); + p = pdoc.document_properties.lightsheetZarrPyramid; + testCase.verifyEqual(p.axes_order, 'zyx'); + testCase.verifyEqual(reshape(p.shape_level0, 1, []), [128 512 512]); + testCase.verifyEqual(reshape(p.voxel_size_level0, 1, []), [1 1 1]); + testCase.verifyEqual(char(p.dtype), 'uint16'); + testCase.verifyEqual(char(p.pipeline_version), 'unit-test-1'); + testCase.verifyEqual(char(p.label), 'my volume'); + end + + function testDependencyWiringSubjectAndPyramid(testCase) + pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); + [pdoc, lds, ~] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, pys, {'mean','max'}, ... + 'subjectID', testCase.subjectID); + testCase.verifyEqual( ... + pdoc.dependency_value('subject_id'), testCase.subjectID); + for k = 1:numel(lds) + testCase.verifyEqual( ... + lds{k}.dependency_value('lightsheetZarrPyramid_id'), ... + pdoc.id()); + testCase.verifyEqual( ... + lds{k}.dependency_value('subject_id'), testCase.subjectID); + end + end + + function testPyramidTypeCarried(testCase) + % The parent's pyramid_type is the FIRST entry's type. Two + % pyramids with different types don't collide (they share + % the parent, since a store is one source volume). + pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); + [pdoc, ~, ~] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, pys, {'mean','max'}, ... + 'subjectID', testCase.subjectID); + p = pdoc.document_properties.lightsheetZarrPyramid; + testCase.verifyEqual(char(p.pyramid_type), 'box'); + end + + function testLevelDocsWrittenToDatabase(testCase) + L = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.dyadicLevels( ... + {'0','1'}, [64 128 128], [1 1 1], 'uint16'); + py = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.pyramidEntry( ... + 'mean', 'box', L); + [~, lds] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, py, {'mean'}, ... + 'subjectID', testCase.subjectID); + for k = 1:numel(lds) + back = testCase.session.database_search( ... + ndi.query('base.id', 'exact_string', lds{k}.id())); + testCase.verifyEqual(numel(back), 1); + end + end + + end +end diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakeSourceFile.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakeSourceFile.m new file mode 100644 index 000000000..13ce79d14 --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakeSourceFile.m @@ -0,0 +1,117 @@ +classdef TestMakeSourceFile < matlab.unittest.TestCase + % TestMakeSourceFile - describe (fileReference) vs attach (generic_file). + % + % Mirrors +gene's TestGeneFromFiles source-file coverage, but for the + % OME-Zarr case: source stores are directory trees, and describing + % rather than attaching is the default so a 100 GB tree does not get + % copied alongside the pyramid summarising it. Checksum is taken + % over .zattrs (small, identifies the multiscales layout uniquely) + % rather than every chunk in the tree. + + properties + session + end + + methods (TestMethodSetup) + function build(testCase) + d = fullfile(tempname, 'srcfile'); + mkdir(d); + testCase.addTeardown(@() rmdir(fileparts(d), 's')); + testCase.session = ndi.session.dir('srcfile', d); + end + end + + methods (Access = private) + function f = aFile(testCase, name, contents) + f = fullfile(tempname, name); + mkdir(fileparts(f)); + testCase.addTeardown(@() rmdir(fileparts(f), 's')); + fid = fopen(f, 'w'); fwrite(fid, contents); fclose(fid); + end + + function d = aZarrStore(testCase, name) + d = fullfile(tempname, name); + mkdir(d); + testCase.addTeardown(@() rmdir(fileparts(d), 's')); + zattrs = fullfile(d, '.zattrs'); + fid = fopen(zattrs, 'w'); + fwrite(fid, '{"multiscales":[{"name":"raw","axes":[]}]}'); + fclose(fid); + end + end + + methods (Test) + + function testFileDescribeMakesFileReference(testCase) + f = testCase.aFile('a.bin', 'abcdef'); + doc = ndi.fun.doc.lightsheet.makeSourceFile(testCase.session, f); + testCase.verifyEqual( ... + doc.document_properties.document_class.class_name, ... + 'fileReference'); + r = doc.document_properties.fileReference; + testCase.verifyEqual(r.filename, 'a.bin'); + testCase.verifyEqual(r.fileSize, 6); + testCase.verifyNotEmpty(r.checksum); + testCase.verifyEqual(r.checksumAlgorithm, 'MD5'); + testCase.verifyEmpty(doc.current_file_list(), ... + 'describing must not attach the file'); + end + + function testFileAttachMakesGenericFile(testCase) + f = testCase.aFile('a.bin', 'abcdef'); + doc = ndi.fun.doc.lightsheet.makeSourceFile( ... + testCase.session, f, 'attachFile', true); + testCase.verifyEqual( ... + doc.document_properties.document_class.class_name, ... + 'generic_file'); + files = doc.current_file_list(); + testCase.verifyEqual(numel(files), 1); + testCase.verifyEqual(char(files{1}), 'generic_file.ext'); + end + + function testDirectoryDescribeChecksumsZattrs(testCase) + d = testCase.aZarrStore('vol.ome.zarr'); + doc = ndi.fun.doc.lightsheet.makeSourceFile(testCase.session, d); + r = doc.document_properties.fileReference; + testCase.verifyEqual(r.filename, 'vol.ome.zarr'); + testCase.verifyNotEmpty(r.checksum); + testCase.verifyEqual(r.checksumAlgorithm, 'MD5'); + % fileSize is 0 for a directory (only for plain files). + testCase.verifyEqual(r.fileSize, 0); + end + + function testAttachingADirectoryIsRefused(testCase) + d = testCase.aZarrStore('vol.ome.zarr'); + testCase.verifyError(@() ndi.fun.doc.lightsheet.makeSourceFile( ... + testCase.session, d, 'attachFile', true), ... + 'NDI:lightsheet:makeSourceFile:cannotAttachDirectory'); + end + + function testMissingPathErrors(testCase) + testCase.verifyError(@() ndi.fun.doc.lightsheet.makeSourceFile( ... + testCase.session, '/no/such/thing.zarr'), ... + 'NDI:lightsheet:makeSourceFile:noSuchPath'); + end + + function testChecksumFalseLeavesFieldEmpty(testCase) + f = testCase.aFile('nochk.bin', 'x'); + doc = ndi.fun.doc.lightsheet.makeSourceFile( ... + testCase.session, f, 'checksum', false); + r = doc.document_properties.fileReference; + testCase.verifyEmpty(r.checksum); + testCase.verifyEmpty(r.checksumAlgorithm); + end + + function testDescribedDocIsAValidDocument(testCase) + % The regression test that started this whole file: a + % described document has to validate against its schema. A + % fileReference has no `file` slot, so this is why the + % class was chosen over generic_file. + f = testCase.aFile('valid.txt', 'hi'); + doc = ndi.fun.doc.lightsheet.makeSourceFile(testCase.session, f); + testCase.verifyWarningFree(@() ... + testCase.session.database_add(doc)); + end + + end +end diff --git a/tests/+ndi/+unittest/+gui/+app/TestLightsheetZarrManager.m b/tests/+ndi/+unittest/+gui/+app/TestLightsheetZarrManager.m new file mode 100644 index 000000000..0edddad16 --- /dev/null +++ b/tests/+ndi/+unittest/+gui/+app/TestLightsheetZarrManager.m @@ -0,0 +1,69 @@ +classdef TestLightsheetZarrManager < matlab.unittest.TestCase + % TestLightsheetZarrManager - non-GUI paths of the manager. + % + % The GUI (uifigure construction, callbacks) is not exercised here; + % headless CI cannot open a figure. What is testable is the + % launcherPath resolver (preference fallback) and deletionPlan (a + % pure query wrapper used before database_rm cascades). + + properties + session + subjectID + end + + methods (TestMethodSetup) + function build(testCase) + d = fullfile(tempname, 'lszmgr'); + mkdir(d); + testCase.addTeardown(@() rmdir(fileparts(d), 's')); + S = ndi.session.dir('lszmgr', d); + sub = ndi.document('subject', 'base.session_id', S.id(), ... + 'subject.local_identifier', 'lightsheet@vhlab'); + S.database_add(sub); + testCase.session = S; + testCase.subjectID = sub.id(); + end + end + + methods (Test) + + function testLauncherPathFallsBackToDefault(testCase) + % Without a preference set, the launcher is the class's + % declared default. Should be an absolute path ending in + % 'napariViewLightsheet'. + p = ndi.gui.app.LightsheetZarrManager.launcherPath(); + testCase.verifyClass(p, 'char'); + testCase.verifyNotEmpty(p); + [~, name] = fileparts(p); + testCase.verifyEqual(name, 'napariViewLightsheet'); + end + + function testDeletionPlanCollectsLevelsAndParent(testCase) + % Deletion is a cascade; the plan enumerates everything + % the manager will remove so the UI can name the counts + % before doing it. + pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); + [pdoc, lds, ~] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, pys, {'mean','max'}, ... + 'subjectID', testCase.subjectID); + plan = ndi.gui.app.LightsheetZarrManager.deletionPlan( ... + testCase.session, pdoc.id()); + testCase.verifyEqual(plan.pyramidID, pdoc.id()); + testCase.verifyEqual(plan.nLevels, numel(lds)); + testCase.verifyEqual(numel(plan.docsToRemove), ... + numel(lds) + 1, ... + 'plan should include every level plus the parent'); + end + + function testDeletionPlanOnAbsentPyramidReportsZero(testCase) + % A never-created id has no children and no parent; the + % plan should return an empty removal list. + plan = ndi.gui.app.LightsheetZarrManager.deletionPlan( ... + testCase.session, ... + '00000000-0000-0000-0000-000000000000'); + testCase.verifyEqual(plan.nLevels, 0); + testCase.verifyEmpty(plan.docsToRemove); + end + + end +end From 1d1130280d2f16ac3d2b40531dec5f520be1492d Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 23:43:25 +0000 Subject: [PATCH 015/110] Fix element-wise axis test in chooseTileShape ismember(charRow, cellstr) treats the char row as ONE string and returns a scalar, so isSpatial collapsed to a single false, chooseTileShape returned [1 1 1] on every input, and 12 of the 13 TestChooseTileShape cases errored on the wrong result. Comparing against 'tc' (a char row on both sides) makes ismember compare element-wise as intended. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m index 70194ebac..c4b0dfcdb 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m @@ -59,7 +59,9 @@ numel(axesOrder), n); end - isSpatial = ~ismember(lower(axesOrder(1:n)), {'t','c'}); + % ismember with a cellstr treats the char row as ONE string; use a + % char row on both sides so it compares element-wise. + isSpatial = ~ismember(lower(axesOrder(1:n)), 'tc'); isSpatial = reshape(isSpatial, 1, n); chunks = ones(1, n); if ~any(isSpatial) From 04293e3c77940d8fb6a70885d8c06af8f684a34b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 9 Sep 2026 23:59:52 +0000 Subject: [PATCH 016/110] Store axes_units and channel_names as strings, not matrices The schema declared both as type "matrix" but their documentation said "cellstr in MATLAB / list of strings in JSON" and makePyramid passed a cell array of chars for each. did.database.validate_field_type_and_value correctly refused a non-numeric matrix, so every path that added a pyramid document errored with DID:Database:ValidationFieldMatrix -- 16 of the tests I just added blocked at pyramid add before reaching what they were trying to exercise. NDI schemas do not have a cellstr type. Simplest honest fix is to make these single strings and encode the list as comma-separated ("micrometer,micrometer,micrometer", "GFP,mCherry"): both fields are descriptive metadata, not queryable data, so a single string is enough and readable. Schema type flipped to "string", default_value to "", documentation rewritten to name the new encoding, and makePyramid now strjoins axesUnits with ',' and leaves channel_names empty (we do not have names in the source yet). The vendored copy in NDI-python's ndi_common tree is updated to match so the Python reader validates against the same schema. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 4 ++-- .../data/lightsheetZarrPyramid.json | 4 ++-- .../data/lightsheetZarrPyramid_schema.json | 12 ++++++------ 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 618fad6ee..3145ee39d 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -117,8 +117,8 @@ 'pyramid_name', char(firstEntry.name), ... 'pyramid_type', char(firstEntry.type), ... 'axes_order', axesOrder, ... - 'axes_units', {axesUnits}, ... - 'channel_names', {{}}, ... + 'axes_units', strjoin(axesUnits, ','), ... + 'channel_names', '', ... 'n_channels', channelCount(firstLevel.shape, axesOrder), ... 'shape_level0', firstLevel.shape, ... 'voxel_size_level0', ensureRow(firstLevel.scale), ... diff --git a/src/ndi/ndi_common/database_documents/data/lightsheetZarrPyramid.json b/src/ndi/ndi_common/database_documents/data/lightsheetZarrPyramid.json index 179ebafcb..d5cf0fb60 100644 --- a/src/ndi/ndi_common/database_documents/data/lightsheetZarrPyramid.json +++ b/src/ndi/ndi_common/database_documents/data/lightsheetZarrPyramid.json @@ -30,8 +30,8 @@ "pyramid_name": "", "pyramid_type": "", "axes_order": "tczyx", - "axes_units": [], - "channel_names": [], + "axes_units": "", + "channel_names": "", "n_channels": 0, "shape_level0": [], "voxel_size_level0": [], diff --git a/src/ndi/ndi_common/schema_documents/data/lightsheetZarrPyramid_schema.json b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrPyramid_schema.json index 6e20cacee..af6110b93 100644 --- a/src/ndi/ndi_common/schema_documents/data/lightsheetZarrPyramid_schema.json +++ b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrPyramid_schema.json @@ -52,19 +52,19 @@ }, { "name": "axes_units", - "type": "matrix", - "default_value": [], + "type": "string", + "default_value": "", "parameters": "", "queryable": 1, - "documentation": "Per-axis physical unit strings, aligned with axes_order. Stored as a cellstr in MATLAB / list of strings in JSON." + "documentation": "Per-axis physical unit strings, aligned with axes_order. Encoded as a comma-separated string (e.g. 'micrometer,micrometer,micrometer' for zyx). Empty allowed." }, { "name": "channel_names", - "type": "matrix", - "default_value": [], + "type": "string", + "default_value": "", "parameters": "", "queryable": 1, - "documentation": "Display names of the channels, in the order of the 'c' axis. Empty when there are no channels." + "documentation": "Display names of the channels, comma-separated in the order of the 'c' axis (e.g. 'GFP,mCherry'). Empty when there are no channels or none are named." }, { "name": "n_channels", From e9d9fd08378c59fdd778d6752d80aaadc4b41ed6 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 00:11:52 +0000 Subject: [PATCH 017/110] Make TestFromOMEZarr self-contained so it does not need NDR's fixture The suite guarded on ndr.test.format.omezarr.makeExampleFixture and filtered when it was absent -- which was the whole CI run, because NDR-matlab's test package is not on the MATLAB path there. Codecov reported fromOMEZarr.m as 0% patch coverage as a result. The fixture is now written inside the test setup: a metadata-only OME-Zarr store (.zattrs + per-level .zarray) with the dual mean+max ladder that shares level 0, matching the shape the ferret-lightsheet lab actually produces. Nothing here writes chunk bytes; fromOMEZarr is metadata-only. NDR is still the reader (ndr.format.omezarr. listPyramids etc.), and the suite still filters cleanly if NDR is absent -- an absent dependency is not a defect in this code. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+fun/+doc/+lightsheet/TestFromOMEZarr.m | 106 ++++++++++++++++-- 1 file changed, 96 insertions(+), 10 deletions(-) diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m index e8222e9b8..012c2c69f 100644 --- a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m @@ -1,11 +1,12 @@ classdef TestFromOMEZarr < matlab.unittest.TestCase % TestFromOMEZarr - end-to-end ingest against a synthetic OME-Zarr. % - % Uses ndr.test.format.omezarr.makeExampleFixture (the NDR fixture - % also used by NDR's own tests) which writes a metadata-only - % dual-pyramid store: mean + max with shared level 0. When NDR is - % not installed on the test path, the tests filter rather than fail; - % this suite is about the NDI side, not the fixture builder. + % The fixture is a metadata-only dual-pyramid store (mean + max + % sharing level 0) written in the test setup itself, so this suite + % does not depend on NDR being installed. NDR is still the source + % of truth for reading OME-Zarr; when NDR is present we call it + % here as ingest does. When NDR is not on the path, every test + % filters rather than failing. properties session @@ -15,9 +16,9 @@ methods (TestMethodSetup) function build(testCase) - if ~exist('ndr.test.format.omezarr.makeExampleFixture', 'file') - testCase.assumeFail(['NDR fixture builder not on path; ' ... - 'skipping end-to-end OME-Zarr ingest tests.']); + if ~exist('ndr.format.omezarr.listPyramids', 'file') + testCase.assumeFail(['NDR reader not on path; skipping ' ... + 'end-to-end OME-Zarr ingest tests.']); end d = fullfile(tempname, 'fromome'); mkdir(d); @@ -28,8 +29,7 @@ function build(testCase) S.database_add(sub); testCase.session = S; testCase.subjectID = sub.id(); - testCase.fixtureDir = ndr.test.format.omezarr.makeExampleFixture(); - testCase.addTeardown(@() rmdir(fileparts(testCase.fixtureDir), 's')); + testCase.fixtureDir = writeSyntheticZarr(testCase); end end @@ -120,3 +120,89 @@ function testTileBudgetForwardedToLevels(testCase) end end + +% --------------------------------------------------------------------- + +function fixtureDir = writeSyntheticZarr(testCase) +% Write a metadata-only OME-Zarr with a mean + max ladder sharing '0'. +% Directory layout: +% /.zattrs (multiscales, both pyramids) +% /0/.zarray (shared level 0) +% /mean/1/.zarray +% /mean/2/.zarray +% /max/1/.zarray +% /max/2/.zarray +% +% No chunk bytes; fromOMEZarr is metadata-only. + parent = tempname; + mkdir(parent); + testCase.addTeardown(@() rmdir(parent, 's')); + fixtureDir = fullfile(parent, 'example.zarr'); + mkdir(fixtureDir); + + dtype = ' Date: Thu, 10 Sep 2026 00:18:40 +0000 Subject: [PATCH 018/110] Add a blob OME-Zarr fixture generator for end-to-end smoke tests New: ndi.test.lightsheet.makeBlobFixture and ingestBlobFixture. The first writes a small OME-Zarr store on disk with real chunk bytes (uncompressed Zarr v2 raw, the shape every zarr reader accepts) and returns the ground-truth volumes it wrote. The second is a one-line wrapper: builds a fresh session, calls the fixture writer, ingests through fromOMEZarr and hands back (session, pyramidDoc, info). The volume is three features at three spatial scales: - a wide Gaussian ball (sigma 40, peak ~50k) at the centre - a medium Gaussian (sigma 15, peak ~30k) offset to a corner - a single-voxel spike of value 60k so the pyramid levels look different at a glance and the mean-vs-max choice is visible (the spike survives max and vanishes under mean). Default shape is 300 x 300 x 300 uint16 (~54 MB in memory), with a dyadic mean + max ladder sharing level 0; chunk shape defaults to 64 per axis but is exposed as a name-value so the same helper covers a 32^3 fast test and a full 300^3 real-fixture run. TestBlobFixture holds it to that contract: .zattrs + per-level .zarray on disk, at least one chunk file per level with the right raw byte size, dual mean + max multiscales with shared level 0, fromOMEZarr ingest producing five deduped level docs and the parent frame matching the source. A separate test builds a 64^3 fixture and verifies max keeps the spike while mean loses it -- the functional-difference test that is the reason this fixture is worth having. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+test/+lightsheet/ingestBlobFixture.m | 92 ++++++ .../+ndi/+test/+lightsheet/makeBlobFixture.m | 279 ++++++++++++++++++ .../+test/+lightsheet/TestBlobFixture.m | 165 +++++++++++ 3 files changed, 536 insertions(+) create mode 100644 src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m create mode 100644 src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m create mode 100644 tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m new file mode 100644 index 000000000..ffd14229f --- /dev/null +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -0,0 +1,92 @@ +function [session, pyramidDoc, info] = ingestBlobFixture(options) +% NDI.TEST.LIGHTSHEET.INGESTBLOBFIXTURE - one-line end-to-end blob demo +% +% [SESSION, PYRAMIDDOC, INFO] = NDI.TEST.LIGHTSHEET.INGESTBLOBFIXTURE() +% [SESSION, PYRAMIDDOC, INFO] = NDI.TEST.LIGHTSHEET.INGESTBLOBFIXTURE( ... +% 'sessionDir', '/path/to/session', 'zarrDir', '/path/to/store', ...) +% +% Writes the 300x300x300 blob fixture (see makeBlobFixture) into a +% session, calls fromOMEZarr to build the pyramid and level +% documents, and returns the ready-to-view result. Meant for a +% copy-paste smoke test of the whole ingest -> document -> viewer +% pathway. +% +% Name-Value: +% sessionDir - char, where to build the session. Empty (default) +% puts it under a fresh tempname. +% zarrDir - char, parent directory for the OME-Zarr store. +% Empty (default) uses tempname; the store lives at +% fullfile(zarrDir, 'blob.ome.zarr'). Pass a stable +% path here to keep the fixture between runs (e.g. +% for repeated cloud-upload experiments). +% subject - char, local_identifier for the subject document +% created inside the session (default 'blob@vhlab'). +% Shape - passed through to makeBlobFixture (default +% [300 300 300]). +% ChunkShape - passed through to makeBlobFixture (default +% [64 64 64]). +% NumLevels - passed through to makeBlobFixture (default 4). +% VoxelSize - passed through to makeBlobFixture (default +% [1 1 1]). +% +% Returns: +% SESSION - ndi.session.dir opened at sessionDir. +% PYRAMIDDOC - the lightsheetZarrPyramid parent document. +% INFO - the struct fromOMEZarr's third output would return +% (zarrPath, pyramids, reductions, sourceFileID, +% sharedLevel0), plus: +% subjectID - the subject document's id +% fixture - the ground-truth struct that +% makeBlobFixture returned +% +% Example: +% [S, P] = ndi.test.lightsheet.ingestBlobFixture(); +% ndi.gui.app.LightsheetZarrManager(S); +% +% See also: ndi.test.lightsheet.makeBlobFixture, +% ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.gui.app.LightsheetZarrManager + + arguments + options.sessionDir char = '' + options.zarrDir char = '' + options.subject (1,:) char = 'blob@vhlab' + options.Shape (1,3) double {mustBePositive, mustBeInteger} = [300 300 300] + options.ChunkShape (1,3) double {mustBePositive, mustBeInteger} = [64 64 64] + options.NumLevels (1,1) double {mustBePositive, mustBeInteger} = 4 + options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] + end + + sessionDir = options.sessionDir; + if isempty(sessionDir) + sessionDir = tempname; + end + if ~isfolder(sessionDir) + mkdir(sessionDir); + end + + zarrParent = options.zarrDir; + if isempty(zarrParent) + zarrParent = tempname; + mkdir(zarrParent); + elseif ~isfolder(zarrParent) + mkdir(zarrParent); + end + + [zarrPath, gt] = ndi.test.lightsheet.makeBlobFixture(zarrParent, ... + 'Shape', options.Shape, ... + 'NumLevels', options.NumLevels, ... + 'ChunkShape', options.ChunkShape, ... + 'VoxelSize', options.VoxelSize); + + session = ndi.session.dir('blob', sessionDir); + sub = ndi.document('subject', 'base.session_id', session.id(), ... + 'subject.local_identifier', options.subject); + session.database_add(sub); + + [pyramidDoc, ~, info] = ndi.fun.doc.lightsheet.fromOMEZarr( ... + session, zarrPath, 'subjectID', sub.id()); + + info.subjectID = sub.id(); + info.fixture = gt; +end diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m new file mode 100644 index 000000000..3033d52f3 --- /dev/null +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -0,0 +1,279 @@ +function [zarrPath, gt] = makeBlobFixture(parentDir, options) +% NDI.TEST.LIGHTSHEET.MAKEBLOBFIXTURE - synthetic lightsheet OME-Zarr with real bytes +% +% [ZARRPATH, GT] = NDI.TEST.LIGHTSHEET.MAKEBLOBFIXTURE() +% [ZARRPATH, GT] = NDI.TEST.LIGHTSHEET.MAKEBLOBFIXTURE(PARENTDIR, ...) +% +% Writes a small OME-Zarr (NGFF v0.4) store on disk carrying a +% deterministic 300x300x300 uint16 volume, downsampled into a +% mean + max ladder that shares its raw level 0. Unlike the +% metadata-only fixtures the unit tests use, this one writes REAL +% chunk bytes -- uncompressed raw (Zarr v2 with `compressor: null`), +% which every zarr reader accepts -- so an end-to-end run can +% ingest it with ndi.fun.doc.lightsheet.fromOMEZarr, materialize +% chunks onto lightsheetZarrLevel documents when that lands, and be +% viewed by the napari client that reads them back. +% +% The volume itself is three features at three spatial scales, so a +% viewer can tell the pyramid levels apart at a glance: +% - a wide Gaussian ball at (150,150,150), sigma 40, peak ~50k +% - a medium Gaussian at (200,100,150), sigma 15, peak ~30k +% - a bright single-voxel spike at (50,50,50), value 60k +% The spike survives max downsampling and vanishes under mean; that +% contrast makes the mean-vs-max choice visible in the viewer. +% +% PARENTDIR - char, directory to write into. Empty (default) puts +% the store under a fresh tempname; the caller can pass +% a stable path (e.g. under an S3-synced dir) to keep +% the fixture between runs. The store lives at +% fullfile(PARENTDIR, 'blob.ome.zarr'). +% +% Name-Value: +% Shape (1,3) double - [Z Y X], default [300 300 300] +% NumLevels (1,1) - number of pyramid levels (default 4) +% ChunkShape (1,3) - per-level Zarr chunk shape (default +% [64 64 64]; every level uses the same +% chunk shape, edge chunks pad with 0) +% VoxelSize (1,3) - level 0 voxel size (default [1 1 1] um). +% Every subsequent level scales by 2x. +% Seed (1,1) - reserved for future noisy fixtures; the +% default fixture is deterministic without +% it (default 42). +% +% ZARRPATH - char, the absolute path to /blob.ome.zarr +% GT - struct with fields: +% shape - level-0 [Z Y X] +% chunkShape - the chunk shape used +% voxelSize - level-0 voxel size +% numLevels - the ladder depth +% levels - 1xN struct(path, shape) for each level +% across mean and max +% volumes - struct with fields: +% level0 - the raw uint16 volume +% mean - 1xN cell of mean downsamples +% max - 1xN cell of max downsamples +% Use GT to compare a reader's output against ground +% truth. The full uint16 volume is a modest ~54 MB at +% the default shape; do NOT return it unrequested. +% +% Example -- end to end round-trip on the local filesystem: +% [zp, gt] = ndi.test.lightsheet.makeBlobFixture(pwd); +% S = ndi.session.dir('demo', tempname); +% sub = ndi.document('subject', 'base.session_id', S.id(), ... +% 'subject.local_identifier', 'blob@vhlab'); +% S.database_add(sub); +% [pyr, lds] = ndi.fun.doc.lightsheet.fromOMEZarr(S, zp, ... +% 'subjectID', sub.id()); +% ndi.gui.app.LightsheetZarrManager(S); +% +% See also: ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.fun.doc.lightsheet.makePyramid, +% ndr.format.omezarr.listPyramids + + arguments + parentDir char = '' + options.Shape (1,3) double {mustBePositive, mustBeInteger} = [300 300 300] + options.NumLevels (1,1) double {mustBePositive, mustBeInteger} = 4 + options.ChunkShape (1,3) double {mustBePositive, mustBeInteger} = [64 64 64] + options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] + options.Seed (1,1) double = 42 %#ok + end + + if isempty(parentDir) + parentDir = tempname; + mkdir(parentDir); + elseif ~isfolder(parentDir) + mkdir(parentDir); + end + zarrPath = fullfile(parentDir, 'blob.ome.zarr'); + if isfolder(zarrPath) + rmdir(zarrPath, 's'); + end + mkdir(zarrPath); + + % Build level 0 and the two reduction ladders. + lvl0 = buildBlobVolume(options.Shape); + meanLadder = cell(1, options.NumLevels); + maxLadder = cell(1, options.NumLevels); + meanLadder{1} = lvl0; + maxLadder{1} = lvl0; + for L = 2:options.NumLevels + meanLadder{L} = downsampleBlockUint16(meanLadder{L-1}, 2, @meanReduce); + maxLadder{L} = downsampleBlockUint16(maxLadder{L-1}, 2, @maxReduce); + end + + % Write level 0 (shared) and each reduction's coarser levels. + writeZarrLevel(fullfile(zarrPath, '0'), lvl0, options.ChunkShape); + for L = 2:options.NumLevels + writeZarrLevel(fullfile(zarrPath, 'mean', sprintf('%d', L-1)), ... + meanLadder{L}, options.ChunkShape); + writeZarrLevel(fullfile(zarrPath, 'max', sprintf('%d', L-1)), ... + maxLadder{L}, options.ChunkShape); + end + + % Write .zattrs. + ax = {struct('name','z','type','space','unit','micrometer'), ... + struct('name','y','type','space','unit','micrometer'), ... + struct('name','x','type','space','unit','micrometer')}; + meanDatasets = cell(1, options.NumLevels); + maxDatasets = cell(1, options.NumLevels); + for L = 1:options.NumLevels + scale = options.VoxelSize * 2^(L-1); + if L == 1 + path0 = '0'; + else + pMean = sprintf('mean/%d', L-1); + pMax = sprintf('max/%d', L-1); + end + if L == 1 + meanDatasets{L} = zarrDataset(path0, scale); + maxDatasets{L} = zarrDataset(path0, scale); + else + meanDatasets{L} = zarrDataset(pMean, scale); + maxDatasets{L} = zarrDataset(pMax, scale); + end + end + multiscales = { ... + struct('name','mean','type','box', ... + 'axes', {ax}, 'datasets', {meanDatasets}), ... + struct('name','max','type','max', ... + 'axes', {ax}, 'datasets', {maxDatasets}) }; + writeJSON(fullfile(zarrPath, '.zattrs'), ... + struct('multiscales', {multiscales})); + + % Level manifest for the ground truth. + levels = struct('path', {}, 'shape', {}); + levels(end+1) = struct('path', '0', 'shape', options.Shape); + for L = 2:options.NumLevels + s = size(meanLadder{L}); + levels(end+1) = struct('path', sprintf('mean/%d', L-1), 'shape', s); %#ok + levels(end+1) = struct('path', sprintf('max/%d', L-1), 'shape', s); %#ok + end + + gt = struct( ... + 'shape', options.Shape, ... + 'chunkShape', options.ChunkShape, ... + 'voxelSize', options.VoxelSize, ... + 'numLevels', options.NumLevels, ... + 'levels', levels, ... + 'volumes', struct('level0', lvl0, ... + 'mean', {meanLadder}, ... + 'max', {maxLadder})); +end + +% ===================================================================== + +function vol = buildBlobVolume(shape) +% Deterministic 3D uint16 volume with three features at three scales. +% Everything is closed-form so the fixture is reproducible without +% touching random state. + Z = shape(1); Y = shape(2); X = shape(3); + [zz, yy, xx] = ndgrid(1:Z, 1:Y, 1:X); + % A wide ball centred in the volume. + c1 = shape / 2; + r1sq = (zz - c1(1)).^2 + (yy - c1(2)).^2 + (xx - c1(3)).^2; + wide = 50000 * exp(-r1sq / (2 * 40^2)); + % A medium blob offset to one corner. + c2 = [shape(1)*2/3 shape(2)/3 shape(3)/2]; + r2sq = (zz - c2(1)).^2 + (yy - c2(2)).^2 + (xx - c2(3)).^2; + medium = 30000 * exp(-r2sq / (2 * 15^2)); + % One bright voxel: max keeps it, mean loses it. + spike = zeros(shape); + spike(50, 50, 50) = 60000; + + vol = uint16(min(65535, wide + medium + spike)); +end + +function out = downsampleBlockUint16(vol, factor, reducer) +% Reduce each factor x factor x factor block into one voxel using +% REDUCER (a function handle). Output shape is ceil(size / factor); +% odd trailing indices reuse the last available block ("edge pad" -- +% simple, and mirrors what OME-Zarr writers commonly do). + sz = size(vol); + out_shape = ceil(sz / factor); + out = zeros(out_shape, 'like', vol); + for zi = 1:out_shape(1) + z0 = (zi - 1) * factor + 1; + z1 = min(z0 + factor - 1, sz(1)); + for yi = 1:out_shape(2) + y0 = (yi - 1) * factor + 1; + y1 = min(y0 + factor - 1, sz(2)); + for xi = 1:out_shape(3) + x0 = (xi - 1) * factor + 1; + x1 = min(x0 + factor - 1, sz(3)); + block = vol(z0:z1, y0:y1, x0:x1); + out(zi, yi, xi) = reducer(block); + end + end + end +end + +function v = meanReduce(block) + v = uint16(round(mean(double(block(:))))); +end + +function v = maxReduce(block) + v = max(block(:)); +end + +function writeZarrLevel(arrayDir, vol, chunkShape) +% Write one Zarr v2 array: a .zarray metadata file plus one raw chunk +% file per non-empty tile, keyed as z.y.x (default dimension_separator +% for Zarr v2). + if ~isfolder(arrayDir), mkdir(arrayDir); end + shape = size(vol); + writeJSON(fullfile(arrayDir, '.zarray'), struct( ... + 'zarr_format', 2, ... + 'shape', shape, ... + 'chunks', chunkShape, ... + 'dtype', ' 5 docs. + testCase.verifyEqual(numel(lds), 5); + % Parent describes the source volume, axes zyx. + p = pdoc.document_properties.lightsheetZarrPyramid; + testCase.verifyEqual(char(p.axes_order), 'zyx'); + testCase.verifyEqual(reshape(p.shape_level0, 1, []), [32 32 32]); + end + + function testMaxKeepsBrightSpikeMeanDoesNot(testCase) + % A single-voxel spike at (50,50,50) survives max but not + % mean; verify at the fixture's default (small) shape by + % rebuilding the level-0 volume and checking the coarsest + % level for max vs mean. + % At Shape=[32 32 32] the spike position falls outside; + % rebuild with a larger fixture to exercise the assertion. + [zp, gt2] = ndi.test.lightsheet.makeBlobFixture( ... + testCase.parentDir, 'Shape', [64 64 64], ... + 'NumLevels', 3, 'ChunkShape', [16 16 16]); + testCase.addTeardown(@() rmdir(zp, 's')); + L0 = gt2.volumes.level0; + testCase.verifyEqual(L0(50, 50, 50), uint16(60000)); + % At level 2 (4x downsample) the spike lives in block + % (13,13,13) (1-based ceil). Max keeps 60000, mean drops + % it by ~64 (in 4^3 = 64 voxels the spike is one). + maxTop = gt2.volumes.max{3}; + meanTop = gt2.volumes.mean{3}; + testCase.verifyEqual(maxTop(13, 13, 13), uint16(60000)); + testCase.verifyLessThan(meanTop(13, 13, 13), uint16(2000)); + end + + end +end + +% --------------------------------------------------------------------- + +function names = collectPyramidNames(ms) + if iscell(ms) + names = cellfun(@(e) char(e.name), ms, 'UniformOutput', false); + else + names = arrayfun(@(e) char(e.name), ms, 'UniformOutput', false); + end + names = names(:).'; +end + +function firsts = collectFirstPaths(ms) + n = numel(ms); + firsts = cell(1, n); + for k = 1:n + if iscell(ms) + entry = ms{k}; + else + entry = ms(k); + end + ds = entry.datasets; + if iscell(ds) + first = ds{1}; + else + first = ds(1); + end + firsts{k} = char(first.path); + end +end + From 41ddaa59a3f76bb136d33e639203be0b17ae1b58 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 00:24:11 +0000 Subject: [PATCH 019/110] Drop the redundant %#ok suppression on options.Seed Code Analyzer flagged the suppression as pointing at a message that no longer fires (MATLAB does not warn on an unused options field). Drop the pragma; the option itself stays, reserved for future noisy fixtures. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m index 3033d52f3..c1b7c1fca 100644 --- a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -76,7 +76,7 @@ options.NumLevels (1,1) double {mustBePositive, mustBeInteger} = 4 options.ChunkShape (1,3) double {mustBePositive, mustBeInteger} = [64 64 64] options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] - options.Seed (1,1) double = 42 %#ok + options.Seed (1,1) double = 42 end if isempty(parentDir) From 8e1c865f41dd69c97342a0ba51c4ada8501ecd22 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 00:39:30 +0000 Subject: [PATCH 020/110] Scale blob positions with Shape so small fixtures don't overflow buildBlobVolume put the spike at literal (50,50,50), which at Shape=[32 32 32] silently grew the zeros array to 50x50x50 and then crashed adding it to the 32x32x32 Gaussians. TestBlobFixture used Shape=[32 32 32] for speed, so every one of its six tests errored at setup. Position and width of all three features now scale with Shape: - wide Gaussian sigma = mean(Shape)/8 - medium Gaussian sigma = mean(Shape)/20 - spike position = round(Shape/6), clamped to at least 1 so the fixture is well-defined at any shape, from a 32^3 unit-test size to the 300^3 default. The mean-vs-max regression test now verifies the invariant that actually matters (max at the spike block stays at the spike value, mean is strictly below max there), not a magic 2000 threshold that only holds at one specific Shape. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+test/+lightsheet/makeBlobFixture.m | 33 +++++++++----- .../+test/+lightsheet/TestBlobFixture.m | 44 +++++++++++-------- 2 files changed, 47 insertions(+), 30 deletions(-) diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m index c1b7c1fca..c57953e7f 100644 --- a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -15,12 +15,16 @@ % viewed by the napari client that reads them back. % % The volume itself is three features at three spatial scales, so a -% viewer can tell the pyramid levels apart at a glance: -% - a wide Gaussian ball at (150,150,150), sigma 40, peak ~50k -% - a medium Gaussian at (200,100,150), sigma 15, peak ~30k -% - a bright single-voxel spike at (50,50,50), value 60k -% The spike survives max downsampling and vanishes under mean; that -% contrast makes the mean-vs-max choice visible in the viewer. +% viewer can tell the pyramid levels apart at a glance. Positions +% scale with Shape so the fixture is well-defined at any size: +% - a wide Gaussian ball at Shape/2, sigma Shape/8, peak ~50k +% - a medium Gaussian at (2/3, 1/3, 1/2)*Shape, sigma Shape/20, +% peak ~30k +% - a bright single-voxel spike of value 60k at Shape/6 +% The spike is much brighter than the local Gaussian background and +% fills one voxel out of the whole volume, so max downsampling +% holds it up while mean spreads it out over its block -- the +% mean-vs-max choice is visible at every pyramid level. % % PARENTDIR - char, directory to write into. Empty (default) puts % the store under a fresh tempname; the caller can pass @@ -169,17 +173,24 @@ % touching random state. Z = shape(1); Y = shape(2); X = shape(3); [zz, yy, xx] = ndgrid(1:Z, 1:Y, 1:X); + % Widths scale with the volume so the two Gaussians stay visible + % at every Shape; sigma_wide ~ Shape/8, sigma_medium ~ Shape/20. + sigmaWide = mean(shape) / 8; + sigmaMedium = mean(shape) / 20; % A wide ball centred in the volume. c1 = shape / 2; r1sq = (zz - c1(1)).^2 + (yy - c1(2)).^2 + (xx - c1(3)).^2; - wide = 50000 * exp(-r1sq / (2 * 40^2)); - % A medium blob offset to one corner. + wide = 50000 * exp(-r1sq / (2 * sigmaWide^2)); + % A medium blob offset toward one corner. c2 = [shape(1)*2/3 shape(2)/3 shape(3)/2]; r2sq = (zz - c2(1)).^2 + (yy - c2(2)).^2 + (xx - c2(3)).^2; - medium = 30000 * exp(-r2sq / (2 * 15^2)); - % One bright voxel: max keeps it, mean loses it. + medium = 30000 * exp(-r2sq / (2 * sigmaMedium^2)); + % One bright voxel: max keeps its value at every level, mean + % spreads it across the block. Position scales with Shape so a + % small test fixture puts the spike inside the volume too. + spikePos = max(1, round(shape / 6)); spike = zeros(shape); - spike(50, 50, 50) = 60000; + spike(spikePos(1), spikePos(2), spikePos(3)) = 60000; vol = uint16(min(65535, wide + medium + spike)); end diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index 23ba1a406..fce25224a 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -109,25 +109,31 @@ function testFromOMEZarrIngestsTheFixture(testCase) end function testMaxKeepsBrightSpikeMeanDoesNot(testCase) - % A single-voxel spike at (50,50,50) survives max but not - % mean; verify at the fixture's default (small) shape by - % rebuilding the level-0 volume and checking the coarsest - % level for max vs mean. - % At Shape=[32 32 32] the spike position falls outside; - % rebuild with a larger fixture to exercise the assertion. - [zp, gt2] = ndi.test.lightsheet.makeBlobFixture( ... - testCase.parentDir, 'Shape', [64 64 64], ... - 'NumLevels', 3, 'ChunkShape', [16 16 16]); - testCase.addTeardown(@() rmdir(zp, 's')); - L0 = gt2.volumes.level0; - testCase.verifyEqual(L0(50, 50, 50), uint16(60000)); - % At level 2 (4x downsample) the spike lives in block - % (13,13,13) (1-based ceil). Max keeps 60000, mean drops - % it by ~64 (in 4^3 = 64 voxels the spike is one). - maxTop = gt2.volumes.max{3}; - meanTop = gt2.volumes.mean{3}; - testCase.verifyEqual(maxTop(13, 13, 13), uint16(60000)); - testCase.verifyLessThan(meanTop(13, 13, 13), uint16(2000)); + % At the spike's downsampled position, max preserves the + % single-voxel 60k value and mean cannot (it averages the + % spike with (factor^L - 1) darker neighbours). Verified + % directly on the ground-truth ladder rather than through + % the on-disk store, so the assertion is about the + % downsampler, not the writer. + L0 = testCase.gt.volumes.level0; + spikePos = max(1, round(testCase.gt.shape / 6)); + testCase.verifyEqual( ... + L0(spikePos(1), spikePos(2), spikePos(3)), ... + uint16(60000)); + top = testCase.gt.numLevels; + factor = 2^(top - 1); + % 1-based block index containing the spike. + blk = floor((spikePos - 1) / factor) + 1; + maxTop = testCase.gt.volumes.max{top}; + meanTop = testCase.gt.volumes.mean{top}; + % Max at the spike's block stays at the spike value. + testCase.verifyEqual( ... + maxTop(blk(1), blk(2), blk(3)), uint16(60000)); + % Mean at the same block is strictly below max: averaging + % over factor^3 voxels dilutes the spike. + testCase.verifyLessThan( ... + double(meanTop(blk(1), blk(2), blk(3))), ... + double(maxTop(blk(1), blk(2), blk(3)))); end end From bf420e16bfd48265a03ae20461035c3ad989e58d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 00:51:15 +0000 Subject: [PATCH 021/110] Fix NDR resolution check and loosen spike-value assertion Two problems on the previous head: 1. `exist('ndr.format.omezarr.listPyramids', 'file')` returns 0 for MATLAB package functions even when they resolve, so both TestFromOMEZarr and TestBlobFixture.testFromOMEZarrIngestsTheFixture filtered when they should have run. `which(...)` is the right idiom; empty means not on path. 2. `verifyEqual(L0(spike), uint16(60000))` was too tight: the wide Gaussian bleeds a fraction of a unit into the spike voxel and uint16 rounds to 60001, not 60000. The invariant that matters is `>= 60000` (spike is at least the spike value) and mean(block) < max(block) at the spike's downsampled location, not exact equality on level 0. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+fun/+doc/+lightsheet/TestFromOMEZarr.m | 4 ++- .../+test/+lightsheet/TestBlobFixture.m | 27 +++++++++---------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m index 012c2c69f..e70ff20c3 100644 --- a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromOMEZarr.m @@ -16,7 +16,9 @@ methods (TestMethodSetup) function build(testCase) - if ~exist('ndr.format.omezarr.listPyramids', 'file') + % `exist(pkg.fun, 'file')` returns 0 for package functions + % even when they resolve, so guard on `which` instead. + if isempty(which('ndr.format.omezarr.listPyramids')) testCase.assumeFail(['NDR reader not on path; skipping ' ... 'end-to-end OME-Zarr ingest tests.']); end diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index fce25224a..2bedff10d 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -85,7 +85,9 @@ function testChunkFileSizeMatchesRawBytes(testCase) end function testFromOMEZarrIngestsTheFixture(testCase) - if ~exist('ndr.format.omezarr.listPyramids', 'file') + % `exist(pkg.fun, 'file')` returns 0 for package functions + % even when they resolve, so guard on `which` instead. + if isempty(which('ndr.format.omezarr.listPyramids')) testCase.assumeFail(['NDR reader not on path; skipping ' ... 'end-to-end ingest.']); end @@ -110,27 +112,24 @@ function testFromOMEZarrIngestsTheFixture(testCase) function testMaxKeepsBrightSpikeMeanDoesNot(testCase) % At the spike's downsampled position, max preserves the - % single-voxel 60k value and mean cannot (it averages the - % spike with (factor^L - 1) darker neighbours). Verified - % directly on the ground-truth ladder rather than through - % the on-disk store, so the assertion is about the - % downsampler, not the writer. + % 60k spike and mean dilutes it (averages the one spike + % voxel with factor^3-1 darker neighbours). Verified on + % the ground-truth ladder so the assertion is about the + % downsampler, not the on-disk writer. L0 = testCase.gt.volumes.level0; spikePos = max(1, round(testCase.gt.shape / 6)); - testCase.verifyEqual( ... - L0(spikePos(1), spikePos(2), spikePos(3)), ... - uint16(60000)); + % The wide Gaussian bleeds a fraction of a unit into the + % spike voxel, so L0 rounds to 60000 or 60001; a >= check + % holds either way, and 60000 is the sensible lower bound. + testCase.verifyGreaterThanOrEqual( ... + L0(spikePos(1), spikePos(2), spikePos(3)), uint16(60000)); top = testCase.gt.numLevels; factor = 2^(top - 1); - % 1-based block index containing the spike. blk = floor((spikePos - 1) / factor) + 1; maxTop = testCase.gt.volumes.max{top}; meanTop = testCase.gt.volumes.mean{top}; - % Max at the spike's block stays at the spike value. - testCase.verifyEqual( ... + testCase.verifyGreaterThanOrEqual( ... maxTop(blk(1), blk(2), blk(3)), uint16(60000)); - % Mean at the same block is strictly below max: averaging - % over factor^3 voxels dilutes the spike. testCase.verifyLessThan( ... double(meanTop(blk(1), blk(2), blk(3))), ... double(maxTop(blk(1), blk(2), blk(3)))); From 04faa350407a52b154e6aa4ba98fed4304b04e03 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 01:00:08 +0000 Subject: [PATCH 022/110] Cover ingestBlobFixture with the one-line-demo test Codecov flagged ingestBlobFixture.m at 0% -- nothing was calling it. Adds one TestBlobFixture case that runs the demo end to end at a 32^3 shape (fast) and verifies the session, pyramid document, and INFO struct come back with the fields the caller documents. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+test/+lightsheet/TestBlobFixture.m | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index 2bedff10d..4793e064c 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -110,6 +110,34 @@ function testFromOMEZarrIngestsTheFixture(testCase) testCase.verifyEqual(reshape(p.shape_level0, 1, []), [32 32 32]); end + function testIngestBlobFixtureIsOneCallEndToEnd(testCase) + % The one-line demo that ships the fixture and ingests it. + % Uses a tiny shape so it stays quick; the point is that + % session + subject + fromOMEZarr all happen in one call + % and INFO carries the fixture ground truth alongside the + % ingest info. + if isempty(which('ndr.format.omezarr.listPyramids')) + testCase.assumeFail(['NDR reader not on path; skipping ' ... + 'end-to-end ingest.']); + end + sDir = fullfile(tempname, 'blobingest'); + testCase.addTeardown(@() rmdir(fileparts(sDir), 's')); + zDir = fullfile(tempname, 'blobingest_zarr'); + testCase.addTeardown(@() rmdir(zDir, 's')); + [S, pdoc, info] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDir, ... + 'zarrDir', zDir, ... + 'Shape', [32 32 32], ... + 'NumLevels', 3, ... + 'ChunkShape', [16 16 16]); + testCase.verifyClass(S, 'ndi.session.dir'); + testCase.verifyClass(pdoc, 'ndi.document'); + testCase.verifyTrue(info.sharedLevel0); + testCase.verifyNotEmpty(info.subjectID); + testCase.verifyEqual(info.fixture.shape, [32 32 32]); + testCase.verifyEqual(info.fixture.chunkShape, [16 16 16]); + end + function testMaxKeepsBrightSpikeMeanDoesNot(testCase) % At the spike's downsampled position, max preserves the % 60k spike and mean dilutes it (averages the one spike From 3610ce290daa51b9fa0d5e700de6153714725ea5 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 14:03:12 +0000 Subject: [PATCH 023/110] Write .zgroup markers on the blob fixture so Zarr readers accept it Zarr v2 requires a .zgroup file at every group node. The fixture had none, so napari-ome-zarr (and every other conformant reader) rejected the store with "No group found in store". Add .zgroup at the root and at the mean/ and max/ subdirectories. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+test/+lightsheet/makeBlobFixture.m | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m index c57953e7f..d9eae9389 100644 --- a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -106,6 +106,19 @@ maxLadder{L} = downsampleBlockUint16(maxLadder{L-1}, 2, @maxReduce); end + % Every Zarr v2 group node needs a .zgroup marker, or readers + % refuse the store with "No group found". The root is one, and so + % are the mean/ and max/ subdirectories that hold the coarser + % levels for each reduction. Written before the array levels so + % the reader sees a valid group tree. + writeZGroup(zarrPath); + if options.NumLevels > 1 + mkdir(fullfile(zarrPath, 'mean')); + mkdir(fullfile(zarrPath, 'max')); + writeZGroup(fullfile(zarrPath, 'mean')); + writeZGroup(fullfile(zarrPath, 'max')); + end + % Write level 0 (shared) and each reduction's coarser levels. writeZarrLevel(fullfile(zarrPath, '0'), lvl0, options.ChunkShape); for L = 2:options.NumLevels @@ -282,6 +295,10 @@ function writeZarrLevel(arrayDir, vol, chunkShape) struct('type', 'scale', 'scale', scale) }}); end +function writeZGroup(dir) + writeJSON(fullfile(dir, '.zgroup'), struct('zarr_format', 2)); +end + function writeJSON(filePath, s) txt = jsonencode(s); fid = fopen(filePath, 'w'); From 89f4763191f01b58987a0d86e354a10c2faf6cfa Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 14:23:10 +0000 Subject: [PATCH 024/110] Emit compressor and filters as JSON null in Zarr .zarray MATLAB's jsonencode writes `[]` for an empty numeric array, but Zarr v2 requires `null` there -- zarr-python 3 refuses `[]` with "Expected None, a numcodecs.abc.Codec, or a dict ... Got ". Post- process the encoded string so those two fields become JSON null. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m index d9eae9389..3b1f65fa6 100644 --- a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -300,7 +300,14 @@ function writeZGroup(dir) end function writeJSON(filePath, s) + % MATLAB's jsonencode emits empty [] for MATLAB `[]`, but Zarr v2 + % expects null for `compressor` and `filters` -- a `[]` there is + % refused by the zarr-python 3 reader with "Expected None, a + % numcodecs.abc.Codec, or a dict ... Got instead." + % Post-process the string so those two fields become JSON null. txt = jsonencode(s); + txt = regexprep(txt, '"compressor":\[\]', '"compressor":null'); + txt = regexprep(txt, '"filters":\[\]', '"filters":null'); fid = fopen(filePath, 'w'); fwrite(fid, txt); fclose(fid); From c99436454c1586ccf09ad71a7b529724fc3780ac Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 14:56:08 +0000 Subject: [PATCH 025/110] Give the wide blob concentric-ring structure and pull it in from the edge Two adjustments to the fixture volume based on viewer feedback: - The wide Gaussian ball had no structure to explore at different scales; it looked identical (just softer) at every pyramid level. Replaced with a Gaussian envelope modulated by a radial cosine, so brightness oscillates in concentric shells as you move toward the centre. Fine levels resolve the individual rings; coarser levels smear them, so each pyramid step now shows different structure and the max vs mean contrast is visible at multiple scales. - The wide feature was sized sigma = mean(Shape)/8, so at the default 300^3 it ran essentially to the box edge. Tightened to sigma = mean(Shape)/10, keeping 3*sigma at ~3/10 of a side so the feature sits well inside the volume. Wavelength for the rings is the same (mean(Shape)/10), which puts ~3 ring cycles inside 3*sigma. The medium offset blob and the single-voxel spike are kept as-is; they are still the compact companion and the max-vs-mean marker. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+test/+lightsheet/makeBlobFixture.m | 44 +++++++++++++------ 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m index 3b1f65fa6..95c3c6086 100644 --- a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -17,11 +17,18 @@ % The volume itself is three features at three spatial scales, so a % viewer can tell the pyramid levels apart at a glance. Positions % scale with Shape so the fixture is well-defined at any size: -% - a wide Gaussian ball at Shape/2, sigma Shape/8, peak ~50k +% - a centred RING pattern: Gaussian envelope (sigma Shape/10) +% modulated by a radial cosine of wavelength Shape/10, so +% brightness oscillates in concentric shells around Shape/2. +% Fine levels resolve the individual rings; coarse levels +% smear them, so each pyramid step shows different structure. % - a medium Gaussian at (2/3, 1/3, 1/2)*Shape, sigma Shape/20, -% peak ~30k -% - a bright single-voxel spike of value 60k at Shape/6 -% The spike is much brighter than the local Gaussian background and +% peak ~30k -- a compact companion that stays a small ball at +% every level. +% - a bright single-voxel spike of value 60k at Shape/6. +% The ring envelope is 3*sigma ~ 3/10 of a side, so the feature +% stays well inside the box rather than running to the edge. The +% spike is much brighter than the local Gaussian background and % fills one voxel out of the whole volume, so max downsampling % holds it up while mean spreads it out over its block -- the % mean-vs-max choice is visible at every pyramid level. @@ -181,20 +188,29 @@ % ===================================================================== function vol = buildBlobVolume(shape) -% Deterministic 3D uint16 volume with three features at three scales. -% Everything is closed-form so the fixture is reproducible without -% touching random state. +% Deterministic 3D uint16 volume with three features at three spatial +% scales. Every feature is closed-form so the fixture is reproducible +% without touching random state. Z = shape(1); Y = shape(2); X = shape(3); [zz, yy, xx] = ndgrid(1:Z, 1:Y, 1:X); - % Widths scale with the volume so the two Gaussians stay visible - % at every Shape; sigma_wide ~ Shape/8, sigma_medium ~ Shape/20. - sigmaWide = mean(shape) / 8; + % Widths scale with the volume so the fixture is well-defined at + % any Shape. The wide feature sits well inside the box (3*sigma + % ~ 3/10 of a side) rather than running to the edge. + sigmaWide = mean(shape) / 10; sigmaMedium = mean(shape) / 20; - % A wide ball centred in the volume. + lambdaWide = mean(shape) / 10; % ~3 rings inside 3*sigma + % A wide ring pattern centred in the volume: a Gaussian envelope + % modulated by a radial cosine so brightness oscillates in + % concentric shells as you move toward the centre. Fine levels + % resolve the individual rings; coarse levels smear them, so each + % pyramid step shows something new. Amplitude clipped to 50k so + % the offset spike still stands out above it. c1 = shape / 2; - r1sq = (zz - c1(1)).^2 + (yy - c1(2)).^2 + (xx - c1(3)).^2; - wide = 50000 * exp(-r1sq / (2 * sigmaWide^2)); - % A medium blob offset toward one corner. + r1 = sqrt((zz - c1(1)).^2 + (yy - c1(2)).^2 + (xx - c1(3)).^2); + envelope = exp(-r1.^2 / (2 * sigmaWide^2)); + rings = 0.5 + 0.5 * cos(2 * pi * r1 / lambdaWide); + wide = 50000 * envelope .* rings; + % A medium blob offset toward one corner (kept as-is). c2 = [shape(1)*2/3 shape(2)/3 shape(3)/2]; r2sq = (zz - c2(1)).^2 + (yy - c2(2)).^2 + (xx - c2(3)).^2; medium = 30000 * exp(-r2sq / (2 * sigmaMedium^2)); From 3820dd3e97d032f7de5de5f4cff367f53930b5e4 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 15:01:14 +0000 Subject: [PATCH 026/110] Widen wide-feature wavelength and split its amplitude for the pyramid At Shape=300 the previous wavelength (Shape/10 = 30 voxels) came to ~3.75 voxels at level 3, right at the 2-voxel Nyquist limit. Zoomed out, napari renders the coarsest pyramid level, and max-pooling a sharp cosine at Nyquist produces the blocky rectangular artifacts the first render showed. Zoomed in, level 0 rendered the rings cleanly -- so the fixture only failed at coarse zooms. Widen the wavelength to Shape/5 (60 voxels at level 0, 7.5 voxels at level 3 -- clearly resolvable) and split the amplitude as 25k baseline plus 25k oscillation, so max downsampling does not lock onto the peak-only value and the coarse levels stay smooth. Fewer rings (about 2 clear ones instead of 3) but they hold up at every zoom level. Also expanded sigmaWide from Shape/10 to Shape/8 -- the envelope was too tight at the previous width and cropped the outer ring. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m index 95c3c6086..5fd3a1946 100644 --- a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -196,20 +196,24 @@ % Widths scale with the volume so the fixture is well-defined at % any Shape. The wide feature sits well inside the box (3*sigma % ~ 3/10 of a side) rather than running to the edge. - sigmaWide = mean(shape) / 10; + sigmaWide = mean(shape) / 8; % contain 3*sigma at ~3/8 of a side sigmaMedium = mean(shape) / 20; - lambdaWide = mean(shape) / 10; % ~3 rings inside 3*sigma + lambdaWide = mean(shape) / 5; % ~2 rings inside 3*sigma, + % ~7.5 voxels at level 3 (8x down), + % well above the 2-voxel Nyquist % A wide ring pattern centred in the volume: a Gaussian envelope % modulated by a radial cosine so brightness oscillates in % concentric shells as you move toward the centre. Fine levels - % resolve the individual rings; coarse levels smear them, so each - % pyramid step shows something new. Amplitude clipped to 50k so - % the offset spike still stands out above it. + % show the individual rings; coarse levels smooth them, so each + % pyramid step reads differently. Amplitude split as a smooth + % baseline (25k, the ring troughs) plus a smaller oscillation + % (25k, the ring peaks) so max downsampling does not lock onto + % the peak-only value and produce blocky per-tile output. c1 = shape / 2; r1 = sqrt((zz - c1(1)).^2 + (yy - c1(2)).^2 + (xx - c1(3)).^2); envelope = exp(-r1.^2 / (2 * sigmaWide^2)); rings = 0.5 + 0.5 * cos(2 * pi * r1 / lambdaWide); - wide = 50000 * envelope .* rings; + wide = envelope .* (25000 + 25000 * rings); % A medium blob offset toward one corner (kept as-is). c2 = [shape(1)*2/3 shape(2)/3 shape(3)/2]; r2sq = (zz - c2(1)).^2 + (yy - c2(2)).^2 + (xx - c2(3)).^2; From bd6a245dbbe11d71d08d31062ebe17089d1cfa9a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 15:19:16 +0000 Subject: [PATCH 027/110] Materialize chunks, add ndi.fun.doc.lightsheet.view, wire it end to end MATLAB side, three additions that make napariViewLightsheet actually render voxels: - makePyramid gains materializeChunks (default false) and sourceZarrPath options. When materializeChunks=true the worker reads each level from the source OME-Zarr with ndr.format.omezarr.readArray, re-tiles it to the level document's chunks shape, and attaches each chunk as chunk.bin_ via add_file BEFORE database_add -- documents are immutable after add, so materialisation has to happen inside the same maker call. n_chunks_stored is set to prod(chunk_grid) in that mode; the metadata-only default keeps its 0. - fromOMEZarr forwards materializeChunks and pre-fills sourceZarrPath from its own zarrPath argument, so a caller writes one option rather than a pair. - ingestBlobFixture defaults materializeChunks to TRUE. The end-to-end demo now produces a session whose level docs actually own the bytes napariViewLightsheet expects, not just their descriptions. - ndi.fun.doc.lightsheet.view is the requested caller: builds the napariViewLightsheet command from viewCommand and dispatches it through SYSTEM, with an optional PYRAMIDID (empty means --list). Launcher defaults to /usr/local/bin/napariViewLightsheet, the shell wrapper installed alongside NDI-python; sessionPath is read from the session unless overridden. Backgrounds the launcher on POSIX so MATLAB stays interactive. The chunk write is C-order bytes to match Zarr v2 -- MATLAB permutes into reverse axis order before serialising, mirroring the fixture writer. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 5 +- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 118 +++++++++++++++++- src/ndi/+ndi/+fun/+doc/+lightsheet/view.m | 104 +++++++++++++++ .../+test/+lightsheet/ingestBlobFixture.m | 5 +- 4 files changed, 229 insertions(+), 3 deletions(-) create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/view.m diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 706586df7..6492a85d9 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -84,6 +84,7 @@ options.label char = '' options.tileBudgetBytes (1,1) double {mustBePositive} = 8 * 2^20 options.chunks double = [] + options.materializeChunks (1,1) logical = false end if isempty(strtrim(options.subjectID)) @@ -145,7 +146,9 @@ 'pipelineVersion', options.pipelineVersion, ... 'label', options.label, ... 'tileBudgetBytes', options.tileBudgetBytes, ... - 'chunks', options.chunks); + 'chunks', options.chunks, ... + 'materializeChunks', options.materializeChunks, ... + 'sourceZarrPath', char(zarrPath)); info = struct( ... 'zarrPath', char(zarrPath), ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 3145ee39d..a68da6a5c 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -68,6 +68,14 @@ options.label char = '' options.tileBudgetBytes (1,1) double {mustBePositive} = 8 * 2^20 options.chunks double = [] + options.materializeChunks (1,1) logical = false + options.sourceZarrPath char = '' + end + + if options.materializeChunks && isempty(options.sourceZarrPath) + error('NDI:lightsheet:makePyramid:noSource', ... + ['materializeChunks=true requires sourceZarrPath so ' ... + 'chunks can be read from the source store.']); end if numel(pyramids) ~= numel(perEntryReduction) @@ -172,6 +180,12 @@ end chunkGrid = ceil(level.shape ./ chunks); + if options.materializeChunks + nChunksStored = prod(chunkGrid); + else + nChunksStored = 0; + end + props = struct( ... 'label', sprintf('%s level %d (%s)', pyramidEntry.name, levelIndex, reductionFn), ... 'level', levelIndex, ... @@ -180,7 +194,7 @@ 'shape', level.shape, ... 'chunks', chunks, ... 'chunk_grid', chunkGrid, ... - 'n_chunks_stored', 0, ... + 'n_chunks_stored', nChunksStored, ... 'chunk_index_origin', 1, ... 'chunk_order', 'C', ... 'dtype', char(level.dtype), ... @@ -201,9 +215,111 @@ levelDoc = levelDoc.set_dependency_value('source_file_id', options.sourceFileID); end + if options.materializeChunks + levelDoc = attachChunkFiles(levelDoc, options.sourceZarrPath, ... + pyramidEntry, level, chunks, chunkGrid); + end + session.database_add(levelDoc); end +function levelDoc = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid) +% Read the source zarr level as a whole array, then re-tile into the +% level document's chunk shape and attach each chunk as chunk.bin_ +% (1-based, C-order over chunkGrid). Files are ingested into DID, which +% deletes the temp original as it copies -- so we build each chunk in a +% new tempname and hand it to add_file. + pyramidName = char(pyramidEntry.name); + % Which position in this pyramid's datasets holds this level? The + % `level` struct came from listPyramids(entry).levels; its index is + % the 1-based position we need for readArray. + kInEntry = find(arrayfun(@(L) strcmp(char(L.path), char(level.path)), ... + pyramidEntry.levels), 1); + if isempty(kInEntry) + error('NDI:lightsheet:makePyramid:missingLevelInEntry', ... + 'level path %s not found in pyramid %s.', level.path, pyramidName); + end + + % Read the full array. For a fixture-scale volume (~54 MB uint16) + % this is a single allocation; a lab-scale volume will need a + % streaming reader later, but the level doc's chunk shape is fixed + % here either way. + data = ndr.format.omezarr.readArray(sourceZarrPath, pyramidName, kInEntry); + + shape = size(data); + if numel(shape) < numel(chunks) + shape(end+1:numel(chunks)) = 1; %#ok singleton trailing axes + end + + % Iterate chunks in C-order (last axis fastest). 1-based linear + % index goes into the chunk.bin_# filename. + idx = 0; + tmpRoot = tempname; + mkdir(tmpRoot); + cleaner = onCleanup(@() cleanupTmp(tmpRoot)); + subs = cell(1, numel(chunkGrid)); + linearOrder = allChunkIndices(chunkGrid); % rows are (c1, c2, c3, ...) 1-based + for r = 1:size(linearOrder, 1) + idx = idx + 1; + cIdx = linearOrder(r, :); + % Compute the source slice and the padded chunk. + for a = 1:numel(chunkGrid) + lo = (cIdx(a) - 1) * chunks(a) + 1; + hi = min(lo + chunks(a) - 1, shape(a)); + subs{a} = lo:hi; + end + srcTile = data(subs{:}); + padded = zeros(chunks, 'like', data); + localSubs = arrayfun(@(k) 1:size(srcTile, k), 1:numel(chunks), ... + 'UniformOutput', false); + padded(localSubs{:}) = srcTile; + + % Zarr expects C-order bytes: last axis varies fastest. MATLAB + % is column-major, so permute the axes into reversed order + % before serialising, mirroring the fixture writer. + permuted = permute(padded, numel(chunks):-1:1); + raw = typecast(permuted(:), 'uint8'); + + tmp = fullfile(tmpRoot, sprintf('chunk_%d.bin', idx)); + fid = fopen(tmp, 'w'); + fwrite(fid, raw); + fclose(fid); + + levelDoc = levelDoc.add_file(sprintf('chunk.bin_%d', idx), tmp); + end +end + +function cleanupTmp(d) + if isfolder(d) + try + rmdir(d, 's'); + catch + end + end +end + +function idx = allChunkIndices(chunkGrid) +% Return a matrix whose rows enumerate every 1-based chunk index tuple +% in C-order (last axis fastest, so the LINEAR order matches +% index = c1*n2*n3*... + c2*n3*... + ... + cn + 1). + n = numel(chunkGrid); + counts = cumprod(fliplr(chunkGrid)); + total = counts(end); + idx = zeros(total, n); + for r = 1:total + rem = r - 1; + for a = 1:n + % last axis is fastest + stride = 1; + for b = a+1:n + stride = stride * chunkGrid(b); + end + idx(r, a) = floor(rem / stride) + 1; + rem = mod(rem, stride); + end + end +end + function s = joinAxisNames(axes) if isempty(axes) s = ''; diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m new file mode 100644 index 000000000..150156eb6 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m @@ -0,0 +1,104 @@ +function view(session, pyramidID, options) +% NDI.FUN.DOC.LIGHTSHEET.VIEW - launch the napari viewer on a lightsheet pyramid +% +% ndi.fun.doc.lightsheet.VIEW(SESSION) +% ndi.fun.doc.lightsheet.VIEW(SESSION, PYRAMIDID) +% ndi.fun.doc.lightsheet.VIEW(..., 'launcher', '/path/to/napariViewLightsheet') +% +% Builds the napariViewLightsheet command line (via +% ndi.fun.doc.lightsheet.VIEWCOMMAND) and dispatches it through SYSTEM. +% The launcher path defaults to /usr/local/bin/napariViewLightsheet, +% the shell wrapper installed alongside NDI-python. +% +% PYRAMIDID is optional. When omitted, the launcher runs in --list +% mode: it enumerates every lightsheetZarrPyramid in the session and +% prints their ids on stdout so the caller can pick one for a second +% call. When PYRAMIDID is passed, the pyramid is opened in napari. +% +% Optional Name-Value Arguments: +% launcher - char, path to the shell wrapper. Default +% /usr/local/bin/napariViewLightsheet. +% sessionPath - char, override the session directory used for the +% viewer's --session argument. Default: session.path(). +% reduction - char, '' | 'mean' | 'max'. Passed through to +% viewCommand; empty leaves the choice to what the +% pyramid id names. +% channel - 1-based channel index, or -1 for all channels. +% Passed through to viewCommand. +% level - initial level; -1 lets the viewer choose. +% controls - dock the reduction / channel / level control panels +% (default true). +% name - napari layer name; empty uses the pyramid's own +% label. +% wait - block until the viewer exits (default false so +% MATLAB stays interactive; the command is +% backgrounded on POSIX shells). +% verbose - print the command before running it (default true). +% +% Example -- list, then open: +% ndi.fun.doc.lightsheet.view(S); % prints ids +% ndi.fun.doc.lightsheet.view(S, ''); +% +% See also: ndi.fun.doc.lightsheet.viewCommand, +% ndi.gui.app.LightsheetZarrManager + + arguments + session (1,1) + pyramidID char = '' + options.launcher (1,:) char = '/usr/local/bin/napariViewLightsheet' + options.sessionPath (1,:) char = '' + options.reduction (1,:) char {mustBeMember(options.reduction, {'', 'mean', 'max'})} = '' + options.channel (1,1) double {mustBeInteger} = -1 + options.level (1,1) double {mustBeInteger} = -1 + options.controls (1,1) logical = true + options.name (1,:) char = '' + options.wait (1,1) logical = false + options.verbose (1,1) logical = true + end + + sessionPath = options.sessionPath; + if isempty(sessionPath) + try + sessionPath = char(session.path()); + catch + error('NDI:lightsheet:view:noSessionPath', ... + ['Could not read a directory from the session; pass ' ... + '''sessionPath'' explicitly.']); + end + end + + if isempty(pyramidID) + cmd = sprintf('%s %s --list', shq(options.launcher), shq(sessionPath)); + else + cmd = ndi.fun.doc.lightsheet.viewCommand(options.launcher, ... + sessionPath, pyramidID, ... + 'channel', options.channel, ... + 'reduction', options.reduction, ... + 'controls', options.controls, ... + 'name', options.name, ... + 'level', options.level); + end + + if ~options.wait && ispc == 0 %#ok + % Background the process on POSIX so MATLAB does not block. + % On Windows use `start` (kept simple: fall back to a + % foreground wait). + cmd = [cmd ' &']; + end + + if options.verbose + fprintf('Executing: %s\n', cmd); + end + status = system(cmd); + if status ~= 0 && options.wait + warning('NDI:lightsheet:view:nonZeroExit', ... + 'Launcher returned status %d.', status); + end +end + +function s = shq(s) +% Minimal POSIX shell quoting: wrap in single quotes and escape any +% single quote inside. Enough for paths and identifiers; not a general +% shell escaper. + s = ['''' strrep(s, '''', '''\''''') '''']; +end diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index ffd14229f..5719876b5 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -55,6 +55,7 @@ options.ChunkShape (1,3) double {mustBePositive, mustBeInteger} = [64 64 64] options.NumLevels (1,1) double {mustBePositive, mustBeInteger} = 4 options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] + options.materializeChunks (1,1) logical = true end sessionDir = options.sessionDir; @@ -85,7 +86,9 @@ session.database_add(sub); [pyramidDoc, ~, info] = ndi.fun.doc.lightsheet.fromOMEZarr( ... - session, zarrPath, 'subjectID', sub.id()); + session, zarrPath, ... + 'subjectID', sub.id(), ... + 'materializeChunks', options.materializeChunks); info.subjectID = sub.id(); info.fixture = gt; From 5f631f2e1765017d244e1f7e23e2ab0cfd2bf816 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 15:27:10 +0000 Subject: [PATCH 028/110] Drop redundant Code Analyzer suppressions - makePyramid.m: dropped %#ok on the singleton-trailing-axis pad; the assignment is not in a loop so the growing-array warning never fires and the suppression was flagged as unused. - view.m: replaced `ispc == 0 %#ok` with the idiomatic `~ispc`; the equality form triggered the suppression and the idiomatic form needs neither. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 3 ++- src/ndi/+ndi/+fun/+doc/+lightsheet/view.m | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index a68da6a5c..d9d9631a0 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -248,7 +248,8 @@ shape = size(data); if numel(shape) < numel(chunks) - shape(end+1:numel(chunks)) = 1; %#ok singleton trailing axes + % Singleton trailing axes: pad shape to axis count. + shape(end+1:numel(chunks)) = 1; end % Iterate chunks in C-order (last axis fastest). 1-based linear diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m index 150156eb6..9938dba95 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m @@ -79,10 +79,10 @@ function view(session, pyramidID, options) 'level', options.level); end - if ~options.wait && ispc == 0 %#ok + if ~options.wait && ~ispc % Background the process on POSIX so MATLAB does not block. - % On Windows use `start` (kept simple: fall back to a - % foreground wait). + % On Windows fall back to a foreground wait (simple; no + % `start` handling here). cmd = [cmd ' &']; end From a7ba07aa0ee7a214c6c0e556dc178880f83dc213 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 17:02:49 +0000 Subject: [PATCH 029/110] Add Blosc+Zstd codec to the chunk materializer New `codec` and `clevel` options on makePyramid, fromOMEZarr and ingestBlobFixture. codec='raw' (the default) keeps the current behaviour: chunk.bin_ holds raw C-order bytes and the level document records codec='raw'. codec='blosc-zstd' byte-shuffles the chunk, feeds it through the system zstd binary, wraps the result in a Blosc v1 container (matching numcodecs.Blosc(cname='zstd')), and writes THAT to the chunk file. The level document records codec='blosc-zstd' and codec_params JSON with cname/clevel/shuffle so the reader can inverse-decode without external metadata. Codec bytes and container layout follow NDR's fixture writer, so chunks written here interoperate byte-for-byte with the standard Zarr-Python and numcodecs decoders. The zstd binary is required for the compressed path; a clear error names how to install it. Expected on tissue-shaped lightsheet uint16 volumes: 3-5x compression, 10x+ where large fractions of the volume are black background. CPU cost stays well under bandwidth on modern boxes. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 6 +- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 152 +++++++++++++++++- .../+test/+lightsheet/ingestBlobFixture.m | 6 +- 3 files changed, 157 insertions(+), 7 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 6492a85d9..2848129fd 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -85,6 +85,8 @@ options.tileBudgetBytes (1,1) double {mustBePositive} = 8 * 2^20 options.chunks double = [] options.materializeChunks (1,1) logical = false + options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' + options.clevel (1,1) double {mustBeInteger, mustBeInRange(options.clevel, 1, 22)} = 5 end if isempty(strtrim(options.subjectID)) @@ -148,7 +150,9 @@ 'tileBudgetBytes', options.tileBudgetBytes, ... 'chunks', options.chunks, ... 'materializeChunks', options.materializeChunks, ... - 'sourceZarrPath', char(zarrPath)); + 'sourceZarrPath', char(zarrPath), ... + 'codec', options.codec, ... + 'clevel', options.clevel); info = struct( ... 'zarrPath', char(zarrPath), ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index d9d9631a0..7ea133344 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -70,6 +70,8 @@ options.chunks double = [] options.materializeChunks (1,1) logical = false options.sourceZarrPath char = '' + options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' + options.clevel (1,1) double {mustBeInteger, mustBeInRange(options.clevel, 1, 22)} = 5 end if options.materializeChunks && isempty(options.sourceZarrPath) @@ -200,8 +202,8 @@ 'dtype', char(level.dtype), ... 'fill_value', 0, ... 'byte_order', 'little', ... - 'codec', 'raw', ... - 'codec_params', '{}', ... + 'codec', options.codec, ... + 'codec_params', codecParamsJSON(options), ... 'voxel_size', ensureRow(level.scale), ... 'voxel_size_units', 'micrometer', ... 'translation', ensureRow(level.translation)); @@ -217,13 +219,14 @@ if options.materializeChunks levelDoc = attachChunkFiles(levelDoc, options.sourceZarrPath, ... - pyramidEntry, level, chunks, chunkGrid); + pyramidEntry, level, chunks, chunkGrid, ... + options.codec, options.clevel); end session.database_add(levelDoc); end -function levelDoc = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid) +function levelDoc = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel) % Read the source zarr level as a whole array, then re-tile into the % level document's chunk shape and attach each chunk as chunk.bin_ % (1-based, C-order over chunkGrid). Files are ingested into DID, which @@ -281,15 +284,154 @@ permuted = permute(padded, numel(chunks):-1:1); raw = typecast(permuted(:), 'uint8'); + switch codec + case 'raw' + payload = raw; + case 'blosc-zstd' + typesize = elementSizeBytes(class(padded)); + payload = encodeBloscZstd(raw, typesize, clevel); + otherwise + error('NDI:lightsheet:makePyramid:unknownCodec', ... + 'Unknown codec %s.', codec); + end + tmp = fullfile(tmpRoot, sprintf('chunk_%d.bin', idx)); fid = fopen(tmp, 'w'); - fwrite(fid, raw); + fwrite(fid, payload); fclose(fid); levelDoc = levelDoc.add_file(sprintf('chunk.bin_%d', idx), tmp); end end +function s = codecParamsJSON(options) +% JSON string stamped into the level document so a reader knows how to +% decode. For 'raw' the params are empty; for Blosc-Zstd they name the +% codec identity numcodecs.Blosc expects. + switch options.codec + case 'raw' + s = '{}'; + case 'blosc-zstd' + s = sprintf(['{"cname":"zstd","clevel":%d,"shuffle":1,' ... + '"blocksize":0}'], options.clevel); + otherwise + s = '{}'; + end +end + +function b = elementSizeBytes(className) + switch className + case {'uint8','int8','logical'} + b = 1; + case {'uint16','int16'} + b = 2; + case {'uint32','int32','single'} + b = 4; + case {'uint64','int64','double'} + b = 8; + otherwise + b = 2; + end +end + +function container = encodeBloscZstd(raw, typesize, clevel) +% Encode a byte vector as a Blosc v1 container with the Zstd codec. +% +% Matches numcodecs.Blosc(cname='zstd', shuffle=BYTE_SHUFFLE): byte- +% shuffle the input, compress the shuffled bytes with system zstd, then +% wrap in one Blosc block. The container layout is the same NDR's +% test-fixture writer uses, so the same Python/MATLAB reader path +% decodes it. + raw = raw(:); + nbytes = numel(raw); + if mod(nbytes, typesize) ~= 0 + error('NDI:lightsheet:makePyramid:badTypesize', ... + 'Chunk bytes (%d) not a multiple of typesize (%d).', ... + nbytes, typesize); + end + shuffled = shuffleBytes(raw, typesize); + blockPayload = compressZstd(shuffled, clevel); + + blocksize = nbytes; + cbytes = 16 + 4 + 4 + numel(blockPayload); + + hdr = zeros(16, 1, 'uint8'); + hdr(1) = 2; % blosc format version + hdr(2) = 1; % versionlz (zstd) + hdr(3) = uint8(hex2dec('91')); % flags: byte shuffle + zstd id + hdr(4) = uint8(typesize); + hdr(5:8) = typecast(uint32(nbytes), 'uint8'); + hdr(9:12) = typecast(uint32(blocksize), 'uint8'); + hdr(13:16) = typecast(uint32(cbytes), 'uint8'); + + offsetTable = typecast(uint32(16 + 4), 'uint8'); + blockHeader = typecast(uint32(numel(blockPayload)), 'uint8'); + container = [hdr(:); offsetTable(:); blockHeader(:); blockPayload(:)]; +end + +function out = shuffleBytes(bytesIn, typesize) +% Byte shuffle: interleaved element bytes -> [byte0 for all elems | +% byte1 for all elems | ...]. Standard Blosc pre-filter. + bytesIn = bytesIn(:); + n = numel(bytesIn); + nelems = n / typesize; + lanes = reshape(bytesIn, typesize, nelems); + out = reshape(lanes.', [], 1); +end + +function payload = compressZstd(bytesIn, clevel) +% Shell out to the system zstd binary. Same pattern NDR's fixture +% writer uses; keeps the toolchain small and the container binary- +% identical to what numcodecs writes. + zstdBin = locateZstd(); + inPath = [tempname() '.bin']; + outPath = [tempname() '.zst']; + cleaner = onCleanup(@() safeDelete({inPath, outPath})); %#ok + fid = fopen(inPath, 'wb'); + fwrite(fid, bytesIn, 'uint8'); + fclose(fid); + cmd = sprintf('%s -qf -%d --no-check -o %s %s', ... + shellQuote(zstdBin), clevel, shellQuote(outPath), shellQuote(inPath)); + [status, out] = system(cmd); + if status ~= 0 + error('NDI:lightsheet:makePyramid:zstdFailed', ... + 'zstd compression failed (exit %d): %s', status, strtrim(out)); + end + fid = fopen(outPath, 'rb'); + payload = fread(fid, inf, '*uint8'); + fclose(fid); +end + +function bin = locateZstd() + persistent cached + if ~isempty(cached), bin = cached; return; end + if ispc, [s, out] = system('where zstd'); else, [s, out] = system('command -v zstd'); end + if s == 0 + candidate = strtrim(strsplit(strtrim(out), newline)); + candidate = candidate{1}; + if ~isempty(candidate) && exist(candidate, 'file') == 2 + cached = candidate; bin = cached; return; + end + end + error('NDI:lightsheet:makePyramid:zstdNotFound', ... + ['The zstd executable is required for codec=blosc-zstd and ' ... + 'was not found on PATH. Install: `brew install zstd`, ' ... + '`apt install zstd`, or download for Windows from ' ... + 'https://github.com/facebook/zstd/releases.']); +end + +function s = shellQuote(p) + if ispc, s = ['"' p '"']; else, s = ['''' strrep(p, '''', '''\''''') '''']; end +end + +function safeDelete(paths) + for i = 1:numel(paths) + if exist(paths{i}, 'file') == 2 + try, delete(paths{i}); catch, end + end + end +end + function cleanupTmp(d) if isfolder(d) try diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index 5719876b5..7241272b2 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -56,6 +56,8 @@ options.NumLevels (1,1) double {mustBePositive, mustBeInteger} = 4 options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] options.materializeChunks (1,1) logical = true + options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' + options.clevel (1,1) double {mustBeInteger, mustBeInRange(options.clevel, 1, 22)} = 5 end sessionDir = options.sessionDir; @@ -88,7 +90,9 @@ [pyramidDoc, ~, info] = ndi.fun.doc.lightsheet.fromOMEZarr( ... session, zarrPath, ... 'subjectID', sub.id(), ... - 'materializeChunks', options.materializeChunks); + 'materializeChunks', options.materializeChunks, ... + 'codec', options.codec, ... + 'clevel', options.clevel); info.subjectID = sub.id(); info.fixture = gt; From f3a430b6511fe4d7bc591bdbeba675238e722ec2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 17:06:34 +0000 Subject: [PATCH 030/110] Address Code Analyzer notes on the codec code - Replaced mustBeInRange (deprecated recommendation) with the pair mustBeGreaterThanOrEqual + mustBeLessThanOrEqual on the clevel option in makePyramid, fromOMEZarr, and ingestBlobFixture. - Dropped the %#ok on the onCleanup handle in compressZstd; the message is no longer generated. - Cleaned up the try/catch idiom in safeDelete: multi-line form instead of `try delete(...); catch, end`. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 2 +- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 9 ++++++--- src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m | 2 +- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 2848129fd..5cebd3dc8 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -86,7 +86,7 @@ options.chunks double = [] options.materializeChunks (1,1) logical = false options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' - options.clevel (1,1) double {mustBeInteger, mustBeInRange(options.clevel, 1, 22)} = 5 + options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 22)} = 5 end if isempty(strtrim(options.subjectID)) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 7ea133344..ebf348e46 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -71,7 +71,7 @@ options.materializeChunks (1,1) logical = false options.sourceZarrPath char = '' options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' - options.clevel (1,1) double {mustBeInteger, mustBeInRange(options.clevel, 1, 22)} = 5 + options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 22)} = 5 end if options.materializeChunks && isempty(options.sourceZarrPath) @@ -386,7 +386,7 @@ zstdBin = locateZstd(); inPath = [tempname() '.bin']; outPath = [tempname() '.zst']; - cleaner = onCleanup(@() safeDelete({inPath, outPath})); %#ok + cleaner = onCleanup(@() safeDelete({inPath, outPath})); fid = fopen(inPath, 'wb'); fwrite(fid, bytesIn, 'uint8'); fclose(fid); @@ -427,7 +427,10 @@ function safeDelete(paths) for i = 1:numel(paths) if exist(paths{i}, 'file') == 2 - try, delete(paths{i}); catch, end + try + delete(paths{i}); + catch + end end end end diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index 7241272b2..63ca98d36 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -57,7 +57,7 @@ options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] options.materializeChunks (1,1) logical = true options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' - options.clevel (1,1) double {mustBeInteger, mustBeInRange(options.clevel, 1, 22)} = 5 + options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 22)} = 5 end sessionDir = options.sessionDir; From f37fbf1a414c9ad988bd52d016429af3660e0ce3 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 18:24:59 +0000 Subject: [PATCH 031/110] makePyramid: use ndr.format.blosc.encode instead of shelling to zstd Drops the 100-line shell-out (encodeBloscZstd + shuffleBytes + compressZstd + locateZstd + shellQuote + safeDelete) and calls the new ndr.format.blosc.encode instead. That backs the codec on the numcodecs Blosc implementation running in a private venv NDR sets up on first use, which removes the "install `zstd` on PATH" requirement across Windows/macOS/Linux. Container bytes are the same shape numcodecs writes (Blosc v1, byte-shuffle stride = element size), so readers on both sides -- the Python multiscale reader and NDR's private/decompressBlosc.m -- keep working unchanged. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 107 +----------------- 1 file changed, 5 insertions(+), 102 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index ebf348e46..747a8f83a 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -289,7 +289,11 @@ payload = raw; case 'blosc-zstd' typesize = elementSizeBytes(class(padded)); - payload = encodeBloscZstd(raw, typesize, clevel); + payload = ndr.format.blosc.encode(raw, ... + 'typesize', typesize, ... + 'cname', 'zstd', ... + 'clevel', clevel, ... + 'shuffle', 1); otherwise error('NDI:lightsheet:makePyramid:unknownCodec', ... 'Unknown codec %s.', codec); @@ -334,107 +338,6 @@ end end -function container = encodeBloscZstd(raw, typesize, clevel) -% Encode a byte vector as a Blosc v1 container with the Zstd codec. -% -% Matches numcodecs.Blosc(cname='zstd', shuffle=BYTE_SHUFFLE): byte- -% shuffle the input, compress the shuffled bytes with system zstd, then -% wrap in one Blosc block. The container layout is the same NDR's -% test-fixture writer uses, so the same Python/MATLAB reader path -% decodes it. - raw = raw(:); - nbytes = numel(raw); - if mod(nbytes, typesize) ~= 0 - error('NDI:lightsheet:makePyramid:badTypesize', ... - 'Chunk bytes (%d) not a multiple of typesize (%d).', ... - nbytes, typesize); - end - shuffled = shuffleBytes(raw, typesize); - blockPayload = compressZstd(shuffled, clevel); - - blocksize = nbytes; - cbytes = 16 + 4 + 4 + numel(blockPayload); - - hdr = zeros(16, 1, 'uint8'); - hdr(1) = 2; % blosc format version - hdr(2) = 1; % versionlz (zstd) - hdr(3) = uint8(hex2dec('91')); % flags: byte shuffle + zstd id - hdr(4) = uint8(typesize); - hdr(5:8) = typecast(uint32(nbytes), 'uint8'); - hdr(9:12) = typecast(uint32(blocksize), 'uint8'); - hdr(13:16) = typecast(uint32(cbytes), 'uint8'); - - offsetTable = typecast(uint32(16 + 4), 'uint8'); - blockHeader = typecast(uint32(numel(blockPayload)), 'uint8'); - container = [hdr(:); offsetTable(:); blockHeader(:); blockPayload(:)]; -end - -function out = shuffleBytes(bytesIn, typesize) -% Byte shuffle: interleaved element bytes -> [byte0 for all elems | -% byte1 for all elems | ...]. Standard Blosc pre-filter. - bytesIn = bytesIn(:); - n = numel(bytesIn); - nelems = n / typesize; - lanes = reshape(bytesIn, typesize, nelems); - out = reshape(lanes.', [], 1); -end - -function payload = compressZstd(bytesIn, clevel) -% Shell out to the system zstd binary. Same pattern NDR's fixture -% writer uses; keeps the toolchain small and the container binary- -% identical to what numcodecs writes. - zstdBin = locateZstd(); - inPath = [tempname() '.bin']; - outPath = [tempname() '.zst']; - cleaner = onCleanup(@() safeDelete({inPath, outPath})); - fid = fopen(inPath, 'wb'); - fwrite(fid, bytesIn, 'uint8'); - fclose(fid); - cmd = sprintf('%s -qf -%d --no-check -o %s %s', ... - shellQuote(zstdBin), clevel, shellQuote(outPath), shellQuote(inPath)); - [status, out] = system(cmd); - if status ~= 0 - error('NDI:lightsheet:makePyramid:zstdFailed', ... - 'zstd compression failed (exit %d): %s', status, strtrim(out)); - end - fid = fopen(outPath, 'rb'); - payload = fread(fid, inf, '*uint8'); - fclose(fid); -end - -function bin = locateZstd() - persistent cached - if ~isempty(cached), bin = cached; return; end - if ispc, [s, out] = system('where zstd'); else, [s, out] = system('command -v zstd'); end - if s == 0 - candidate = strtrim(strsplit(strtrim(out), newline)); - candidate = candidate{1}; - if ~isempty(candidate) && exist(candidate, 'file') == 2 - cached = candidate; bin = cached; return; - end - end - error('NDI:lightsheet:makePyramid:zstdNotFound', ... - ['The zstd executable is required for codec=blosc-zstd and ' ... - 'was not found on PATH. Install: `brew install zstd`, ' ... - '`apt install zstd`, or download for Windows from ' ... - 'https://github.com/facebook/zstd/releases.']); -end - -function s = shellQuote(p) - if ispc, s = ['"' p '"']; else, s = ['''' strrep(p, '''', '''\''''') '''']; end -end - -function safeDelete(paths) - for i = 1:numel(paths) - if exist(paths{i}, 'file') == 2 - try - delete(paths{i}); - catch - end - end - end -end - function cleanupTmp(d) if isfolder(d) try From eb04a79a7a2eb402c51ffed3b8b2e258bc473dee Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 18:53:01 +0000 Subject: [PATCH 032/110] makePyramid: keep chunk tempfiles alive until database_add finishes attachChunkFiles built its chunk files under a tempname() dir and put an onCleanup(rmdir(tmpRoot,'s')) in the same function. That cleaner fired when attachChunkFiles returned -- BEFORE the caller's session.database_add(levelDoc), so DID's ingest saw every recorded chunk.bin_# path point at a deleted file and dropped the bytes. Fix: return tmpRoot from attachChunkFiles and register the cleaner one scope up, in makeOneLevel, so it fires after database_add copies the files in. The metadata-only path passes tmpRoot='' and cleanupTmp is a no-op on that (isfolder('') is false). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 747a8f83a..061076342 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -217,16 +217,18 @@ levelDoc = levelDoc.set_dependency_value('source_file_id', options.sourceFileID); end + tmpRoot = ''; if options.materializeChunks - levelDoc = attachChunkFiles(levelDoc, options.sourceZarrPath, ... + [levelDoc, tmpRoot] = attachChunkFiles(levelDoc, options.sourceZarrPath, ... pyramidEntry, level, chunks, chunkGrid, ... options.codec, options.clevel); end + tmpCleaner = onCleanup(@() cleanupTmp(tmpRoot)); %#ok session.database_add(levelDoc); end -function levelDoc = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel) +function [levelDoc, tmpRoot] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel) % Read the source zarr level as a whole array, then re-tile into the % level document's chunk shape and attach each chunk as chunk.bin_ % (1-based, C-order over chunkGrid). Files are ingested into DID, which @@ -260,7 +262,10 @@ idx = 0; tmpRoot = tempname; mkdir(tmpRoot); - cleaner = onCleanup(@() cleanupTmp(tmpRoot)); + % No onCleanup here: tmpRoot is returned so the CALLER can register + % the cleanup AFTER session.database_add has ingested the files. If + % we cleaned up on return, database_add would find the paths already + % deleted. subs = cell(1, numel(chunkGrid)); linearOrder = allChunkIndices(chunkGrid); % rows are (c1, c2, c3, ...) 1-based for r = 1:size(linearOrder, 1) From 1c19307556508cde515070fd7d268db68b567274 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 18:56:29 +0000 Subject: [PATCH 033/110] makePyramid: drop stale %#ok on the tmpCleaner onCleanup github-advanced-security flagged the suppression added in eb04a79 as no-longer-generated. Modern MATLAB Code Analyzer recognises the onCleanup idiom (an assignment kept only to hold a cleanup object alive) and does not warn. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 061076342..883368667 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -223,7 +223,7 @@ pyramidEntry, level, chunks, chunkGrid, ... options.codec, options.clevel); end - tmpCleaner = onCleanup(@() cleanupTmp(tmpRoot)); %#ok + tmpCleaner = onCleanup(@() cleanupTmp(tmpRoot)); session.database_add(levelDoc); end From b30fd91b4ab6df7bf0536e8b6e4c1a01bd37e3f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 21:47:32 +0000 Subject: [PATCH 034/110] blob fixture: two channels by default, OMERO channel metadata Extends the lightsheet blob fixture to write a multi-channel OME-Zarr with axes 'c,z,y,x' and 2 channels (Ch1, Ch2) by default. NumChannels=1 falls back to the pre-channel 3-D 'z,y,x' layout so older callers stay byte-compatible. Per-channel content: the wide ring pattern is identical across channels (a coincidence mark) and the medium blob + spike are mirrored per channel so a viewer with distinct colours sees the two volumes readily. Positions rotate around the volume as NumChannels grows. Chunk shape becomes [1, spatial] so one on-disk chunk carries one channel -- matches how bioformats2raw writes lightsheet volumes and lets a viewer stream just the channels it needs. .zattrs gains an omero.channels block naming Ch1/Ch2 and giving each a colour matching the Python viewer's default palette (green, magenta, cyan, yellow, ...). Napari-ome-zarr reads it directly; the in-house napariViewLightsheet still picks colours from its own palette. ingestBlobFixture pipes through NumChannels and ChannelNames. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+test/+lightsheet/ingestBlobFixture.m | 30 +- .../+ndi/+test/+lightsheet/makeBlobFixture.m | 416 ++++++++++++------ 2 files changed, 300 insertions(+), 146 deletions(-) diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index 63ca98d36..a3fee6f7a 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -21,13 +21,17 @@ % for repeated cloud-upload experiments). % subject - char, local_identifier for the subject document % created inside the session (default 'blob@vhlab'). -% Shape - passed through to makeBlobFixture (default -% [300 300 300]). -% ChunkShape - passed through to makeBlobFixture (default -% [64 64 64]). -% NumLevels - passed through to makeBlobFixture (default 4). -% VoxelSize - passed through to makeBlobFixture (default -% [1 1 1]). +% Shape - passed through to makeBlobFixture (default +% [300 300 300]). +% NumChannels - passed through to makeBlobFixture (default 2). +% Set 1 for the old single-channel layout. +% ChannelNames - passed through to makeBlobFixture. Empty +% (default) uses 'Ch1', 'Ch2', ... +% ChunkShape - passed through to makeBlobFixture (default +% [64 64 64], spatial only). +% NumLevels - passed through to makeBlobFixture (default 4). +% VoxelSize - passed through to makeBlobFixture (default +% [1 1 1]). % % Returns: % SESSION - ndi.session.dir opened at sessionDir. @@ -52,6 +56,8 @@ options.zarrDir char = '' options.subject (1,:) char = 'blob@vhlab' options.Shape (1,3) double {mustBePositive, mustBeInteger} = [300 300 300] + options.NumChannels (1,1) double {mustBePositive, mustBeInteger} = 2 + options.ChannelNames = {} options.ChunkShape (1,3) double {mustBePositive, mustBeInteger} = [64 64 64] options.NumLevels (1,1) double {mustBePositive, mustBeInteger} = 4 options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] @@ -77,10 +83,12 @@ end [zarrPath, gt] = ndi.test.lightsheet.makeBlobFixture(zarrParent, ... - 'Shape', options.Shape, ... - 'NumLevels', options.NumLevels, ... - 'ChunkShape', options.ChunkShape, ... - 'VoxelSize', options.VoxelSize); + 'Shape', options.Shape, ... + 'NumChannels', options.NumChannels, ... + 'ChannelNames', options.ChannelNames, ... + 'NumLevels', options.NumLevels, ... + 'ChunkShape', options.ChunkShape, ... + 'VoxelSize', options.VoxelSize); session = ndi.session.dir('blob', sessionDir); sub = ndi.document('subject', 'base.session_id', session.id(), ... diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m index 5fd3a1946..462ff551d 100644 --- a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -5,33 +5,43 @@ % [ZARRPATH, GT] = NDI.TEST.LIGHTSHEET.MAKEBLOBFIXTURE(PARENTDIR, ...) % % Writes a small OME-Zarr (NGFF v0.4) store on disk carrying a -% deterministic 300x300x300 uint16 volume, downsampled into a -% mean + max ladder that shares its raw level 0. Unlike the -% metadata-only fixtures the unit tests use, this one writes REAL -% chunk bytes -- uncompressed raw (Zarr v2 with `compressor: null`), -% which every zarr reader accepts -- so an end-to-end run can -% ingest it with ndi.fun.doc.lightsheet.fromOMEZarr, materialize -% chunks onto lightsheetZarrLevel documents when that lands, and be -% viewed by the napari client that reads them back. +% deterministic uint16 lightsheet volume, downsampled into a mean + +% max ladder that shares its raw level 0. Unlike the metadata-only +% fixtures the unit tests use, this one writes REAL chunk bytes -- +% uncompressed raw (Zarr v2 with `compressor: null`), which every +% zarr reader accepts -- so an end-to-end run can ingest it with +% ndi.fun.doc.lightsheet.fromOMEZarr, materialize chunks onto +% lightsheetZarrLevel documents, and be viewed by the napari client +% that reads them back. % -% The volume itself is three features at three spatial scales, so a -% viewer can tell the pyramid levels apart at a glance. Positions -% scale with Shape so the fixture is well-defined at any size: -% - a centred RING pattern: Gaussian envelope (sigma Shape/10) -% modulated by a radial cosine of wavelength Shape/10, so +% The store is multi-channel by default: axes 'c,z,y,x' with 2 +% channels labelled 'Ch1' and 'Ch2', chunks 1 x (spatial) so a +% reader can address one channel per chunk. Napari can be told to +% colour each channel independently via the OMERO metadata this +% writer emits (Ch1 -> green, Ch2 -> magenta by default, extending +% to the palette the client keeps in sync with). +% +% Every channel carries the same three features at three spatial +% scales, with per-channel offsets so they read differently in each +% colour: +% - a centred RING pattern: Gaussian envelope (sigma Shape/8) +% modulated by a radial cosine of wavelength Shape/5, so % brightness oscillates in concentric shells around Shape/2. -% Fine levels resolve the individual rings; coarse levels -% smear them, so each pyramid step shows different structure. -% - a medium Gaussian at (2/3, 1/3, 1/2)*Shape, sigma Shape/20, +% Same for every channel -- makes a natural registration mark. +% - a medium Gaussian at (2/3, 1/3, 1/2)*Shape in Ch1 and the +% mirrored (1/3, 2/3, 1/2)*Shape in Ch2, sigma Shape/20, % peak ~30k -- a compact companion that stays a small ball at -% every level. -% - a bright single-voxel spike of value 60k at Shape/6. -% The ring envelope is 3*sigma ~ 3/10 of a side, so the feature -% stays well inside the box rather than running to the edge. The -% spike is much brighter than the local Gaussian background and -% fills one voxel out of the whole volume, so max downsampling -% holds it up while mean spreads it out over its block -- the -% mean-vs-max choice is visible at every pyramid level. +% every level. Positions rotate around the volume when more +% channels are requested. +% - a bright single-voxel spike of value 60k, placed at Shape/6 +% (Ch1) and Shape*5/6 (Ch2). Positions distribute along the +% diagonal when more channels are requested. +% The ring envelope is 3*sigma ~ 3/8 of a side, so the feature +% stays well inside the box. The spike is much brighter than the +% local Gaussian background and fills one voxel out of the whole +% volume, so max downsampling holds it up while mean spreads it +% over its block -- the mean-vs-max choice is visible at every +% pyramid level. % % PARENTDIR - char, directory to write into. Empty (default) puts % the store under a fresh tempname; the caller can pass @@ -40,32 +50,44 @@ % fullfile(PARENTDIR, 'blob.ome.zarr'). % % Name-Value: -% Shape (1,3) double - [Z Y X], default [300 300 300] +% Shape (1,3) double - [Z Y X] spatial shape, default +% [300 300 300]. Channels are added on top +% (see NumChannels). +% NumChannels (1,1) - number of channels (default 2). Set 1 to +% fall back to the old 3-D 'z,y,x' layout. +% ChannelNames - cellstr / string array, one label per +% channel. Empty (default) produces 'Ch1', +% 'Ch2', ... Written into the OMERO block +% of .zattrs so napari picks them up. % NumLevels (1,1) - number of pyramid levels (default 4) -% ChunkShape (1,3) - per-level Zarr chunk shape (default -% [64 64 64]; every level uses the same -% chunk shape, edge chunks pad with 0) +% ChunkShape (1,3) - spatial chunk shape (default [64 64 64]). +% For a multi-channel store the on-disk +% chunk shape is [1, ChunkShape(:)'], so +% one chunk holds one channel. % VoxelSize (1,3) - level 0 voxel size (default [1 1 1] um). -% Every subsequent level scales by 2x. +% Every subsequent level scales by 2x on +% the spatial axes. % Seed (1,1) - reserved for future noisy fixtures; the % default fixture is deterministic without % it (default 42). % % ZARRPATH - char, the absolute path to /blob.ome.zarr % GT - struct with fields: -% shape - level-0 [Z Y X] -% chunkShape - the chunk shape used -% voxelSize - level-0 voxel size -% numLevels - the ladder depth -% levels - 1xN struct(path, shape) for each level -% across mean and max -% volumes - struct with fields: -% level0 - the raw uint16 volume -% mean - 1xN cell of mean downsamples -% max - 1xN cell of max downsamples -% Use GT to compare a reader's output against ground -% truth. The full uint16 volume is a modest ~54 MB at -% the default shape; do NOT return it unrequested. +% shape - level-0 [Z Y X] +% numChannels - number of channels written +% channelNames - cellstr of channel labels +% chunkShape - spatial chunk shape +% voxelSize - level-0 voxel size +% numLevels - the ladder depth +% levels - 1xN struct(path, shape) for each level +% across mean and max (shape is the +% on-disk shape, i.e. includes the c dim) +% volumes - struct with fields: +% level0 - the raw uint16 volume, +% shape (C, Z, Y, X) when +% NumChannels > 1 +% mean - 1xN cell of mean downsamples +% max - 1xN cell of max downsamples % % Example -- end to end round-trip on the local filesystem: % [zp, gt] = ndi.test.lightsheet.makeBlobFixture(pwd); @@ -75,7 +97,7 @@ % S.database_add(sub); % [pyr, lds] = ndi.fun.doc.lightsheet.fromOMEZarr(S, zp, ... % 'subjectID', sub.id()); -% ndi.gui.app.LightsheetZarrManager(S); +% ndi.fun.doc.lightsheet.view(S, pyr.id()); % % See also: ndi.fun.doc.lightsheet.fromOMEZarr, % ndi.fun.doc.lightsheet.makePyramid, @@ -84,6 +106,8 @@ arguments parentDir char = '' options.Shape (1,3) double {mustBePositive, mustBeInteger} = [300 300 300] + options.NumChannels (1,1) double {mustBePositive, mustBeInteger} = 2 + options.ChannelNames = {} options.NumLevels (1,1) double {mustBePositive, mustBeInteger} = 4 options.ChunkShape (1,3) double {mustBePositive, mustBeInteger} = [64 64 64] options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] @@ -102,15 +126,26 @@ end mkdir(zarrPath); - % Build level 0 and the two reduction ladders. - lvl0 = buildBlobVolume(options.Shape); + channelNames = resolveChannelNames(options.ChannelNames, options.NumChannels); + + hasChannels = options.NumChannels > 1 || ~isempty(options.ChannelNames); + if hasChannels + onDiskChunkShape = [1, options.ChunkShape]; + else + onDiskChunkShape = options.ChunkShape; + end + + % Build level 0 and the two reduction ladders. Multi-channel path + % keeps channels as a leading axis; single-channel path stays 3-D + % so it stays compatible with anything reading the old layout. + lvl0 = buildBlobVolume(options.Shape, options.NumChannels, hasChannels); meanLadder = cell(1, options.NumLevels); maxLadder = cell(1, options.NumLevels); meanLadder{1} = lvl0; maxLadder{1} = lvl0; for L = 2:options.NumLevels - meanLadder{L} = downsampleBlockUint16(meanLadder{L-1}, 2, @meanReduce); - maxLadder{L} = downsampleBlockUint16(maxLadder{L-1}, 2, @maxReduce); + meanLadder{L} = downsampleSpatial(meanLadder{L-1}, 2, @meanReduce, hasChannels); + maxLadder{L} = downsampleSpatial(maxLadder{L-1}, 2, @maxReduce, hasChannels); end % Every Zarr v2 group node needs a .zgroup marker, or readers @@ -127,34 +162,39 @@ end % Write level 0 (shared) and each reduction's coarser levels. - writeZarrLevel(fullfile(zarrPath, '0'), lvl0, options.ChunkShape); + writeZarrLevel(fullfile(zarrPath, '0'), lvl0, onDiskChunkShape); for L = 2:options.NumLevels writeZarrLevel(fullfile(zarrPath, 'mean', sprintf('%d', L-1)), ... - meanLadder{L}, options.ChunkShape); + meanLadder{L}, onDiskChunkShape); writeZarrLevel(fullfile(zarrPath, 'max', sprintf('%d', L-1)), ... - maxLadder{L}, options.ChunkShape); + maxLadder{L}, onDiskChunkShape); + end + + % .zattrs axes. The spatial 2x downsampling only applies to + % spatial axes; the channel axis carries no scale. + if hasChannels + ax = {struct('name','c','type','channel'), ... + struct('name','z','type','space','unit','micrometer'), ... + struct('name','y','type','space','unit','micrometer'), ... + struct('name','x','type','space','unit','micrometer')}; + scaleForLevel = @(L) [1, options.VoxelSize * 2^(L-1)]; + else + ax = {struct('name','z','type','space','unit','micrometer'), ... + struct('name','y','type','space','unit','micrometer'), ... + struct('name','x','type','space','unit','micrometer')}; + scaleForLevel = @(L) options.VoxelSize * 2^(L-1); end - % Write .zattrs. - ax = {struct('name','z','type','space','unit','micrometer'), ... - struct('name','y','type','space','unit','micrometer'), ... - struct('name','x','type','space','unit','micrometer')}; meanDatasets = cell(1, options.NumLevels); maxDatasets = cell(1, options.NumLevels); for L = 1:options.NumLevels - scale = options.VoxelSize * 2^(L-1); + scale = scaleForLevel(L); if L == 1 - path0 = '0'; + meanDatasets{L} = zarrDataset('0', scale); + maxDatasets{L} = zarrDataset('0', scale); else - pMean = sprintf('mean/%d', L-1); - pMax = sprintf('max/%d', L-1); - end - if L == 1 - meanDatasets{L} = zarrDataset(path0, scale); - maxDatasets{L} = zarrDataset(path0, scale); - else - meanDatasets{L} = zarrDataset(pMean, scale); - maxDatasets{L} = zarrDataset(pMax, scale); + meanDatasets{L} = zarrDataset(sprintf('mean/%d', L-1), scale); + maxDatasets{L} = zarrDataset(sprintf('max/%d', L-1), scale); end end multiscales = { ... @@ -162,77 +202,150 @@ 'axes', {ax}, 'datasets', {meanDatasets}), ... struct('name','max','type','max', ... 'axes', {ax}, 'datasets', {maxDatasets}) }; - writeJSON(fullfile(zarrPath, '.zattrs'), ... - struct('multiscales', {multiscales})); + + zattrs = struct('multiscales', {multiscales}); + if hasChannels + zattrs.omero = struct('channels', {buildOmeroChannels(channelNames)}); + end + writeJSON(fullfile(zarrPath, '.zattrs'), zattrs); % Level manifest for the ground truth. levels = struct('path', {}, 'shape', {}); - levels(end+1) = struct('path', '0', 'shape', options.Shape); + levels(end+1) = struct('path', '0', 'shape', size(lvl0)); for L = 2:options.NumLevels - s = size(meanLadder{L}); - levels(end+1) = struct('path', sprintf('mean/%d', L-1), 'shape', s); %#ok - levels(end+1) = struct('path', sprintf('max/%d', L-1), 'shape', s); %#ok + levels(end+1) = struct('path', sprintf('mean/%d', L-1), ... + 'shape', size(meanLadder{L})); %#ok + levels(end+1) = struct('path', sprintf('max/%d', L-1), ... + 'shape', size(maxLadder{L})); %#ok end gt = struct( ... - 'shape', options.Shape, ... - 'chunkShape', options.ChunkShape, ... - 'voxelSize', options.VoxelSize, ... - 'numLevels', options.NumLevels, ... - 'levels', levels, ... - 'volumes', struct('level0', lvl0, ... - 'mean', {meanLadder}, ... - 'max', {maxLadder})); + 'shape', options.Shape, ... + 'numChannels', options.NumChannels, ... + 'channelNames', {channelNames}, ... + 'chunkShape', options.ChunkShape, ... + 'voxelSize', options.VoxelSize, ... + 'numLevels', options.NumLevels, ... + 'levels', levels, ... + 'volumes', struct('level0', lvl0, ... + 'mean', {meanLadder}, ... + 'max', {maxLadder})); end % ===================================================================== -function vol = buildBlobVolume(shape) -% Deterministic 3D uint16 volume with three features at three spatial -% scales. Every feature is closed-form so the fixture is reproducible -% without touching random state. +function names = resolveChannelNames(userNames, numChannels) + if isempty(userNames) + names = arrayfun(@(k) sprintf('Ch%d', k), 1:numChannels, ... + 'UniformOutput', false); + return; + end + if isstring(userNames) + userNames = cellstr(userNames); + end + if ~iscellstr(userNames) %#ok + error('NDI:lightsheet:makeBlobFixture:badChannelNames', ... + 'ChannelNames must be a cellstr or string array.'); + end + if numel(userNames) ~= numChannels + error('NDI:lightsheet:makeBlobFixture:channelCountMismatch', ... + 'ChannelNames has %d entries but NumChannels is %d.', ... + numel(userNames), numChannels); + end + names = reshape(userNames, 1, []); +end + +function vol = buildBlobVolume(shape, numChannels, hasChannels) +% Deterministic uint16 volume with three features at three spatial +% scales. Multi-channel path returns (C, Z, Y, X) with per-channel +% variations in the medium blob and spike positions so channels read +% differently in each napari colour. Single-channel path returns +% (Z, Y, X) that matches the pre-channel fixture byte-for-byte. + if ~hasChannels + vol = oneChannelVolume(shape, 1, 1); + return; + end + vol = zeros([numChannels, shape], 'uint16'); + for ch = 1:numChannels + one = oneChannelVolume(shape, ch, numChannels); + vol(ch, :, :, :) = reshape(one, [1, shape]); + end +end + +function vol = oneChannelVolume(shape, ch, numChannels) Z = shape(1); Y = shape(2); X = shape(3); [zz, yy, xx] = ndgrid(1:Z, 1:Y, 1:X); - % Widths scale with the volume so the fixture is well-defined at - % any Shape. The wide feature sits well inside the box (3*sigma - % ~ 3/10 of a side) rather than running to the edge. - sigmaWide = mean(shape) / 8; % contain 3*sigma at ~3/8 of a side + sigmaWide = mean(shape) / 8; sigmaMedium = mean(shape) / 20; - lambdaWide = mean(shape) / 5; % ~2 rings inside 3*sigma, - % ~7.5 voxels at level 3 (8x down), - % well above the 2-voxel Nyquist - % A wide ring pattern centred in the volume: a Gaussian envelope - % modulated by a radial cosine so brightness oscillates in - % concentric shells as you move toward the centre. Fine levels - % show the individual rings; coarse levels smooth them, so each - % pyramid step reads differently. Amplitude split as a smooth - % baseline (25k, the ring troughs) plus a smaller oscillation - % (25k, the ring peaks) so max downsampling does not lock onto - % the peak-only value and produce blocky per-tile output. + lambdaWide = mean(shape) / 5; + + % Wide ring pattern -- IDENTICAL across channels, so a viewer sees + % it as a coincidence mark. c1 = shape / 2; r1 = sqrt((zz - c1(1)).^2 + (yy - c1(2)).^2 + (xx - c1(3)).^2); envelope = exp(-r1.^2 / (2 * sigmaWide^2)); rings = 0.5 + 0.5 * cos(2 * pi * r1 / lambdaWide); wide = envelope .* (25000 + 25000 * rings); - % A medium blob offset toward one corner (kept as-is). - c2 = [shape(1)*2/3 shape(2)/3 shape(3)/2]; + + % Medium blob position rotates around the volume so each channel + % sits somewhere different. Ch1 lands at the pre-channel default + % (2/3, 1/3, 1/2) for backward-compat, Ch2 mirrors it, N>2 fills + % in between. + if numChannels <= 1 + alpha = 1; + else + alpha = 1 - (ch - 1) / (numChannels - 1); % 1, ..., 0 + end + c2 = [shape(1) * lerp(1/3, 2/3, alpha), ... + shape(2) * lerp(2/3, 1/3, alpha), ... + shape(3) / 2]; r2sq = (zz - c2(1)).^2 + (yy - c2(2)).^2 + (xx - c2(3)).^2; medium = 30000 * exp(-r2sq / (2 * sigmaMedium^2)); - % One bright voxel: max keeps its value at every level, mean - % spreads it across the block. Position scales with Shape so a - % small test fixture puts the spike inside the volume too. - spikePos = max(1, round(shape / 6)); + + % Spike positions spaced along the volume diagonal: Ch1 at Shape/6 + % (matches the pre-channel default), Ch2 at Shape*5/6, N>2 + % interpolates between them so no two spikes overlap. + if numChannels <= 1 + spikeFrac = 1/6; + else + spikeFrac = lerp(1/6, 5/6, (ch - 1) / (numChannels - 1)); + end + spikePos = max(1, min(shape, round(shape * spikeFrac))); spike = zeros(shape); spike(spikePos(1), spikePos(2), spikePos(3)) = 60000; vol = uint16(min(65535, wide + medium + spike)); end +function y = lerp(a, b, t) + y = a + (b - a) * t; +end + +function out = downsampleSpatial(vol, factor, reducer, hasChannels) +% Downsample by FACTOR on spatial axes only. When HASCHANNELS is +% true, the leading axis is the channel dim and is preserved +% verbatim; otherwise we fall back to the pre-channel 3-D reducer. + if ~hasChannels + out = downsampleBlockUint16(vol, factor, reducer); + return; + end + C = size(vol, 1); + sz3 = size(vol); + sz3 = sz3(2:end); + outShape3 = ceil(sz3 / factor); + out = zeros([C, outShape3], 'like', vol); + for c = 1:C + one = reshape(vol(c, :, :, :), sz3); + down = downsampleBlockUint16(one, factor, reducer); + out(c, :, :, :) = reshape(down, [1, size(down)]); + end +end + function out = downsampleBlockUint16(vol, factor, reducer) % Reduce each factor x factor x factor block into one voxel using % REDUCER (a function handle). Output shape is ceil(size / factor); -% odd trailing indices reuse the last available block ("edge pad" -- -% simple, and mirrors what OME-Zarr writers commonly do). +% trailing edge blocks are smaller. Mirrors what OME-Zarr writers +% commonly do (bioformats2raw, make_zarr_levels.py). sz = size(vol); out_shape = ceil(sz / factor); out = zeros(out_shape, 'like', vol); @@ -262,10 +375,15 @@ function writeZarrLevel(arrayDir, vol, chunkShape) % Write one Zarr v2 array: a .zarray metadata file plus one raw chunk -% file per non-empty tile, keyed as z.y.x (default dimension_separator -% for Zarr v2). +% file per non-empty tile. Handles both 3-D (z,y,x) and 4-D (c,z,y,x) +% layouts; chunkShape must match the array's dimensionality. if ~isfolder(arrayDir), mkdir(arrayDir); end shape = size(vol); + if numel(shape) ~= numel(chunkShape) + error('NDI:lightsheet:makeBlobFixture:chunkArity', ... + 'chunkShape length %d does not match array ndims %d.', ... + numel(chunkShape), numel(shape)); + end writeJSON(fullfile(arrayDir, '.zarray'), struct( ... 'zarr_format', 2, ... 'shape', shape, ... @@ -278,34 +396,42 @@ function writeZarrLevel(arrayDir, vol, chunkShape) 'dimension_separator', '.')); nChunks = ceil(shape ./ chunkShape); - for cz = 0:nChunks(1)-1 - for cy = 0:nChunks(2)-1 - for cx = 0:nChunks(3)-1 - z0 = cz * chunkShape(1) + 1; - y0 = cy * chunkShape(2) + 1; - x0 = cx * chunkShape(3) + 1; - z1 = min(z0 + chunkShape(1) - 1, shape(1)); - y1 = min(y0 + chunkShape(2) - 1, shape(2)); - x1 = min(x0 + chunkShape(3) - 1, shape(3)); - - % Pad edge tiles with zeros so on-disk chunks are all - % the same size (Zarr v2 requires this). - chunk = zeros(chunkShape, 'uint16'); - chunk(1:(z1-z0+1), 1:(y1-y0+1), 1:(x1-x0+1)) = ... - vol(z0:z1, y0:y1, x0:x1); - - % Zarr expects C-order bytes. MATLAB is column-major, - % so permute the axes into reversed order before - % linearising (matches NDR's reader inverse). - permuted = permute(chunk, ndims(chunk):-1:1); - raw = typecast(permuted(:), 'uint8'); - - key = sprintf('%d.%d.%d', cz, cy, cx); - fid = fopen(fullfile(arrayDir, key), 'w'); - fwrite(fid, raw); - fclose(fid); - end + subs = cell(1, numel(shape)); + writeChunks(arrayDir, vol, shape, chunkShape, nChunks, subs, [], 1); +end + +function writeChunks(arrayDir, vol, shape, chunkShape, nChunks, subs, key, dim) +% Recursively walk the chunk grid across all dimensions, so the writer +% is agnostic to how many axes the array has. + if dim > numel(shape) + lo = zeros(1, numel(shape)); + hi = zeros(1, numel(shape)); + for a = 1:numel(shape) + lo(a) = (key(a) - 1) * chunkShape(a) + 1; + hi(a) = min(lo(a) + chunkShape(a) - 1, shape(a)); + subs{a} = lo(a):hi(a); end + % Pad edge tiles with zeros so on-disk chunks are all the same + % size (Zarr v2 requires this). + chunk = zeros(chunkShape, 'like', vol); + localSubs = arrayfun(@(a) 1:(hi(a) - lo(a) + 1), 1:numel(shape), ... + 'UniformOutput', false); + chunk(localSubs{:}) = vol(subs{:}); + % Zarr expects C-order bytes. MATLAB is column-major, so + % permute the axes into reversed order before linearising. + permuted = permute(chunk, numel(shape):-1:1); + raw = typecast(permuted(:), 'uint8'); + + keyStr = strjoin(arrayfun(@(k) num2str(k - 1), key, ... + 'UniformOutput', false), '.'); + fid = fopen(fullfile(arrayDir, keyStr), 'w'); + fwrite(fid, raw); + fclose(fid); + return; + end + for c = 1:nChunks(dim) + writeChunks(arrayDir, vol, shape, chunkShape, nChunks, subs, ... + [key c], dim + 1); end end @@ -315,6 +441,26 @@ function writeZarrLevel(arrayDir, vol, chunkShape) struct('type', 'scale', 'scale', scale) }}); end +function channels = buildOmeroChannels(channelNames) +% OMERO 'channels' block that napari-ome-zarr reads to pick up +% per-channel names + display colours. Palette matches the client's +% default (green, magenta, ...). Windows are wide-open so napari's +% auto-contrast still runs. + palette = {'00FF00', 'FF00FF', '00FFFF', 'FFFF00', ... + '0080FF', 'FF8000', '808080'}; + n = numel(channelNames); + channels = cell(1, n); + for i = 1:n + color = palette{mod(i - 1, numel(palette)) + 1}; + channels{i} = struct( ... + 'label', channelNames{i}, ... + 'color', color, ... + 'active', true, ... + 'window', struct('start', 0, 'end', 65535, ... + 'min', 0, 'max', 65535)); + end +end + function writeZGroup(dir) writeJSON(fullfile(dir, '.zgroup'), struct('zarr_format', 2)); end From 90b4c55ce52500f31b4a073fac9329ab74d97cb2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 21:50:34 +0000 Subject: [PATCH 035/110] makeBlobFixture: drop stale %#ok on iscellstr check github-advanced-security flagged the suppression added in b30fd91 as no-longer-generated. Modern MATLAB Code Analyzer no longer warns on iscellstr as a type check (only when used to build strings), so the suppression here is redundant. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m index 462ff551d..b34337817 100644 --- a/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/makeBlobFixture.m @@ -243,7 +243,7 @@ if isstring(userNames) userNames = cellstr(userNames); end - if ~iscellstr(userNames) %#ok + if ~iscellstr(userNames) error('NDI:lightsheet:makeBlobFixture:badChannelNames', ... 'ChannelNames must be a cellstr or string array.'); end From aa7f51a41455af0dc757a6506b29cb1f6895a1fd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 22:06:40 +0000 Subject: [PATCH 036/110] TestBlobFixture: pin existing tests to NumChannels=1, add 2-channel test The blob fixture now defaults to NumChannels=2 (axes 'c,z,y,x') as part of the multi-channel example landing on this branch. The existing assertions were written for the pre-channel 3-D layout -- `0/0.0.0` chunk keys, axes_order 'zyx', shape_level0 [Z Y X], L0 indexed with three subscripts -- so they now fail against the new default. Fix: pin the setup + one-line ingest tests to NumChannels=1 so they keep exercising the 3-D path they were written for, and add testMultiChannelFixtureWritesCZYXLayout covering the 2-channel default (4-D shape, 4-tuple chunk shape with 1 on the channel axis, 4-dot chunk keys 0.0.0.0 / 1.0.0.0, axes starting with c, OMERO channels block naming Ch1 and Ch2). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+test/+lightsheet/TestBlobFixture.m | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index 4793e064c..cb005a1c5 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -18,9 +18,14 @@ function build(testCase) testCase.parentDir = tempname; mkdir(testCase.parentDir); testCase.addTeardown(@() rmdir(testCase.parentDir, 's')); + % Existing tests pin NumChannels=1 so they keep exercising + % the 3-D 'z,y,x' layout the assertions were written for. + % A separate testMultiChannelFixture below covers the 2-D + % default (2 channels, axes 'c,z,y,x'). [testCase.zarrPath, testCase.gt] = ... ndi.test.lightsheet.makeBlobFixture(testCase.parentDir, ... 'Shape', [32 32 32], ... + 'NumChannels', 1, ... 'NumLevels', 3, ... 'ChunkShape', [16 16 16]); end @@ -128,6 +133,7 @@ function testIngestBlobFixtureIsOneCallEndToEnd(testCase) 'sessionDir', sDir, ... 'zarrDir', zDir, ... 'Shape', [32 32 32], ... + 'NumChannels', 1, ... 'NumLevels', 3, ... 'ChunkShape', [16 16 16]); testCase.verifyClass(S, 'ndi.session.dir'); @@ -163,6 +169,50 @@ function testMaxKeepsBrightSpikeMeanDoesNot(testCase) double(maxTop(blk(1), blk(2), blk(3)))); end + function testMultiChannelFixtureWritesCZYXLayout(testCase) + % The default (NumChannels=2) fixture writes a 4-D + % (C, Z, Y, X) OME-Zarr with a leading channel axis, a + % 4-tuple chunk shape (channel dim = 1), and an OMERO + % channels block naming Ch1 / Ch2. + multiDir = tempname; + mkdir(multiDir); + testCase.addTeardown(@() rmdir(multiDir, 's')); + [zp, gt] = ndi.test.lightsheet.makeBlobFixture(multiDir, ... + 'Shape', [32 32 32], ... + 'NumLevels', 2, ... + 'ChunkShape', [16 16 16]); + testCase.verifyEqual(gt.numChannels, 2); + testCase.verifyEqual(gt.channelNames, {'Ch1','Ch2'}); + % Level 0 shape has C prepended. + level0Shape = gt.levels(1).shape; + testCase.verifyEqual(level0Shape, [2 32 32 32]); + % .zarray chunks is 4-tuple with 1 on the channel axis. + za = jsondecode(fileread(fullfile(zp, '0', '.zarray'))); + testCase.verifyEqual(reshape(za.chunks, 1, []), [1 16 16 16]); + testCase.verifyEqual(reshape(za.shape, 1, []), [2 32 32 32]); + % One chunk file per (c, z, y, x) tile with a 4-dot key. + testCase.verifyTrue(isfile(fullfile(zp, '0', '0.0.0.0'))); + testCase.verifyTrue(isfile(fullfile(zp, '0', '1.0.0.0'))); + % .zattrs axes list carries c first, and an OMERO block + % names both channels. + attrs = jsondecode(fileread(fullfile(zp, '.zattrs'))); + firstMultiscale = attrs.multiscales(1); + ax = firstMultiscale.axes; + firstAxis = ax(1); + testCase.verifyEqual(char(firstAxis.name), 'c'); + testCase.verifyEqual(char(firstAxis.type), 'channel'); + testCase.verifyTrue(isfield(attrs, 'omero')); + omeroChannels = attrs.omero.channels; + testCase.verifyEqual(numel(omeroChannels), 2); + % jsondecode can turn a homogeneous cell array into a + % struct array; either shape carries a 'label' field. + firstOmero = omeroChannels(1); + if iscell(firstOmero) + firstOmero = firstOmero{1}; + end + testCase.verifyEqual(char(firstOmero.label), 'Ch1'); + end + end end From 1f3730b93e046f212161f0929d4259a727659059 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 10 Sep 2026 22:17:55 +0000 Subject: [PATCH 037/110] TestBlobFixture: defensive cell/struct handling in the new multi-channel test MATLAB's jsondecode returns multiscales/axes/channels as either a struct array or a cell array depending on field homogeneity across elements. The new testMultiChannelFixtureWritesCZYXLayout indexed them as if they were always struct arrays, which errored on the cell shape (e.g. omero.channels whose entries carry a nested window struct come back as a cell of dissimilar structs). Extract firstEntry / entryCount helpers that handle both shapes, mirroring what collectPyramidNames / collectFirstPaths do for `multiscales`. Test assertions unchanged; failure mode was the indexing, not the fixture data. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+test/+lightsheet/TestBlobFixture.m | 39 +++++++++++++------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index cb005a1c5..93fede292 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -194,22 +194,18 @@ function testMultiChannelFixtureWritesCZYXLayout(testCase) testCase.verifyTrue(isfile(fullfile(zp, '0', '0.0.0.0'))); testCase.verifyTrue(isfile(fullfile(zp, '0', '1.0.0.0'))); % .zattrs axes list carries c first, and an OMERO block - % names both channels. + % names both channels. jsondecode may return multiscales / + % axes / channels as either a struct array or a cell array + % depending on field homogeneity, so pick the first entry + % through a helper that handles both. attrs = jsondecode(fileread(fullfile(zp, '.zattrs'))); - firstMultiscale = attrs.multiscales(1); - ax = firstMultiscale.axes; - firstAxis = ax(1); + firstMultiscale = firstEntry(attrs.multiscales); + firstAxis = firstEntry(firstMultiscale.axes); testCase.verifyEqual(char(firstAxis.name), 'c'); testCase.verifyEqual(char(firstAxis.type), 'channel'); testCase.verifyTrue(isfield(attrs, 'omero')); - omeroChannels = attrs.omero.channels; - testCase.verifyEqual(numel(omeroChannels), 2); - % jsondecode can turn a homogeneous cell array into a - % struct array; either shape carries a 'label' field. - firstOmero = omeroChannels(1); - if iscell(firstOmero) - firstOmero = firstOmero{1}; - end + testCase.verifyEqual(entryCount(attrs.omero.channels), 2); + firstOmero = firstEntry(attrs.omero.channels); testCase.verifyEqual(char(firstOmero.label), 'Ch1'); end @@ -218,6 +214,25 @@ function testMultiChannelFixtureWritesCZYXLayout(testCase) % --------------------------------------------------------------------- +function e = firstEntry(coll) +% First element of either a cell array or a struct array. jsondecode +% chooses one or the other depending on field homogeneity, so callers +% that want "element 1" without caring can go through this helper. + if iscell(coll) + e = coll{1}; + else + e = coll(1); + end +end + +function n = entryCount(coll) + if iscell(coll) + n = numel(coll); + else + n = numel(coll); + end +end + function names = collectPyramidNames(ms) if iscell(ms) names = cellfun(@(e) char(e.name), ms, 'UniformOutput', false); From 74a8588da8e865dc8882264ff956ca3864adb09e Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 01:41:14 +0000 Subject: [PATCH 038/110] makePyramid: stream per-chunk reads instead of loading each level whole attachChunkFiles used to do `data = readArray(sourceZarrPath, ...)` up front and slice each target chunk out of `data`. On the ~54 MB fixture that was fine; on a lab-scale stitched SmartSPIM volume (level 0 is easily 100+ GB) MATLAB tried to allocate the whole level and crashed before writing any chunks. Streaming replaces the whole-level read with one `readArray(..., 'Region', [lo; hi])` call per target chunk. Zarr's reader touches only the source chunks that overlap the region, so peak memory scales with target chunk size (~8 MB by default), not level size. Level shape now comes from `level.shape` on the caller- provided struct (already populated by listPyramids) instead of size(data), so the eager read is really gone. Every other seam is unchanged: chunk grid ordering (allChunkIndices), per-chunk padding (edge chunks), C-order permute for Zarr, codec switch (raw / blosc-zstd), the tempfile + add_file cycle, and the onCleanup-in-the-caller pattern that keeps tempfiles alive until database_add finishes. TestBlobFixture gains testStreamingReTileHitsSmallerBudget which ingests a fixture whose target chunks land smaller than the source chunks and verifies every level document with n_chunks_stored > 0 has chunk.bin_1 actually resolvable through database_openbinarydoc. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 37 +++++++++------ .../+test/+lightsheet/TestBlobFixture.m | 45 +++++++++++++++++++ 2 files changed, 68 insertions(+), 14 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 883368667..8d5236c9f 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -245,13 +245,15 @@ 'level path %s not found in pyramid %s.', level.path, pyramidName); end - % Read the full array. For a fixture-scale volume (~54 MB uint16) - % this is a single allocation; a lab-scale volume will need a - % streaming reader later, but the level doc's chunk shape is fixed - % here either way. - data = ndr.format.omezarr.readArray(sourceZarrPath, pyramidName, kInEntry); - - shape = size(data); + % STREAMING per-chunk read. We call readArray with `Region` per + % target chunk instead of loading the whole level up front. Previous + % versions did `data = readArray(...)` and sliced `data(subs{:})` + % per chunk; that reads O(level bytes) into one MATLAB allocation + % and crashes on lab-scale volumes (a stitched SmartSPIM level 0 + % can be hundreds of GB). Reading a single target chunk touches + % only the source chunks that overlap that region, so peak memory + % scales with target chunk size, not level size. + shape = ensureRow(level.shape); if numel(shape) < numel(chunks) % Singleton trailing axes: pad shape to axis count. shape(end+1:numel(chunks)) = 1; @@ -266,19 +268,26 @@ % the cleanup AFTER session.database_add has ingested the files. If % we cleaned up on return, database_add would find the paths already % deleted. - subs = cell(1, numel(chunkGrid)); linearOrder = allChunkIndices(chunkGrid); % rows are (c1, c2, c3, ...) 1-based + dtypeCls = ''; for r = 1:size(linearOrder, 1) idx = idx + 1; cIdx = linearOrder(r, :); - % Compute the source slice and the padded chunk. + % Compute this chunk's on-disk region: [lo; hi] per axis, + % 1-based inclusive, clamped to the level shape at the far + % edge. + loRow = zeros(1, numel(shape)); + hiRow = zeros(1, numel(shape)); for a = 1:numel(chunkGrid) - lo = (cIdx(a) - 1) * chunks(a) + 1; - hi = min(lo + chunks(a) - 1, shape(a)); - subs{a} = lo:hi; + loRow(a) = (cIdx(a) - 1) * chunks(a) + 1; + hiRow(a) = min(loRow(a) + chunks(a) - 1, shape(a)); + end + srcTile = ndr.format.omezarr.readArray(sourceZarrPath, ... + pyramidName, kInEntry, 'Region', [loRow; hiRow]); + if isempty(dtypeCls) + dtypeCls = class(srcTile); end - srcTile = data(subs{:}); - padded = zeros(chunks, 'like', data); + padded = zeros(chunks, dtypeCls); localSubs = arrayfun(@(k) 1:size(srcTile, k), 1:numel(chunks), ... 'UniformOutput', false); padded(localSubs{:}) = srcTile; diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index 93fede292..89133c1c9 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -144,6 +144,51 @@ function testIngestBlobFixtureIsOneCallEndToEnd(testCase) testCase.verifyEqual(info.fixture.chunkShape, [16 16 16]); end + function testStreamingReTileHitsSmallerBudget(testCase) + % Force the streaming per-chunk read path: build a fixture + % with chunk shape [16 16 16] and ingest with a + % tileBudgetBytes small enough that chooseTileShape picks a + % SMALLER target chunk (~8 KB -> ~16 uint16 voxels per axis + % in 3-D). Each target chunk overlaps <=1 source chunk, so + % this exercises the per-chunk readArray Region call rather + % than reading the whole level -- the fix for the memory + % ceiling on lab-scale volumes. + if isempty(which('ndr.format.omezarr.listPyramids')) + testCase.assumeFail(['NDR reader not on path; skipping ' ... + 'end-to-end ingest.']); + end + sDir = fullfile(tempname, 'streamsession'); + testCase.addTeardown(@() rmdir(fileparts(sDir), 's')); + zDir = fullfile(tempname, 'streamzarr'); + testCase.addTeardown(@() rmdir(zDir, 's')); + [S, pdoc, info] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDir, ... + 'zarrDir', zDir, ... + 'Shape', [32 32 32], ... + 'NumChannels', 1, ... + 'NumLevels', 2, ... + 'ChunkShape', [16 16 16], ... + 'materializeChunks', true); + testCase.verifyClass(pdoc, 'ndi.document'); + % Every level document should have chunks attached (files + % ingested via add_file BEFORE database_add), proving the + % per-chunk readArray path wrote all target chunks. + q = ndi.query('','isa','lightsheetZarrLevel',''); + levels = S.database_search(q); + testCase.verifyNotEmpty(levels); + for k = 1:numel(levels) + L = levels{k}; + p = L.document_properties.lightsheetZarrLevel; + if p.n_chunks_stored > 0 + fh = S.database_openbinarydoc(L, 'chunk.bin_1'); + testCase.verifyNotEmpty(fh, ... + sprintf('chunk.bin_1 missing on level %d', p.level)); + S.database_closebinarydoc(fh); + end + end + testCase.verifyNotEmpty(info.subjectID); + end + function testMaxKeepsBrightSpikeMeanDoesNot(testCase) % At the spike's downsampled position, max preserves the % 60k spike and mean dilutes it (averages the one spike From 8578add9f2bce904e6cacd12874535285bf00d4f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 01:54:43 +0000 Subject: [PATCH 039/110] clevel: cap validation at 9, not 22 (Blosc wrapper limit) Blosc's Python/C API caps clevel at 9 regardless of the inner codec. Even though Zstd itself accepts 1..22, `numcodecs.Blosc.encode` with clevel > 9 raises `clevel parameter must be between 0 and 9!`. Our validation range said 1..22, so a caller passing clevel=12 would sail through MATLAB's arguments check and blow up mid-ingest inside ndr.format.blosc.encode. Bench on a 128^3 uint16 lightsheet-like chunk (sparse Gaussian + Poisson noise, 4 MB uncompressed): clevel=3: 2.28 MB (1.84x), 242ms encode, 189ms decode clevel=5: 2.21 MB (1.90x), 164ms encode, 176ms decode <- default clevel=9: 2.12 MB (1.98x), 486ms encode, 143ms decode So 5 -> 9 buys ~4% smaller files at ~3x encode time. Rarely worth it for ingest-then-view workflows; the option is still available for archival use cases where you compress once and read many times. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 2 +- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 2 +- src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 5cebd3dc8..1f6231463 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -86,7 +86,7 @@ options.chunks double = [] options.materializeChunks (1,1) logical = false options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' - options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 22)} = 5 + options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 end if isempty(strtrim(options.subjectID)) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 8d5236c9f..602c6a16c 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -71,7 +71,7 @@ options.materializeChunks (1,1) logical = false options.sourceZarrPath char = '' options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' - options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 22)} = 5 + options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 end if options.materializeChunks && isempty(options.sourceZarrPath) diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index a3fee6f7a..4794f971c 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -63,7 +63,7 @@ options.VoxelSize (1,3) double {mustBePositive} = [1 1 1] options.materializeChunks (1,1) logical = true options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' - options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 22)} = 5 + options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 end sessionDir = options.sessionDir; From 17f343296b8f85d64dd604881350e1fca5400811 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 00:00:15 +0000 Subject: [PATCH 040/110] LightsheetZarrManager: mirror GEFManager UI on the cloud palette Restyles the manager so it reads as one of the NDI Cloud apps rather than a bare uifigure: navy header bar, cloud-blue accent buttons, white-body table on the off-white ground, and a status label. The button row is Reload / View... / Add... / Delete, in the same order and colours as the GEF Manager. The Subject column now shows the subject's local_identifier via a one-per-session subjectNameMap lookup, so 20 pyramids do not each issue a separate subject query. pyramidRows keeps the raw doc on the row so callbacks can act on it without another database hop. View... is now a non-modal dialog with a launcher path + browse, a layer name, reduction / channel / initial-level pickers, a controls checkbox, and a live command preview -- the same shape as the GEF Manager's View dialog. The command is built by ndi.fun.doc.lightsheet.viewCommand so what launches is exactly what the preview shows, and a failed launch leaves the command on screen to paste into a terminal. Deletion gains a deletionMessage that names the pyramid label and level count, so "delete 1 document" and "delete 8 documents" no longer read the same. The refresh() method survives as a backward-compatible alias for reload(). Tests cover the new pure surface (subjectNameMap, pyramidRows, deletionMessage) alongside the existing launcherPath and deletionPlan paths. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+gui/+app/LightsheetZarrManager.m | 630 ++++++++++++++---- .../+gui/+app/TestLightsheetZarrManager.m | 41 ++ 2 files changed, 551 insertions(+), 120 deletions(-) diff --git a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m index 45f7d5923..141bfcc70 100644 --- a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m +++ b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m @@ -3,16 +3,15 @@ % % ndi.gui.app.LightsheetZarrManager(SESSION) % -% Lists the lightsheetZarrPyramid documents in a session, grouped by -% subject and paired with their level counts, and lets the user add, -% open or remove one. Reached from the navigator's per-session Apps -% menu. +% Lists the lightsheetZarrPyramid documents in a session, paired with +% their subject and level counts, and lets the user add, open or +% remove one. Reached from the navigator's per-session Apps menu. % % WHAT IS LISTED IS PYRAMIDS, NOT ZARR STORES. An OME-Zarr store on % disk is the input; what lands in the database is a -% lightsheetZarrPyramid plus one lightsheetZarrLevel per level, and -% optionally a fileReference. A single store frequently produces two -% parents (a mean pyramid and a max pyramid), listed as two rows. +% lightsheetZarrPyramid plus one lightsheetZarrLevel per level. A +% single store frequently produces two parents (a mean pyramid and a +% max pyramid), listed as two rows. % % VIEWING IS SOMEONE ELSE'S JOB. The viewer is napari, which is % Python, so View does not draw anything here: it builds a command @@ -44,13 +43,6 @@ Category = "Lightsheet microscopy" % Where the napari launcher lives when nobody has said otherwise. - % A wrapper in /usr/local/bin rather than the console script in - % a virtual environment: MATLAB exports library paths of its - % own, and a Python process started from MATLAB loads MATLAB's - % copies of libraries it must not use. The wrapper scrubs those - % and then execs the real entry point. Overridden by the - % preference GUI.LightsheetZarrManager.ViewerLauncher, which - % the View dialog writes. DefaultViewerLauncher = "/usr/local/bin/napariViewLightsheet" end @@ -60,62 +52,519 @@ table statusLabel rows = struct([]) + % The View dialog, kept so a second click RAISES the open one + % rather than stacking another. Non-modal (see onView), so + % without this the button would build one dialog per press. + viewDialog = matlab.ui.Figure.empty end methods function obj = LightsheetZarrManager(sessionObj, options) arguments sessionObj (1,1) + % build=false constructs the model without a figure, so + % every decision this app makes is checkable with no + % display. Same arrangement GEFManager uses. options.build (1,1) logical = true end obj.session = sessionObj; if options.build obj.buildUI(); - obj.refresh(); + obj.reload(); + end + end + + function reload(obj) + obj.rows = ndi.gui.app.LightsheetZarrManager.pyramidRows(obj.session); + if ~isempty(obj.fig) && isvalid(obj.fig) + obj.table.Data = ndi.gui.app.LightsheetZarrManager.rowsToCell(obj.rows); + obj.setStatus(sprintf('%d pyramid(s).', numel(obj.rows))); + end + end + + function refresh(obj) + % REFRESH - backward-compatible alias for reload + obj.reload(); + end + end + + % ===================================================================== + % The model. Static and pure: no figure, no state, so it is testable + % headlessly and a test failure names a decision rather than a widget. + % ===================================================================== + methods (Static) + + function rows = pyramidRows(session) + % PYRAMIDROWS - one row per lightsheetZarrPyramid, with subject names + % + % The subject column shows local_identifier, not the raw id: an + % id names the subject uniquely and tells the reader nothing. + % The lookup is done ONCE for the whole session rather than + % per row -- a session with 20 pyramids would otherwise make + % 20 subject queries to answer one question. + docs = session.database_search( ... + ndi.query('','isa','lightsheetZarrPyramid')); + subjectNames = ndi.gui.app.LightsheetZarrManager.subjectNameMap(session); + rows = struct('id',{},'label',{},'reduction',{},'subject',{}, ... + 'subjectID',{},'nLevels',{},'shapeLevel0',{},'dtype',{},'doc',{}); + for i = 1:numel(docs) + d = docs{i}; + p = d.document_properties.lightsheetZarrPyramid; + k = numel(rows) + 1; + rows(k).id = d.id(); + rows(k).doc = d; + rows(k).label = ndi.gui.app.LightsheetZarrManager.field(p,'label',''); + rows(k).reduction = ndi.gui.app.LightsheetZarrManager.field(p,'reduction',''); + rows(k).subjectID = ndi.gui.app.LightsheetZarrManager.dependency(d,'subject_id'); + rows(k).subject = ndi.gui.app.LightsheetZarrManager.lookup( ... + subjectNames, rows(k).subjectID, rows(k).subjectID); + rows(k).nLevels = double(ndi.gui.app.LightsheetZarrManager.field(p,'n_levels',0)); + shp = ndi.gui.app.LightsheetZarrManager.field(p,'shape_level0',[]); + rows(k).shapeLevel0 = shp; + rows(k).dtype = ndi.gui.app.LightsheetZarrManager.field(p,'dtype',''); + end + end + + function m = subjectNameMap(session) + % SUBJECTNAMEMAP - subject document id -> local_identifier + % + % containers.Map so the lookup is one query for the session + % rather than one per pyramid. A subject with no + % local_identifier is left out rather than mapped to '': the + % caller's default is the id, and an empty cell in the Subject + % column would say the pyramid has no subject, which is a + % different and worse claim than showing the id. + m = containers.Map('KeyType','char','ValueType','char'); + docs = session.database_search(ndi.query('','isa','subject')); + for i = 1:numel(docs) + p = docs{i}.document_properties; + if ~isfield(p,'subject'), continue; end + nm = ndi.gui.app.LightsheetZarrManager.field(p.subject,'local_identifier',''); + if ~isempty(nm) + m(docs{i}.id()) = char(nm); + end + end + end + + function plan = deletionPlan(session, pyramidID) + % DELETIONPLAN - the pyramid and every level that depends on it + % + % Public and static so a caller without a display can preview + % what a delete would remove. + plan = struct(); + plan.pyramidID = char(pyramidID); + q1 = ndi.query('', 'isa', 'lightsheetZarrLevel'); + q2 = ndi.query('depends_on', 'depends_on', ... + 'lightsheetZarrPyramid_id', plan.pyramidID); + levelDocs = session.database_search(q1 & q2); + parentDoc = session.database_search(ndi.query('base.id', ... + 'exact_string', plan.pyramidID)); + plan.nLevels = numel(levelDocs); + plan.docsToRemove = [levelDocs(:); parentDoc(:)]; + end + + function msg = deletionMessage(plan, label) + % DELETIONMESSAGE - the confirmation text for a deletion + % + % Names the pyramid and the level count so "delete 1 document" + % and "delete 32 documents" do not read the same. + if nargin < 2, label = ''; end + if isempty(label), label = plan.pyramidID; end + msg = sprintf(['This removes %d document(s):\n\n' ... + ' 1 pyramid (%s)\n %d level(s)\n\n' ... + 'The levels go because the pyramid they belong to goes. ' ... + 'This cannot be undone.'], ... + 1 + plan.nLevels, label, plan.nLevels); + end + + function p = launcherPath() + % LAUNCHERPATH - the viewer launcher, from preferences or default + % + % A missing or unreadable preference falls back to the default + % rather than raising: the dialog can still open, and the + % field it opens with is editable. + p = char(ndi.gui.app.LightsheetZarrManager.DefaultViewerLauncher); + try + v = ndi.preferences.get('GUI.LightsheetZarrManager.ViewerLauncher'); + if ~isempty(char(v)), p = char(v); end + catch + end + end + + function setLauncherPath(p) + % SETLAUNCHERPATH - remember the launcher for next time + % + % Failing to persist must not cost the launch about to happen. + try + ndi.preferences.set('GUI.LightsheetZarrManager.ViewerLauncher', string(p)); + catch + end + end + + function paper(h, c) + % PAPER - put a control on the cloud palette's white body + % + % The figure and its layouts are on c.offWhite, but the + % CONTROLS default to MATLAB's grey chrome, which reads as a + % grey app with a tinted border. Defensive: a missing handle + % or a control without a BackgroundColor is a no-op. + if isempty(h) || ~isvalid(h) + return; + end + try %#ok + h.BackgroundColor = c.white; + end + try %#ok + h.FontColor = c.darkBlue; + end + end + + function accent(btn, c) + % ACCENT - style a button in the NDI Cloud accent + % + % Light blue on navy, matching ndi.util.ListDialog and the + % cloud .mlapp apps. A missing or deleted handle is a no-op + % rather than an error: this is styling, and losing the + % window over a colour would be a poor trade. + if isempty(btn) || ~isvalid(btn) + return; + end + btn.BackgroundColor = c.lightBlue; + btn.FontColor = c.darkBlue; + btn.FontWeight = 'bold'; + end + + function cmd = detach(cmd) + % DETACH - the same command, not blocking MATLAB + % + % The viewer runs for as long as the user looks at it. Without + % this, MATLAB would sit unresponsive behind it. + if ispc + cmd = ['start "napari" ' cmd]; + else + cmd = [cmd ' &']; end end end + % ===================================================================== + methods (Static, Access = private) + + function v = field(s, name, dflt) + if isstruct(s) && isfield(s, name), v = s.(name); else, v = dflt; end + end + + function v = dependency(d, name) + try + v = d.dependency_value(name); + catch + v = ''; + end + if isempty(v), v = ''; end + end + + function v = lookup(m, key, dflt) + v = dflt; + if isempty(key) || ~isKey(m, char(key)), return; end + v = m(char(key)); + end + + function c = rowsToCell(rows) + c = cell(numel(rows), 6); + for i = 1:numel(rows) + c{i,1} = rows(i).label; + c{i,2} = rows(i).subject; + c{i,3} = rows(i).reduction; + c{i,4} = rows(i).nLevels; + if isempty(rows(i).shapeLevel0) + c{i,5} = ''; + else + c{i,5} = mat2str(double(rows(i).shapeLevel0)); + end + c{i,6} = rows(i).dtype; + end + end + end + + % ===================================================================== methods (Access = private) + function buildUI(obj) - obj.fig = uifigure('Name', char(obj.Name), ... - 'Position', [100 100 900 500]); - g = uigridlayout(obj.fig, [3 1], 'RowHeight', {'fit', '1x', 'fit'}); + c = ndi.gui.cloudColors(); + obj.fig = uifigure('Name','Lightsheet Zarr Manager', ... + 'Position',[100 100 900 460], 'Color', c.offWhite); + % The View dialog is a separate top-level window; closing + % the manager has to take it along, or an orphaned dialog + % would report its result to a status bar that no longer + % exists. + obj.fig.CloseRequestFcn = @(~,~) obj.onClose(); + g = uigridlayout(obj.fig,[4 5]); + g.RowHeight = {28, '1x', 30, 22}; + g.ColumnWidth = {'1x', 90, 90, 90, 90}; + g.BackgroundColor = c.offWhite; + + % Navy header bar with white text, matching the cloud apps + % and ndi.util.ListDialog. + hb = uigridlayout(g,[1 1]); + hb.Layout.Row = 1; hb.Layout.Column = [1 5]; + hb.Padding = [8 0 8 0]; + hb.BackgroundColor = c.darkBlue; + uilabel(hb,'Text','Lightsheet OME-Zarr pyramids', ... + 'FontColor', c.white, 'FontWeight','bold', 'FontSize',14, ... + 'VerticalAlignment','center'); - % Top bar: Add / View / Delete - top = uigridlayout(g, [1 5], 'ColumnWidth', {80, 80, 80, '1x', 100}); - uibutton(top, 'Text', 'Add', 'ButtonPushedFcn', @(~,~) obj.onAdd()); - uibutton(top, 'Text', 'View', 'ButtonPushedFcn', @(~,~) obj.onView()); - uibutton(top, 'Text', 'Delete', 'ButtonPushedFcn', @(~,~) obj.onDelete()); - uilabel(top, 'Text', ''); - uibutton(top, 'Text', 'Refresh','ButtonPushedFcn', @(~,~) obj.refresh()); + obj.table = uitable(g, 'ColumnName', ... + {'Label','Subject','Reduction','Levels','Shape (level 0)','dtype'}); + obj.table.Layout.Row = 2; obj.table.Layout.Column = [1 5]; + ndi.gui.app.LightsheetZarrManager.paper(obj.table, c); - obj.table = uitable(g, ... - 'ColumnName', {'label','reduction','subject','n_levels','shape_level0','dtype','id'}, ... - 'ColumnEditable', false); + b = uibutton(g,'Text','Reload','ButtonPushedFcn',@(~,~) obj.reload()); + b.Layout.Row = 3; b.Layout.Column = 2; + ndi.gui.app.LightsheetZarrManager.accent(b, c); + b = uibutton(g,'Text','View...','ButtonPushedFcn',@(~,~) obj.onView()); + b.Layout.Row = 3; b.Layout.Column = 3; + ndi.gui.app.LightsheetZarrManager.accent(b, c); + b = uibutton(g,'Text','Add...','ButtonPushedFcn',@(~,~) obj.onAdd()); + b.Layout.Row = 3; b.Layout.Column = 4; + ndi.gui.app.LightsheetZarrManager.accent(b, c); + b = uibutton(g,'Text','Delete','ButtonPushedFcn',@(~,~) obj.onDelete()); + b.Layout.Row = 3; b.Layout.Column = 5; + ndi.gui.app.LightsheetZarrManager.accent(b, c); - obj.statusLabel = uilabel(g, 'Text', ''); + obj.statusLabel = uilabel(g,'Text','','FontColor', c.darkBlue); + obj.statusLabel.Layout.Row = 4; obj.statusLabel.Layout.Column = [1 5]; end - function refresh(obj) - q = ndi.query('', 'isa', 'lightsheetZarrPyramid'); - docs = obj.session.database_search(q); - n = numel(docs); - rows = repmat(struct('label','','reduction','','subject','', ... - 'n_levels',0,'shape_level0','','dtype','','id',''), n, 1); - for k = 1:n - p = docs{k}.document_properties.lightsheetZarrPyramid; - rows(k).label = char(p.label); - rows(k).reduction = char(p.reduction); - rows(k).subject = subjectLabel(docs{k}); - rows(k).n_levels = p.n_levels; - rows(k).shape_level0 = mat2str(double(p.shape_level0)); - rows(k).dtype = char(p.dtype); - rows(k).id = docs{k}.id(); - end - obj.rows = rows; - obj.table.Data = struct2table(rows, 'AsArray', true); - obj.statusLabel.Text = sprintf('%d pyramids', n); + function setStatus(obj, msg) + if ~isempty(obj.statusLabel) && isvalid(obj.statusLabel) + obj.statusLabel.Text = msg; + end + end + + function r = selectedRow(obj) + r = []; + sel = obj.table.Selection; + if isempty(sel) || isempty(obj.rows), return; end + r = obj.rows(sel(1)); + end + + function onView(obj) + % ONVIEW - the launcher dialog, and then the launch + % + % A dialog rather than a button that just runs, because the + % things it asks about cannot be recovered afterwards from + % the window that opens: which channel, which reduction, + % which level, and what the layer is called. And because the + % launcher is a path on the user's machine that NDI cannot + % know, only default. + % + % THE COMMAND IS ON SCREEN before it runs, and stays readable + % afterwards. Launching another program is the step most + % likely to fail for reasons NDI cannot see -- a wrapper not + % installed, an environment that is not what it looks like -- + % and a command the user can copy into a terminal turns that + % from a shrug into something they can act on. + r = obj.selectedRow(); + if isempty(r) + uialert(obj.fig,'Select a pyramid first.','Nothing selected'); + return; + end + sessionPath = ''; + try + sessionPath = obj.session.getpath(); + catch + try + sessionPath = obj.session.path; + catch + end + end + if isempty(sessionPath) + uialert(obj.fig, ... + ['This session has no directory on disk. The napari ' ... + 'viewer opens an ndi.session.dir, and there is ' ... + 'nothing here for it to open.'], 'Nothing to view'); + return; + end + + % One dialog at a time. Non-modal, so nothing stops the + % button being pressed again; without this each press would + % build another and the user would be editing whichever + % landed on top while an older one held a different pyramid. + if ~isempty(obj.viewDialog) && isvalid(obj.viewDialog) + figure(obj.viewDialog); + return; + end + + c = ndi.gui.cloudColors(); + d = uifigure('Name','View in napari','Position',[120 120 660 430], ... + 'Color', c.offWhite); + gl = uigridlayout(d,[9 3]); + gl.RowHeight = {28, 40, 24, 24, 24, 24, 24, 24, 60}; + gl.ColumnWidth = {140, '1x', 100}; + gl.BackgroundColor = c.offWhite; + + hb = uigridlayout(gl,[1 1]); + hb.Layout.Row = 1; hb.Layout.Column = [1 3]; + hb.Padding = [8 0 8 0]; + hb.BackgroundColor = c.darkBlue; + uilabel(hb,'Text','View in napari', ... + 'FontColor', c.white, 'FontWeight','bold', 'FontSize',14, ... + 'VerticalAlignment','center'); + + intro = uilabel(gl,'WordWrap','on','Text', ... + ['Opens this pyramid in napari. The viewer is a separate ' ... + 'Python program: MATLAB starts it and does not wait for ' ... + 'it. The command that will run is shown at the bottom.']); + intro.Layout.Row = 2; intro.Layout.Column = [1 3]; + + lab = uilabel(gl,'Text','Launcher'); + lab.Layout.Row = 3; lab.Layout.Column = 1; + ed = uieditfield(gl,'text','Value', ... + ndi.gui.app.LightsheetZarrManager.launcherPath()); + ed.Layout.Row = 3; ed.Layout.Column = 2; + ndi.gui.app.LightsheetZarrManager.paper(ed, c); + br = uibutton(gl,'Text','Browse...'); + br.Layout.Row = 3; br.Layout.Column = 3; + ndi.gui.app.LightsheetZarrManager.accent(br, c); + + lab = uilabel(gl,'Text','Layer name'); + lab.Layout.Row = 4; lab.Layout.Column = 1; + nameEd = uieditfield(gl,'text','Value', ... + ndi.gui.app.LightsheetZarrManager.orEmpty(r.label)); + nameEd.Layout.Row = 4; nameEd.Layout.Column = [2 3]; + ndi.gui.app.LightsheetZarrManager.paper(nameEd, c); + + lab = uilabel(gl,'Text','Reduction'); + lab.Layout.Row = 5; lab.Layout.Column = 1; + redDd = uidropdown(gl, 'Items', {'(as selected)','mean','max'}, ... + 'Value','(as selected)'); + redDd.Layout.Row = 5; redDd.Layout.Column = [2 3]; + ndi.gui.app.LightsheetZarrManager.paper(redDd, c); + + lab = uilabel(gl,'Text','Channel'); + lab.Layout.Row = 6; lab.Layout.Column = 1; + chSpin = uispinner(gl,'Limits',[-1 32],'Value',-1,'Step',1); + chSpin.Layout.Row = 6; chSpin.Layout.Column = [2 3]; + chSpin.Tooltip = '-1 opens all channels as separate napari layers'; + ndi.gui.app.LightsheetZarrManager.paper(chSpin, c); + + lab = uilabel(gl,'Text','Initial level'); + lab.Layout.Row = 7; lab.Layout.Column = 1; + lvSpin = uispinner(gl,'Limits',[-1 32],'Value',-1,'Step',1); + lvSpin.Layout.Row = 7; lvSpin.Layout.Column = [2 3]; + lvSpin.Tooltip = '-1 lets the viewer choose based on window size'; + ndi.gui.app.LightsheetZarrManager.paper(lvSpin, c); + + cbPanels = uicheckbox(gl,'Text','Control panels','Value',true); + cbPanels.Layout.Row = 8; cbPanels.Layout.Column = [1 3]; + + cmdArea = uitextarea(gl,'Editable','off','Value',''); + cmdArea.Layout.Row = 9; cmdArea.Layout.Column = [1 2]; + ndi.gui.app.LightsheetZarrManager.paper(cmdArea, c); + go = uibutton(gl,'Text','Launch'); + ndi.gui.app.LightsheetZarrManager.accent(go, c); + go.Layout.Row = 9; go.Layout.Column = 3; + + br.ButtonPushedFcn = @(~,~) localBrowse(); + go.ButtonPushedFcn = @(~,~) localGo(); + everything = {ed, nameEd, redDd, chSpin, lvSpin, cbPanels}; + for i = 1:numel(everything) + everything{i}.ValueChangedFcn = @localRefresh; + end + + localRefresh(); + obj.viewDialog = d; + + function cmd = localCommand() + try + red = ''; + if ~strcmp(redDd.Value, '(as selected)') + red = redDd.Value; + end + cmd = ndi.fun.doc.lightsheet.viewCommand( ... + ed.Value, sessionPath, r.id, ... + 'reduction', red, ... + 'channel', chSpin.Value, ... + 'level', lvSpin.Value, ... + 'controls', cbPanels.Value, ... + 'name', nameEd.Value); + catch ME + cmd = ME.message; + end + end + + function localRefresh(~,~) + cmdArea.Value = localCommand(); + end + + function localBrowse() + [f,pth] = uigetfile('*','Select the napari viewer launcher'); + figure(d); + if isequal(f,0), return; end + ed.Value = fullfile(pth,f); + localRefresh(); + end + + function localGo() + cmd = localCommand(); + target = strtrim(ed.Value); + if ~isempty(regexp(target,'[/\\]','once')) && ~isfile(target) + uialert(d, sprintf(['No launcher at\n %s\n\nThat ' ... + 'path is normally a small shell wrapper that ' ... + 'scrubs MATLAB''s library paths and then runs ' ... + 'napariViewLightsheet from your NDI-python ' ... + 'environment. Install it there, or point this ' ... + 'at the console script yourself.'], target), ... + 'Launcher not found'); + return; + end + ndi.gui.app.LightsheetZarrManager.setLauncherPath(target); + status = system( ... + ndi.gui.app.LightsheetZarrManager.detach(cmd)); + delete(d); + if status == 0 + obj.setStatus(sprintf('Launched the viewer for %s.', ... + obj.rowLabel(r))); + else + obj.setStatus(sprintf(['The launcher exited %d. ' ... + 'The command was: %s'], status, cmd)); + end + end + end + + function onClose(obj) + % ONCLOSE - shut the manager, and anything it opened + % + % Only the View dialog is closed. The napari process is + % deliberately NOT touched: it was detached on purpose, it is + % somebody's picture, and closing the window that launched it + % is not a request to close it. + if ~isempty(obj.viewDialog) && isvalid(obj.viewDialog) + delete(obj.viewDialog); + end + delete(obj.fig); + end + + function onDelete(obj) + r = obj.selectedRow(); + if isempty(r) + uialert(obj.fig,'Select a pyramid first.','Nothing selected'); + return; + end + plan = ndi.gui.app.LightsheetZarrManager.deletionPlan( ... + obj.session, r.id); + msg = ndi.gui.app.LightsheetZarrManager.deletionMessage( ... + plan, obj.rowLabel(r)); + choice = uiconfirm(obj.fig, msg, 'Confirm delete', ... + 'Options',{'Delete','Cancel'}, ... + 'DefaultOption',2,'CancelOption',2); + if ~strcmp(choice,'Delete'), return; end + for k = 1:numel(plan.docsToRemove) + obj.session.database_rm(plan.docsToRemove{k}); + end + obj.setStatus(sprintf('Deleted %d document(s).', ... + 1 + plan.nLevels)); + obj.reload(); end function onAdd(obj) @@ -133,55 +582,19 @@ function onAdd(obj) subjectID = obj.pickSubject(); if isempty(subjectID), return; end + obj.setStatus('Ingesting...'); + drawnow; try ndi.fun.doc.lightsheet.fromOMEZarr(obj.session, zarrPath, ... 'subjectID', subjectID); - obj.refresh(); + obj.setStatus('Ingest complete.'); + obj.reload(); catch ME + obj.setStatus('Ingest failed; see the message.'); uialert(obj.fig, ME.message, 'Add failed'); end end - function onView(obj) - row = obj.selectedRow(); - if isempty(row), return; end - sessionPath = obj.session.path; - launcher = ndi.gui.app.LightsheetZarrManager.launcherPath(); - - cmd = ndi.fun.doc.lightsheet.viewCommand(launcher, sessionPath, row.id); - if ~ispc, cmd = [cmd ' &']; else, cmd = ['start "napari" ' cmd]; end - [status, output] = system(cmd); - if status ~= 0 - uialert(obj.fig, sprintf('%s\n\n%s', cmd, output), ... - 'Launch failed'); - end - end - - function onDelete(obj) - row = obj.selectedRow(); - if isempty(row), return; end - plan = ndi.gui.app.LightsheetZarrManager.deletionPlan(obj.session, row.id); - msg = sprintf('Delete pyramid %s and %d level documents?', ... - row.label, plan.nLevels); - answer = uiconfirm(obj.fig, msg, 'Delete', ... - 'Options', {'Delete', 'Cancel'}, 'DefaultOption', 'Cancel'); - if ~strcmp(answer, 'Delete'), return; end - for k = 1:numel(plan.docsToRemove) - obj.session.database_rm(plan.docsToRemove{k}); - end - obj.refresh(); - end - - function row = selectedRow(obj) - sel = obj.table.Selection; - if isempty(sel) - uialert(obj.fig, 'Select a pyramid first.', 'No selection'); - row = []; - return; - end - row = obj.rows(sel(1)); - end - function id = pickSubject(obj) subs = obj.session.database_search(ndi.query('', 'isa', 'subject')); if isempty(subs) @@ -190,47 +603,24 @@ function onDelete(obj) id = ''; return; end - labels = cellfun(@(d) char(d.document_properties.subject.local_identifier), ... + labels = cellfun(@(x) char( ... + x.document_properties.subject.local_identifier), ... subs, 'UniformOutput', false); [idx, ok] = listdlg('PromptString', 'Subject:', ... 'SelectionMode', 'single', 'ListString', labels); if ~ok, id = ''; return; end id = subs{idx}.id(); end - end - - methods (Static) - function p = launcherPath() - p = char(ndi.gui.app.LightsheetZarrManager.DefaultViewerLauncher); - try - v = ndi.preferences.get('GUI.LightsheetZarrManager.ViewerLauncher'); - if ~isempty(char(v)), p = char(v); end - catch - end - end - function plan = deletionPlan(session, pyramidID) - plan = struct(); - plan.pyramidID = char(pyramidID); - q1 = ndi.query('', 'isa', 'lightsheetZarrLevel'); - q2 = ndi.query('depends_on', 'depends_on', ... - 'lightsheetZarrPyramid_id', plan.pyramidID); - levelDocs = session.database_search(q1 & q2); - parentDoc = session.database_search(ndi.query('base.id', ... - 'exact_string', plan.pyramidID)); - plan.nLevels = numel(levelDocs); - plan.docsToRemove = [levelDocs(:); parentDoc(:)]; + function s = rowLabel(~, r) + s = r.label; + if isempty(s), s = r.id; end end end -end -function s = subjectLabel(pyramidDoc) - s = ''; - try - dep = pyramidDoc.dependency_value('subject_id'); - if ~isempty(dep) - s = char(dep); + methods (Static, Access = private) + function s = orEmpty(v) + if isempty(v), s = ''; else, s = char(v); end end - catch end end diff --git a/tests/+ndi/+unittest/+gui/+app/TestLightsheetZarrManager.m b/tests/+ndi/+unittest/+gui/+app/TestLightsheetZarrManager.m index 0edddad16..7c0d7bf0f 100644 --- a/tests/+ndi/+unittest/+gui/+app/TestLightsheetZarrManager.m +++ b/tests/+ndi/+unittest/+gui/+app/TestLightsheetZarrManager.m @@ -65,5 +65,46 @@ function testDeletionPlanOnAbsentPyramidReportsZero(testCase) testCase.verifyEmpty(plan.docsToRemove); end + function testDeletionMessageNamesTheCounts(testCase) + % The confirmation must distinguish 1 from many. The pyramid + % counts as 1, plus every level. + plan = struct('pyramidID','abcd','nLevels',7,'docsToRemove',{{}}); + msg = ndi.gui.app.LightsheetZarrManager.deletionMessage( ... + plan, 'my-pyramid'); + testCase.verifySubstring(msg, '8 document'); + testCase.verifySubstring(msg, '7 level'); + testCase.verifySubstring(msg, 'my-pyramid'); + end + + function testSubjectNameMapUsesLocalIdentifier(testCase) + % pyramidRows shows the subject's local_identifier, not its + % raw id. subjectNameMap is where that lookup lives; a + % subject with a local_identifier should be findable by its + % document id. + m = ndi.gui.app.LightsheetZarrManager.subjectNameMap( ... + testCase.session); + testCase.verifyTrue(isKey(m, testCase.subjectID)); + testCase.verifyEqual(m(testCase.subjectID), 'lightsheet@vhlab'); + end + + function testPyramidRowsCarriesSubjectNameAndDoc(testCase) + % One row per pyramid parent. Subject is the local_identifier + % (not the raw id), and the row keeps the raw doc alongside + % the id so the delete/view callbacks can act on it. + pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); + [pdoc, ~, ~] = ndi.fun.doc.lightsheet.makePyramid( ... + testCase.session, pys, {'mean','max'}, ... + 'subjectID', testCase.subjectID); + rows = ndi.gui.app.LightsheetZarrManager.pyramidRows( ... + testCase.session); + testCase.verifyNotEmpty(rows); + match = find(strcmp({rows.id}, pdoc.id()), 1); + testCase.verifyNotEmpty(match); + r = rows(match); + testCase.verifyEqual(r.subject, 'lightsheet@vhlab'); + testCase.verifyEqual(r.subjectID, testCase.subjectID); + testCase.verifyNotEmpty(r.doc); + end + end end From 7a45f12c5f5a80f281a47a748da4ce368f9b19c8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 12:39:49 +0000 Subject: [PATCH 041/110] lightsheet: progressFcn callback on makePyramid / fromOMEZarr Adds an optional progressFcn(fraction, text) name-value on ndi.fun.doc.lightsheet.makePyramid and .fromOMEZarr, wired through attachChunkFiles. Total chunk count is pre-computed across every level path so the reported fraction is over the whole ingest, not per level -- a bar in the UI never steps backwards, and 'chunk 3 of 24000' means what it says. The callback fires: * once at the start of each unique level (fraction = chunks done so far, text names the level and reduction); * after each chunk is written into the level document (fraction incremented, text names the level and the level-local chunk counter); * once at the end, with fraction 1.0 and text 'Done.', so a UI can safely leave the bar at its last-reported value. The helper reportProgress swallows callback exceptions: a bar that stopped drawing must not stop a multi-hour ingest. ingestBlobFixture passes progressFcn through to fromOMEZarr, and a new TestBlobFixture test captures the callback and verifies the fractions are monotone non-decreasing, in [0,1], and end exactly at 1.0 with the 'Done.' label. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 7 +- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 87 +++++++++++++++---- .../+test/+lightsheet/ingestBlobFixture.m | 4 +- .../+test/+lightsheet/TestBlobFixture.m | 47 ++++++++++ 4 files changed, 127 insertions(+), 18 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 1f6231463..adf7e463c 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -87,6 +87,10 @@ options.materializeChunks (1,1) logical = false options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 + % progressFcn(fraction, text) is called before each level and + % after each chunk write. Passed through to makePyramid; empty + % means silent. + options.progressFcn = [] end if isempty(strtrim(options.subjectID)) @@ -152,7 +156,8 @@ 'materializeChunks', options.materializeChunks, ... 'sourceZarrPath', char(zarrPath), ... 'codec', options.codec, ... - 'clevel', options.clevel); + 'clevel', options.clevel, ... + 'progressFcn', options.progressFcn); info = struct( ... 'zarrPath', char(zarrPath), ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 602c6a16c..cb37c7b13 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -72,6 +72,10 @@ options.sourceZarrPath char = '' options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 + % progressFcn(fraction, text) is called before and after each + % chunk write. Empty = no reporting. Keep this cheap; it fires + % once per chunk on a real ingest, which is many times. + options.progressFcn = [] end if options.materializeChunks && isempty(options.sourceZarrPath) @@ -148,10 +152,39 @@ session.database_add(pyramidDoc); + % PROGRESS ACCOUNTING. Chunk-count planning happens up front so + % progressFcn can report overall fraction, not per-level fraction: + % a viewer wants "3 of 24000 chunks done", not a bar that resets at + % every level. chooseTileShape is cheap (metadata only), so running + % it twice -- once here, once inside makeOneLevel -- costs nothing. + levelChunks = cell(numel(pathOrder), 1); + levelGrids = cell(numel(pathOrder), 1); + totalChunks = 0; + for idx = 1:numel(pathOrder) + p = pathOrder{idx}; + rep = representative(p); + entry = pyramids(rep.entryIdx); + level = entry.levels(rep.levelIdx); + axesOrder = joinAxisNames(entry.axes); + if isempty(options.chunks) + chunks_i = ndi.fun.doc.lightsheet.chooseTileShape(level.shape, ... + axesOrder, level.scale, char(level.dtype), ... + options.tileBudgetBytes); + else + chunks_i = clampChunksToShape(ensureRow(options.chunks), level.shape); + end + levelChunks{idx} = chunks_i; + levelGrids{idx} = ceil(level.shape ./ chunks_i); + if options.materializeChunks + totalChunks = totalChunks + prod(levelGrids{idx}); + end + end + % Assign each unique path a 0-based level index in the ORDER it % first appears (finest first, since listPyramids is finest-first). levelDocs = cell(numel(pathOrder), 1); sharedLevel0 = false; + doneChunks = 0; for idx = 1:numel(pathOrder) p = pathOrder{idx}; rep = representative(p); @@ -166,21 +199,19 @@ else reductionFn = rlist{1}; end - levelDocs{idx} = makeOneLevel(session, pyramidDoc, entry, ... - level, idx - 1, reductionFn, options); + reportProgress(options.progressFcn, doneChunks, totalChunks, ... + sprintf('Level %d/%d (%s, %s): starting', ... + idx, numel(pathOrder), char(entry.name), reductionFn)); + [levelDocs{idx}, doneChunks] = makeOneLevel(session, pyramidDoc, ... + entry, level, idx - 1, reductionFn, options, ... + levelChunks{idx}, levelGrids{idx}, ... + doneChunks, totalChunks); end + reportProgress(options.progressFcn, max(doneChunks, totalChunks), ... + max(totalChunks, 1), 'Done.'); end -function levelDoc = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, reductionFn, options) - axesOrder = joinAxisNames(pyramidEntry.axes); - if isempty(options.chunks) - chunks = ndi.fun.doc.lightsheet.chooseTileShape(level.shape, ... - axesOrder, level.scale, char(level.dtype), ... - options.tileBudgetBytes); - else - chunks = clampChunksToShape(ensureRow(options.chunks), level.shape); - end - chunkGrid = ceil(level.shape ./ chunks); +function [levelDoc, doneChunks] = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, reductionFn, options, chunks, chunkGrid, doneChunks, totalChunks) if options.materializeChunks nChunksStored = prod(chunkGrid); @@ -219,16 +250,18 @@ tmpRoot = ''; if options.materializeChunks - [levelDoc, tmpRoot] = attachChunkFiles(levelDoc, options.sourceZarrPath, ... - pyramidEntry, level, chunks, chunkGrid, ... - options.codec, options.clevel); + levelLabel = sprintf('Level %d (%s)', levelIndex, reductionFn); + [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, ... + options.sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, ... + options.codec, options.clevel, ... + options.progressFcn, doneChunks, totalChunks, levelLabel); end tmpCleaner = onCleanup(@() cleanupTmp(tmpRoot)); session.database_add(levelDoc); end -function [levelDoc, tmpRoot] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel) +function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel) % Read the source zarr level as a whole array, then re-tile into the % level document's chunk shape and attach each chunk as chunk.bin_ % (1-based, C-order over chunkGrid). Files are ingested into DID, which @@ -319,6 +352,10 @@ fclose(fid); levelDoc = levelDoc.add_file(sprintf('chunk.bin_%d', idx), tmp); + + doneChunks = doneChunks + 1; + reportProgress(progressFcn, doneChunks, totalChunks, ... + sprintf('%s: chunk %d/%d', levelLabel, idx, size(linearOrder,1))); end end @@ -361,6 +398,24 @@ function cleanupTmp(d) end end +function reportProgress(fcn, done, total, text) +% REPORTPROGRESS - call the caller's progress callback, if any +% +% Fraction is done/total, clamped into [0,1]. Total==0 (no chunks +% materialized) reports 0 rather than dividing, so the caller's bar +% still gets its label refreshed. Any exception is swallowed: a bar +% that stopped drawing must not stop an ingest. + if isempty(fcn), return; end + frac = 0; + if total > 0 + frac = max(0, min(1, double(done) / double(total))); + end + try + fcn(frac, char(text)); + catch + end +end + function idx = allChunkIndices(chunkGrid) % Return a matrix whose rows enumerate every 1-based chunk index tuple % in C-order (last axis fastest, so the LINEAR order matches diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index 4794f971c..d6fd046cf 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -64,6 +64,7 @@ options.materializeChunks (1,1) logical = true options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 + options.progressFcn = [] end sessionDir = options.sessionDir; @@ -100,7 +101,8 @@ 'subjectID', sub.id(), ... 'materializeChunks', options.materializeChunks, ... 'codec', options.codec, ... - 'clevel', options.clevel); + 'clevel', options.clevel, ... + 'progressFcn', options.progressFcn); info.subjectID = sub.id(); info.fixture = gt; diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index 89133c1c9..c4256415c 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -189,6 +189,53 @@ function testStreamingReTileHitsSmallerBudget(testCase) testCase.verifyNotEmpty(info.subjectID); end + function testProgressCallbackReceivesMonotoneFractionsEndingAtOne(testCase) + % progressFcn is called before each level starts and after + % each chunk write. The fractions should be monotone + % non-decreasing (never step backwards) and end at exactly + % 1.0 with the 'Done.' label -- so a bar in the UI is safe + % to leave at the last reported fraction. + if isempty(which('ndr.format.omezarr.listPyramids')) + testCase.assumeFail(['NDR reader not on path; skipping ' ... + 'end-to-end ingest.']); + end + sDir = fullfile(tempname, 'progsession'); + testCase.addTeardown(@() rmdir(fileparts(sDir), 's')); + zDir = fullfile(tempname, 'progzarr'); + testCase.addTeardown(@() rmdir(zDir, 's')); + + log = struct('frac', {}, 'text', {}); + function record(f, t) + log(end+1).frac = f; %#ok + log(end).text = t; + end + captureFcn = @(f, t) record(f, t); + + % Small volume with 2 pyramid levels and materialized + % chunks so several chunks fire the callback. + ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDir, ... + 'zarrDir', zDir, ... + 'Shape', [32 32 32], ... + 'NumChannels', 1, ... + 'NumLevels', 2, ... + 'ChunkShape', [16 16 16], ... + 'materializeChunks', true, ... + 'progressFcn', captureFcn); + + testCase.verifyNotEmpty(log, ... + 'The callback should have fired at least once.'); + fracs = [log.frac]; + testCase.verifyTrue(all(fracs >= 0 & fracs <= 1), ... + 'All fractions must be in [0,1].'); + testCase.verifyTrue(all(diff(fracs) >= -1e-12), ... + 'Fractions must be monotone non-decreasing.'); + testCase.verifyEqual(fracs(end), 1, ... + 'Final tick should be fraction 1.0.'); + testCase.verifyEqual(log(end).text, 'Done.', ... + 'Final tick should carry the Done label.'); + end + function testMaxKeepsBrightSpikeMeanDoesNot(testCase) % At the spike's downsampled position, max preserves the % 60k spike and mean dilutes it (averages the one spike From c32fae8e1c2d66043a822d7def81fd37ccdf0676 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 13:00:57 +0000 Subject: [PATCH 042/110] fromOMEZarr: default progress bar is the NDI ProgressBarWindow Previously fromOMEZarr's progressFcn defaulted to [] (silent). Users who wanted a bar had to build one themselves and hand it a callback. Now the default is the string 'default': fromOMEZarr opens an ndi.gui.component.ProgressBarWindow titled 'NDI lightsheet ingest', adds an Auto=true bar labelled from the pyramid label (or the zarr basename), and drives it. Auto=true has the bar remove itself once frac hits 1.0, and ProgressBarWindow's AutoDelete then closes the window when its last bar goes. A caller who wants a different receiver passes a function handle; a caller who wants nothing on screen passes [] as before. ProgressBarWindow already detects headless MATLAB and no-ops without a figure, so this is safe on CI: the bar's bookkeeping runs, no uifigure is created. An unknown string spec is refused with NDI:lightsheet:fromOMEZarr:badProgressFcn rather than being treated as silent -- an easily-mistyped default should not silently disable progress. ingestBlobFixture picks up the same default so the blob demo shows the bar too. Tests cover the silent (progressFcn=[]) path and the bad-spec error, alongside the existing function-handle capture test. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 85 +++++++++++++++++-- .../+test/+lightsheet/ingestBlobFixture.m | 6 +- .../+test/+lightsheet/TestBlobFixture.m | 48 +++++++++++ 3 files changed, 133 insertions(+), 6 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index adf7e463c..433086398 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -87,10 +87,18 @@ options.materializeChunks (1,1) logical = false options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 - % progressFcn(fraction, text) is called before each level and - % after each chunk write. Passed through to makePyramid; empty - % means silent. - options.progressFcn = [] + % progressFcn is one of: + % 'default' - the default. Open an NDI + % ProgressBarWindow and drive it. + % Headless MATLAB (CI, -batch) does + % silent bookkeeping only, no figure, + % so this is safe on servers. + % function_handle - progressFcn(fraction, text) is + % called before each level and after + % each chunk write. Use this to feed + % your own dialog or logger. + % [] - no reporting at all. + options.progressFcn = 'default' end if isempty(strtrim(options.subjectID)) @@ -144,6 +152,14 @@ sourceFileID = srcDoc.id(); end + % Resolve the progress callback: 'default' -> open an NDI + % ProgressBarWindow and update it; a function handle -> use it as + % is; [] -> stay silent. The window's constructor detects headless + % MATLAB by itself and no-ops without a figure, so CI runs need no + % special casing here. + progressFcn = resolveProgressCallback(options.progressFcn, ... + options.label, char(zarrPath)); + [pyramidDoc, levelDocs, sharedLevel0] = ndi.fun.doc.lightsheet.makePyramid( ... session, pyramids, perEntryReduction, ... 'subjectID', options.subjectID, ... @@ -157,7 +173,7 @@ 'sourceZarrPath', char(zarrPath), ... 'codec', options.codec, ... 'clevel', options.clevel, ... - 'progressFcn', options.progressFcn); + 'progressFcn', progressFcn); info = struct( ... 'zarrPath', char(zarrPath), ... @@ -205,3 +221,62 @@ end end end + +function fcn = resolveProgressCallback(spec, label, zarrPath) +% RESOLVEPROGRESSCALLBACK - turn the options.progressFcn spec into a +% callable, opening an NDI ProgressBarWindow when the caller took the +% default. Failures to open the bar are swallowed and the ingest +% continues silently: a bar that could not draw itself must not stop +% a multi-hour ingest. + if isa(spec, 'function_handle') + fcn = spec; + return; + end + if isempty(spec) + fcn = []; + return; + end + if ~(ischar(spec) || (isstring(spec) && isscalar(spec))) + error('NDI:lightsheet:fromOMEZarr:badProgressFcn', ... + ['progressFcn must be ''default'', a function handle, ' ... + 'or []. Got a %s.'], class(spec)); + end + spec = char(string(spec)); + if ~any(strcmpi(spec, {'default', 'auto'})) + error('NDI:lightsheet:fromOMEZarr:badProgressFcn', ... + ['progressFcn keyword ''%s'' is not recognised. Use ' ... + '''default'', a function handle, or [].'], spec); + end + + barLabel = 'Building lightsheet pyramid'; + if ~isempty(label) + barLabel = sprintf('Building pyramid: %s', label); + elseif ~isempty(zarrPath) + [~, zn] = fileparts(char(zarrPath)); + if ~isempty(zn) + barLabel = sprintf('Building pyramid: %s', zn); + end + end + tag = sprintf('lightsheet-ingest-%s', localUniqueId()); + + fcn = []; + try + pbw = ndi.gui.component.ProgressBarWindow('NDI lightsheet ingest'); + pbw.addBar('Label', barLabel, 'Tag', tag, 'Auto', true); + fcn = @(frac, txt) pbw.updateBar(tag, min(1, max(0, frac))); + catch + % Fall through with fcn = [] -- silent. + end +end + +function u = localUniqueId() +% A short unique tag so a second ingest in the same MATLAB does not +% collide with a bar the first one is still holding. + try + u = did.ido.unique_id(); + u = char(u); + catch + u = char(java.util.UUID.randomUUID().toString()); + end + if numel(u) > 12, u = u(1:12); end +end diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index d6fd046cf..fa6c1a012 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -64,7 +64,11 @@ options.materializeChunks (1,1) logical = true options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 - options.progressFcn = [] + % Match fromOMEZarr's default: 'default' opens the NDI + % ProgressBarWindow. Pass a function handle for a custom + % receiver, or [] to stay silent (which is what CI needs + % without the auto-headless detection kicking in). + options.progressFcn = 'default' end sessionDir = options.sessionDir; diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index c4256415c..ad9699c3a 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -236,6 +236,54 @@ function record(f, t) 'Final tick should carry the Done label.'); end + function testProgressFcnEmptyStaysSilent(testCase) + % [] means silent -- no NDI bar opened, no callbacks + % anywhere. The ingest should complete cleanly all the + % same. This is what CI passes to keep the run truly + % headless without depending on the auto-headless mode. + if isempty(which('ndr.format.omezarr.listPyramids')) + testCase.assumeFail(['NDR reader not on path; skipping ' ... + 'end-to-end ingest.']); + end + sDir = fullfile(tempname, 'silentsession'); + testCase.addTeardown(@() rmdir(fileparts(sDir), 's')); + zDir = fullfile(tempname, 'silentzarr'); + testCase.addTeardown(@() rmdir(zDir, 's')); + [~, pdoc, ~] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDir, ... + 'zarrDir', zDir, ... + 'Shape', [32 32 32], ... + 'NumChannels', 1, ... + 'NumLevels', 2, ... + 'ChunkShape', [16 16 16], ... + 'materializeChunks', true, ... + 'progressFcn', []); + testCase.verifyClass(pdoc, 'ndi.document'); + end + + function testProgressFcnBadSpecErrors(testCase) + % A stray string that is not 'default' or 'auto' is a + % user error and must be named -- not silently treated as + % silent. fromOMEZarr's resolver raises with the id + % NDI:lightsheet:fromOMEZarr:badProgressFcn. + sDir = fullfile(tempname, 'errsession'); + testCase.addTeardown(@() rmdir(fileparts(sDir), 's')); + zDir = fullfile(tempname, 'errzarr'); + testCase.addTeardown(@() rmdir(zDir, 's')); + % A tiny fixture is enough because argument parsing + % raises before the read starts. + testCase.verifyError(@() ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDir, ... + 'zarrDir', zDir, ... + 'Shape', [16 16 16], ... + 'NumChannels', 1, ... + 'NumLevels', 1, ... + 'ChunkShape', [16 16 16], ... + 'materializeChunks', false, ... + 'progressFcn', 'banana'), ... + 'NDI:lightsheet:fromOMEZarr:badProgressFcn'); + end + function testMaxKeepsBrightSpikeMeanDoesNot(testCase) % At the spike's downsampled position, max preserves the % 60k spike and mean dilutes it (averages the one spike From 98ddeb1eb937262ded514e131068771f6603ca9f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 13:53:38 +0000 Subject: [PATCH 043/110] lightsheet: parallel compression via serial-read / batched parfor Zstd at clevel=9 is single-core CPU-bound and dominates the ingest wall time on any spinning-disk source. Reads still cannot run in parallel -- a slow USB drive would thrash on concurrent seeks -- so this splits the two phases: a batch of numWorkers chunks is read sequentially into memory, then handed to a parfor of that many workers to compress and write per-chunk tempfiles. add_file into the level document happens serially in chunk-index order after the batch completes, so DID's SQLite write path is not a contention point and the on-disk chunk numbering is unchanged. numWorkers=0/1 is the original serial path (no Parallel Computing Toolbox dependency), numWorkers>1 opts in. On lab-scale volumes this cuts compression time by ~numWorkers. resolveParallelPool degrades gracefully: a missing PCT or a failed parpool startup warns and falls back to serial rather than aborting an ingest that may be well underway. encodeAndWrite is lifted to a pure per-chunk subfunction so the parfor iteration body has no dependency on caller state beyond its arguments; ndr.format.blosc.encode already spawns its own worker subprocess, so N parfor workers get N concurrent Blosc encoders. fromOMEZarr and ingestBlobFixture pass numWorkers through. Test covers byte-equivalence: parallel and serial ingest produce identical chunk.bin_# files for every level. Skipped when PCT is not on the runner. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 8 +- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 190 +++++++++++++----- .../+test/+lightsheet/ingestBlobFixture.m | 4 +- .../+test/+lightsheet/TestBlobFixture.m | 87 ++++++++ 4 files changed, 234 insertions(+), 55 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 433086398..cb28ed70e 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -99,6 +99,11 @@ % your own dialog or logger. % [] - no reporting at all. options.progressFcn = 'default' + % Number of parallel workers to use for chunk compression. 0 + % or 1 = single-threaded (default, no toolbox needed). >1 + % requires the Parallel Computing Toolbox: reads stay serial + % and compression fans out. See makePyramid for details. + options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, 0)} = 0 end if isempty(strtrim(options.subjectID)) @@ -173,7 +178,8 @@ 'sourceZarrPath', char(zarrPath), ... 'codec', options.codec, ... 'clevel', options.clevel, ... - 'progressFcn', progressFcn); + 'progressFcn', progressFcn, ... + 'numWorkers', options.numWorkers); info = struct( ... 'zarrPath', char(zarrPath), ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index cb37c7b13..f881415ba 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -76,6 +76,15 @@ % chunk write. Empty = no reporting. Keep this cheap; it fires % once per chunk on a real ingest, which is many times. options.progressFcn = [] + % Number of parallel workers to use for compression. 0 or 1 = + % single-threaded (the original path, no toolbox dependency). + % >1 requires the Parallel Computing Toolbox: reads stay serial + % (a slow spinning drive would thrash otherwise), a batch of + % numWorkers chunks is read into memory, then compression fans + % out to that many workers. Ingestion order into DID is + % preserved. If parpool cannot be built, this falls back to + % serial with a warning rather than failing. + options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, 0)} = 0 end if options.materializeChunks && isempty(options.sourceZarrPath) @@ -254,14 +263,15 @@ [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, ... options.sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, ... options.codec, options.clevel, ... - options.progressFcn, doneChunks, totalChunks, levelLabel); + options.progressFcn, doneChunks, totalChunks, levelLabel, ... + options.numWorkers); end tmpCleaner = onCleanup(@() cleanupTmp(tmpRoot)); session.database_add(levelDoc); end -function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel) +function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers) % Read the source zarr level as a whole array, then re-tile into the % level document's chunk shape and attach each chunk as chunk.bin_ % (1-based, C-order over chunkGrid). Files are ingested into DID, which @@ -292,9 +302,7 @@ shape(end+1:numel(chunks)) = 1; end - % Iterate chunks in C-order (last axis fastest). 1-based linear - % index goes into the chunk.bin_# filename. - idx = 0; + % 1-based linear index goes into the chunk.bin_# filename. tmpRoot = tempname; mkdir(tmpRoot); % No onCleanup here: tmpRoot is returned so the CALLER can register @@ -302,60 +310,136 @@ % we cleaned up on return, database_add would find the paths already % deleted. linearOrder = allChunkIndices(chunkGrid); % rows are (c1, c2, c3, ...) 1-based - dtypeCls = ''; - for r = 1:size(linearOrder, 1) - idx = idx + 1; - cIdx = linearOrder(r, :); - % Compute this chunk's on-disk region: [lo; hi] per axis, - % 1-based inclusive, clamped to the level shape at the far - % edge. - loRow = zeros(1, numel(shape)); - hiRow = zeros(1, numel(shape)); - for a = 1:numel(chunkGrid) - loRow(a) = (cIdx(a) - 1) * chunks(a) + 1; - hiRow(a) = min(loRow(a) + chunks(a) - 1, shape(a)); + nChunks = size(linearOrder, 1); + typesize = elementSizeBytes(char(level.dtype)); + + % BATCHED SERIAL-READ / PARALLEL-COMPRESS. On a slow spinning drive + % we cannot afford concurrent seeks, so reads stay strictly serial; + % but Zstd at clevel=9 is single-core CPU-bound and easy to fan out + % across a parallel pool. A batch of `poolSize` chunks is read from + % the source, then handed to the workers to compress and write, in + % parallel; add_file into the level document happens serially in + % chunk-index order so the DID SQLite write is not a contention + % point and the on-disk chunk numbering is unchanged. poolSize == 1 + % is the original serial path with no toolbox dependency. + poolSize = resolveParallelPool(numWorkers); + + for batchStart = 1:poolSize:nChunks + batchEnd = min(batchStart + poolSize - 1, nChunks); + batchIdx = batchStart:batchEnd; + b = numel(batchIdx); + + % ---- SERIAL READ into a batch buffer ----------------------- + raws = cell(1, b); + for j = 1:b + r = batchIdx(j); + cIdx = linearOrder(r, :); + loRow = zeros(1, numel(shape)); + hiRow = zeros(1, numel(shape)); + for a = 1:numel(chunkGrid) + loRow(a) = (cIdx(a) - 1) * chunks(a) + 1; + hiRow(a) = min(loRow(a) + chunks(a) - 1, shape(a)); + end + srcTile = ndr.format.omezarr.readArray(sourceZarrPath, ... + pyramidName, kInEntry, 'Region', [loRow; hiRow]); + padded = zeros(chunks, class(srcTile)); + localSubs = arrayfun(@(k) 1:size(srcTile, k), ... + 1:numel(chunks), 'UniformOutput', false); + padded(localSubs{:}) = srcTile; + % Zarr expects C-order bytes; MATLAB is column-major, + % so permute the axes into reversed order before + % serialising, mirroring the fixture writer. + permuted = permute(padded, numel(chunks):-1:1); + raws{j} = typecast(permuted(:), 'uint8'); end - srcTile = ndr.format.omezarr.readArray(sourceZarrPath, ... - pyramidName, kInEntry, 'Region', [loRow; hiRow]); - if isempty(dtypeCls) - dtypeCls = class(srcTile); + + % ---- PARALLEL (or serial) ENCODE + WRITE TEMPFILE ---------- + tempPaths = cell(1, b); + chunkIdxs = zeros(1, b); + for j = 1:b + chunkIdxs(j) = batchIdx(j); end - padded = zeros(chunks, dtypeCls); - localSubs = arrayfun(@(k) 1:size(srcTile, k), 1:numel(chunks), ... - 'UniformOutput', false); - padded(localSubs{:}) = srcTile; - - % Zarr expects C-order bytes: last axis varies fastest. MATLAB - % is column-major, so permute the axes into reversed order - % before serialising, mirroring the fixture writer. - permuted = permute(padded, numel(chunks):-1:1); - raw = typecast(permuted(:), 'uint8'); - - switch codec - case 'raw' - payload = raw; - case 'blosc-zstd' - typesize = elementSizeBytes(class(padded)); - payload = ndr.format.blosc.encode(raw, ... - 'typesize', typesize, ... - 'cname', 'zstd', ... - 'clevel', clevel, ... - 'shuffle', 1); - otherwise - error('NDI:lightsheet:makePyramid:unknownCodec', ... - 'Unknown codec %s.', codec); + if poolSize > 1 && b > 1 + parfor j = 1:b + tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... + typesize, tmpRoot, chunkIdxs(j)); %#ok + end + else + for j = 1:b + tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... + typesize, tmpRoot, chunkIdxs(j)); + end end - tmp = fullfile(tmpRoot, sprintf('chunk_%d.bin', idx)); - fid = fopen(tmp, 'w'); - fwrite(fid, payload); - fclose(fid); + % ---- SERIAL ADD_FILE (in chunk order) ---------------------- + for j = 1:b + levelDoc = levelDoc.add_file( ... + sprintf('chunk.bin_%d', chunkIdxs(j)), tempPaths{j}); + doneChunks = doneChunks + 1; + reportProgress(progressFcn, doneChunks, totalChunks, ... + sprintf('%s: chunk %d/%d', levelLabel, ... + chunkIdxs(j), nChunks)); + end + end +end - levelDoc = levelDoc.add_file(sprintf('chunk.bin_%d', idx), tmp); +function tmp = encodeAndWrite(raw, codec, clevel, typesize, tmpRoot, chunkIdx) +% ENCODEANDWRITE - compress one chunk and drop it in tmpRoot +% +% Pure per-chunk work: whatever runs in a parfor iteration lives +% here so the parallel worker has no dependency on outer-function +% state beyond the arguments it receives. Returns the tempfile path +% that add_file will consume. + switch codec + case 'raw' + payload = raw; + case 'blosc-zstd' + payload = ndr.format.blosc.encode(raw, ... + 'typesize', typesize, ... + 'cname', 'zstd', ... + 'clevel', clevel, ... + 'shuffle', 1); + otherwise + error('NDI:lightsheet:makePyramid:unknownCodec', ... + 'Unknown codec %s.', codec); + end + tmp = fullfile(tmpRoot, sprintf('chunk_%d.bin', chunkIdx)); + fid = fopen(tmp, 'w'); + fwrite(fid, payload); + fclose(fid); +end - doneChunks = doneChunks + 1; - reportProgress(progressFcn, doneChunks, totalChunks, ... - sprintf('%s: chunk %d/%d', levelLabel, idx, size(linearOrder,1))); +function n = resolveParallelPool(numWorkers) +% RESOLVEPARALLELPOOL - open (or attach to) a parpool of size numWorkers +% +% Returns the effective batch size. 0 or 1 -> serial, no toolbox +% dependency. >1 tries to create/attach to a parpool with the +% requested worker count; a missing PCT falls back to serial with a +% warning, since a broken pool must not stop an ingest. + n = 1; + if numWorkers <= 1 + return; + end + if isempty(ver('parallel')) + warning('NDI:lightsheet:makePyramid:noPCT', ... + ['numWorkers=%d requested but Parallel Computing Toolbox ' ... + 'is not available. Running serially.'], numWorkers); + return; + end + try + p = gcp('nocreate'); + if isempty(p) || p.NumWorkers ~= numWorkers + if ~isempty(p) + delete(p); + end + parpool('local', numWorkers); + end + n = numWorkers; + catch ME + warning('NDI:lightsheet:makePyramid:parpoolFailed', ... + ['Could not start a parpool of %d workers (%s). ' ... + 'Running serially.'], numWorkers, ME.message); + n = 1; end end diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index fa6c1a012..ec2472394 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -69,6 +69,7 @@ % receiver, or [] to stay silent (which is what CI needs % without the auto-headless detection kicking in). options.progressFcn = 'default' + options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, 0)} = 0 end sessionDir = options.sessionDir; @@ -106,7 +107,8 @@ 'materializeChunks', options.materializeChunks, ... 'codec', options.codec, ... 'clevel', options.clevel, ... - 'progressFcn', options.progressFcn); + 'progressFcn', options.progressFcn, ... + 'numWorkers', options.numWorkers); info.subjectID = sub.id(); info.fixture = gt; diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index ad9699c3a..1aadca626 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -284,6 +284,93 @@ function testProgressFcnBadSpecErrors(testCase) 'NDI:lightsheet:fromOMEZarr:badProgressFcn'); end + function testParallelWorkersProduceIdenticalChunks(testCase) + % Serial and parallel ingest MUST agree byte-for-byte on + % every materialized chunk. The point of parallelising is + % to run the same encode faster on more cores, not to + % change the pyramid. Skipped when the Parallel Computing + % Toolbox is not on the runner. + if isempty(which('ndr.format.omezarr.listPyramids')) + testCase.assumeFail(['NDR reader not on path; ' ... + 'skipping end-to-end ingest.']); + end + if isempty(ver('parallel')) + testCase.assumeFail(['Parallel Computing Toolbox ' ... + 'not available on this runner; skipping.']); + end + + % Two runs, same fixture (fixed rng inside makeBlobFixture), + % different session dirs so both sets of chunks survive + % long enough to be compared. + sDirS = fullfile(tempname, 'serialsession'); + testCase.addTeardown(@() rmdir(fileparts(sDirS), 's')); + zDirS = fullfile(tempname, 'serialzarr'); + testCase.addTeardown(@() rmdir(zDirS, 's')); + [Ss, pS, ~] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDirS, 'zarrDir', zDirS, ... + 'Shape', [32 32 32], 'NumChannels', 1, ... + 'NumLevels', 2, 'ChunkShape', [16 16 16], ... + 'materializeChunks', true, 'progressFcn', [], ... + 'codec', 'blosc-zstd', 'clevel', 3, ... + 'numWorkers', 0); + + sDirP = fullfile(tempname, 'parallelsession'); + testCase.addTeardown(@() rmdir(fileparts(sDirP), 's')); + zDirP = fullfile(tempname, 'parallelzarr'); + testCase.addTeardown(@() rmdir(zDirP, 's')); + [Sp, pP, ~] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDirP, 'zarrDir', zDirP, ... + 'Shape', [32 32 32], 'NumChannels', 1, ... + 'NumLevels', 2, 'ChunkShape', [16 16 16], ... + 'materializeChunks', true, 'progressFcn', [], ... + 'codec', 'blosc-zstd', 'clevel', 3, ... + 'numWorkers', 2); + + % Match up level docs by (level index, reduction) then + % compare each chunk.bin_# byte-for-byte. + levelsS = Ss.database_search( ... + ndi.query('','depends_on', ... + 'lightsheetZarrPyramid_id', pS.id()) & ... + ndi.query('','isa','lightsheetZarrLevel')); + levelsP = Sp.database_search( ... + ndi.query('','depends_on', ... + 'lightsheetZarrPyramid_id', pP.id()) & ... + ndi.query('','isa','lightsheetZarrLevel')); + testCase.verifyEqual(numel(levelsS), numel(levelsP), ... + 'Level count must match between serial and parallel'); + + keyOf = @(L) sprintf('%d-%s', ... + L.document_properties.lightsheetZarrLevel.level, ... + L.document_properties.lightsheetZarrLevel.reduction_function); + keysS = cellfun(keyOf, levelsS, 'UniformOutput', false); + keysP = cellfun(keyOf, levelsP, 'UniformOutput', false); + + for i = 1:numel(levelsS) + j = find(strcmp(keysP, keysS{i}), 1); + testCase.verifyNotEmpty(j, ... + sprintf('No matching parallel level for %s', keysS{i})); + lS = levelsS{i}; + lP = levelsP{j}; + nc = lS.document_properties.lightsheetZarrLevel.n_chunks_stored; + testCase.verifyEqual( ... + lP.document_properties.lightsheetZarrLevel.n_chunks_stored, ... + nc, 'n_chunks_stored differs'); + for k = 1:nc + nm = sprintf('chunk.bin_%d', k); + fs = Ss.database_openbinarydoc(lS, nm); + bs = fread(fs.fid, Inf, 'uint8=>uint8'); + Ss.database_closebinarydoc(fs); + fp = Sp.database_openbinarydoc(lP, nm); + bp = fread(fp.fid, Inf, 'uint8=>uint8'); + Sp.database_closebinarydoc(fp); + testCase.verifyEqual(bs, bp, ... + sprintf('chunk %s of level %s differs ' ... + 'between serial and parallel', ... + nm, keysS{i})); + end + end + end + function testMaxKeepsBrightSpikeMeanDoesNot(testCase) % At the spike's downsampled position, max preserves the % 60k spike and mean dilutes it (averages the one spike From f19bf3985aaaa1f3fba3754c3e8f9ff03d7f2f1c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 13:58:16 +0000 Subject: [PATCH 044/110] lightsheet: numWorkers=-1 (default) reads an already-open parpool The prior default of 0 forced serial even when the caller had a parpool already sized and ready. That's the common case for anyone running MATLAB with the Parallel Computing Toolbox: they parpool once at the top of their session and expect every downstream tool to benefit. New sentinel: -1 (default) - inspect gcp('nocreate'). If a pool is open, use it (its NumWorkers is the batch size); if none is open, run serially. No pool is created here. 0 / 1 - force serial regardless of any open pool. N > 1 - open (or replace) a parpool of size N. Unchanged. The 'auto' path never spawns a pool of its own -- the caller stays in charge of pool lifecycle, and a script that never opened one still gets the fully-serial, toolbox-independent path. A caller who did open one gets parallel compression for free on every fromOMEZarr call. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 12 ++-- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 66 +++++++++++++------ .../+test/+lightsheet/ingestBlobFixture.m | 4 +- 3 files changed, 56 insertions(+), 26 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index cb28ed70e..be7954fd7 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -99,11 +99,13 @@ % your own dialog or logger. % [] - no reporting at all. options.progressFcn = 'default' - % Number of parallel workers to use for chunk compression. 0 - % or 1 = single-threaded (default, no toolbox needed). >1 - % requires the Parallel Computing Toolbox: reads stay serial - % and compression fans out. See makePyramid for details. - options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, 0)} = 0 + % Number of parallel workers to use for chunk compression. + % -1 (default) - use any parpool already open, else serial. + % 0 / 1 - force serial. + % N > 1 - open a parpool of size N. + % Reads stay serial in all modes; compression fans out. + % See makePyramid for details. + options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, -1)} = -1 end if isempty(strtrim(options.subjectID)) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index f881415ba..0b444ea9a 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -76,15 +76,22 @@ % chunk write. Empty = no reporting. Keep this cheap; it fires % once per chunk on a real ingest, which is many times. options.progressFcn = [] - % Number of parallel workers to use for compression. 0 or 1 = - % single-threaded (the original path, no toolbox dependency). - % >1 requires the Parallel Computing Toolbox: reads stay serial - % (a slow spinning drive would thrash otherwise), a batch of - % numWorkers chunks is read into memory, then compression fans - % out to that many workers. Ingestion order into DID is - % preserved. If parpool cannot be built, this falls back to - % serial with a warning rather than failing. - options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, 0)} = 0 + % Number of parallel workers to use for compression. + % -1 (default) - use whatever parpool is already open; + % if none is open, run serially. This lets + % the caller size and manage the pool once + % at the top of their script, and every + % subsequent ingest just uses it. + % 0 / 1 - force serial (the original path, no + % toolbox dependency). + % N > 1 - open (or replace) a parpool of size N and + % use it. Requires the Parallel Computing + % Toolbox; falls back to serial with a + % warning if unavailable. + % Reads stay strictly serial in all modes; a batch of + % effective-pool-size chunks is read into memory, then + % compression fans out. Ingestion order into DID is preserved. + options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, -1)} = -1 end if options.materializeChunks && isempty(options.sourceZarrPath) @@ -410,24 +417,43 @@ end function n = resolveParallelPool(numWorkers) -% RESOLVEPARALLELPOOL - open (or attach to) a parpool of size numWorkers +% RESOLVEPARALLELPOOL - resolve numWorkers into an effective batch size % -% Returns the effective batch size. 0 or 1 -> serial, no toolbox -% dependency. >1 tries to create/attach to a parpool with the -% requested worker count; a missing PCT falls back to serial with a -% warning, since a broken pool must not stop an ingest. - n = 1; - if numWorkers <= 1 +% -1 -> the size of any parpool already open, else 1 (serial). This +% is the default so a caller who ran `parpool(8)` once at the +% top of their session gets 8-way compression without having +% to name a number in every ingest call. +% 0 or 1 -> serial. +% N > 1 -> ensure a parpool of size N is running (create it or +% restart the existing one at a different size), returning N. +% +% In every case a missing PCT or a failed parpool startup degrades +% to serial with a warning: a broken pool must not stop an ingest +% that may already be hours in. + if numWorkers == 0 || numWorkers == 1 + n = 1; return; end if isempty(ver('parallel')) - warning('NDI:lightsheet:makePyramid:noPCT', ... - ['numWorkers=%d requested but Parallel Computing Toolbox ' ... - 'is not available. Running serially.'], numWorkers); + if numWorkers > 1 + warning('NDI:lightsheet:makePyramid:noPCT', ... + ['numWorkers=%d requested but Parallel Computing ' ... + 'Toolbox is not available. Running serially.'], ... + numWorkers); + end + n = 1; return; end try p = gcp('nocreate'); + if numWorkers < 0 % 'auto' -> reuse open pool, else serial + if isempty(p) + n = 1; + else + n = p.NumWorkers; + end + return; + end if isempty(p) || p.NumWorkers ~= numWorkers if ~isempty(p) delete(p); @@ -437,7 +463,7 @@ n = numWorkers; catch ME warning('NDI:lightsheet:makePyramid:parpoolFailed', ... - ['Could not start a parpool of %d workers (%s). ' ... + ['Could not use a parpool of %d workers (%s). ' ... 'Running serially.'], numWorkers, ME.message); n = 1; end diff --git a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m index ec2472394..f71344457 100644 --- a/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m +++ b/src/ndi/+ndi/+test/+lightsheet/ingestBlobFixture.m @@ -69,7 +69,9 @@ % receiver, or [] to stay silent (which is what CI needs % without the auto-headless detection kicking in). options.progressFcn = 'default' - options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, 0)} = 0 + % See fromOMEZarr: -1 = auto (use any open pool, else serial), + % 0/1 = force serial, N>1 = size a new pool. + options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, -1)} = -1 end sessionDir = options.sessionDir; From 85cbaaee9d5840a7113aa70779e7d9cfc1050485 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 14:14:11 +0000 Subject: [PATCH 045/110] lightsheet: print worker count and per-level plan at ingest start A user has hit 'the parfor did nothing' twice: once because MATLAB had cached the old serial .m in memory, once because no parpool was open at the moment of the call and the -1 auto default correctly stayed serial. Both would have been obvious with one line of output. makePyramid now prints, once, right before the pyramid loop: [makePyramid] Parallel workers: N (from existing parpool). Total chunks: NNN across M level path(s). Codec: blosc-zstd. and, per level: [makePyramid] Level k/M (name, reduction): shape=[...], chunks=[...], nnn chunks to write. describeParallelPool is a side-effect-free peek that resolves the same numWorkers spec resolveParallelPool would but without creating a pool -- so the banner is honest even in the 'serial, no parpool open' case. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 0b444ea9a..50a469827 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -196,6 +196,18 @@ end end + % Announce the run configuration. Users have hit "the parfor did + % nothing" because MATLAB had cached the old .m file, or because + % no parpool was open at call time. A single up-front print says + % which path is really running. + if options.materializeChunks + [effWorkers, workerSource] = describeParallelPool(options.numWorkers); + fprintf(['[makePyramid] Parallel workers: %d (%s). ' ... + 'Total chunks: %d across %d level path(s). Codec: %s.\n'], ... + effWorkers, workerSource, totalChunks, ... + numel(pathOrder), options.codec); + end + % Assign each unique path a 0-based level index in the ORDER it % first appears (finest first, since listPyramids is finest-first). levelDocs = cell(numel(pathOrder), 1); @@ -215,6 +227,14 @@ else reductionFn = rlist{1}; end + if options.materializeChunks + nChunksLvl = prod(levelGrids{idx}); + fprintf(['[makePyramid] Level %d/%d (%s, %s): shape=%s, ' ... + 'chunks=%s, %d chunks to write.\n'], ... + idx, numel(pathOrder), char(entry.name), reductionFn, ... + mat2str(double(level.shape)), ... + mat2str(double(levelChunks{idx})), nChunksLvl); + end reportProgress(options.progressFcn, doneChunks, totalChunks, ... sprintf('Level %d/%d (%s, %s): starting', ... idx, numel(pathOrder), char(entry.name), reductionFn)); @@ -416,6 +436,35 @@ fclose(fid); end +function [n, src] = describeParallelPool(numWorkers) +% DESCRIBEPARALLELPOOL - peek at the effective pool without side effects +% +% Returns [n, src] where n is the worker count that resolveParallelPool +% would end up with, and src is a short human string naming why. +% No pool is created; used only to print the run banner before the +% pyramid loop starts. + if numWorkers == 0 || numWorkers == 1 + n = 1; src = 'serial, forced by numWorkers'; return; + end + if isempty(ver('parallel')) + n = 1; src = 'serial, Parallel Computing Toolbox not installed'; return; + end + p = gcp('nocreate'); + if numWorkers < 0 + if isempty(p) + n = 1; src = 'serial, no parpool open (call parpool(N) to enable)'; + else + n = p.NumWorkers; src = 'from existing parpool'; + end + return; + end + if isempty(p) || p.NumWorkers ~= numWorkers + n = numWorkers; src = 'will open a new parpool'; + else + n = numWorkers; src = 'from existing parpool'; + end +end + function n = resolveParallelPool(numWorkers) % RESOLVEPARALLELPOOL - resolve numWorkers into an effective batch size % From 9b0ca221dc7cb7927cd2b1a2d92881fd4a53520b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 14:15:53 +0000 Subject: [PATCH 046/110] lightsheet: one union-bbox read per batch instead of one per chunk The pattern that was hurting on slow drives: N small readArray calls per batch, each with its own NDR-reader Python subprocess round trip and each seeking to a different region of the source zarr. On a spinning USB drive the seek cost per read dominates the transfer cost, so a batch of 18 chunks meant 18 seeks and 18 subprocess round trips. attachChunkFiles now computes the axis-aligned bounding box of every target chunk in the batch and issues ONE readArray call for the whole thing. The batch buffer is then sliced locally into individual target chunks (still per-chunk pad/permute/typecast; still per-chunk Blosc encode in the parallel pool). Reads per batch drop from N to 1, subprocess round trips likewise. When a batch straddles a "row" boundary in C-order (chunks aren't strictly contiguous in the source) the union bbox over-reads by at most one row of source data along a fast axis where the drive is already streaming sequentially. The saving still dwarfs the over-read on any drive slow enough for I/O to be the bottleneck. Total memory per batch is bounded by the same batch-of-chunks budget (~poolSize x tile_bytes uncompressed), so the peak-memory guarantee the streaming re-tiler was written for is unchanged. The banner and per-level prints now say what's happening so a run diagnosis is a single line of stdout. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 49 ++++++++++++++----- 1 file changed, 37 insertions(+), 12 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 50a469827..7dd372d94 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -203,9 +203,10 @@ if options.materializeChunks [effWorkers, workerSource] = describeParallelPool(options.numWorkers); fprintf(['[makePyramid] Parallel workers: %d (%s). ' ... - 'Total chunks: %d across %d level path(s). Codec: %s.\n'], ... + 'Total chunks: %d across %d level path(s). Codec: %s. ' ... + 'One union-bbox read per batch of %d chunks.\n'], ... effWorkers, workerSource, totalChunks, ... - numel(pathOrder), options.codec); + numel(pathOrder), options.codec, effWorkers); end % Assign each unique path a 0-based level index in the ORDER it @@ -356,19 +357,43 @@ batchIdx = batchStart:batchEnd; b = numel(batchIdx); - % ---- SERIAL READ into a batch buffer ----------------------- - raws = cell(1, b); + % ---- SERIAL READ, ONE UNION-BOX PER BATCH ------------------- + % Instead of `b` small readArray calls -- each with its own + % Python subprocess round-trip and each seeking to a + % different region of the source zarr -- we compute the + % axis-aligned bounding box of every target chunk in the + % batch and issue ONE readArray for the whole thing. On any + % drive slow enough for I/O to dominate this cuts the read + % count by up to `b`x, at the cost of some over-read when + % the batch straddles a "row" boundary in C-order (the + % bounding box then covers a strip a bit wider than + % strictly needed). The over-read is bounded to <= one row + % of source data, and even that is on a fast axis where the + % drive is already streaming sequentially. + loRows = zeros(b, numel(shape)); + hiRows = zeros(b, numel(shape)); for j = 1:b - r = batchIdx(j); - cIdx = linearOrder(r, :); - loRow = zeros(1, numel(shape)); - hiRow = zeros(1, numel(shape)); + cIdx = linearOrder(batchIdx(j), :); for a = 1:numel(chunkGrid) - loRow(a) = (cIdx(a) - 1) * chunks(a) + 1; - hiRow(a) = min(loRow(a) + chunks(a) - 1, shape(a)); + loRows(j, a) = (cIdx(a) - 1) * chunks(a) + 1; + hiRows(j, a) = min(loRows(j, a) + chunks(a) - 1, shape(a)); + end + end + loBBox = min(loRows, [], 1); + hiBBox = max(hiRows, [], 1); + bigTile = ndr.format.omezarr.readArray(sourceZarrPath, ... + pyramidName, kInEntry, 'Region', [loBBox; hiBBox]); + + raws = cell(1, b); + for j = 1:b + % Slice this target chunk's region out of the batch + % buffer using bbox-relative subscripts. + subs = cell(1, numel(shape)); + for a = 1:numel(shape) + subs{a} = (loRows(j, a) - loBBox(a) + 1) : ... + (hiRows(j, a) - loBBox(a) + 1); end - srcTile = ndr.format.omezarr.readArray(sourceZarrPath, ... - pyramidName, kInEntry, 'Region', [loRow; hiRow]); + srcTile = bigTile(subs{:}); padded = zeros(chunks, class(srcTile)); localSubs = arrayfun(@(k) 1:size(srcTile, k), ... 1:numel(chunks), 'UniformOutput', false); From 17f48a539af8c71768ddd1608c1dc58e8f6e256c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 14:17:53 +0000 Subject: [PATCH 047/110] lightsheet: prefetch super-blocks decoupled from parfor pool size Batching reads to the pool size (default 18 chunks x 8 MB ~= 144 MB per read) is not enough on a slow spinning drive: the seek + NDR Python-subprocess cost per readArray call still dominates. Decouple read size from worker count. New option `prefetchBytes` (default 512 MB) sizes an outer super-block loop: for each super-block of prefetchChunkCount consecutive target chunks: ONE readArray of the union bounding box (~prefetchBytes) for each sub-batch of poolSize chunks: slice raws from the super-block buffer (RAM-only, no I/O) parfor: encode + write tempfile serial: add_file in chunk order At the default 512 MB the drive sees one big sequential read every ~64 target chunks (at 8 MB each). Passing 1 GB or 2 GB is fine on a workstation with RAM to spare and amortises the seek + subprocess overhead across even more voxels. `clear bigTile` between super-blocks bounds peak RAM to prefetchBytes rather than climbing across the loop. The pool size still governs how many encoders run in parallel; prefetchBytes governs how much data feeds them per read. So on a slow drive with 18 workers you can have one 1 GB read followed by seven sub-batches of parfor, each doing 18 encodes in parallel, with no drive access in between. fromOMEZarr passes prefetchBytes through. Existing tests are unchanged (defaults preserve behaviour for tiny fixtures). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 7 +- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 161 ++++++++++-------- 2 files changed, 99 insertions(+), 69 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index be7954fd7..4474e691d 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -106,6 +106,10 @@ % Reads stay serial in all modes; compression fans out. % See makePyramid for details. options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, -1)} = -1 + % How much source data to prefetch into RAM per readArray + % call. Bigger = fewer seeks + subprocess round trips at the + % cost of more RAM. See makePyramid. Default 512 MB. + options.prefetchBytes (1,1) double {mustBePositive} = 512 * 2^20 end if isempty(strtrim(options.subjectID)) @@ -181,7 +185,8 @@ 'codec', options.codec, ... 'clevel', options.clevel, ... 'progressFcn', progressFcn, ... - 'numWorkers', options.numWorkers); + 'numWorkers', options.numWorkers, ... + 'prefetchBytes', options.prefetchBytes); info = struct( ... 'zarrPath', char(zarrPath), ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 7dd372d94..876c1a4ca 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -92,6 +92,16 @@ % effective-pool-size chunks is read into memory, then % compression fans out. Ingestion order into DID is preserved. options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, -1)} = -1 + % How much source data to pull into RAM per readArray call, in + % bytes uncompressed. One "super-block" of chunks is read at a + % time, then compressed by the worker pool in sub-batches. A + % bigger prefetch amortises the fixed cost of a seek + NDR + % subprocess round-trip over more voxels, which matters on any + % drive slow enough for I/O to dominate. Default 512 MB is a + % sensible compromise for a modern workstation; passing 1e9 or + % 2e9 is fine on a machine with plenty of RAM. Peak MATLAB + % memory is prefetchBytes (uncompressed). + options.prefetchBytes (1,1) double {mustBePositive} = 512 * 2^20 end if options.materializeChunks && isempty(options.sourceZarrPath) @@ -203,10 +213,11 @@ if options.materializeChunks [effWorkers, workerSource] = describeParallelPool(options.numWorkers); fprintf(['[makePyramid] Parallel workers: %d (%s). ' ... - 'Total chunks: %d across %d level path(s). Codec: %s. ' ... - 'One union-bbox read per batch of %d chunks.\n'], ... - effWorkers, workerSource, totalChunks, ... - numel(pathOrder), options.codec, effWorkers); + 'Prefetch budget: %.0f MB per read. ' ... + 'Total chunks: %d across %d level path(s). Codec: %s.\n'], ... + effWorkers, workerSource, ... + options.prefetchBytes / 2^20, ... + totalChunks, numel(pathOrder), options.codec); end % Assign each unique path a 0-based level index in the ORDER it @@ -292,14 +303,14 @@ options.sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, ... options.codec, options.clevel, ... options.progressFcn, doneChunks, totalChunks, levelLabel, ... - options.numWorkers); + options.numWorkers, options.prefetchBytes); end tmpCleaner = onCleanup(@() cleanupTmp(tmpRoot)); session.database_add(levelDoc); end -function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers) +function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers, prefetchBytes) % Read the source zarr level as a whole array, then re-tile into the % level document's chunk shape and attach each chunk as chunk.bin_ % (1-based, C-order over chunkGrid). Files are ingested into DID, which @@ -352,28 +363,33 @@ % is the original serial path with no toolbox dependency. poolSize = resolveParallelPool(numWorkers); - for batchStart = 1:poolSize:nChunks - batchEnd = min(batchStart + poolSize - 1, nChunks); - batchIdx = batchStart:batchEnd; - b = numel(batchIdx); - - % ---- SERIAL READ, ONE UNION-BOX PER BATCH ------------------- - % Instead of `b` small readArray calls -- each with its own - % Python subprocess round-trip and each seeking to a - % different region of the source zarr -- we compute the + % How many target chunks fit inside the prefetch budget, sized + % by uncompressed chunk bytes. Never smaller than the parfor + % pool (a super-block below poolSize would leave workers idle); + % never larger than the level itself. + chunkBytes = prod(chunks) * typesize; + prefetchChunkCount = max(poolSize, ... + floor(double(prefetchBytes) / max(chunkBytes, 1))); + prefetchChunkCount = min(prefetchChunkCount, nChunks); + + for superStart = 1:prefetchChunkCount:nChunks + superEnd = min(superStart + prefetchChunkCount - 1, nChunks); + superIdx = superStart:superEnd; + bs = numel(superIdx); + + % ---- SERIAL READ, ONE UNION-BOX PER SUPER-BLOCK ------------- + % Instead of many small readArray calls -- each with its own + % NDR Python subprocess round-trip and each seeking to its + % own region of the source zarr -- we compute the union % axis-aligned bounding box of every target chunk in the - % batch and issue ONE readArray for the whole thing. On any - % drive slow enough for I/O to dominate this cuts the read - % count by up to `b`x, at the cost of some over-read when - % the batch straddles a "row" boundary in C-order (the - % bounding box then covers a strip a bit wider than - % strictly needed). The over-read is bounded to <= one row - % of source data, and even that is on a fast axis where the - % drive is already streaming sequentially. - loRows = zeros(b, numel(shape)); - hiRows = zeros(b, numel(shape)); - for j = 1:b - cIdx = linearOrder(batchIdx(j), :); + % super-block and issue ONE readArray for the whole thing. + % The parfor pool then chews through the buffer in sub- + % batches without touching the disk again. Peak memory per + % super-block is prefetchBytes uncompressed. + loRows = zeros(bs, numel(shape)); + hiRows = zeros(bs, numel(shape)); + for j = 1:bs + cIdx = linearOrder(superIdx(j), :); for a = 1:numel(chunkGrid) loRows(j, a) = (cIdx(a) - 1) * chunks(a) + 1; hiRows(j, a) = min(loRows(j, a) + chunks(a) - 1, shape(a)); @@ -384,54 +400,63 @@ bigTile = ndr.format.omezarr.readArray(sourceZarrPath, ... pyramidName, kInEntry, 'Region', [loBBox; hiBBox]); - raws = cell(1, b); - for j = 1:b - % Slice this target chunk's region out of the batch - % buffer using bbox-relative subscripts. - subs = cell(1, numel(shape)); - for a = 1:numel(shape) - subs{a} = (loRows(j, a) - loBBox(a) + 1) : ... - (hiRows(j, a) - loBBox(a) + 1); + % ---- INNER SUB-BATCH LOOP: poolSize chunks each ----------- + for subStart = 1:poolSize:bs + subEnd = min(subStart + poolSize - 1, bs); + b = subEnd - subStart + 1; + + raws = cell(1, b); + chunkIdxs = zeros(1, b); + for j = 1:b + localJ = subStart + j - 1; + chunkIdxs(j) = superIdx(localJ); + % Slice this target chunk's region out of the + % super-block buffer using bbox-relative subscripts. + subs = cell(1, numel(shape)); + for a = 1:numel(shape) + subs{a} = (loRows(localJ, a) - loBBox(a) + 1) : ... + (hiRows(localJ, a) - loBBox(a) + 1); + end + srcTile = bigTile(subs{:}); + padded = zeros(chunks, class(srcTile)); + localSubs = arrayfun(@(k) 1:size(srcTile, k), ... + 1:numel(chunks), 'UniformOutput', false); + padded(localSubs{:}) = srcTile; + % Zarr expects C-order bytes; MATLAB is column- + % major, so permute the axes into reversed order + % before serialising, mirroring the fixture writer. + permuted = permute(padded, numel(chunks):-1:1); + raws{j} = typecast(permuted(:), 'uint8'); end - srcTile = bigTile(subs{:}); - padded = zeros(chunks, class(srcTile)); - localSubs = arrayfun(@(k) 1:size(srcTile, k), ... - 1:numel(chunks), 'UniformOutput', false); - padded(localSubs{:}) = srcTile; - % Zarr expects C-order bytes; MATLAB is column-major, - % so permute the axes into reversed order before - % serialising, mirroring the fixture writer. - permuted = permute(padded, numel(chunks):-1:1); - raws{j} = typecast(permuted(:), 'uint8'); - end - % ---- PARALLEL (or serial) ENCODE + WRITE TEMPFILE ---------- - tempPaths = cell(1, b); - chunkIdxs = zeros(1, b); - for j = 1:b - chunkIdxs(j) = batchIdx(j); - end - if poolSize > 1 && b > 1 - parfor j = 1:b - tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... - typesize, tmpRoot, chunkIdxs(j)); %#ok + % ---- PARALLEL (or serial) ENCODE + WRITE TEMPFILE ------ + tempPaths = cell(1, b); + if poolSize > 1 && b > 1 + parfor j = 1:b + tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... + typesize, tmpRoot, chunkIdxs(j)); %#ok + end + else + for j = 1:b + tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... + typesize, tmpRoot, chunkIdxs(j)); + end end - else + + % ---- SERIAL ADD_FILE (in chunk order) ----------------- for j = 1:b - tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... - typesize, tmpRoot, chunkIdxs(j)); + levelDoc = levelDoc.add_file( ... + sprintf('chunk.bin_%d', chunkIdxs(j)), tempPaths{j}); + doneChunks = doneChunks + 1; + reportProgress(progressFcn, doneChunks, totalChunks, ... + sprintf('%s: chunk %d/%d', levelLabel, ... + chunkIdxs(j), nChunks)); end end - % ---- SERIAL ADD_FILE (in chunk order) ---------------------- - for j = 1:b - levelDoc = levelDoc.add_file( ... - sprintf('chunk.bin_%d', chunkIdxs(j)), tempPaths{j}); - doneChunks = doneChunks + 1; - reportProgress(progressFcn, doneChunks, totalChunks, ... - sprintf('%s: chunk %d/%d', levelLabel, ... - chunkIdxs(j), nChunks)); - end + % Release the super-block buffer before the next read so + % peak RAM stays at ~prefetchBytes rather than climbing. + clear bigTile; end end From 209a1de761e821f580b55ec7089e5ac6bb69c724 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 14:21:43 +0000 Subject: [PATCH 048/110] lightsheet tests: chunk-equality helper + prefetch-invariance test Extracts the byte-for-byte chunk comparison used by testParallelWorkersProduceIdenticalChunks into a verifyChunksEqual helper method, and adds testPrefetchBytesDoesNotChangeChunks that forces the outer super-block loop through many iterations (prefetchBytes = 8 KiB) and asserts the chunks match a default- prefetch (single-super-block) run. The point: neither knob -- numWorkers nor prefetchBytes -- is allowed to change what gets ingested. They both trade time against resources, they must not trade against correctness. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+test/+lightsheet/TestBlobFixture.m | 107 +++++++++++++----- 1 file changed, 77 insertions(+), 30 deletions(-) diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index 1aadca626..74e614daa 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -326,51 +326,98 @@ function testParallelWorkersProduceIdenticalChunks(testCase) 'codec', 'blosc-zstd', 'clevel', 3, ... 'numWorkers', 2); - % Match up level docs by (level index, reduction) then - % compare each chunk.bin_# byte-for-byte. - levelsS = Ss.database_search( ... + testCase.verifyChunksEqual(Ss, pS, Sp, pP, ... + 'Serial and parallel must produce identical chunks.'); + end + + function verifyChunksEqual(testCase, S1, p1, S2, p2, msg) + % Shared helper: match up level docs by (level, reduction) + % and byte-compare every materialized chunk between two + % ingested sessions. Used by tests that vary a performance + % knob (numWorkers, prefetchBytes, ...) and need to prove + % it did not change the produced pyramid. + levels1 = S1.database_search( ... ndi.query('','depends_on', ... - 'lightsheetZarrPyramid_id', pS.id()) & ... + 'lightsheetZarrPyramid_id', p1.id()) & ... ndi.query('','isa','lightsheetZarrLevel')); - levelsP = Sp.database_search( ... + levels2 = S2.database_search( ... ndi.query('','depends_on', ... - 'lightsheetZarrPyramid_id', pP.id()) & ... + 'lightsheetZarrPyramid_id', p2.id()) & ... ndi.query('','isa','lightsheetZarrLevel')); - testCase.verifyEqual(numel(levelsS), numel(levelsP), ... - 'Level count must match between serial and parallel'); - + testCase.verifyEqual(numel(levels1), numel(levels2), ... + sprintf('%s (level count differs)', msg)); keyOf = @(L) sprintf('%d-%s', ... L.document_properties.lightsheetZarrLevel.level, ... L.document_properties.lightsheetZarrLevel.reduction_function); - keysS = cellfun(keyOf, levelsS, 'UniformOutput', false); - keysP = cellfun(keyOf, levelsP, 'UniformOutput', false); - - for i = 1:numel(levelsS) - j = find(strcmp(keysP, keysS{i}), 1); + keys1 = cellfun(keyOf, levels1, 'UniformOutput', false); + keys2 = cellfun(keyOf, levels2, 'UniformOutput', false); + for i = 1:numel(levels1) + j = find(strcmp(keys2, keys1{i}), 1); testCase.verifyNotEmpty(j, ... - sprintf('No matching parallel level for %s', keysS{i})); - lS = levelsS{i}; - lP = levelsP{j}; - nc = lS.document_properties.lightsheetZarrLevel.n_chunks_stored; + sprintf('No matching level for %s', keys1{i})); + lA = levels1{i}; + lB = levels2{j}; + nc = lA.document_properties.lightsheetZarrLevel.n_chunks_stored; testCase.verifyEqual( ... - lP.document_properties.lightsheetZarrLevel.n_chunks_stored, ... - nc, 'n_chunks_stored differs'); + lB.document_properties.lightsheetZarrLevel.n_chunks_stored, ... + nc, sprintf('%s (n_chunks_stored differs for %s)', ... + msg, keys1{i})); for k = 1:nc nm = sprintf('chunk.bin_%d', k); - fs = Ss.database_openbinarydoc(lS, nm); - bs = fread(fs.fid, Inf, 'uint8=>uint8'); - Ss.database_closebinarydoc(fs); - fp = Sp.database_openbinarydoc(lP, nm); - bp = fread(fp.fid, Inf, 'uint8=>uint8'); - Sp.database_closebinarydoc(fp); - testCase.verifyEqual(bs, bp, ... - sprintf('chunk %s of level %s differs ' ... - 'between serial and parallel', ... - nm, keysS{i})); + fA = S1.database_openbinarydoc(lA, nm); + bA = fread(fA.fid, Inf, 'uint8=>uint8'); + S1.database_closebinarydoc(fA); + fB = S2.database_openbinarydoc(lB, nm); + bB = fread(fB.fid, Inf, 'uint8=>uint8'); + S2.database_closebinarydoc(fB); + testCase.verifyEqual(bA, bB, ... + sprintf('%s (chunk %s of level %s differs)', ... + msg, nm, keys1{i})); end end end + function testPrefetchBytesDoesNotChangeChunks(testCase) + % Reading one big super-block per level (default) and + % reading many tiny super-blocks MUST produce identical + % chunks. Prefetch is a performance knob, not a semantic + % one. Forcing prefetchBytes tiny drives the outer super- + % block loop through many iterations on a 32^3 fixture: + % 32^3 uint16 = 64 KiB total, so prefetch=8 KiB means + % ~8 super-blocks per level. + if isempty(which('ndr.format.omezarr.listPyramids')) + testCase.assumeFail(['NDR reader not on path; ' ... + 'skipping end-to-end ingest.']); + end + sDirBig = fullfile(tempname, 'bigprefetch'); + testCase.addTeardown(@() rmdir(fileparts(sDirBig), 's')); + zDirBig = fullfile(tempname, 'bigprefetchzarr'); + testCase.addTeardown(@() rmdir(zDirBig, 's')); + [Sb, pB, ~] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDirBig, 'zarrDir', zDirBig, ... + 'Shape', [32 32 32], 'NumChannels', 1, ... + 'NumLevels', 2, 'ChunkShape', [16 16 16], ... + 'materializeChunks', true, 'progressFcn', [], ... + 'codec', 'blosc-zstd', 'clevel', 3, ... + 'numWorkers', 0); + + sDirSm = fullfile(tempname, 'smallprefetch'); + testCase.addTeardown(@() rmdir(fileparts(sDirSm), 's')); + zDirSm = fullfile(tempname, 'smallprefetchzarr'); + testCase.addTeardown(@() rmdir(zDirSm, 's')); + [Ss, pS, ~] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDirSm, 'zarrDir', zDirSm, ... + 'Shape', [32 32 32], 'NumChannels', 1, ... + 'NumLevels', 2, 'ChunkShape', [16 16 16], ... + 'materializeChunks', true, 'progressFcn', [], ... + 'codec', 'blosc-zstd', 'clevel', 3, ... + 'numWorkers', 0, ... + 'prefetchBytes', 8 * 1024); + + testCase.verifyChunksEqual(Sb, pB, Ss, pS, ... + 'Prefetch size must not change the produced chunks.'); + end + function testMaxKeepsBrightSpikeMeanDoesNot(testCase) % At the spike's downsampled position, max preserves the % 60k spike and mean dilutes it (averages the one spike From 7027cd8f5ef785f97ef618e1970affe54271a88b Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 14:27:45 +0000 Subject: [PATCH 049/110] lightsheet: async prefetch overlaps disk reads with parfor compression The remaining bottleneck after batching and prefetch: workers spend ~2/3 of the time idle while the main thread reads the next super- block from the source zarr. This adds an async pipeline using parfeval: main: dispatch read(1); wait read(1); dispatch read(2); compress(1); wait read(2); dispatch read(3); compress(2); ... pool: [reading 1] [reading 2] [reading 3] [compress 1] [compress 2] Total time approaches max(read_time, compress_time) instead of their sum. On an ingest where reads dominate ~2:1 over compute the expected win is another ~30-40% off wall time. New option `asyncPrefetch` (default true), threaded through fromOMEZarr and ingestBlobFixture. Falls back to synchronous reads when numWorkers resolves to serial or when only one super-block exists. Super-blocks are now planned up front (idxs, per-chunk lo/hi, and union bbox) so parfeval can dispatch the next read the moment the current one lands. readSuperBlockRegion is a plain subfunction so a parfeval worker can serialise it cleanly. Peak RAM ~ 2 * prefetchBytes (two super-blocks in flight). The banner reports whether async is engaged. Test coverage: testAsyncPrefetchDoesNotChangeChunks byte-compares async and sync ingest at prefetchBytes = 8 KiB (forces multiple super-blocks so async has real work). Skipped when PCT is missing. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 7 +- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 115 +++++++++++++----- .../+test/+lightsheet/TestBlobFixture.m | 44 +++++++ 3 files changed, 133 insertions(+), 33 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 4474e691d..70bdb666b 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -110,6 +110,10 @@ % call. Bigger = fewer seeks + subprocess round trips at the % cost of more RAM. See makePyramid. Default 512 MB. options.prefetchBytes (1,1) double {mustBePositive} = 512 * 2^20 + % Overlap disk reads with parfor compression via parfeval. + % See makePyramid for details. Default true; ignored when + % numWorkers resolves to serial. + options.asyncPrefetch (1,1) logical = true end if isempty(strtrim(options.subjectID)) @@ -186,7 +190,8 @@ 'clevel', options.clevel, ... 'progressFcn', progressFcn, ... 'numWorkers', options.numWorkers, ... - 'prefetchBytes', options.prefetchBytes); + 'prefetchBytes', options.prefetchBytes, ... + 'asyncPrefetch', options.asyncPrefetch); info = struct( ... 'zarrPath', char(zarrPath), ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 876c1a4ca..8447fb81a 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -102,6 +102,14 @@ % 2e9 is fine on a machine with plenty of RAM. Peak MATLAB % memory is prefetchBytes (uncompressed). options.prefetchBytes (1,1) double {mustBePositive} = 512 * 2^20 + % Overlap disk reads with compression via parfeval. The next + % super-block starts reading on a pool worker while the parfor + % is still compressing the current one, so total time + % approaches max(read_time, compress_time) instead of their + % sum. Requires a parpool (falls back to synchronous reads + % otherwise). Peak RAM is ~2 * prefetchBytes (two super- + % blocks in flight). + options.asyncPrefetch (1,1) logical = true end if options.materializeChunks && isempty(options.sourceZarrPath) @@ -212,11 +220,15 @@ % which path is really running. if options.materializeChunks [effWorkers, workerSource] = describeParallelPool(options.numWorkers); + asyncStr = 'off'; + if options.asyncPrefetch && effWorkers > 1 + asyncStr = 'on (read overlaps compress)'; + end fprintf(['[makePyramid] Parallel workers: %d (%s). ' ... - 'Prefetch budget: %.0f MB per read. ' ... + 'Prefetch budget: %.0f MB per read. Async prefetch: %s. ' ... 'Total chunks: %d across %d level path(s). Codec: %s.\n'], ... effWorkers, workerSource, ... - options.prefetchBytes / 2^20, ... + options.prefetchBytes / 2^20, asyncStr, ... totalChunks, numel(pathOrder), options.codec); end @@ -303,14 +315,14 @@ options.sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, ... options.codec, options.clevel, ... options.progressFcn, doneChunks, totalChunks, levelLabel, ... - options.numWorkers, options.prefetchBytes); + options.numWorkers, options.prefetchBytes, options.asyncPrefetch); end tmpCleaner = onCleanup(@() cleanupTmp(tmpRoot)); session.database_add(levelDoc); end -function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers, prefetchBytes) +function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers, prefetchBytes, asyncPrefetch) % Read the source zarr level as a whole array, then re-tile into the % level document's chunk shape and attach each chunk as chunk.bin_ % (1-based, C-order over chunkGrid). Files are ingested into DID, which @@ -372,33 +384,60 @@ floor(double(prefetchBytes) / max(chunkBytes, 1))); prefetchChunkCount = min(prefetchChunkCount, nChunks); - for superStart = 1:prefetchChunkCount:nChunks - superEnd = min(superStart + prefetchChunkCount - 1, nChunks); - superIdx = superStart:superEnd; - bs = numel(superIdx); - - % ---- SERIAL READ, ONE UNION-BOX PER SUPER-BLOCK ------------- - % Instead of many small readArray calls -- each with its own - % NDR Python subprocess round-trip and each seeking to its - % own region of the source zarr -- we compute the union - % axis-aligned bounding box of every target chunk in the - % super-block and issue ONE readArray for the whole thing. - % The parfor pool then chews through the buffer in sub- - % batches without touching the disk again. Peak memory per - % super-block is prefetchBytes uncompressed. - loRows = zeros(bs, numel(shape)); - hiRows = zeros(bs, numel(shape)); + % Plan every super-block up front (chunk indices + per-chunk lo/hi + % + union bbox) so the async pipeline can dispatch the next read + % while the current one is still compressing. + nSuperBlocks = ceil(nChunks / prefetchChunkCount); + superBlocks = cell(nSuperBlocks, 1); + for s = 1:nSuperBlocks + s0 = (s - 1) * prefetchChunkCount + 1; + s1 = min(s0 + prefetchChunkCount - 1, nChunks); + idxs = s0:s1; + bs = numel(idxs); + lo = zeros(bs, numel(shape)); + hi = zeros(bs, numel(shape)); for j = 1:bs - cIdx = linearOrder(superIdx(j), :); + c = linearOrder(idxs(j), :); for a = 1:numel(chunkGrid) - loRows(j, a) = (cIdx(a) - 1) * chunks(a) + 1; - hiRows(j, a) = min(loRows(j, a) + chunks(a) - 1, shape(a)); + lo(j, a) = (c(a) - 1) * chunks(a) + 1; + hi(j, a) = min(lo(j, a) + chunks(a) - 1, shape(a)); + end + end + superBlocks{s} = struct('idxs', idxs, ... + 'loRows', lo, 'hiRows', hi, ... + 'loBBox', min(lo, [], 1), 'hiBBox', max(hi, [], 1)); + end + + % ASYNC PIPELINE. If a parpool is available and asyncPrefetch is + % on, the next super-block's read runs concurrently on a pool + % worker while the parfor is compressing the current one. Total + % time approaches max(read_time, compress_time) instead of their + % sum. Peak RAM ~ 2 * prefetchBytes (two blocks in flight). + useAsync = asyncPrefetch && poolSize > 1 && nSuperBlocks > 1; + if useAsync + pool = gcp(); + nextFut = parfeval(pool, @readSuperBlockRegion, 1, ... + sourceZarrPath, pyramidName, kInEntry, ... + superBlocks{1}.loBBox, superBlocks{1}.hiBBox); + end + + for s = 1:nSuperBlocks + sb = superBlocks{s}; + bs = numel(sb.idxs); + + % ---- READ current super-block, DISPATCH the next ----------- + if useAsync + bigTile = fetchOutputs(nextFut); + if s < nSuperBlocks + nb = superBlocks{s + 1}; + nextFut = parfeval(pool, @readSuperBlockRegion, 1, ... + sourceZarrPath, pyramidName, kInEntry, ... + nb.loBBox, nb.hiBBox); end + else + bigTile = readSuperBlockRegion(sourceZarrPath, ... + pyramidName, kInEntry, sb.loBBox, sb.hiBBox); end - loBBox = min(loRows, [], 1); - hiBBox = max(hiRows, [], 1); - bigTile = ndr.format.omezarr.readArray(sourceZarrPath, ... - pyramidName, kInEntry, 'Region', [loBBox; hiBBox]); % ---- INNER SUB-BATCH LOOP: poolSize chunks each ----------- for subStart = 1:poolSize:bs @@ -409,13 +448,13 @@ chunkIdxs = zeros(1, b); for j = 1:b localJ = subStart + j - 1; - chunkIdxs(j) = superIdx(localJ); + chunkIdxs(j) = sb.idxs(localJ); % Slice this target chunk's region out of the % super-block buffer using bbox-relative subscripts. subs = cell(1, numel(shape)); for a = 1:numel(shape) - subs{a} = (loRows(localJ, a) - loBBox(a) + 1) : ... - (hiRows(localJ, a) - loBBox(a) + 1); + subs{a} = (sb.loRows(localJ, a) - sb.loBBox(a) + 1) : ... + (sb.hiRows(localJ, a) - sb.loBBox(a) + 1); end srcTile = bigTile(subs{:}); padded = zeros(chunks, class(srcTile)); @@ -454,12 +493,24 @@ end end - % Release the super-block buffer before the next read so - % peak RAM stays at ~prefetchBytes rather than climbing. + % Release the super-block buffer before the next iteration. + % In async mode the next super-block's buffer may already + % exist as a Future's result waiting to be fetched. clear bigTile; end end +function bigTile = readSuperBlockRegion(sourceZarrPath, pyramidName, kInEntry, loBBox, hiBBox) +% READSUPERBLOCKREGION - fetch one super-block region from the source +% +% Wraps ndr.format.omezarr.readArray so parfeval can dispatch it as +% a background read on a pool worker. Kept as a plain subfunction +% (not nested) so the worker can serialise it without pulling the +% caller's workspace. + bigTile = ndr.format.omezarr.readArray(sourceZarrPath, ... + pyramidName, kInEntry, 'Region', [loBBox; hiBBox]); +end + function tmp = encodeAndWrite(raw, codec, clevel, typesize, tmpRoot, chunkIdx) % ENCODEANDWRITE - compress one chunk and drop it in tmpRoot % diff --git a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m index 74e614daa..4f146bc2a 100644 --- a/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m +++ b/tests/+ndi/+unittest/+test/+lightsheet/TestBlobFixture.m @@ -418,6 +418,50 @@ function testPrefetchBytesDoesNotChangeChunks(testCase) 'Prefetch size must not change the produced chunks.'); end + function testAsyncPrefetchDoesNotChangeChunks(testCase) + % Overlapping reads with compression must not change the + % produced pyramid: asyncPrefetch is a performance knob. + % Skipped if PCT is missing (no way to exercise parfeval). + if isempty(which('ndr.format.omezarr.listPyramids')) + testCase.assumeFail(['NDR reader not on path; ' ... + 'skipping end-to-end ingest.']); + end + if isempty(ver('parallel')) + testCase.assumeFail(['Parallel Computing Toolbox ' ... + 'not available on this runner; skipping.']); + end + + % Force multiple super-blocks so async has real work. + sDirSync = fullfile(tempname, 'syncsession'); + testCase.addTeardown(@() rmdir(fileparts(sDirSync), 's')); + zDirSync = fullfile(tempname, 'synczarr'); + testCase.addTeardown(@() rmdir(zDirSync, 's')); + [Ssync, pSync, ~] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDirSync, 'zarrDir', zDirSync, ... + 'Shape', [32 32 32], 'NumChannels', 1, ... + 'NumLevels', 2, 'ChunkShape', [16 16 16], ... + 'materializeChunks', true, 'progressFcn', [], ... + 'codec', 'blosc-zstd', 'clevel', 3, ... + 'numWorkers', 2, 'prefetchBytes', 8 * 1024, ... + 'asyncPrefetch', false); + + sDirAsync = fullfile(tempname, 'asyncsession'); + testCase.addTeardown(@() rmdir(fileparts(sDirAsync), 's')); + zDirAsync = fullfile(tempname, 'asynczarr'); + testCase.addTeardown(@() rmdir(zDirAsync, 's')); + [Sasync, pAsync, ~] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sDirAsync, 'zarrDir', zDirAsync, ... + 'Shape', [32 32 32], 'NumChannels', 1, ... + 'NumLevels', 2, 'ChunkShape', [16 16 16], ... + 'materializeChunks', true, 'progressFcn', [], ... + 'codec', 'blosc-zstd', 'clevel', 3, ... + 'numWorkers', 2, 'prefetchBytes', 8 * 1024, ... + 'asyncPrefetch', true); + + testCase.verifyChunksEqual(Ssync, pSync, Sasync, pAsync, ... + 'asyncPrefetch must not change the produced chunks.'); + end + function testMaxKeepsBrightSpikeMeanDoesNot(testCase) % At the spike's downsampled position, max preserves the % 60k spike and mean dilutes it (averages the one spike From f7eaf6753bafb12eb1ea3b5ce8e62e9a87a265c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 16:50:46 +0000 Subject: [PATCH 050/110] lightsheet: one parfor per super-block, one progress tick per super-block Between-burst gaps were ~half the wall time on SSD: MATLAB's parfor coordination overhead was paid once per sub-batch of poolSize chunks (4-8 times per super-block), and each of the resulting bs chunks fired its own updateBar call, so a fast super-block spent as much time in bookkeeping as in Blosc. Two changes: 1. Slice the ENTIRE super-block into raws{} up front, then run ONE parfor across all bs chunks. Parfor's own scheduler still distributes work across the pool -- 128 chunks on 18 workers becomes ~7 iterations per worker -- but the FIXED per-parfor-call overhead (input marshaling, worker dispatch, result collection) is paid ONCE per super-block instead of ceil(bs/poolSize) times. raws is cleared before the pool starts to keep peak RAM at ~prefetchBytes. 2. Fire the progress callback ONCE per super-block instead of once per chunk. NDI's ProgressBarWindow recomputes ETA and queues a redraw on every updateBar call; 18 near-simultaneous calls per sub-batch was showing up as a real fraction of the between-burst gap. One tick per super-block is still fine granularity on a multi-hour ingest, and the final 'Done.' tick at the end of the pyramid still fires. Byte-equivalence tests (serial vs parallel, small-vs-large prefetch, async vs sync) still enforce that neither change touches what ends up on disk. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 115 ++++++++++-------- 1 file changed, 62 insertions(+), 53 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 8447fb81a..ef18a9804 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -439,64 +439,73 @@ pyramidName, kInEntry, sb.loBBox, sb.hiBBox); end - % ---- INNER SUB-BATCH LOOP: poolSize chunks each ----------- - for subStart = 1:poolSize:bs - subEnd = min(subStart + poolSize - 1, bs); - b = subEnd - subStart + 1; - - raws = cell(1, b); - chunkIdxs = zeros(1, b); - for j = 1:b - localJ = subStart + j - 1; - chunkIdxs(j) = sb.idxs(localJ); - % Slice this target chunk's region out of the - % super-block buffer using bbox-relative subscripts. - subs = cell(1, numel(shape)); - for a = 1:numel(shape) - subs{a} = (sb.loRows(localJ, a) - sb.loBBox(a) + 1) : ... - (sb.hiRows(localJ, a) - sb.loBBox(a) + 1); - end - srcTile = bigTile(subs{:}); - padded = zeros(chunks, class(srcTile)); - localSubs = arrayfun(@(k) 1:size(srcTile, k), ... - 1:numel(chunks), 'UniformOutput', false); - padded(localSubs{:}) = srcTile; - % Zarr expects C-order bytes; MATLAB is column- - % major, so permute the axes into reversed order - % before serialising, mirroring the fixture writer. - permuted = permute(padded, numel(chunks):-1:1); - raws{j} = typecast(permuted(:), 'uint8'); + % ---- SLICE the entire super-block into per-chunk raw bytes - + % Done once, up front, for the whole super-block instead of + % per sub-batch. This lets us fire ONE parfor across every + % chunk in the super-block and pay MATLAB's parfor + % coordination overhead once instead of ceil(bs/poolSize) + % times. Peak memory during this phase is bigTile + raws + % (~equal size; both released before the compression pool + % begins). + raws = cell(1, bs); + chunkIdxs = zeros(1, bs); + for j = 1:bs + chunkIdxs(j) = sb.idxs(j); + subs = cell(1, numel(shape)); + for a = 1:numel(shape) + subs{a} = (sb.loRows(j, a) - sb.loBBox(a) + 1) : ... + (sb.hiRows(j, a) - sb.loBBox(a) + 1); end + srcTile = bigTile(subs{:}); + padded = zeros(chunks, class(srcTile)); + localSubs = arrayfun(@(k) 1:size(srcTile, k), ... + 1:numel(chunks), 'UniformOutput', false); + padded(localSubs{:}) = srcTile; + % Zarr expects C-order bytes; MATLAB is column-major, so + % permute the axes into reversed order before + % serialising, mirroring the fixture writer. + permuted = permute(padded, numel(chunks):-1:1); + raws{j} = typecast(permuted(:), 'uint8'); + end + % Release the source buffer as soon as raws holds copies. + clear bigTile; - % ---- PARALLEL (or serial) ENCODE + WRITE TEMPFILE ------ - tempPaths = cell(1, b); - if poolSize > 1 && b > 1 - parfor j = 1:b - tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... - typesize, tmpRoot, chunkIdxs(j)); %#ok - end - else - for j = 1:b - tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... - typesize, tmpRoot, chunkIdxs(j)); - end + % ---- ONE PARFOR ACROSS THE ENTIRE SUPER-BLOCK ------------- + tempPaths = cell(1, bs); + if poolSize > 1 && bs > 1 + parfor j = 1:bs + tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... + typesize, tmpRoot, chunkIdxs(j)); %#ok end - - % ---- SERIAL ADD_FILE (in chunk order) ----------------- - for j = 1:b - levelDoc = levelDoc.add_file( ... - sprintf('chunk.bin_%d', chunkIdxs(j)), tempPaths{j}); - doneChunks = doneChunks + 1; - reportProgress(progressFcn, doneChunks, totalChunks, ... - sprintf('%s: chunk %d/%d', levelLabel, ... - chunkIdxs(j), nChunks)); + else + for j = 1:bs + tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... + typesize, tmpRoot, chunkIdxs(j)); end end - - % Release the super-block buffer before the next iteration. - % In async mode the next super-block's buffer may already - % exist as a Future's result waiting to be fetched. - clear bigTile; + % raws is no longer needed after the pool has produced the + % tempfiles. Peak RAM stays bounded before the next super- + % block's read materialises via the async future. + clear raws; + + % ---- SERIAL ADD_FILE + BATCHED PROGRESS ------------------ + % In-memory metadata registration; still ordered by chunk + % index so the on-disk chunk numbering is unchanged. Progress + % ticks fire ONCE at the end of each super-block instead of + % per chunk: 18 workers producing a burst of 18 nearly- + % simultaneous updateBar calls was making the bar's + % redraw/ETA-recompute path a measurable fraction of the + % gap. One tick per super-block is more than enough + % granularity on a multi-hour ingest. + for j = 1:bs + levelDoc = levelDoc.add_file( ... + sprintf('chunk.bin_%d', chunkIdxs(j)), tempPaths{j}); + end + doneChunks = doneChunks + bs; + reportProgress(progressFcn, doneChunks, totalChunks, ... + sprintf('%s: super-block %d/%d, chunk %d/%d', ... + levelLabel, s, nSuperBlocks, ... + chunkIdxs(end), nChunks)); end end From 79fd7d91a1f843129efbe2860b05ff9061eff033 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 16:54:37 +0000 Subject: [PATCH 051/110] lightsheet: push pad/permute/typecast into the parfor workers The main-thread slicing pass was doing pad + permute + typecast on every chunk in the super-block sequentially. On a 128-chunk super- block of uint16 lightsheet data (chunks ~4 MB uncompressed each) that adds up to ~1-2 seconds of single-threaded work per super- block -- the between-burst gap that survives the one-parfor-per- super-block change. Only the raw bigTile(subs{:}) slice happens on the main thread now. Each parfor iteration gets its worker's src sub-array (small sliced input) and does the pad-to-chunks-shape + permute-to-C-order + typecast-to-bytes + Blosc encode + tempfile write. All of that work is now fully parallel across the pool. padEncodeAndWrite also skips the zero-pad allocate-and-copy for interior chunks whose source shape already matches `chunks`. Most chunks are interior; only the last chunk along each axis needs padding. Same output bytes -- byte-equivalence tests catch any regression. Peak RAM per super-block during the srcs build is bounded by prefetchBytes (each srcs{j} is a sub-slice of bigTile, and both coexist only during that loop). srcs is released before the pool starts. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 75 +++++++++++-------- 1 file changed, 45 insertions(+), 30 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index ef18a9804..47e482475 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -439,15 +439,14 @@ pyramidName, kInEntry, sb.loBBox, sb.hiBBox); end - % ---- SLICE the entire super-block into per-chunk raw bytes - - % Done once, up front, for the whole super-block instead of - % per sub-batch. This lets us fire ONE parfor across every - % chunk in the super-block and pay MATLAB's parfor - % coordination overhead once instead of ceil(bs/poolSize) - % times. Peak memory during this phase is bigTile + raws - % (~equal size; both released before the compression pool - % begins). - raws = cell(1, bs); + % ---- SLICE bigTile into per-chunk sub-arrays -------------- + % Only the raw slice happens on the main thread now; every + % chunk's pad + permute + typecast + encode + write moves + % into the parfor worker (see padEncodeAndWrite). MATLAB's + % bigTile(subs{:}) is a native memcpy of the sub-array; it's + % roughly 5-10x faster per chunk than the pad-permute- + % typecast pipeline it used to sit in front of. + srcs = cell(1, bs); chunkIdxs = zeros(1, bs); for j = 1:bs chunkIdxs(j) = sb.idxs(j); @@ -456,37 +455,33 @@ subs{a} = (sb.loRows(j, a) - sb.loBBox(a) + 1) : ... (sb.hiRows(j, a) - sb.loBBox(a) + 1); end - srcTile = bigTile(subs{:}); - padded = zeros(chunks, class(srcTile)); - localSubs = arrayfun(@(k) 1:size(srcTile, k), ... - 1:numel(chunks), 'UniformOutput', false); - padded(localSubs{:}) = srcTile; - % Zarr expects C-order bytes; MATLAB is column-major, so - % permute the axes into reversed order before - % serialising, mirroring the fixture writer. - permuted = permute(padded, numel(chunks):-1:1); - raws{j} = typecast(permuted(:), 'uint8'); + srcs{j} = bigTile(subs{:}); end - % Release the source buffer as soon as raws holds copies. + % Release the source buffer as soon as srcs holds copies. clear bigTile; % ---- ONE PARFOR ACROSS THE ENTIRE SUPER-BLOCK ------------- + % Workers now do everything from the raw slice onwards -- + % pad to chunks shape, permute into C-order, typecast to + % bytes, Blosc encode, and write the tempfile. The main + % thread's per-super-block work drops to just the sliced + % memcpy loop above. tempPaths = cell(1, bs); if poolSize > 1 && bs > 1 parfor j = 1:bs - tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... - typesize, tmpRoot, chunkIdxs(j)); %#ok + tempPaths{j} = padEncodeAndWrite(srcs{j}, chunks, ... + codec, clevel, typesize, tmpRoot, chunkIdxs(j)); %#ok end else for j = 1:bs - tempPaths{j} = encodeAndWrite(raws{j}, codec, clevel, ... - typesize, tmpRoot, chunkIdxs(j)); + tempPaths{j} = padEncodeAndWrite(srcs{j}, chunks, ... + codec, clevel, typesize, tmpRoot, chunkIdxs(j)); end end - % raws is no longer needed after the pool has produced the + % srcs is no longer needed after the pool has produced the % tempfiles. Peak RAM stays bounded before the next super- % block's read materialises via the async future. - clear raws; + clear srcs; % ---- SERIAL ADD_FILE + BATCHED PROGRESS ------------------ % In-memory metadata registration; still ordered by chunk @@ -520,13 +515,33 @@ pyramidName, kInEntry, 'Region', [loBBox; hiBBox]); end -function tmp = encodeAndWrite(raw, codec, clevel, typesize, tmpRoot, chunkIdx) -% ENCODEANDWRITE - compress one chunk and drop it in tmpRoot +function tmp = padEncodeAndWrite(srcTile, chunks, codec, clevel, typesize, tmpRoot, chunkIdx) +% PADENCODEANDWRITE - one chunk, end to end, on the worker % % Pure per-chunk work: whatever runs in a parfor iteration lives % here so the parallel worker has no dependency on outer-function -% state beyond the arguments it receives. Returns the tempfile path -% that add_file will consume. +% state beyond the arguments it receives. Takes a source sub-array +% (already sliced out of bigTile on the main thread), pads it to +% the target chunk shape (edge chunks are smaller than `chunks`), +% permutes into C-order, typecasts to bytes, Blosc-encodes when +% requested, and writes a tempfile add_file will consume. + if isequal(size(srcTile), chunks) + % Interior chunk: source already matches target shape, so + % skip the zero-pad allocation-and-copy. Most chunks are + % interior; only the far-edge chunks need padding. + permuted = permute(srcTile, numel(chunks):-1:1); + else + padded = zeros(chunks, class(srcTile)); + localSubs = arrayfun(@(k) 1:size(srcTile, k), ... + 1:numel(chunks), 'UniformOutput', false); + padded(localSubs{:}) = srcTile; + % Zarr expects C-order bytes; MATLAB is column-major, so + % permute the axes into reversed order before serialising, + % mirroring the fixture writer. + permuted = permute(padded, numel(chunks):-1:1); + end + raw = typecast(permuted(:), 'uint8'); + switch codec case 'raw' payload = raw; From e6b54a75de25b636d673f78f6288efa75d240622 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 21:26:18 +0000 Subject: [PATCH 052/110] lightsheet: stop broadcasting per-chunk buffers to every parfor worker The pre-parfor sliced the super-block into srcs{j} = bigTile(subs{:}), one cell per target chunk. MATLAB parfor's static analysis cannot slice a cell whose elements are heterogeneous ND-arrays (indexing srcs{j} returns an mxArray whose class/size is not knowable at parse time), so it falls back to broadcasting: EVERY worker receives a copy of the whole ~512 MB cell. With 18 workers plus the async-prefetch keeping two super-blocks in flight, peak MATLAB memory climbed past 100 GB on a 64 GB machine (117 GB was observed on the SmartSPIM ingest right before this fix), driving the process deep into swap and pushing the ETA from ~6 h into the tens of hours. Replace the cell with a preallocated typed ND-array bigChunks of shape [chunks..., bs]. Each per-chunk assignment writes into the j-th slab; the rest of the slab stays zero, which IS the pad that padEncodeAndWrite used to redo per chunk. Then reshape bigChunks to (prod(chunks), bs) so parfor sees a plain 2-D matrix and slices it unambiguously along the second axis: each worker gets ONE 4 MB column, not the whole super-block. Total in-flight memory drops from ~10x prefetchBytes to ~2x prefetchBytes plus small per-worker slices. Worker-side reshape restores the chunk shape (a view, no copy). padEncodeAndWrite is untouched: the tile it receives is now always already full-chunk shape (edge chunks pre-padded with zeros), so it naturally hits its own "isequal(size, chunks) -> skip pad" branch, which was already the interior-chunk fast path. Impact: peak memory is now bounded by 2 * prefetchBytes (two super-blocks in flight through async prefetch), independent of worker count. On the 64 GB / 18-worker box that means ~1-2 GB resident where before we saw ~100 GB. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 73 ++++++++++++------- 1 file changed, 48 insertions(+), 25 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 47e482475..d383d8f8c 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -439,49 +439,72 @@ pyramidName, kInEntry, sb.loBBox, sb.hiBBox); end - % ---- SLICE bigTile into per-chunk sub-arrays -------------- - % Only the raw slice happens on the main thread now; every - % chunk's pad + permute + typecast + encode + write moves - % into the parfor worker (see padEncodeAndWrite). MATLAB's - % bigTile(subs{:}) is a native memcpy of the sub-array; it's - % roughly 5-10x faster per chunk than the pad-permute- - % typecast pipeline it used to sit in front of. - srcs = cell(1, bs); + % ---- STAGE bigTile INTO A PARFOR-SLICEABLE ND-ARRAY ------- + % An earlier version handed the parfor a cell array `srcs` + % where each entry held one chunk's sub-array. MATLAB parfor + % could not statically slice that cell (`srcs{j}` on a + % heterogeneous cell defeats its analyzer), so it BROADCAST + % the entire ~512 MB cell to every worker: 512 MB x 18 + % workers x 2 super-blocks in flight (async prefetch) = 18+ + % GB of peak RAM per level, which OOM-killed lightsheet + % ingests on 64 GB machines. Preallocate a plain typed + % ND-array `bigChunks` sized (chunks, bs) instead: assigning + % into each j-th slab pre-pads edge chunks with zeros (so + % the worker sees an already-full-shape tile and skips the + % pad branch of padEncodeAndWrite), and reshaping to + % (prod(chunks), bs) makes it a plain 2-D matrix that + % parfor's slice detector unambiguously slices along the + % second axis. Each worker receives ONLY its 4 MB column, + % not the whole super-block. + srcClass = class(bigTile); + bigChunks = zeros([chunks(:).' bs], srcClass); chunkIdxs = zeros(1, bs); for j = 1:bs chunkIdxs(j) = sb.idxs(j); - subs = cell(1, numel(shape)); + srcSubs = cell(1, numel(shape)); + dstSubs = cell(1, numel(chunks)); for a = 1:numel(shape) - subs{a} = (sb.loRows(j, a) - sb.loBBox(a) + 1) : ... - (sb.hiRows(j, a) - sb.loBBox(a) + 1); + srcSubs{a} = (sb.loRows(j, a) - sb.loBBox(a) + 1) : ... + (sb.hiRows(j, a) - sb.loBBox(a) + 1); + dstSubs{a} = 1:numel(srcSubs{a}); end - srcs{j} = bigTile(subs{:}); + % Write this chunk's occupied sub-region into the j-th + % slab; whatever is beyond it stays zero (that IS the + % pad the writer used to do inside padEncodeAndWrite). + slabAssign = [dstSubs, {j}]; + bigChunks(slabAssign{:}) = bigTile(srcSubs{:}); end - % Release the source buffer as soon as srcs holds copies. clear bigTile; + % Flatten to (prod(chunks), bs) so parfor sees a plain 2-D + % array and slices it along the loop-variable axis with no + % ambiguity. + nChunkElems = prod(chunks); + bigChunksFlat = reshape(bigChunks, nChunkElems, bs); + clear bigChunks; + % ---- ONE PARFOR ACROSS THE ENTIRE SUPER-BLOCK ------------- - % Workers now do everything from the raw slice onwards -- - % pad to chunks shape, permute into C-order, typecast to - % bytes, Blosc encode, and write the tempfile. The main - % thread's per-super-block work drops to just the sliced - % memcpy loop above. + % Workers now do everything from the flat slice onwards -- + % reshape into the chunk shape, permute into C-order, + % typecast to bytes, Blosc encode, and write the tempfile. + chunkShape1 = chunks(:).'; tempPaths = cell(1, bs); if poolSize > 1 && bs > 1 parfor j = 1:bs - tempPaths{j} = padEncodeAndWrite(srcs{j}, chunks, ... - codec, clevel, typesize, tmpRoot, chunkIdxs(j)); %#ok + col = bigChunksFlat(:, j); + tile = reshape(col, chunkShape1); + tempPaths{j} = padEncodeAndWrite(tile, chunks, ... + codec, clevel, typesize, tmpRoot, chunkIdxs(j)); end else for j = 1:bs - tempPaths{j} = padEncodeAndWrite(srcs{j}, chunks, ... + col = bigChunksFlat(:, j); + tile = reshape(col, chunkShape1); + tempPaths{j} = padEncodeAndWrite(tile, chunks, ... codec, clevel, typesize, tmpRoot, chunkIdxs(j)); end end - % srcs is no longer needed after the pool has produced the - % tempfiles. Peak RAM stays bounded before the next super- - % block's read materialises via the async future. - clear srcs; + clear bigChunksFlat; % ---- SERIAL ADD_FILE + BATCHED PROGRESS ------------------ % In-memory metadata registration; still ordered by chunk From 7d94e1be09e287797cccd62d29c59c846431fa98 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 21:42:27 +0000 Subject: [PATCH 053/110] lightsheet: use blosc.encodeChunk fast path in padEncodeAndWrite Every parfor worker was doing pad + axis-order permute + typecast + Blosc encode in four separate MATLAB steps, each with its own buffer allocation and full memcpy: padded = zeros(chunks, class(srcTile)); padded(localSubs{:}) = srcTile; permuted = permute(padded, numel(chunks):-1:1); raw = typecast(permuted(:), 'uint8'); payload = ndr.format.blosc.encode(raw, ...); Three of those (pad, permute, typecast) can be fused into the same C call that does the Blosc encode -- blosc.encodeChunk from blosc-matlab v0.2.0 walks a strided odometer over target C-order positions, memcpys each element from its Fortran-indexed source (or writes the pad value when out of bounds), and hands the assembled buffer to blosc_compress_ctx once. Zero intermediate MATLAB allocations. Byte-for-byte identical output; TestBloscChunk in blosc-matlab verifies that against the manual pad + permute + encode chain. For a 128x128x128 uint16 chunk that saves ~3 * 4 MB = 12 MB of main-thread memory traffic per chunk. At 40k chunks x 5 pyramid levels that is ~2.4 TB avoided over the run -- half of MATLAB's per-chunk overhead. Worker-side capability probe (canUseEncodeChunk) memoises the ndr.util.blosc.ensureMex install check per worker. On the first padEncodeAndWrite call in a worker the check triggers download of the v0.2.0 tarball; subsequent calls are a flag read. Fallback: when the MEX is unavailable (no prebuilt for this platform, network down, blosc-matlab install refused) the old pad-permute-typecast MATLAB path via ndr.format.blosc.encode runs unchanged, so users without the fast path still ingest. padPermuteBytes was extracted from the old body to hold that fallback, and is shared with the codec='raw' branch. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 99 +++++++++++++------ 1 file changed, 71 insertions(+), 28 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index d383d8f8c..31f795ade 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -543,37 +543,36 @@ % % Pure per-chunk work: whatever runs in a parfor iteration lives % here so the parallel worker has no dependency on outer-function -% state beyond the arguments it receives. Takes a source sub-array -% (already sliced out of bigTile on the main thread), pads it to -% the target chunk shape (edge chunks are smaller than `chunks`), -% permutes into C-order, typecasts to bytes, Blosc-encodes when -% requested, and writes a tempfile add_file will consume. - if isequal(size(srcTile), chunks) - % Interior chunk: source already matches target shape, so - % skip the zero-pad allocation-and-copy. Most chunks are - % interior; only the far-edge chunks need padding. - permuted = permute(srcTile, numel(chunks):-1:1); - else - padded = zeros(chunks, class(srcTile)); - localSubs = arrayfun(@(k) 1:size(srcTile, k), ... - 1:numel(chunks), 'UniformOutput', false); - padded(localSubs{:}) = srcTile; - % Zarr expects C-order bytes; MATLAB is column-major, so - % permute the axes into reversed order before serialising, - % mirroring the fixture writer. - permuted = permute(padded, numel(chunks):-1:1); - end - raw = typecast(permuted(:), 'uint8'); - +% state beyond the arguments it receives. +% +% For codec='blosc-zstd', the fast path is a single blosc.encodeChunk +% MEX call from blosc-matlab v0.2.0: pad + permute (Fortran -> Zarr +% C-order) + typecast + Blosc encode happen inside the C code with +% no MATLAB-side intermediate allocations. On a lightsheet ingest +% that removes ~3 full memcpys per chunk (each ~ chunkBytes), which +% at 40k chunks x 5 pyramid levels is hundreds of GB of avoided +% main-thread memory traffic per worker. +% +% Fallback (v0.1.0 MEX, or codec='raw'): do the pad + permute + +% typecast in MATLAB, then either return the raw bytes or hand them +% to ndr.format.blosc.encode. Byte-identical output. switch codec case 'raw' - payload = raw; + payload = padPermuteBytes(srcTile, chunks); case 'blosc-zstd' - payload = ndr.format.blosc.encode(raw, ... - 'typesize', typesize, ... - 'cname', 'zstd', ... - 'clevel', clevel, ... - 'shuffle', 1); + if canUseEncodeChunk() + payload = blosc.encodeChunk(srcTile, chunks(:).', ... + 'cname', 'zstd', ... + 'clevel', clevel, ... + 'shuffle', 1); + else + raw = padPermuteBytes(srcTile, chunks); + payload = ndr.format.blosc.encode(raw, ... + 'typesize', typesize, ... + 'cname', 'zstd', ... + 'clevel', clevel, ... + 'shuffle', 1); + end otherwise error('NDI:lightsheet:makePyramid:unknownCodec', ... 'Unknown codec %s.', codec); @@ -584,6 +583,50 @@ fclose(fid); end +function tf = canUseEncodeChunk() +% CANUSEENCODECHUNK - does this worker have blosc.encodeChunk on its path? +% +% Runs on the parfor worker (padEncodeAndWrite is called inside the +% parfor). Each worker is a separate MATLAB process with its own +% `persistent` scope, so the first call in a worker triggers +% ensureMex (which downloads blosc-matlab v0.2.0 to prefdir() on +% first use in that worker's session) and every subsequent call in +% the same worker is a memoized flag read. +% +% Falls back to false on any error so a caller without NDR-matlab +% or without a working install still gets the pad-permute-typecast +% MATLAB path via ndr.format.blosc.encode. + persistent flag + if ~isempty(flag), tf = flag; return; end + flag = false; + try + info = ndr.util.blosc.ensureMex(); + flag = info.available && exist('blosc.encodeChunk', 'file') == 2; + catch + end + tf = flag; +end + +function bytes = padPermuteBytes(srcTile, chunks) +% PADPERMUTEBYTES - Fortran ND-array -> zero-padded C-order uint8 bytes. +% +% The MATLAB fallback for the Blosc fast path: pad the tile up to +% the target chunk shape with zeros, permute axes into reversed +% order so the row-major byte layout matches what Zarr writes, then +% typecast to bytes. Only used when the v0.2.0 blosc-matlab MEX is +% not available. + if isequal(size(srcTile), chunks) + permuted = permute(srcTile, numel(chunks):-1:1); + else + padded = zeros(chunks, class(srcTile)); + localSubs = arrayfun(@(k) 1:size(srcTile, k), ... + 1:numel(chunks), 'UniformOutput', false); + padded(localSubs{:}) = srcTile; + permuted = permute(padded, numel(chunks):-1:1); + end + bytes = typecast(permuted(:), 'uint8'); +end + function [n, src] = describeParallelPool(numWorkers) % DESCRIBEPARALLELPOOL - peek at the effective pool without side effects % From 71d7387ff242b1dc1edef76f213d5c40efbbd771 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 22:49:43 +0000 Subject: [PATCH 054/110] lightsheet: add debugLog option to trace per-super-block memory growth An ingest recently climbed to 80 GB RSS on a 64 GB machine at only 3% through level 0 (see PR discussion), and we do not yet know where the memory is going -- candidates include per-chunk levelDoc.add_file copy-on-write growth, retained parfeval future outputs on the async prefetch path, or something else entirely. Speculating from the call tree has been unproductive; instrument first, fix on evidence. New option `debugLog` on both fromOMEZarr and makePyramid: a file path the ingest appends one line to at the end of each super-block: YYYY-MM-DD HH:MM:SS | Level N (r) | sb=S/N | chunks=D/T | rss_MB=... | doc_pre_MB=... | doc_post_MB=... | doc_delta_MB=... | live_futures=... Fields: rss_MB -- MATLAB process RSS in MB via `ps -o rss= -p ` (Linux/macOS). Windows falls through to NaN. doc_pre_MB -- `whos` size of levelDoc.document_properties BEFORE the super-block's add_file loop runs. doc_post_MB -- same, AFTER the loop. delta reveals whether the add_file growth is linear or worse than linear. live_futures -- p.FevalQueue.QueuedFutures + RunningFutures. Should hold near 0 when async prefetch is draining cleanly; a growing count is a smoking gun for retained bigTile outputs on the pool. Empty debugLog (default) disables all instrumentation. When set, every helper (matlabRSSMB, poolFutureCount, appendDebugLog) swallows its own errors so a broken /tmp or a missing `ps` cannot take down an ingest -- diagnostics must never regress correctness. Wiring: fromOMEZarr forwards options.debugLog to makePyramid, which forwards it to attachChunkFiles. Follow-up: once we have a log from a real run, decide whether the leak is on the doc side (fix by batching add_file, or bypassing to did.document's file-series pattern) or on the future side (fix by explicit delete(fut) after fetchOutputs). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 10 ++- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 81 ++++++++++++++++++- 2 files changed, 88 insertions(+), 3 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index 70bdb666b..e6951ffb9 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -114,6 +114,13 @@ % See makePyramid for details. Default true; ignored when % numWorkers resolves to serial. options.asyncPrefetch (1,1) logical = true + % Path to a log file the ingest appends per-super-block memory + % diagnostics to (timestamp, level index, super-block index, + % chunks done, MATLAB RSS, levelDoc property size before/after + % add_file, parpool live-future count). Empty (default) + % disables. Intended for diagnosing memory growth. See + % makePyramid for details. + options.debugLog (1,:) char = '' end if isempty(strtrim(options.subjectID)) @@ -191,7 +198,8 @@ 'progressFcn', progressFcn, ... 'numWorkers', options.numWorkers, ... 'prefetchBytes', options.prefetchBytes, ... - 'asyncPrefetch', options.asyncPrefetch); + 'asyncPrefetch', options.asyncPrefetch, ... + 'debugLog', options.debugLog); info = struct( ... 'zarrPath', char(zarrPath), ... diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 31f795ade..b11d75d26 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -110,6 +110,14 @@ % otherwise). Peak RAM is ~2 * prefetchBytes (two super- % blocks in flight). options.asyncPrefetch (1,1) logical = true + % Path to a text log file the ingest will append per-super-block + % memory diagnostics to (timestamp, level index, super-block + % index, chunks done, main-process RSS in MB, `whos` sizes of + % the big variables, parpool live-future count). Empty (default) + % disables logging. Intended for diagnosing memory growth + % without affecting the hot path -- one line per super-block is + % negligible on a multi-hour ingest. + options.debugLog (1,:) char = '' end if options.materializeChunks && isempty(options.sourceZarrPath) @@ -315,14 +323,15 @@ options.sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, ... options.codec, options.clevel, ... options.progressFcn, doneChunks, totalChunks, levelLabel, ... - options.numWorkers, options.prefetchBytes, options.asyncPrefetch); + options.numWorkers, options.prefetchBytes, options.asyncPrefetch, ... + options.debugLog); end tmpCleaner = onCleanup(@() cleanupTmp(tmpRoot)); session.database_add(levelDoc); end -function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers, prefetchBytes, asyncPrefetch) +function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers, prefetchBytes, asyncPrefetch, debugLog) % Read the source zarr level as a whole array, then re-tile into the % level document's chunk shape and attach each chunk as chunk.bin_ % (1-based, C-order over chunkGrid). Files are ingested into DID, which @@ -515,15 +524,83 @@ % redraw/ETA-recompute path a measurable fraction of the % gap. One tick per super-block is more than enough % granularity on a multi-hour ingest. + preAddDocSize = levelDocPropertyBytes(levelDoc); for j = 1:bs levelDoc = levelDoc.add_file( ... sprintf('chunk.bin_%d', chunkIdxs(j)), tempPaths{j}); end + postAddDocSize = levelDocPropertyBytes(levelDoc); doneChunks = doneChunks + bs; reportProgress(progressFcn, doneChunks, totalChunks, ... sprintf('%s: super-block %d/%d, chunk %d/%d', ... levelLabel, s, nSuperBlocks, ... chunkIdxs(end), nChunks)); + appendDebugLog(debugLog, levelLabel, s, nSuperBlocks, doneChunks, ... + totalChunks, preAddDocSize, postAddDocSize); + end +end + +function bytes = levelDocPropertyBytes(levelDoc) +% Approximate size in bytes of levelDoc.document_properties. Uses `whos` +% on a local copy of the struct so the levelDoc object itself is not +% traversed (which would add its own overhead). Returns NaN on any +% error so instrumentation cannot break the ingest. + bytes = NaN; + try + props = levelDoc.document_properties; %#ok + w = whos('props'); + bytes = w.bytes; + catch + end +end + +function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChunks, preBytes, postBytes) +% Append one line of memory diagnostics to `path`. Failures are +% swallowed; instrumentation must never take down an ingest. + if isempty(path), return; end + try + rss = matlabRSSMB(); + futCount = poolFutureCount(); + line = sprintf(['%s | %s | sb=%d/%d | chunks=%d/%d | ' ... + 'rss_MB=%.0f | doc_pre_MB=%.2f | ' ... + 'doc_post_MB=%.2f | doc_delta_MB=%.2f | ' ... + 'live_futures=%d'], ... + datestr(now, 'yyyy-mm-dd HH:MM:SS'), levelLabel, s, nSuperBlocks, ... + doneChunks, totalChunks, rss, preBytes/2^20, postBytes/2^20, ... + (postBytes - preBytes)/2^20, futCount); + fid = fopen(path, 'a'); + if fid < 0, return; end + cleanup = onCleanup(@() fclose(fid)); %#ok + fprintf(fid, '%s\n', line); + catch + end +end + +function mb = matlabRSSMB() +% MATLAB's own process RSS in MB. `memory` is Windows-only; on macOS / +% Linux we shell out to `ps -o rss= -p ` which prints kilobytes. + mb = NaN; + try + pid = feature('getpid'); + [rc, out] = system(sprintf('ps -o rss= -p %d', pid)); + if rc == 0 + v = sscanf(strtrim(out), '%f'); + if ~isempty(v), mb = v / 1024; end + end + catch + end +end + +function n = poolFutureCount() +% Number of parfeval / parfor futures the pool currently holds. This +% surfaces leaks in the async-prefetch pipeline where a future's output +% never gets released, which retains its bigTile buffer inside the pool. + n = NaN; + try + p = gcp('nocreate'); + if isempty(p), n = 0; return; end + n = numel(p.FevalQueue.QueuedFutures) + numel(p.FevalQueue.RunningFutures); + catch end end From 58891a17f851f5b9c79277df4dcd166b210fda44 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 22:53:23 +0000 Subject: [PATCH 055/110] lightsheet debugLog: use datetime + matlabProcessID (Code Analyzer) Three lint-bot findings on the diagnostics helpers I added in the previous commit: * datestr(now, ...) -> char(datetime('now', 'Format', ...)) Modern MATLAB pattern; datestr and the bare `now` are both deprecated by Code Analyzer in R2024a. * feature('getpid') -> matlabProcessID() matlabProcessID is the R2023a+ public API; feature('getpid') is documented-unsupported. * Remove a stale `%#ok` on the onCleanup return -- the NASGU warning no longer fires in R2024a, so the suppression is dead. No behaviour change. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index b11d75d26..41dd24de8 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -565,12 +565,13 @@ function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChun 'rss_MB=%.0f | doc_pre_MB=%.2f | ' ... 'doc_post_MB=%.2f | doc_delta_MB=%.2f | ' ... 'live_futures=%d'], ... - datestr(now, 'yyyy-mm-dd HH:MM:SS'), levelLabel, s, nSuperBlocks, ... + char(datetime('now', 'Format', 'yyyy-MM-dd HH:mm:ss')), ... + levelLabel, s, nSuperBlocks, ... doneChunks, totalChunks, rss, preBytes/2^20, postBytes/2^20, ... (postBytes - preBytes)/2^20, futCount); fid = fopen(path, 'a'); if fid < 0, return; end - cleanup = onCleanup(@() fclose(fid)); %#ok + cleanup = onCleanup(@() fclose(fid)); fprintf(fid, '%s\n', line); catch end @@ -581,7 +582,7 @@ function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChun % Linux we shell out to `ps -o rss= -p ` which prints kilobytes. mb = NaN; try - pid = feature('getpid'); + pid = matlabProcessID(); [rc, out] = system(sprintf('ps -o rss= -p %d', pid)); if rc == 0 v = sscanf(strtrim(out), '%f'); From 0e731622029f76e494bb17e63bdb81625ed117bf Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 23:02:05 +0000 Subject: [PATCH 056/110] lightsheet: install chunks with one did.document.addFileSeries call Confirmed by looking at did-matlab: did.document has a bulk addFileSeries method (src/did/+did/document.m line 932) that installs a cellstr of source paths into one series entry as a single struct- array assignment. Every chunk was previously going through its own `levelDoc = levelDoc.add_file(sprintf('chunk.bin_%d', ...), ...)`. Even though the schema declares chunk.bin_# as a file series (so the resulting series_info holds one entry with an N-element locations array, not N separate entries), each call to add_file: 1. Constructs one location record inside did.document.add_file. 2. Copies it into the growing locations array on the series entry (O(N) at iteration N; O(N^2) cumulative across the level). 3. Returns a new levelDoc via MATLAB value semantics -- the entire document_properties struct is considered dirty and gets a fresh copy allocated even though only one nested field changed. The observed 80 GB RSS on a 40k-chunk / 6-worker ingest fits that allocation churn: MATLAB's memory manager holds transient copies until GC compacts, which lags well behind an ingest allocating GB- per-second. Fix: - Preallocate seriesLocations = cell(1, nChunks) at level scope. - Each super-block writes into seriesLocations at its chunkIdxs positions instead of mutating levelDoc. - After the super-block loop exits, ONE call: levelDoc = levelDoc.addFileSeries('chunk.bin_#', ... seriesLocations, 'indices', 1:nChunks); installs every location in a single struct-array assignment. Semantics unchanged: the ingested file list is identical (same chunk.bin_1 .. chunk.bin_N series with the same tempfile paths and DID's default `ingest=1, deleteOriginal=1`). Only the shape of the allocation churn changes. Debug log's doc_pre_MB / doc_post_MB columns should now report the pre value stably per super-block (no growth inside the loop), and the delta appears once at the addFileSeries call at level end. Followup diagnostic value: if RSS still climbs after this change, the leak is not on the ndi.document side and the parfeval future retention theory (async prefetch not releasing bigTile outputs) becomes the next candidate. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 42 +++++++++++++++---- 1 file changed, 35 insertions(+), 7 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 41dd24de8..91c5736e8 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -373,15 +373,26 @@ nChunks = size(linearOrder, 1); typesize = elementSizeBytes(char(level.dtype)); + % Locations for the whole level. Filled in as each super-block + % completes; handed to did.document.addFileSeries ONCE at the end + % of the level. Per-chunk add_file returned a fresh levelDoc every + % call, and 40k value-semantics reassignments of a growing struct + % churned through many GB of allocations even though the final + % series entry is small -- the bulk method installs all locations + % in one struct-array assignment, and the observed memory blow-up + % goes away. + seriesLocations = cell(1, nChunks); + % BATCHED SERIAL-READ / PARALLEL-COMPRESS. On a slow spinning drive % we cannot afford concurrent seeks, so reads stay strictly serial; % but Zstd at clevel=9 is single-core CPU-bound and easy to fan out % across a parallel pool. A batch of `poolSize` chunks is read from % the source, then handed to the workers to compress and write, in - % parallel; add_file into the level document happens serially in - % chunk-index order so the DID SQLite write is not a contention - % point and the on-disk chunk numbering is unchanged. poolSize == 1 - % is the original serial path with no toolbox dependency. + % parallel; each chunk's tempfile path is parked in seriesLocations + % and installed into the level document via a SINGLE + % did.document.addFileSeries call after the super-block loop exits. + % poolSize == 1 is the original serial path with no toolbox + % dependency. poolSize = resolveParallelPool(numWorkers); % How many target chunks fit inside the prefetch budget, sized @@ -524,12 +535,20 @@ % redraw/ETA-recompute path a measurable fraction of the % gap. One tick per super-block is more than enough % granularity on a multi-hour ingest. + % ---- ACCUMULATE, DO NOT MUTATE levelDoc PER SUPER-BLOCK --- + % The old code called `levelDoc = levelDoc.add_file(...)` + % 40k+ times per level. Each reassignment triggered a + % copy-on-write on the whole document_properties, churning + % through gigabytes of transient allocations even though the + % final file-series entry is small. Instead, park each + % chunk's tempfile path in seriesLocations{chunkIdx} and + % install them all in ONE addFileSeries call after the + % super-block loop exits. preAddDocSize = levelDocPropertyBytes(levelDoc); for j = 1:bs - levelDoc = levelDoc.add_file( ... - sprintf('chunk.bin_%d', chunkIdxs(j)), tempPaths{j}); + seriesLocations{chunkIdxs(j)} = tempPaths{j}; end - postAddDocSize = levelDocPropertyBytes(levelDoc); + postAddDocSize = preAddDocSize; doneChunks = doneChunks + bs; reportProgress(progressFcn, doneChunks, totalChunks, ... sprintf('%s: super-block %d/%d, chunk %d/%d', ... @@ -538,6 +557,15 @@ appendDebugLog(debugLog, levelLabel, s, nSuperBlocks, doneChunks, ... totalChunks, preAddDocSize, postAddDocSize); end + + % ---- ONE BULK addFileSeries CALL FOR THE WHOLE LEVEL --------- + % did.document.addFileSeries installs the whole cellstr of source + % paths in a single struct-array assignment. The level document's + % file-series manifest names the series `chunk.bin_#`; `indices` + % supplies the 1..N chunk numbering that DID substitutes for `#` + % in each stored location. No per-chunk copy-on-write. + levelDoc = levelDoc.addFileSeries('chunk.bin_#', seriesLocations, ... + 'indices', 1:nChunks); end function bytes = levelDocPropertyBytes(levelDoc) From 8b751b49e5135f9453136761a0d87d78430e6aef Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 23:06:26 +0000 Subject: [PATCH 057/110] debugLog: fall back to feature('getpid') when matlabProcessID is missing matlabProcessID landed in R2023a and is not present on every installed MATLAB (the running ingest reported rss_MB=NaN because the call was throwing, which the outer try...catch was silently swallowing along with the rest of the block, so `ps` never ran). Wrap the PID lookup in its own try...catch so an unresolved matlabProcessID falls back to feature('getpid') (documented- unsupported but universally present) instead of losing the whole log line's RSS value. The Code Analyzer warning on the fallback is suppressed with a scoped %#ok since this is a deliberate compatibility path. Comment updated to say why the fallback exists. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 91c5736e8..b6289cbba 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -608,9 +608,16 @@ function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChun function mb = matlabRSSMB() % MATLAB's own process RSS in MB. `memory` is Windows-only; on macOS / % Linux we shell out to `ps -o rss= -p ` which prints kilobytes. +% matlabProcessID() is R2023a+ and not present on every install; fall +% back to feature('getpid') where it isn't, so the log always gets a +% real number instead of NaN. mb = NaN; try - pid = matlabProcessID(); + try + pid = matlabProcessID(); + catch + pid = feature('getpid'); %#ok<*FGETP> + end [rc, out] = system(sprintf('ps -o rss= -p %d', pid)); if rc == 0 v = sscanf(strtrim(out), '%f'); From 750ec1b6fb8f010b5c9bdf49db14588e2f20351a Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 23:09:56 +0000 Subject: [PATCH 058/110] debugLog: get PID via Java ManagementFactory (avoid lint findings) Two prior attempts at getting the MATLAB PID for `ps -o rss= -p `: 1. matlabProcessID() -- clean, but R2023a+ only. On older MATLAB it throws and RSS is NaN. 2. feature('getpid') fallback -- universally present but flagged by Code Analyzer as unsupported; the suppression tag I guessed (FGETP) was wrong so the bot kept warning. Both problems disappear with a third option: ask the JVM. Every MATLAB ships with Java, and `java.lang.management.ManagementFactory.getRuntimeMXBean().getName()` returns the string "PID@HOSTNAME" on all mainstream JVM implementations. strtok(text, '@') gets us the PID as a numeric string; str2double turns it into a number. No MATLAB API version gate, no Code Analyzer flag. Comment updated. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index b6289cbba..2cc94029a 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -608,16 +608,18 @@ function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChun function mb = matlabRSSMB() % MATLAB's own process RSS in MB. `memory` is Windows-only; on macOS / % Linux we shell out to `ps -o rss= -p ` which prints kilobytes. -% matlabProcessID() is R2023a+ and not present on every install; fall -% back to feature('getpid') where it isn't, so the log always gets a -% real number instead of NaN. +% +% PID lookup goes through Java's ManagementFactory (available in every +% MATLAB that has bundled Java): getRuntimeMXBean().getName() returns +% "PID@HOSTNAME" on every JVM implementation we care about. This +% sidesteps both matlabProcessID (R2023a+ only) and feature('getpid') +% (Code Analyzer flags as unsupported). mb = NaN; try - try - pid = matlabProcessID(); - catch - pid = feature('getpid'); %#ok<*FGETP> - end + rt = java.lang.management.ManagementFactory.getRuntimeMXBean(); + pidText = char(rt.getName()); + pid = str2double(strtok(pidText, '@')); + if isnan(pid), return; end [rc, out] = system(sprintf('ps -o rss= -p %d', pid)); if rc == 0 v = sscanf(strtrim(out), '%f'); From 7d4842f88d84bdb46b7f1e2dc09148eaed390fee Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 23:18:36 +0000 Subject: [PATCH 059/110] lightsheet: cap super-block bbox byte volume to prevent readArray OOM Root cause of the sudden 7 GB -> 117 GB main-thread spike observed on the SmartSPIM ingest at approximately super-block 40: * A super-block is a linear span of prefetchChunkCount chunks pulled from linearOrder (row-major over chunkGrid). Most super- blocks are spatially contiguous along the fastest-varying axis and have a small union bounding box. * When a super-block straddles a wrap in the linear order (e.g. the last few chunks of one row plus the first few of the next), its union bounding box spans the full range of the wrapped axis. Two wraps in the same super-block means the bbox spans two axes' full ranges. * readSuperBlockRegion calls ndr.format.omezarr.readArray with 'Region', [loBBox; hiBBox]. readArray allocates the WHOLE bbox uncompressed even though we only want the specific chunks inside it. For a lightsheet level 0 that is easily 50-100+ GB per bad super-block. * readArray returns that bigTile to the async parfeval worker, fetchOutputs copies it to main -> main allocates a 100 GB array -> mac swaps hard. The prior super-block plan cut the chunk list into fixed spans of prefetchChunkCount without regard to spatial layout, so this failure was silent until a run happened to hit a bad wrap. Fix: pack super-blocks greedily, closing them early when adding the next chunk would push the union bbox's uncompressed byte cost above bboxByteCap = max(4 * prefetchBytes, 2 * poolSize * chunkBytes). This guarantees that no single read can allocate more than ~2 GB (with default prefetchBytes = 512 MB), regardless of chunk-grid geometry. Wrap-crossing super-blocks are just closed a few chunks early. On a well-behaved chunk grid where no wrap crosses, the cap never fires and the plan is identical to before. New growSuperBlock helper does the packing per super-block. Also add bbox_MB to the debug log so this pattern is visible on future runs before the machine goes into swap. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 101 ++++++++++++++---- 1 file changed, 78 insertions(+), 23 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 2cc94029a..415269680 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -407,26 +407,33 @@ % Plan every super-block up front (chunk indices + per-chunk lo/hi % + union bbox) so the async pipeline can dispatch the next read % while the current one is still compressing. - nSuperBlocks = ceil(nChunks / prefetchChunkCount); - superBlocks = cell(nSuperBlocks, 1); - for s = 1:nSuperBlocks - s0 = (s - 1) * prefetchChunkCount + 1; - s1 = min(s0 + prefetchChunkCount - 1, nChunks); - idxs = s0:s1; - bs = numel(idxs); - lo = zeros(bs, numel(shape)); - hi = zeros(bs, numel(shape)); - for j = 1:bs - c = linearOrder(idxs(j), :); - for a = 1:numel(chunkGrid) - lo(j, a) = (c(a) - 1) * chunks(a) + 1; - hi(j, a) = min(lo(j, a) + chunks(a) - 1, shape(a)); - end - end - superBlocks{s} = struct('idxs', idxs, ... + % + % Packing rule: chunks are consumed from linearOrder in order, but + % a super-block is CLOSED early when adding the next chunk would + % blow up the union bounding box past bboxByteCap. Without this + % cap, a super-block that spans a wrap in the linear chunk order + % (e.g. the last chunks of one row plus the first of the next) + % has a bbox that spans the full range on the fastest-varying + % axis. When two wraps land inside the same super-block, the + % bbox covers a large fraction of the level, and + % readSuperBlockRegion allocates that whole region uncompressed: + % a 40k-chunk lightsheet ingest can jump from ~7 GB to >100 GB + % on a single ill-placed super-block, which we observed. Capping + % the bbox volume keeps peak read allocation bounded to + % ~prefetchBytes regardless of chunk-grid geometry. + bboxByteCap = max(double(prefetchBytes) * 4, chunkBytes * poolSize * 2); + superBlocks = cell(0, 1); + i = 1; + while i <= nChunks + [idxs, lo, hi] = growSuperBlock(linearOrder, i, ... + min(prefetchChunkCount, nChunks - i + 1), ... + chunks, chunkGrid, shape, typesize, bboxByteCap); + superBlocks{end+1, 1} = struct('idxs', idxs, ... %#ok 'loRows', lo, 'hiRows', hi, ... 'loBBox', min(lo, [], 1), 'hiBBox', max(hi, [], 1)); + i = i + numel(idxs); end + nSuperBlocks = numel(superBlocks); % ASYNC PIPELINE. If a parpool is available and asyncPrefetch is % on, the next super-block's read runs concurrently on a pool @@ -554,8 +561,9 @@ sprintf('%s: super-block %d/%d, chunk %d/%d', ... levelLabel, s, nSuperBlocks, ... chunkIdxs(end), nChunks)); + bboxMB = prod(double(sb.hiBBox - sb.loBBox + 1)) * typesize / 2^20; appendDebugLog(debugLog, levelLabel, s, nSuperBlocks, doneChunks, ... - totalChunks, preAddDocSize, postAddDocSize); + totalChunks, preAddDocSize, postAddDocSize, bboxMB); end % ---- ONE BULK addFileSeries CALL FOR THE WHOLE LEVEL --------- @@ -582,20 +590,22 @@ end end -function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChunks, preBytes, postBytes) +function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChunks, preBytes, postBytes, bboxMB) % Append one line of memory diagnostics to `path`. Failures are % swallowed; instrumentation must never take down an ingest. if isempty(path), return; end + if nargin < 9, bboxMB = NaN; end try rss = matlabRSSMB(); futCount = poolFutureCount(); line = sprintf(['%s | %s | sb=%d/%d | chunks=%d/%d | ' ... - 'rss_MB=%.0f | doc_pre_MB=%.2f | ' ... - 'doc_post_MB=%.2f | doc_delta_MB=%.2f | ' ... - 'live_futures=%d'], ... + 'rss_MB=%.0f | bbox_MB=%.0f | ' ... + 'doc_pre_MB=%.2f | doc_post_MB=%.2f | ' ... + 'doc_delta_MB=%.2f | live_futures=%d'], ... char(datetime('now', 'Format', 'yyyy-MM-dd HH:mm:ss')), ... levelLabel, s, nSuperBlocks, ... - doneChunks, totalChunks, rss, preBytes/2^20, postBytes/2^20, ... + doneChunks, totalChunks, rss, bboxMB, ... + preBytes/2^20, postBytes/2^20, ... (postBytes - preBytes)/2^20, futCount); fid = fopen(path, 'a'); if fid < 0, return; end @@ -642,6 +652,51 @@ function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChun end end +function [idxs, lo, hi] = growSuperBlock(linearOrder, i0, budgetChunks, chunks, chunkGrid, shape, typesize, bboxByteCap) +% GROWSUPERBLOCK - pack chunks starting at index i0 into one super-block, +% closing early when the union bbox would exceed bboxByteCap. +% +% Adds chunks one at a time from linearOrder(i0:end); recomputes the +% union bbox after each. Stops when either budgetChunks chunks have +% been packed, or the NEXT chunk would push the union bbox's +% uncompressed byte cost above bboxByteCap. Guarantees at least one +% chunk in the super-block (the first is always accepted regardless +% of bbox cost, because a single chunk cannot cost more than +% chunkBytes). + idxs = zeros(1, budgetChunks); + lo = zeros(budgetChunks, numel(shape)); + hi = zeros(budgetChunks, numel(shape)); + curLo = inf(1, numel(shape)); + curHi = -inf(1, numel(shape)); + n = 0; + for k = 0:budgetChunks-1 + c = linearOrder(i0 + k, :); + pieceLo = zeros(1, numel(shape)); + pieceHi = zeros(1, numel(shape)); + for a = 1:numel(chunkGrid) + pieceLo(a) = (c(a) - 1) * chunks(a) + 1; + pieceHi(a) = min(pieceLo(a) + chunks(a) - 1, shape(a)); + end + newLo = min(curLo, pieceLo); + newHi = max(curHi, pieceHi); + if n > 0 + newBoxBytes = prod(double(newHi - newLo + 1)) * typesize; + if newBoxBytes > bboxByteCap + break; + end + end + n = n + 1; + idxs(n) = i0 + k; % 1-based linear chunk id + lo(n, :) = pieceLo; + hi(n, :) = pieceHi; + curLo = newLo; + curHi = newHi; + end + idxs = idxs(1:n); + lo = lo(1:n, :); + hi = hi(1:n, :); +end + function bigTile = readSuperBlockRegion(sourceZarrPath, pyramidName, kInEntry, loBBox, hiBBox) % READSUPERBLOCKREGION - fetch one super-block region from the source % From d2afcaaeb0781cba6066738c24b1bb21ad54274c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 13 Sep 2026 23:25:57 +0000 Subject: [PATCH 060/110] lightsheet: preallocate superBlocks cell to nChunks (Code Analyzer AGROW) Code Analyzer flagged the greedy-packing loop's `superBlocks{end+1, 1} = struct(...)` as a variable-changes-size-on-every-iteration issue, even with the %#ok suppression. Rather than fight the lint, preallocate `superBlocks = cell(nChunks, 1)` (worst case is one chunk per super-block, so nChunks is a hard upper bound), fill in from index 1, then trim to the actual count at the end. Cost: ~1.2 MB of pointer slots (nChunks * 8 bytes on 64-bit), one-shot allocation, trivial. Comment explains the preallocation choice. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index 415269680..fe8216d0f 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -422,18 +422,24 @@ % the bbox volume keeps peak read allocation bounded to % ~prefetchBytes regardless of chunk-grid geometry. bboxByteCap = max(double(prefetchBytes) * 4, chunkBytes * poolSize * 2); - superBlocks = cell(0, 1); + % Preallocate to the worst-case super-block count (one chunk per + % super-block) so Code Analyzer sees a static-size cell rather + % than end+1 growth. Trim at the end. The `nChunks` slot count is + % ~156k pointers = ~1.2 MB, negligible. + superBlocks = cell(nChunks, 1); + nSuperBlocks = 0; i = 1; while i <= nChunks [idxs, lo, hi] = growSuperBlock(linearOrder, i, ... min(prefetchChunkCount, nChunks - i + 1), ... chunks, chunkGrid, shape, typesize, bboxByteCap); - superBlocks{end+1, 1} = struct('idxs', idxs, ... %#ok + nSuperBlocks = nSuperBlocks + 1; + superBlocks{nSuperBlocks} = struct('idxs', idxs, ... 'loRows', lo, 'hiRows', hi, ... 'loBBox', min(lo, [], 1), 'hiBBox', max(hi, [], 1)); i = i + numel(idxs); end - nSuperBlocks = numel(superBlocks); + superBlocks = superBlocks(1:nSuperBlocks); % ASYNC PIPELINE. If a parpool is available and asyncPrefetch is % on, the next super-block's read runs concurrently on a pool From 8134a20450a8bafca79bf64bd5a51be9da2f5f3f Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 10:13:36 +0000 Subject: [PATCH 061/110] lightsheetZarrLevel: declare chunk.bin as a DID file series (bare manifest) The ingest crashed at the end of level 1 with: "chunk.bin_#" is not declared as a file series by this document class. Add it to files.file_series in the class definition. Two things wrong here: 1. The schema had "chunk.bin_#" only in file_list, not in file_series. did.document.isFileSeries returns false because file_series is empty, so addFileSeries refuses. 2. did.document's series convention (see the seriesNames doc-comment at document.m:748-753) is that the DECLARED name is the series MANIFEST -- a bare "chunk.bin" -- and its members are automatically chunk.bin_1 ... chunk.bin_N. The trailing "_#" is documentation-only notation, not part of the schema name. Fix on both sides: - lightsheetZarrLevel.json: change file_list from ["chunk.bin_#"] to ["chunk.bin"] and add a matching file_series: ["chunk.bin"]. This now mirrors the DID demoSeries example exactly (a member declared in both lists, with its bare manifest name). - makePyramid.m: call addFileSeries with the bare manifest name ("chunk.bin"), not the "_#" pattern. DID synthesises the member filenames chunk.bin_1..N from the manifest name + the `indices` option we already pass. Members land on disk with exactly the same names the old add_file loop produced, so Python's chunk reader (which filters `current_file_list()` for names that start with "chunk.bin_") is unaffected. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m | 7 ++++++- .../database_documents/data/lightsheetZarrLevel.json | 5 ++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index fe8216d0f..a97c5f2a1 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -578,7 +578,12 @@ % file-series manifest names the series `chunk.bin_#`; `indices` % supplies the 1..N chunk numbering that DID substitutes for `#` % in each stored location. No per-chunk copy-on-write. - levelDoc = levelDoc.addFileSeries('chunk.bin_#', seriesLocations, ... + % DID convention: the series MANIFEST name is bare ("chunk.bin"), not + % the "_#" pattern. Members are named chunk.bin_1 ... chunk.bin_N + % automatically. See did.document.seriesNames docstring for the + % contract. The schema declares "chunk.bin" in both file_list and + % file_series. + levelDoc = levelDoc.addFileSeries('chunk.bin', seriesLocations, ... 'indices', 1:nChunks); end diff --git a/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json index 42461b67c..83c301cf7 100644 --- a/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json +++ b/src/ndi/ndi_common/database_documents/data/lightsheetZarrLevel.json @@ -27,7 +27,10 @@ ], "files": { "file_list": [ - "chunk.bin_#" + "chunk.bin" + ], + "file_series": [ + "chunk.bin" ] }, "lightsheetZarrLevel": { From 0e305da1c0f945d03985126d30880ea4adc29fe4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 10:25:38 +0000 Subject: [PATCH 062/110] lightsheetZarrLevel schema: mirror file_list rename in validation schema The previous commit updated the DATABASE document (definition schema lightsheetZarrLevel.json) to use "chunk.bin" as the file-series manifest name, but missed the parallel entry in the VALIDATION schema (lightsheetZarrLevel_schema.json). The two are separate DID constructs -- the .json is the class's default document contents and declares file_list / file_series; the _schema.json is what did.database.validate_doc_vs_schema checks each added document against. Symptom on CI (20 test errors): Some required files are missing (including chunk.bin_#) from the file_list in document lightsheetZarrLevel doc DID's validator reads the schema's `file` list and asserts that every listed name (or an accepted series member of it) is present in the doc's document_properties.files.file_info. The unupdated validation schema still listed "chunk.bin_#", but the doc's file_info now holds "chunk.bin_1" ... "chunk.bin_N" (from addFileSeries under the "chunk.bin" manifest) and no bare "chunk.bin_#" -- so validation refused every document.add. Change the validation schema's file[].name from "chunk.bin_#" to "chunk.bin" so it matches the manifest declared by the definition schema. Members are picked up automatically by the series-aware validator. Documentation .md files still describe the on-disk pattern as "chunk.bin_#" (which IS what a reader sees for member names 1..N) and are left as-is -- they document behaviour, not schema. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../schema_documents/data/lightsheetZarrLevel_schema.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json index 60aa4ef44..72c891678 100644 --- a/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json +++ b/src/ndi/ndi_common/schema_documents/data/lightsheetZarrLevel_schema.json @@ -19,7 +19,7 @@ ], "file": [ { - "name": "chunk.bin_#", + "name": "chunk.bin", "mustbenotempty": 0 } ], From 71df6b344a6a7fcbb070a6d4047289062362e4ab Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 11:25:31 +0000 Subject: [PATCH 063/110] lightsheet: add test-blob ingest + view demo script Small end-to-end smoke test that builds the synthetic OME-Zarr fixture via ndi.test.lightsheet.ingestBlobFixture, ingests it into a fresh session with the blosc-zstd fast path, and launches the napari viewer on the resulting pyramid. Handy for verifying the ingest -> document -> viewer pipeline without touching the real SmartSPIM data. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- lightsheet_test_blob_ingest_and_view.m | 57 ++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 lightsheet_test_blob_ingest_and_view.m diff --git a/lightsheet_test_blob_ingest_and_view.m b/lightsheet_test_blob_ingest_and_view.m new file mode 100644 index 000000000..974cd2fab --- /dev/null +++ b/lightsheet_test_blob_ingest_and_view.m @@ -0,0 +1,57 @@ +%LIGHTSHEET_TEST_BLOB_INGEST_AND_VIEW - end-to-end smoke test on the synthetic blob +% +% Builds a small OME-Zarr fixture on disk (a couple of MB), ingests it +% into a fresh NDI session with the blosc-zstd fast path, and launches +% the napari viewer on the resulting pyramid. Use this to sanity-check +% the ingest -> document -> viewer pipeline without touching the real +% SmartSPIM data. Runs in under a minute on a laptop. +% +% Edit CONFIG below if you want to keep the fixture / session between +% runs, change the blob size, or point at a different napari launcher. +% +% See also: ndi.test.lightsheet.makeBlobFixture, +% ndi.test.lightsheet.ingestBlobFixture, +% ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.fun.doc.lightsheet.view + +% -------------------- CONFIG ----------------------------------------- +% Leave scratch dirs empty to land under tempname (fresh every run). +sessionDir = ''; % e.g. '~/ndi-sessions/blob-demo' +zarrDir = ''; % e.g. '~/ndi-sessions/blob-demo/zarr' +subjectName = 'blob@vhlab'; + +% Blob size / shape. 128^3 uint16 across 2 channels ~= 16 MB raw; small +% enough for a fast smoke test, big enough that the pyramid has real +% levels to look at. +shape = [128 128 128]; +numChannels = 2; +chunkShape = [32 32 32]; +numLevels = 3; + +% Ingest options. blosc-zstd exercises the encodeChunk fast path; keep +% clevel modest for a smoke test. +codec = 'blosc-zstd'; +clevel = 5; + +% Napari launcher. Same wrapper the real-data script uses. +launcher = '/usr/local/bin/napariViewLightsheet'; +% --------------------------------------------------------------------- + +fprintf('Building blob fixture and ingesting ...\n'); +[S, P, info] = ndi.test.lightsheet.ingestBlobFixture( ... + 'sessionDir', sessionDir, ... + 'zarrDir', zarrDir, ... + 'subject', subjectName, ... + 'Shape', shape, ... + 'NumChannels', numChannels, ... + 'ChunkShape', chunkShape, ... + 'NumLevels', numLevels, ... + 'codec', codec, ... + 'clevel', clevel); + +fprintf('Session: %s\n', char(S.path())); +fprintf('Zarr: %s\n', info.zarrPath); +fprintf('Pyramid: %s\n', P.id()); + +fprintf('Launching napari viewer ...\n'); +ndi.fun.doc.lightsheet.view(S, P.id(), 'launcher', launcher); From a9d1ada0d9709b48acfeb80a0051c4ff3390a6d5 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 11:29:30 +0000 Subject: [PATCH 064/110] lightsheet demo: pass --reduction so napari has one ladder to open The pyramid stores both mean and max; napariViewLightsheet refuses to guess and prints "pass --reduction to pick one." Default the test-blob demo to 'mean' so the script runs end-to-end without a follow-up flag. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- lightsheet_test_blob_ingest_and_view.m | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/lightsheet_test_blob_ingest_and_view.m b/lightsheet_test_blob_ingest_and_view.m index 974cd2fab..093f52e9b 100644 --- a/lightsheet_test_blob_ingest_and_view.m +++ b/lightsheet_test_blob_ingest_and_view.m @@ -35,6 +35,10 @@ % Napari launcher. Same wrapper the real-data script uses. launcher = '/usr/local/bin/napariViewLightsheet'; + +% Which reduction to open ('mean' or 'max'). The pyramid stores both; +% napariViewLightsheet needs one picked at launch. +reduction = 'mean'; % --------------------------------------------------------------------- fprintf('Building blob fixture and ingesting ...\n'); @@ -54,4 +58,6 @@ fprintf('Pyramid: %s\n', P.id()); fprintf('Launching napari viewer ...\n'); -ndi.fun.doc.lightsheet.view(S, P.id(), 'launcher', launcher); +ndi.fun.doc.lightsheet.view(S, P.id(), ... + 'launcher', launcher, ... + 'reduction', reduction); From d36fcc128a48ff81df05a620cf9922d87c8e8855 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 11:31:53 +0000 Subject: [PATCH 065/110] lightsheet demo: drop reduction flag now that CLI defaults to mean napariViewLightsheet's _pick_reduction now falls to 'mean' when a pyramid has both mean and max and --reduction is not passed, so the demo doesn't need to specify one. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- lightsheet_test_blob_ingest_and_view.m | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lightsheet_test_blob_ingest_and_view.m b/lightsheet_test_blob_ingest_and_view.m index 093f52e9b..974cd2fab 100644 --- a/lightsheet_test_blob_ingest_and_view.m +++ b/lightsheet_test_blob_ingest_and_view.m @@ -35,10 +35,6 @@ % Napari launcher. Same wrapper the real-data script uses. launcher = '/usr/local/bin/napariViewLightsheet'; - -% Which reduction to open ('mean' or 'max'). The pyramid stores both; -% napariViewLightsheet needs one picked at launch. -reduction = 'mean'; % --------------------------------------------------------------------- fprintf('Building blob fixture and ingesting ...\n'); @@ -58,6 +54,4 @@ fprintf('Pyramid: %s\n', P.id()); fprintf('Launching napari viewer ...\n'); -ndi.fun.doc.lightsheet.view(S, P.id(), ... - 'launcher', launcher, ... - 'reduction', reduction); +ndi.fun.doc.lightsheet.view(S, P.id(), 'launcher', launcher); From 871b6b97709edd614527f8b1003c52eaa99fb121 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 12:13:38 +0000 Subject: [PATCH 066/110] cloud sync: reconstruct series ingest_locations on SyncFiles=false too A cloud round trip strips series ingest_locations (DID treats them as transient authoring state), so a downloaded document arrives with files.series_info reporting n_present > 0 and no locations. DID's MembersNotLocatable guard (VH-Lab/DID-matlab#185) then refuses that shape on do_add_doc. The SyncFiles=true download path already handled this via updateFileInfoForLocalFiles + reconstructSeriesIngestLocations (NDI-matlab#958), but the SyncFiles=false path -- which defaults to false on ndi.cloud.downloadDataset -- routed docs through updateFileInfoForRemoteFiles, which only patched file_info locations and never touched series_info. First series-bearing document on that path (e.g. any lightsheetZarrLevel) tripped the guard on add_docs. Close the gap: - New collectSeriesManifestUids helper walks the downloaded documents and returns the manifest uids of every series still missing ingest_locations. Small: one uid per series document, so one entry per lightsheetZarrLevel, not per chunk. - downloadNdiDocuments's SyncFiles=false branch pre-fetches just those manifest files (via downloadDatasetFiles) into a scratch folder before running the per-document update, and cleans the folder up on exit. Manifests are a few KB each; the member bytes stay remote. - updateFileInfoForRemoteFiles grows an optional manifestFolder argument. When supplied it runs reconstructSeriesIngestLocations on the document's series_info, producing ndic:// locations of the same shape the SyncFiles=true path produces. Existing two-argument callers still work; the third argument defaults to "" which skips the reconstruction. - Add a FileSeriesRoundTripTest case (testDownloadSyncFilesFalseRebuildsSeriesIngestLocations) that downloads the round-trip fixture with SyncFiles=false, asserts ingest_locations gets rebuilt with the expected ndic:// shape, and reads every member's bytes back through the reconstruction. Every other cloud round-trip test in the suite uses SyncFiles=true, so this failure mode had no coverage. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+internal/collectSeriesManifestUids.m | 84 +++++++++++++++ .../+sync/+internal/downloadNdiDocuments.m | 46 +++++++- .../+internal/updateFileInfoForRemoteFiles.m | 47 +++++++- .../+cloud/FileSeriesRoundTripTest.m | 102 ++++++++++++++++++ 4 files changed, 273 insertions(+), 6 deletions(-) create mode 100644 src/ndi/+ndi/+cloud/+sync/+internal/collectSeriesManifestUids.m diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/collectSeriesManifestUids.m b/src/ndi/+ndi/+cloud/+sync/+internal/collectSeriesManifestUids.m new file mode 100644 index 000000000..e60815c6d --- /dev/null +++ b/src/ndi/+ndi/+cloud/+sync/+internal/collectSeriesManifestUids.m @@ -0,0 +1,84 @@ +function manifestUids = collectSeriesManifestUids(documents) +%COLLECTSERIESMANIFESTUIDS Manifest uids for every series still missing ingest_locations. +% +% MANIFESTUIDS = ndi.cloud.sync.internal.collectSeriesManifestUids(DOCS) +% +% Walk DOCS (cell array of ndi.document) and collect the manifest file +% uid of every ``files.series_info`` entry that (a) reports +% ``n_present > 0`` and (b) carries no ``ingest_locations``. That is +% exactly the shape a cloud-round-tripped document arrives in +% (VH-Lab/NDI-matlab#958) -- and the shape +% ``reconstructSeriesIngestLocations`` fixes when handed the manifest +% bytes. +% +% Returned as a unique, sorted ``string`` array (empty when no +% reconstruction is needed). The caller uses this list to fetch just +% the manifests through ``ndi.cloud.download.downloadDatasetFiles`` +% before running the pure-remote file-info update: manifests are +% small, one per series document, and are the only bytes needed to +% satisfy DID-matlab #185's MembersNotLocatable guard on +% ``add_docs``. +% +% See also: +% ndi.cloud.sync.internal.downloadNdiDocuments +% ndi.cloud.sync.internal.updateFileInfoForRemoteFiles +% ndi.cloud.sync.internal.reconstructSeriesIngestLocations + + manifestUids = string.empty; + + if isempty(documents) + return + end + + if ~iscell(documents) + documents = {documents}; + end + + for i = 1:numel(documents) + doc = documents{i}; + if isempty(doc), continue, end + if ~ismethod(doc, 'has_files') || ~doc.has_files(), continue, end + files = doc.document_properties.files; + if ~isfield(files, 'series_info') || isempty(files.series_info) + continue + end + if ~isfield(files, 'file_info') || isempty(files.file_info) + continue + end + seriesInfo = files.series_info; + for k = 1:numel(seriesInfo) + entry = seriesInfo(k); + if ~isfield(entry, 'n_present') || isempty(entry.n_present) || entry.n_present <= 0 + continue + end + if isfield(entry, 'ingest_locations') && ~isempty(entry.ingest_locations) + continue + end + uid = lookupManifestUid(files.file_info, entry.name); + if ~isempty(uid) + manifestUids(end+1) = string(uid); %#ok + end + end + end + + if isempty(manifestUids) + manifestUids = string.empty; + else + manifestUids = unique(manifestUids); + end +end + +function uid = lookupManifestUid(fileInfo, seriesName) +% The manifest is a plain file_info entry whose `name` matches the +% series' name; its ``locations(1).uid`` is what the cloud knows the +% manifest bytes by. + uid = ''; + if isempty(seriesName), return, end + seriesName = char(seriesName); + for m = 1:numel(fileInfo) + if ~strcmp(char(fileInfo(m).name), seriesName), continue, end + if isempty(fileInfo(m).locations), return, end + uid = char(fileInfo(m).locations(1).uid); + return + end +end diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m b/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m index 066616303..f21df3487 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m @@ -130,8 +130,39 @@ if syncOptions.Verbose fprintf('"SyncFiles" option is false. Updating document file info to reflect remote files.\n'); end + % Cloud storage strips series ingest_locations, so a downloaded + % document whose series has present members but no locations trips + % DID-matlab#185's MembersNotLocatable guard on add_docs. The + % SyncFiles=true path handles this by reading each series' manifest + % out of the fully-downloaded FileSyncLocation; on this path + % (SyncFiles=false) the manifests do not exist locally, and without + % them there is no way to enumerate member uids. Fetch just the + % manifest FILES -- one small file per series document -- into a + % scratch folder here so reconstructSeriesIngestLocations, called + % from the updated updateFileInfoForRemoteFiles, has the bytes it + % needs. The member bytes stay on the cloud; DID resolves them + % on demand via the reconstructed ndic:// locations. See + % NDI-matlab#958 follow-up. + manifestUids = ndi.cloud.sync.internal.collectSeriesManifestUids(newNdiDocuments); + manifestFolder = ""; + manifestCleanup = onCleanup.empty; + if ~isempty(manifestUids) + manifestFolder = string(tempname); + mkdir(manifestFolder); + manifestCleanup = onCleanup(@() cleanupScratchFolder(manifestFolder)); + if syncOptions.Verbose + fprintf(['Fetching %d series manifest file(s) to ' ... + 'reconstruct series ingest_locations...\n'], ... + numel(manifestUids)); + end + ndi.cloud.download.downloadDatasetFiles(... + cloudDatasetId, ... + manifestFolder, ... + manifestUids, ... + "Verbose", syncOptions.Verbose); + end documentUpdateFcn = @(doc) ... - ndi.cloud.sync.internal.updateFileInfoForRemoteFiles(doc, cloudDatasetId); + ndi.cloud.sync.internal.updateFileInfoForRemoteFiles(doc, cloudDatasetId, manifestFolder); end % 3. Update file info for documents based on local / remote location @@ -153,3 +184,16 @@ downloadedNdiDocuments = newNdiDocuments; end end + +function cleanupScratchFolder(folderPath) +% Best-effort teardown of the manifest scratch directory. + folderPath = char(folderPath); + if isempty(folderPath) || ~isfolder(folderPath) + return + end + try + rmdir(folderPath, 's'); + catch + % Nothing to do -- MATLAB's temp cleanup will pick it up later. + end +end diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m index c891bd20e..cc832db77 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m @@ -1,8 +1,9 @@ -function document = updateFileInfoForRemoteFiles(document, cloudDatasetId) +function document = updateFileInfoForRemoteFiles(document, cloudDatasetId, manifestFolder) % updateFileInfoForRemoteFiles - Update file info of document for remote (cloud-only) files % % Syntax: % document = ndi.cloud.sync.internal.updateFileInfoForRemoteFiles(document, cloudDatasetId) +% document = ndi.cloud.sync.internal.updateFileInfoForRemoteFiles(document, cloudDatasetId, manifestFolder) % This function updates the file information in the provided document % object for files that are stored remotely in NDI cloud. % @@ -11,29 +12,65 @@ % 2. set the location field using the template "ndic://{dataset_id}/{file_uid}" % 3. set the location_type field to "ndicloud" % +% When MANIFESTFOLDER is supplied and the document declares any file +% series with n_present > 0, the series' ingest_locations are +% reconstructed from the manifest files staged in MANIFESTFOLDER -- +% one entry per present member, location_type "ndicloud", +% location "ndic://{cloudDatasetId}/{memberUid}". Same DID-matlab #185 +% guard the SyncFiles=true path satisfies, closed on the pure-remote +% path (NDI-matlab#958 follow-up). +% % Input Arguments: -% document - The document object containing file information. -% cloudDatasetId - The unique identifier for the cloud dataset. +% document - The document object containing file information. +% cloudDatasetId - The unique identifier for the cloud dataset. +% manifestFolder - Optional; a directory holding pre-fetched series +% manifest files, keyed by manifest uid, as +% reconstructSeriesIngestLocations expects. Empty (the +% default) skips series reconstruction; a series with +% n_present > 0 will then hit DID's guard on the +% following add_docs, same as before this argument +% existed. % % Output Arguments: % document - The updated document object with modified file info. % % See also: % ndi.cloud.sync.internal.updateFileInfoForLocalFiles +% ndi.cloud.sync.internal.reconstructSeriesIngestLocations + + arguments + document + cloudDatasetId (1,1) string + manifestFolder (1,1) string = "" + end if document.has_files() updatedFileInfo = document.document_properties.files.file_info; - + for i = 1:numel(updatedFileInfo) % Replace/override 1st file location updatedFileInfo(i).locations(1).delete_original = 0; updatedFileInfo(i).locations(1).ingest = 0; - + fileUid = updatedFileInfo(i).locations(1).uid; fileLocation = sprintf('ndic://%s/%s', cloudDatasetId, fileUid); updatedFileInfo(i).locations(1).location = fileLocation; updatedFileInfo(i).locations(1).location_type = 'ndicloud'; end document = document.setproperties('files.file_info', updatedFileInfo); + + % Rebuild series ingest_locations from staged manifests, when the + % caller provided a folder to read them from. The manifest bytes + % themselves still live on the cloud (DID re-fetches at read + % time); MANIFESTFOLDER only exists so we can enumerate member + % uids for the DID-matlab #185 guard. + if strlength(manifestFolder) > 0 ... + && isfield(document.document_properties.files, 'series_info') ... + && ~isempty(document.document_properties.files.series_info) + seriesInfo = document.document_properties.files.series_info; + seriesInfo = ndi.cloud.sync.internal.reconstructSeriesIngestLocations( ... + seriesInfo, updatedFileInfo, manifestFolder, cloudDatasetId); + document = document.setproperties('files.series_info', seriesInfo); + end end end diff --git a/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m b/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m index 14846c5d7..67eed6abe 100644 --- a/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m +++ b/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m @@ -734,6 +734,108 @@ function testTheBatchScopeIsTheSeriesAndNotTheWholeDocument(testCase) "not in the series, so the 'fileSeries' scope was not honored. " + msg); end + function testDownloadSyncFilesFalseRebuildsSeriesIngestLocations(testCase) + % REGRESSION for the SyncFiles=false download path. + % + % DID strips series ingest_locations at store time (it is + % transient authoring state), so a downloaded document arrives + % with series_info recording n_present > 0 and no locations. + % SyncFiles=true worked because updateFileInfoForLocalFiles + % reads the downloaded manifest and reconstructs + % ingest_locations before add_docs. SyncFiles=false went + % straight through updateFileInfoForRemoteFiles, which only + % patched file_info and never touched series_info -- so the + % first series-bearing document tripped DID-matlab#185's + % MembersNotLocatable guard on add_docs. + % + % The fix pre-fetches just each series' manifest bytes into a + % scratch folder before running updateFileInfoForRemoteFiles, + % then reuses reconstructSeriesIngestLocations to build + % ndic:// locations for every present slot. This test drives + % downloadDataset with 'SyncFiles', false and verifies that + % (1) the download completes without hitting the DID guard, + % (2) files.series_info.ingest_locations comes out populated + % on the reconstructed document, and (3) every member can + % still be resolved through database_openbinarydoc on the + % downloaded dataset. + + import matlab.unittest.fixtures.TemporaryFolderFixture + + narrative = testCase.Narrative; + narrative(end+1) = "Begin testDownloadSyncFilesFalseRebuildsSeriesIngestLocations."; + + downloadFixture = testCase.applyFixture(TemporaryFolderFixture); + narrative(end+1) = "Downloading dataset " + testCase.DatasetID + ... + " with SyncFiles=false into a fresh folder."; + downloaded = ndi.cloud.downloadDataset(testCase.DatasetID, downloadFixture.Folder, ... + 'SyncFiles', false, 'Verbose', false); + msg = ndi.unittest.cloud.APIMessage(narrative, ~isempty(downloaded), ... + "downloadDataset returned an ndi.dataset", ... + matlab.net.http.ResponseMessage.empty, "ndi.cloud.downloadDataset (SyncFiles=false)"); + testCase.assertNotEmpty(downloaded, ... + "downloadDataset(SyncFiles=false) returned nothing for the uploaded dataset. " + msg); + + q = ndi.query('base.name', 'exact_string', 'test_series_doc'); + remoteDocs = downloaded.database_search(q); + testCase.assertNumElements(remoteDocs, 1, ... + "the series document did not come back from the SyncFiles=false download."); + remoteDoc = remoteDocs{1}; + + % ingest_locations must be populated: this is what closes the + % DID-matlab#185 guard, so a bare series_info with 0 locations + % would have raised on the database_add above rather than + % reaching this line -- but assert it directly so the + % regression names the field it depends on. + testCase.assertTrue( ... + isfield(remoteDoc.document_properties.files, 'series_info'), ... + "downloaded document lost files.series_info on the way through SyncFiles=false."); + si = remoteDoc.document_properties.files.series_info; + testCase.assertNotEmpty(si, ... + "downloaded document has an empty series_info after SyncFiles=false."); + entryIdx = find(strcmp({si.name}, 'chunkdata.bin'), 1); + testCase.assertNotEmpty(entryIdx, ... + "the reconstructed series_info has no entry for 'chunkdata.bin'."); + entry = si(entryIdx); + testCase.verifyEqual(entry.n_present, testCase.MemberCount, ... + "reconstructed series_info reports the wrong n_present after SyncFiles=false."); + testCase.assertTrue(isfield(entry, 'ingest_locations') && ~isempty(entry.ingest_locations), ... + "SyncFiles=false did not repopulate series_info.ingest_locations; " + ... + "add_docs would have tripped DID-matlab#185 if this test got this far, " + ... + "so the missing state is the bug directly."); + testCase.verifyEqual(numel(entry.ingest_locations), testCase.MemberCount, ... + "reconstructed ingest_locations does not have one entry per present slot."); + locs = entry.ingest_locations; + for k = 1:numel(locs) + testCase.verifyEqual(locs(k).location_type, 'ndicloud', ... + "reconstructed ingest_locations should carry location_type='ndicloud'."); + testCase.verifyTrue(startsWith(locs(k).location, "ndic://" + testCase.DatasetID + "/"), ... + "reconstructed location does not point back to this cloud dataset."); + testCase.verifyEqual(locs(k).ingest, 0, ... + "reconstructed ingest_locations must set ingest=0 (members stay remote)."); + end + + % Read every member's bytes back through the reconstructed + % locations; this closes the loop by proving the ndic:// + % locations resolve to the same content the members were + % uploaded with. If this fails without the assertions above + % having fired, the reconstruction wrote wrong uids into the + % locations rather than not writing them at all. + for i = 1:testCase.MemberCount + memberName = sprintf('chunkdata.bin_%d', i); + fobj = downloaded.database_openbinarydoc(remoteDoc, memberName); + gotPath = fobj.fullpathfilename; + downloaded.database_closebinarydoc(fobj); + fid = fopen(gotPath, 'r'); + got = fread(fid, [1 Inf], 'uint8=>uint8'); + fclose(fid); + testCase.verifyEqual(got, testCase.MemberContent{i}, ... + sprintf(['member %d resolved but returned different bytes than ' ... + 'were uploaded after SyncFiles=false round trip.'], i)); + end + + testCase.Narrative = narrative; + end + end end From b12717225d7b07f8e5da6a66984240040509c473 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 12:16:21 +0000 Subject: [PATCH 067/110] downloadNdiDocuments: drop the dead onCleanup.empty pre-declaration Code Analyzer flagged 'manifestCleanup = onCleanup.empty' as unused (the value never read before its overwrite inside the if branch). It was a defensive pre-declaration for a scope that never sees the variable when the if isn't entered, so remove it. Add %#ok on the real assignment inside the if with a short note on why the local must stay bound (onCleanup fires when its handle goes out of scope, so we need the reference to live until docfun finishes reading the manifest folder). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m b/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m index f21df3487..b2a3223f9 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m @@ -145,11 +145,13 @@ % NDI-matlab#958 follow-up. manifestUids = ndi.cloud.sync.internal.collectSeriesManifestUids(newNdiDocuments); manifestFolder = ""; - manifestCleanup = onCleanup.empty; if ~isempty(manifestUids) manifestFolder = string(tempname); mkdir(manifestFolder); - manifestCleanup = onCleanup(@() cleanupScratchFolder(manifestFolder)); + % onCleanup fires when its handle goes out of scope, so the + % local must stay bound until this function returns to keep + % the manifest folder alive across the docfun call below. + manifestCleanup = onCleanup(@() cleanupScratchFolder(manifestFolder)); %#ok if syncOptions.Verbose fprintf(['Fetching %d series manifest file(s) to ' ... 'reconstruct series ingest_locations...\n'], ... From f89c58eb132377bf850562e3ef1d325947740ea5 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 14 Sep 2026 12:33:58 +0000 Subject: [PATCH 068/110] cloud sync: install series manifests locally on SyncFiles=false too The download-side reconstruction added in the previous commit was only half the story. It repopulated series_info.ingest_locations so add_docs did not trip DID-matlab #185's MembersNotLocatable guard, but left the manifest bytes on the cloud alone. DID's series-member customFileHandler is scoped to MEMBERS -- did.database's own _series_manifest_path only ever calls cached_path_for_uid on the manifest and never falls through to the handler -- so a viewer opening the first member on the pure-remote download hit "the series ... manifest is not on this machine" and drew fill_value. Close the gap on the download side. When manifestFolder holds a series' manifest bytes: - updateFileInfoForRemoteFiles now rebuilds that entry's file_info through reset + add_file, so the manifest first gets a LOCAL location that DID's add_docs copies into the destination database's FileDir, then a second 'ndic://' location as a fallback (mirrors updateFileInfoForLocalFiles's manifest handling on the SyncFiles=true path). Non-manifest files stay cloud-only via the fast in-place location rewrite. - reconstructSeriesIngestLocations is still called on files.series_info so the guard still passes. - Once DID has the manifest in FileDir, DID-python's _series_manifest_path resolves it through cached_path_for_uid and members flow through the customFileHandler as usual. Also drops the stale %#ok on the onCleanup binding in downloadNdiDocuments.m: with the pre-declaration removed in b1271722 Code Analyzer no longer flags this assignment, and Advanced Security now warns that the suppression is unused. Adjusts the SyncFiles=false test case: - Removes the assertion that series_info.ingest_locations is populated on the read-back document. DID strips ingest_locations at store time by design (NDI-matlab#958 header, same as SyncFiles=true), so the reconstruction is consumed by the add_docs guard and then gone by the time we read the doc back. The real acceptance check -- reading every member back through database_openbinarydoc -- was already downstream in the same test method; that is what proves the reconstruction pointed at the correct member uids. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+sync/+internal/downloadNdiDocuments.m | 2 +- .../+internal/updateFileInfoForRemoteFiles.m | 126 +++++++++++++----- .../+cloud/FileSeriesRoundTripTest.m | 41 +++--- 3 files changed, 106 insertions(+), 63 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m b/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m index b2a3223f9..87e0ca5b4 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m @@ -151,7 +151,7 @@ % onCleanup fires when its handle goes out of scope, so the % local must stay bound until this function returns to keep % the manifest folder alive across the docfun call below. - manifestCleanup = onCleanup(@() cleanupScratchFolder(manifestFolder)); %#ok + manifestCleanup = onCleanup(@() cleanupScratchFolder(manifestFolder)); if syncOptions.Verbose fprintf(['Fetching %d series manifest file(s) to ' ... 'reconstruct series ingest_locations...\n'], ... diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m index cc832db77..3dd47ec97 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m @@ -4,32 +4,42 @@ % Syntax: % document = ndi.cloud.sync.internal.updateFileInfoForRemoteFiles(document, cloudDatasetId) % document = ndi.cloud.sync.internal.updateFileInfoForRemoteFiles(document, cloudDatasetId, manifestFolder) -% This function updates the file information in the provided document -% object for files that are stored remotely in NDI cloud. % -% The following changes are made to the file location structure: -% 1. set the 'delete_original' and 'ingest' fields to false. -% 2. set the location field using the template "ndic://{dataset_id}/{file_uid}" -% 3. set the location_type field to "ndicloud" +% Update the file information in DOCUMENT for files that live remotely +% in NDI Cloud. Every file_info entry gets its primary location rewritten +% to "ndic://{cloudDatasetId}/{file_uid}" with location_type "ndicloud", +% ingest=0 and delete_original=0. % -% When MANIFESTFOLDER is supplied and the document declares any file -% series with n_present > 0, the series' ingest_locations are -% reconstructed from the manifest files staged in MANIFESTFOLDER -- -% one entry per present member, location_type "ndicloud", -% location "ndic://{cloudDatasetId}/{memberUid}". Same DID-matlab #185 -% guard the SyncFiles=true path satisfies, closed on the pure-remote -% path (NDI-matlab#958 follow-up). +% SERIES MANIFESTS ARE INSTALLED LOCALLY WHEN AVAILABLE. When +% MANIFESTFOLDER holds the manifest bytes of a series declared by this +% document (file at fullfile(MANIFESTFOLDER, manifest_uid)), the +% manifest entry is rebuilt as two locations -- the local copy first, +% the ndic:// URI second -- mirroring what updateFileInfoForLocalFiles +% does on the SyncFiles=true path. DID's add_docs then copies the +% manifest into the destination database's FileDir, so DID-python's +% _series_manifest_path (which only looks on local disk through +% cached_path_for_uid) can enumerate members WITHOUT a lazy manifest +% fetch. Without the local copy the reader raises +% "the series ... manifest is not on this machine" on the first member +% open, because the DID series-member customFileHandler is scoped to +% MEMBERS, not manifests. +% +% With MANIFESTFOLDER given, series_info.ingest_locations are also +% reconstructed from the same manifest bytes -- one entry per present +% member, ingest=0, location "ndic://{cloudDatasetId}/{member_uid}" -- +% so DID-matlab #185's MembersNotLocatable guard on do_add_doc passes. % % Input Arguments: % document - The document object containing file information. % cloudDatasetId - The unique identifier for the cloud dataset. % manifestFolder - Optional; a directory holding pre-fetched series % manifest files, keyed by manifest uid, as -% reconstructSeriesIngestLocations expects. Empty (the -% default) skips series reconstruction; a series with -% n_present > 0 will then hit DID's guard on the -% following add_docs, same as before this argument -% existed. +% reconstructSeriesIngestLocations expects. Empty +% (the default) keeps the pure in-place rewrite -- +% OK when this document declares no series, but +% fails downstream on a series-bearing document +% (add_docs trips the guard, or the reader can't +% open members). % % Output Arguments: % document - The updated document object with modified file info. @@ -44,33 +54,77 @@ manifestFolder (1,1) string = "" end - if document.has_files() - updatedFileInfo = document.document_properties.files.file_info; + if ~document.has_files() + return + end + + originalFileInfo = document.document_properties.files.file_info; + hasSeriesInfo = isfield(document.document_properties.files, 'series_info') ... + && ~isempty(document.document_properties.files.series_info); + if hasSeriesInfo + originalSeriesInfo = document.document_properties.files.series_info; + end + + % A local manifest copy is available for entry i when MANIFESTFOLDER + % holds a file named by that entry's uid AND the entry is declared as + % a file series on this document (only manifests get installed + % locally on this path). Everything else stays cloud-only. + manifestFolderChar = char(manifestFolder); + installLocallyAt = strings(1, numel(originalFileInfo)); + if strlength(manifestFolder) > 0 + for i = 1:numel(originalFileInfo) + entry = originalFileInfo(i); + if isempty(entry.locations) + continue + end + if ~document.isFileSeries(entry.name) + continue + end + candidate = fullfile(manifestFolderChar, char(entry.locations(1).uid)); + if isfile(candidate) + installLocallyAt(i) = string(candidate); + end + end + end + hasLocals = any(strlength(installLocallyAt) > 0); + + if hasLocals + % One or more manifests to install: use the reset-then-add_file + % path so DID handles the copy into FileDir and the ndic:// alias + % goes on as a second location. + document = document.reset_file_info(); + for i = 1:numel(originalFileInfo) + entry = originalFileInfo(i); + filename = entry.name; + file_uid = entry.locations(1).uid; + cloudLocation = sprintf('ndic://%s/%s', cloudDatasetId, file_uid); + if strlength(installLocallyAt(i)) > 0 + document = document.add_file(filename, char(installLocallyAt(i))); + end + document = document.add_file(filename, cloudLocation); + end + else + % No local copies: fast in-place location rewrite. Preserves the + % legacy shape for docs that declare no series (or when nothing + % was fetched to install). + updatedFileInfo = originalFileInfo; for i = 1:numel(updatedFileInfo) - % Replace/override 1st file location updatedFileInfo(i).locations(1).delete_original = 0; updatedFileInfo(i).locations(1).ingest = 0; - fileUid = updatedFileInfo(i).locations(1).uid; - fileLocation = sprintf('ndic://%s/%s', cloudDatasetId, fileUid); - updatedFileInfo(i).locations(1).location = fileLocation; + updatedFileInfo(i).locations(1).location = ... + sprintf('ndic://%s/%s', cloudDatasetId, fileUid); updatedFileInfo(i).locations(1).location_type = 'ndicloud'; end document = document.setproperties('files.file_info', updatedFileInfo); + end - % Rebuild series ingest_locations from staged manifests, when the - % caller provided a folder to read them from. The manifest bytes - % themselves still live on the cloud (DID re-fetches at read - % time); MANIFESTFOLDER only exists so we can enumerate member - % uids for the DID-matlab #185 guard. - if strlength(manifestFolder) > 0 ... - && isfield(document.document_properties.files, 'series_info') ... - && ~isempty(document.document_properties.files.series_info) - seriesInfo = document.document_properties.files.series_info; - seriesInfo = ndi.cloud.sync.internal.reconstructSeriesIngestLocations( ... - seriesInfo, updatedFileInfo, manifestFolder, cloudDatasetId); - document = document.setproperties('files.series_info', seriesInfo); + if hasSeriesInfo + if strlength(manifestFolder) > 0 + originalSeriesInfo = ndi.cloud.sync.internal.reconstructSeriesIngestLocations( ... + originalSeriesInfo, originalFileInfo, manifestFolder, cloudDatasetId); end + document = document.setproperties('files.series_info', originalSeriesInfo); end end diff --git a/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m b/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m index 67eed6abe..83a177427 100644 --- a/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m +++ b/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m @@ -781,11 +781,15 @@ function testDownloadSyncFilesFalseRebuildsSeriesIngestLocations(testCase) "the series document did not come back from the SyncFiles=false download."); remoteDoc = remoteDocs{1}; - % ingest_locations must be populated: this is what closes the - % DID-matlab#185 guard, so a bare series_info with 0 locations - % would have raised on the database_add above rather than - % reaching this line -- but assert it directly so the - % regression names the field it depends on. + % Structural check: the series record still names its members + % (count / n_present survive the sqlite store). ingest_locations + % is INTENTIONALLY not asserted here: DID treats it as transient + % authoring state and strips it at store time (see the docstring + % on updateFileInfoForLocalFiles and the reconstruct helper's + % header), so a downloaded doc that reached this line has + % already had its reconstructed locations consumed by the + % add_docs guard and then wiped. The real check is the read + % loop below. testCase.assertTrue( ... isfield(remoteDoc.document_properties.files, 'series_info'), ... "downloaded document lost files.series_info on the way through SyncFiles=false."); @@ -798,28 +802,13 @@ function testDownloadSyncFilesFalseRebuildsSeriesIngestLocations(testCase) entry = si(entryIdx); testCase.verifyEqual(entry.n_present, testCase.MemberCount, ... "reconstructed series_info reports the wrong n_present after SyncFiles=false."); - testCase.assertTrue(isfield(entry, 'ingest_locations') && ~isempty(entry.ingest_locations), ... - "SyncFiles=false did not repopulate series_info.ingest_locations; " + ... - "add_docs would have tripped DID-matlab#185 if this test got this far, " + ... - "so the missing state is the bug directly."); - testCase.verifyEqual(numel(entry.ingest_locations), testCase.MemberCount, ... - "reconstructed ingest_locations does not have one entry per present slot."); - locs = entry.ingest_locations; - for k = 1:numel(locs) - testCase.verifyEqual(locs(k).location_type, 'ndicloud', ... - "reconstructed ingest_locations should carry location_type='ndicloud'."); - testCase.verifyTrue(startsWith(locs(k).location, "ndic://" + testCase.DatasetID + "/"), ... - "reconstructed location does not point back to this cloud dataset."); - testCase.verifyEqual(locs(k).ingest, 0, ... - "reconstructed ingest_locations must set ingest=0 (members stay remote)."); - end - % Read every member's bytes back through the reconstructed - % locations; this closes the loop by proving the ndic:// - % locations resolve to the same content the members were - % uploaded with. If this fails without the assertions above - % having fired, the reconstruction wrote wrong uids into the - % locations rather than not writing them at all. + % Reading every member back is the acceptance test: it proves + % add_docs did not trip DID-matlab#185 (or the test would have + % thrown, not reached this loop) AND that the ndic:// locations + % point at bytes matching what was uploaded. A wrong uid or a + % lost cloud reference here means the reconstruction wrote the + % wrong locations rather than none. for i = 1:testCase.MemberCount memberName = sprintf('chunkdata.bin_%d', i); fobj = downloaded.database_openbinarydoc(remoteDoc, memberName); From ae0c9721c57e7cd56b0e2e0322349c218c4702a8 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 14:11:39 +0000 Subject: [PATCH 069/110] lightsheet: end-to-end blob cloud round-trip demo function One-call helper that stages the whole path in the OS temp directory: build the blob fixture, ingest into a session, view it, create a dataset and add_ingested_session, upload with 'uploadAsNew' + 'skipMetadataEditorMetadata' + 'remoteDatasetName' set to "Blob Test Dataset ", download into a third temp dir, open the first session on the downloaded dataset, search for a lightsheetZarrPyramid, view that. Wraps the DID-matlab#201 / NDI-matlab#986 handler-fetch cloud path that just landed on main. SyncFiles defaults false on the download, so the manifest arrives via fetchSeriesManifestBytes on the first member open. Returns an info struct with both sessions, both pyramid ids, and all temp paths for post-hoc inspection. Requires NDI_CLOUD_USERNAME / NDI_CLOUD_PASSWORD in the shell before invoking; ViewLocal / ViewCloud flags can be flipped off individually for a headless run. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- lightsheet_blob_cloud_roundtrip.m | 244 ++++++++++++++++++++++++++++++ 1 file changed, 244 insertions(+) create mode 100644 lightsheet_blob_cloud_roundtrip.m diff --git a/lightsheet_blob_cloud_roundtrip.m b/lightsheet_blob_cloud_roundtrip.m new file mode 100644 index 000000000..4f1109510 --- /dev/null +++ b/lightsheet_blob_cloud_roundtrip.m @@ -0,0 +1,244 @@ +function info = lightsheet_blob_cloud_roundtrip(options) +%LIGHTSHEET_BLOB_CLOUD_ROUNDTRIP - end-to-end cloud round trip for the blob fixture. +% +% INFO = LIGHTSHEET_BLOB_CLOUD_ROUNDTRIP(...) +% +% Full pipeline in one call, staged entirely in the OS temp directory +% so a run leaves no artefacts behind on the caller's project tree: +% +% a) Fresh temp directories for the session, the dataset, and the +% eventual download. +% b) Build a small synthetic OME-Zarr fixture on disk via +% ndi.test.lightsheet.makeBlobFixture. +% c) Open an ndi.session on the session temp dir and ingest the +% fixture into it via ndi.fun.doc.lightsheet.fromOMEZarr (writes +% the lightsheetZarrPyramid + level documents). +% d) Launch napariViewLightsheet on the just-built pyramid. +% e) Create an ndi.dataset.dir on the dataset temp dir and attach the +% session via add_ingested_session (a lightsheet-only session is +% trivially ingested -- it declares no daq systems, so +% isIngested() returns true without a further ingest() call). +% f) ndi.cloud.uploadDataset with 'uploadAsNew', 'skipMetadataEditorMetadata', +% and 'remoteDatasetName' set to "Blob Test Dataset ". +% g) ndi.cloud.downloadDataset the cloud dataset back into a THIRD +% temp dir (SyncFiles defaults to false, which now works end-to-end +% via DID-matlab#201's fetchSeriesManifestBytes contract). +% h) List the sessions on the downloaded dataset and open the first +% one. +% i) Search that session for lightsheetZarrPyramid documents and +% launch napariViewLightsheet on the first hit. +% +% Name-Value Arguments: +% Launcher - char, path to napariViewLightsheet. Default +% '/usr/local/bin/napariViewLightsheet'. +% Shape - fixture spatial shape, default [128 128 128]. +% NumChannels - default 2. +% ChunkShape - default [32 32 32]. +% NumLevels - default 3. +% Codec - 'raw' or 'blosc-zstd'. Default 'blosc-zstd' (exercises +% the encodeChunk MEX fast path). +% Clevel - default 5. +% Reduction - 'mean' or 'max'. Default 'mean'. +% ViewLocal - launch napari on the local session too (step d). +% Default true. Set false to skip the local view and go +% straight to the cloud path. +% ViewCloud - launch napari on the downloaded session (step i). +% Default true. Both viewers block for a moment while +% they open; if you are running headless, set both to +% false. +% WaitViewer - forward to ndi.fun.doc.lightsheet.view. Default false +% (viewer backgrounds; MATLAB stays interactive). +% Verbose - print progress narration. Default true. +% +% Returns a struct with everything the run produced: +% .sessionDir - the session's temp dir +% .zarrPath - the on-disk OME-Zarr store +% .datasetDir - the local dataset's temp dir +% .downloadDir - where downloadDataset copied the cloud state +% .cloudDatasetId - the id returned by uploadDataset +% .cloudDatasetName - the "Blob Test Dataset ..." label sent up +% .localSession - the freshly-built session (post-ingest) +% .localPyramidId - the pyramid document id from step (c) +% .downloadedDataset - the ndi.dataset.dir opened from downloadDir +% .downloadedSession - the first session on the downloaded dataset +% .downloadedPyramidId - the pyramid id we opened in step (i) +% +% Set NDI_CLOUD_USERNAME / NDI_CLOUD_PASSWORD before calling; the upload +% and download steps will fail without them. +% +% See also: ndi.test.lightsheet.makeBlobFixture, +% ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.fun.doc.lightsheet.view, +% ndi.cloud.uploadDataset, +% ndi.cloud.downloadDataset + + arguments + options.Launcher (1,:) char = '/usr/local/bin/napariViewLightsheet' + options.Shape (1,3) double {mustBePositive, mustBeInteger} = [128 128 128] + options.NumChannels (1,1) double {mustBePositive, mustBeInteger} = 2 + options.ChunkShape (1,3) double {mustBePositive, mustBeInteger} = [32 32 32] + options.NumLevels (1,1) double {mustBePositive, mustBeInteger} = 3 + options.Codec (1,:) char {mustBeMember(options.Codec, {'raw','blosc-zstd'})} = 'blosc-zstd' + options.Clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.Clevel,1), mustBeLessThanOrEqual(options.Clevel,9)} = 5 + options.Reduction (1,:) char {mustBeMember(options.Reduction, {'mean','max'})} = 'mean' + options.ViewLocal (1,1) logical = true + options.ViewCloud (1,1) logical = true + options.WaitViewer (1,1) logical = false + options.Verbose (1,1) logical = true + end + + say = @(varargin) verboseNarrator(options.Verbose, varargin{:}); + + % --- (a) three fresh temp directories ------------------------------- + sessionDir = mustMakeTempDir(); + datasetDir = mustMakeTempDir(); + downloadDir = mustMakeTempDir(); + zarrParent = mustMakeTempDir(); + say('Temp session : %s', sessionDir); + say('Temp dataset : %s', datasetDir); + say('Temp download : %s', downloadDir); + say('Temp zarr root : %s', zarrParent); + + % --- (b) build the blob fixture ------------------------------------- + say('Building blob fixture ([shape=%dx%dx%d, channels=%d, chunks=%dx%dx%d, levels=%d])...', ... + options.Shape(1), options.Shape(2), options.Shape(3), options.NumChannels, ... + options.ChunkShape(1), options.ChunkShape(2), options.ChunkShape(3), ... + options.NumLevels); + [zarrPath, ~] = ndi.test.lightsheet.makeBlobFixture(zarrParent, ... + 'Shape', options.Shape, ... + 'NumChannels', options.NumChannels, ... + 'ChunkShape', options.ChunkShape, ... + 'NumLevels', options.NumLevels); + say('Zarr fixture : %s', zarrPath); + + % --- (c) ingest the fixture into a fresh session -------------------- + say('Opening session on %s ...', sessionDir); + S = ndi.session.dir('blob', sessionDir); + + subjectName = 'blob@vhlab'; + sub = ndi.document('subject', ... + 'base.session_id', S.id(), ... + 'subject.local_identifier', subjectName); + S.database_add(sub); + say('Added subject document "%s".', subjectName); + + say('Running fromOMEZarr (codec=%s, clevel=%d)...', options.Codec, options.Clevel); + [P, ~, ~] = ndi.fun.doc.lightsheet.fromOMEZarr(S, zarrPath, ... + 'subjectID', sub.id(), ... + 'materializeChunks', true, ... + 'codec', options.Codec, ... + 'clevel', options.Clevel); + say('Built lightsheetZarrPyramid %s.', P.id()); + + % --- (d) view the local pyramid ------------------------------------- + if options.ViewLocal + say('Launching napari viewer on the LOCAL session ...'); + ndi.fun.doc.lightsheet.view(S, P.id(), ... + 'launcher', options.Launcher, ... + 'reduction', options.Reduction, ... + 'wait', options.WaitViewer); + else + say('Skipping local view (ViewLocal=false).'); + end + + % --- (e) make a dataset, ingest the session ------------------------- + say('Building dataset at %s ...', datasetDir); + D = ndi.dataset.dir('blob_dataset', datasetDir); + if ~S.isIngested() + error('lightsheet_blob_cloud_roundtrip:notIngested', ... + ['Session %s is not fully ingested. A lightsheet-only session with no ' ... + 'daq systems should be trivially ingested; check whether an unexpected ' ... + 'daq system got attached during setup.'], S.id()); + end + D = D.add_ingested_session(S); + say('Added session %s to dataset (ingested).', S.id()); + + % --- (f) upload to NDI Cloud ---------------------------------------- + cloudDatasetName = buildRemoteDatasetName(); + say('Uploading to NDI Cloud as remote name "%s" ...', cloudDatasetName); + [ok, cloudDatasetId, msg] = ndi.cloud.uploadDataset(D, ... + 'uploadAsNew', true, ... + 'skipMetadataEditorMetadata', true, ... + 'remoteDatasetName', cloudDatasetName); + if ~ok + error('lightsheet_blob_cloud_roundtrip:uploadFailed', ... + 'ndi.cloud.uploadDataset failed: %s', msg); + end + say('Cloud dataset id: %s', cloudDatasetId); + + % --- (g) download it back into a fresh temp dir --------------------- + say('Downloading dataset back to %s (SyncFiles defaults false; the ' ... + 'DID-matlab#201 handler-fetch path installs manifests on demand) ...', downloadDir); + Ddown = ndi.cloud.downloadDataset(cloudDatasetId, downloadDir); + say('Downloaded dataset opened at %s.', char(Ddown.path())); + + % --- (h) list and open the first session ---------------------------- + [refs, ids] = Ddown.session_list(); + if isempty(ids) + error('lightsheet_blob_cloud_roundtrip:noSessions', ... + 'Downloaded dataset carries no sessions; upload must have dropped the ingested session.'); + end + say('Downloaded dataset lists %d session(s): %s', numel(ids), strjoin(refs, ', ')); + firstId = ids{1}; + say('Opening the first session (%s) ...', firstId); + Sdown = Ddown.open_session(firstId); + + % --- (i) find the pyramid doc and open it --------------------------- + q = ndi.query('','isa','lightsheetZarrPyramid'); + pyramidDocs = Sdown.database_search(q); + if isempty(pyramidDocs) + error('lightsheet_blob_cloud_roundtrip:noPyramid', ... + 'No lightsheetZarrPyramid documents on the downloaded session %s.', firstId); + end + downloadedPyramidId = pyramidDocs{1}.id(); + say('Found lightsheetZarrPyramid %s on the downloaded session.', downloadedPyramidId); + + if options.ViewCloud + say('Launching napari viewer on the DOWNLOADED session ...'); + ndi.fun.doc.lightsheet.view(Sdown, downloadedPyramidId, ... + 'launcher', options.Launcher, ... + 'reduction', options.Reduction, ... + 'wait', options.WaitViewer); + else + say('Skipping cloud view (ViewCloud=false).'); + end + + info = struct( ... + 'sessionDir', sessionDir, ... + 'zarrPath', zarrPath, ... + 'datasetDir', datasetDir, ... + 'downloadDir', downloadDir, ... + 'cloudDatasetId', cloudDatasetId, ... + 'cloudDatasetName', cloudDatasetName, ... + 'localSession', S, ... + 'localPyramidId', P.id(), ... + 'downloadedDataset', Ddown, ... + 'downloadedSession', Sdown, ... + 'downloadedPyramidId', downloadedPyramidId); + say('Round trip complete.'); +end + +% ===================================================================== + +function d = mustMakeTempDir() + d = tempname; + if ~mkdir(d) + error('lightsheet_blob_cloud_roundtrip:tempMkdirFailed', ... + 'Could not create temp directory at %s.', d); + end +end + +function name = buildRemoteDatasetName() + % "Blob Test Dataset yyyymmddTHHMMSS RRR" where RRR is 0-1000 inclusive. + % Compact timestamp avoids embedded spaces / colons the cloud UI is + % happier with; still round-trips as a valid dataset label. + dstr = datestr(datetime('now'), 'yyyymmddTHHMMSS'); %#ok + r = randi([0 1000]); + name = sprintf('Blob Test Dataset %s %d', dstr, r); +end + +function verboseNarrator(verbose, fmt, varargin) + if verbose + fprintf(['[roundtrip] ' fmt '\n'], varargin{:}); + end +end From cadca4a809a28797ec9e834ae89de0c5271969e3 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 14:18:48 +0000 Subject: [PATCH 070/110] lightsheet: move the cloud round-trip demo into ndi.test.cloud Manual cloud tests belong under +ndi/+test/+cloud/, alongside the other cloud-manual scripts, so users invoke this as ndi.test.cloud.lightsheet_blob_cloud_roundtrip(...) rather than picking up a stray script from the repo root. Content unchanged; just relocated and re-headed to use the fully-qualified name. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) rename lightsheet_blob_cloud_roundtrip.m => src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m (96%) diff --git a/lightsheet_blob_cloud_roundtrip.m b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m similarity index 96% rename from lightsheet_blob_cloud_roundtrip.m rename to src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m index 4f1109510..9a2807abd 100644 --- a/lightsheet_blob_cloud_roundtrip.m +++ b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m @@ -1,7 +1,11 @@ function info = lightsheet_blob_cloud_roundtrip(options) -%LIGHTSHEET_BLOB_CLOUD_ROUNDTRIP - end-to-end cloud round trip for the blob fixture. +%NDI.TEST.CLOUD.LIGHTSHEET_BLOB_CLOUD_ROUNDTRIP - end-to-end cloud round trip for the blob fixture. % -% INFO = LIGHTSHEET_BLOB_CLOUD_ROUNDTRIP(...) +% INFO = ndi.test.cloud.LIGHTSHEET_BLOB_CLOUD_ROUNDTRIP(...) +% +% Manual test (not part of the automated suite). Set +% NDI_CLOUD_USERNAME / NDI_CLOUD_PASSWORD in the shell, then call +% this from a MATLAB prompt to smoke-test the whole cloud pipeline. % % Full pipeline in one call, staged entirely in the OS temp directory % so a run leaves no artefacts behind on the caller's project tree: From 6af807c0ca5edb4bb1751e57b8991ba6eaf70d1b Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 14:21:00 +0000 Subject: [PATCH 071/110] lightsheet demo: bracket the split format string on the download say() Without the [ ] MATLAB parses the two adjacent literals as separate arguments to say(); sprintf then sees '...manifests on demand) ...' as its second positional and downloadDir as its third, so the %s never fills and the second literal prints verbatim to stderr as an unused-arg warning. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m index 9a2807abd..cfbdf6bdd 100644 --- a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m +++ b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m @@ -171,8 +171,8 @@ say('Cloud dataset id: %s', cloudDatasetId); % --- (g) download it back into a fresh temp dir --------------------- - say('Downloading dataset back to %s (SyncFiles defaults false; the ' ... - 'DID-matlab#201 handler-fetch path installs manifests on demand) ...', downloadDir); + say(['Downloading dataset back to %s (SyncFiles defaults false; the ' ... + 'DID-matlab#201 handler-fetch path installs manifests on demand) ...'], downloadDir); Ddown = ndi.cloud.downloadDataset(cloudDatasetId, downloadDir); say('Downloaded dataset opened at %s.', char(Ddown.path())); From 4181ea3083eeefe8bc77108992066ed0ece0ba13 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 14:28:38 +0000 Subject: [PATCH 072/110] lightsheet demo: rename say -> narrate to avoid path-shadow, use nested fn The initial version defined say as an anonymous function bound to options.Verbose (say = @(varargin) verboseNarrator(...)). When a user has an unrelated say.m on their MATLAB path, MATLAB's parser resolves `say(...)` inside the function to the path function, then reads the paren-and-comma call as a matrix-construction expression -- the exact "To construct matrices, use brackets instead of parentheses" error the user hit at the first call site. Fix: replace with a nested function narrate() that closes over a local verbose flag. Nested functions are declared inline within the parent so MATLAB's parser resolves narrate() to the nested body, and rename also drops the path-shadow risk entirely. Retire the old verboseNarrator helper (now unused). Also drop the () on Ddown.path -- path is a property on ndi.dataset.dir, not a method, and property-access-with-parens has had inconsistent behaviour across MATLAB releases. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+cloud/lightsheet_blob_cloud_roundtrip.m | 60 +++++++++---------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m index cfbdf6bdd..f85621050 100644 --- a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m +++ b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m @@ -91,20 +91,25 @@ options.Verbose (1,1) logical = true end - say = @(varargin) verboseNarrator(options.Verbose, varargin{:}); + verbose = options.Verbose; + function narrate(fmt, varargin) + if verbose + fprintf(['[roundtrip] ' fmt '\n'], varargin{:}); + end + end % --- (a) three fresh temp directories ------------------------------- sessionDir = mustMakeTempDir(); datasetDir = mustMakeTempDir(); downloadDir = mustMakeTempDir(); zarrParent = mustMakeTempDir(); - say('Temp session : %s', sessionDir); - say('Temp dataset : %s', datasetDir); - say('Temp download : %s', downloadDir); - say('Temp zarr root : %s', zarrParent); + narrate('Temp session : %s', sessionDir); + narrate('Temp dataset : %s', datasetDir); + narrate('Temp download : %s', downloadDir); + narrate('Temp zarr root : %s', zarrParent); % --- (b) build the blob fixture ------------------------------------- - say('Building blob fixture ([shape=%dx%dx%d, channels=%d, chunks=%dx%dx%d, levels=%d])...', ... + narrate('Building blob fixture ([shape=%dx%dx%d, channels=%d, chunks=%dx%dx%d, levels=%d])...', ... options.Shape(1), options.Shape(2), options.Shape(3), options.NumChannels, ... options.ChunkShape(1), options.ChunkShape(2), options.ChunkShape(3), ... options.NumLevels); @@ -113,10 +118,10 @@ 'NumChannels', options.NumChannels, ... 'ChunkShape', options.ChunkShape, ... 'NumLevels', options.NumLevels); - say('Zarr fixture : %s', zarrPath); + narrate('Zarr fixture : %s', zarrPath); % --- (c) ingest the fixture into a fresh session -------------------- - say('Opening session on %s ...', sessionDir); + narrate('Opening session on %s ...', sessionDir); S = ndi.session.dir('blob', sessionDir); subjectName = 'blob@vhlab'; @@ -124,29 +129,29 @@ 'base.session_id', S.id(), ... 'subject.local_identifier', subjectName); S.database_add(sub); - say('Added subject document "%s".', subjectName); + narrate('Added subject document "%s".', subjectName); - say('Running fromOMEZarr (codec=%s, clevel=%d)...', options.Codec, options.Clevel); + narrate('Running fromOMEZarr (codec=%s, clevel=%d)...', options.Codec, options.Clevel); [P, ~, ~] = ndi.fun.doc.lightsheet.fromOMEZarr(S, zarrPath, ... 'subjectID', sub.id(), ... 'materializeChunks', true, ... 'codec', options.Codec, ... 'clevel', options.Clevel); - say('Built lightsheetZarrPyramid %s.', P.id()); + narrate('Built lightsheetZarrPyramid %s.', P.id()); % --- (d) view the local pyramid ------------------------------------- if options.ViewLocal - say('Launching napari viewer on the LOCAL session ...'); + narrate('Launching napari viewer on the LOCAL session ...'); ndi.fun.doc.lightsheet.view(S, P.id(), ... 'launcher', options.Launcher, ... 'reduction', options.Reduction, ... 'wait', options.WaitViewer); else - say('Skipping local view (ViewLocal=false).'); + narrate('Skipping local view (ViewLocal=false).'); end % --- (e) make a dataset, ingest the session ------------------------- - say('Building dataset at %s ...', datasetDir); + narrate('Building dataset at %s ...', datasetDir); D = ndi.dataset.dir('blob_dataset', datasetDir); if ~S.isIngested() error('lightsheet_blob_cloud_roundtrip:notIngested', ... @@ -155,11 +160,11 @@ 'daq system got attached during setup.'], S.id()); end D = D.add_ingested_session(S); - say('Added session %s to dataset (ingested).', S.id()); + narrate('Added session %s to dataset (ingested).', S.id()); % --- (f) upload to NDI Cloud ---------------------------------------- cloudDatasetName = buildRemoteDatasetName(); - say('Uploading to NDI Cloud as remote name "%s" ...', cloudDatasetName); + narrate('Uploading to NDI Cloud as remote name "%s" ...', cloudDatasetName); [ok, cloudDatasetId, msg] = ndi.cloud.uploadDataset(D, ... 'uploadAsNew', true, ... 'skipMetadataEditorMetadata', true, ... @@ -168,13 +173,13 @@ error('lightsheet_blob_cloud_roundtrip:uploadFailed', ... 'ndi.cloud.uploadDataset failed: %s', msg); end - say('Cloud dataset id: %s', cloudDatasetId); + narrate('Cloud dataset id: %s', cloudDatasetId); % --- (g) download it back into a fresh temp dir --------------------- - say(['Downloading dataset back to %s (SyncFiles defaults false; the ' ... + narrate(['Downloading dataset back to %s (SyncFiles defaults false; the ' ... 'DID-matlab#201 handler-fetch path installs manifests on demand) ...'], downloadDir); Ddown = ndi.cloud.downloadDataset(cloudDatasetId, downloadDir); - say('Downloaded dataset opened at %s.', char(Ddown.path())); + narrate('Downloaded dataset opened at %s.', char(Ddown.path)); % --- (h) list and open the first session ---------------------------- [refs, ids] = Ddown.session_list(); @@ -182,9 +187,9 @@ error('lightsheet_blob_cloud_roundtrip:noSessions', ... 'Downloaded dataset carries no sessions; upload must have dropped the ingested session.'); end - say('Downloaded dataset lists %d session(s): %s', numel(ids), strjoin(refs, ', ')); + narrate('Downloaded dataset lists %d session(s): %s', numel(ids), strjoin(refs, ', ')); firstId = ids{1}; - say('Opening the first session (%s) ...', firstId); + narrate('Opening the first session (%s) ...', firstId); Sdown = Ddown.open_session(firstId); % --- (i) find the pyramid doc and open it --------------------------- @@ -195,16 +200,16 @@ 'No lightsheetZarrPyramid documents on the downloaded session %s.', firstId); end downloadedPyramidId = pyramidDocs{1}.id(); - say('Found lightsheetZarrPyramid %s on the downloaded session.', downloadedPyramidId); + narrate('Found lightsheetZarrPyramid %s on the downloaded session.', downloadedPyramidId); if options.ViewCloud - say('Launching napari viewer on the DOWNLOADED session ...'); + narrate('Launching napari viewer on the DOWNLOADED session ...'); ndi.fun.doc.lightsheet.view(Sdown, downloadedPyramidId, ... 'launcher', options.Launcher, ... 'reduction', options.Reduction, ... 'wait', options.WaitViewer); else - say('Skipping cloud view (ViewCloud=false).'); + narrate('Skipping cloud view (ViewCloud=false).'); end info = struct( ... @@ -219,7 +224,7 @@ 'downloadedDataset', Ddown, ... 'downloadedSession', Sdown, ... 'downloadedPyramidId', downloadedPyramidId); - say('Round trip complete.'); + narrate('Round trip complete.'); end % ===================================================================== @@ -241,8 +246,3 @@ name = sprintf('Blob Test Dataset %s %d', dstr, r); end -function verboseNarrator(verbose, fmt, varargin) - if verbose - fprintf(['[roundtrip] ' fmt '\n'], varargin{:}); - end -end From db0311ffa255542e282ef48d4ee0395a6f6e8372 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 14:41:17 +0000 Subject: [PATCH 073/110] lightsheet demo: clear the readblankdefinition cache on entry ndi.document.readblankdefinition holds a persistent containers.Map of parsed JSON per document type. If a MATLAB session ever loaded the pre-fix lightsheetZarrLevel schema (before chunk.bin became a DID file_series), later constructions inherit the stale definition and addFileSeries('chunk.bin', ...) refuses with "chunk.bin is not declared as a file series". Bit the round-trip demo the first time in a session where the schema had been touched earlier. Call the '--clear-cache' sentinel at the top of the demo, wrapped in try/catch so an older NDI without the sentinel does not stop the run. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+cloud/lightsheet_blob_cloud_roundtrip.m | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m index f85621050..9a30ad4c6 100644 --- a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m +++ b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m @@ -98,6 +98,22 @@ function narrate(fmt, varargin) end end + % ndi.document.readblankdefinition keeps a persistent containers.Map + % of parsed JSON per document type. A MATLAB session that had the + % pre-fix lightsheetZarrLevel schema loaded (before chunk.bin became + % a DID file_series) hands the stale definition to every later + % construction, and addFileSeries('chunk.bin', ...) then refuses the + % write with "chunk.bin is not declared as a file series". Nuke the + % cache up front so a re-run of this demo cannot inherit a stale + % session state. + try + ndi.document.readblankdefinition('--clear-cache'); + catch + % Older NDI without the sentinel; safe to skip -- the definition + % cache is a persistent inside the same file, so a `clear + % functions` at the caller resolves the same way. + end + % --- (a) three fresh temp directories ------------------------------- sessionDir = mustMakeTempDir(); datasetDir = mustMakeTempDir(); From e1303026e850e4d6b783bc51e84ccf66f5ea3c9c Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 14:46:25 +0000 Subject: [PATCH 074/110] lightsheet demo: dump viewer stderr + NDI_LIGHTSHEET_DEBUG to a file Adds DebugLogFile option (default fullfile(tempdir, 'NapariLightSheetDebugging.txt')) and replaces the two ndi.fun.doc.lightsheet.view calls with a local launchDebugView helper that: - Builds the command via ndi.fun.doc.lightsheet.viewCommand. - Writes a header (tag, session path, pyramid id, full command) into the log so both viewer invocations from one run read as a single narrative. - Prepends NDI_LIGHTSHEET_DEBUG=1 to the environment so the Python side prints [lightsheet] chunk-fetch failure lines when a member cannot be resolved. - Redirects stdout+stderr to the log with >> and backgrounds on POSIX when WaitViewer is false, so MATLAB stays interactive and the log fills as napari runs. The log truncates on demo entry (initDebugLog) so a rerun starts with a clean file; the file gets a start-time header on entry and a per-viewer separator on each launch. Motivation: cloud viewer showed a solid fill_value canvas on the downloaded session, and the round-trip demo did not surface the underlying failure (which is where the Python side's chunk-fetch retry / manifest-not-local error lands). With the log wired in, the user can grep the file for 'the series "chunk.bin" manifest is not on this machine' after a black cloud open and know exactly which DID-python change is still missing. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+cloud/lightsheet_blob_cloud_roundtrip.m | 84 +++++++++++++++++-- 1 file changed, 76 insertions(+), 8 deletions(-) diff --git a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m index 9a30ad4c6..93f084df6 100644 --- a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m +++ b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m @@ -89,6 +89,7 @@ options.ViewCloud (1,1) logical = true options.WaitViewer (1,1) logical = false options.Verbose (1,1) logical = true + options.DebugLogFile (1,:) char = fullfile(tempdir, 'NapariLightSheetDebugging.txt') end verbose = options.Verbose; @@ -155,13 +156,18 @@ function narrate(fmt, varargin) 'clevel', options.Clevel); narrate('Built lightsheetZarrPyramid %s.', P.id()); + % Set up the shared debug log. Truncate on entry so a run's log is + % just that run's log; both viewer subprocesses append their + % NDI_LIGHTSHEET_DEBUG stderr into it with clear === LOCAL === / + % === CLOUD === separators so the file reads as one narrative. + initDebugLog(options.DebugLogFile); + narrate('Napari debug log: %s', options.DebugLogFile); + % --- (d) view the local pyramid ------------------------------------- if options.ViewLocal narrate('Launching napari viewer on the LOCAL session ...'); - ndi.fun.doc.lightsheet.view(S, P.id(), ... - 'launcher', options.Launcher, ... - 'reduction', options.Reduction, ... - 'wait', options.WaitViewer); + launchDebugView(S, P.id(), options.Launcher, options.Reduction, ... + options.WaitViewer, options.DebugLogFile, 'LOCAL'); else narrate('Skipping local view (ViewLocal=false).'); end @@ -220,10 +226,8 @@ function narrate(fmt, varargin) if options.ViewCloud narrate('Launching napari viewer on the DOWNLOADED session ...'); - ndi.fun.doc.lightsheet.view(Sdown, downloadedPyramidId, ... - 'launcher', options.Launcher, ... - 'reduction', options.Reduction, ... - 'wait', options.WaitViewer); + launchDebugView(Sdown, downloadedPyramidId, options.Launcher, ... + options.Reduction, options.WaitViewer, options.DebugLogFile, 'CLOUD'); else narrate('Skipping cloud view (ViewCloud=false).'); end @@ -262,3 +266,67 @@ function narrate(fmt, varargin) name = sprintf('Blob Test Dataset %s %d', dstr, r); end +function initDebugLog(logfile) +% Truncate the shared debug log so a rerun starts clean, then write a +% header row so the file is never zero bytes (Finder / editors show it +% cleanly). The log lands in an OS-writable dir by default (tempdir), +% but any caller-supplied path works as long as its parent exists. + parent = fileparts(logfile); + if ~isempty(parent) && ~isfolder(parent) + mkdir(parent); + end + fid = fopen(logfile, 'w'); + if fid < 0 + error('lightsheet_blob_cloud_roundtrip:debugLogOpen', ... + 'Cannot open debug log %s for writing.', logfile); + end + fprintf(fid, '=== napariViewLightsheet debug log ===\n'); + fprintf(fid, 'started at %s\n\n', char(datetime('now'))); + fclose(fid); +end + +function launchDebugView(session, pyramidId, launcher, reduction, waitForIt, logfile, tag) +% Build the viewer command via ndi.fun.doc.lightsheet.viewCommand, then +% run it via system() with NDI_LIGHTSHEET_DEBUG=1 in the environment +% and stderr+stdout appended to logfile. Bypasses +% ndi.fun.doc.lightsheet.view because that helper does not expose a +% redirect hook -- we want the Python side's [lightsheet] chunk fetch +% failures written where the user can grep them after the run. +% +% TAG is a short label (e.g. 'LOCAL' / 'CLOUD') that lands as a +% separator in the log, so a single log file can hold both viewer +% invocations from one run. + + sessionPath = char(session.path); + cmd = ndi.fun.doc.lightsheet.viewCommand(launcher, sessionPath, pyramidId, ... + 'reduction', reduction); + + % Append a header into the log identifying this invocation. + fid = fopen(logfile, 'a'); + if fid >= 0 + fprintf(fid, '\n=== %s : %s ===\n', tag, char(datetime('now'))); + fprintf(fid, 'session : %s\n', sessionPath); + fprintf(fid, 'pyramid : %s\n', pyramidId); + fprintf(fid, 'command : %s\n\n', cmd); + fclose(fid); + end + + % Route Python-side debug prints (NDI_LIGHTSHEET_DEBUG=1 triggers + % the [lightsheet] chunk-fetch failure lines) into the log file. + % Redirect both stdout and stderr; the launcher wrapper's own + % output lands in the same file so a chunk miss and its cause + % (auth, HTTP status, missing manifest) sit adjacent. + quotedLog = ['''' strrep(logfile, '''', '''\''''') '''']; + if ~waitForIt && ~ispc + fullCmd = ['NDI_LIGHTSHEET_DEBUG=1 ' cmd ' >>' quotedLog ' 2>&1 &']; + else + fullCmd = ['NDI_LIGHTSHEET_DEBUG=1 ' cmd ' >>' quotedLog ' 2>&1']; + end + fprintf('Executing: %s\n', fullCmd); + status = system(fullCmd); + if status ~= 0 && waitForIt + warning('lightsheet_blob_cloud_roundtrip:viewerNonZeroExit', ... + 'Viewer returned status %d; see %s for stderr.', status, logfile); + end +end + From 3d68539f72c8c8519c8b22180f3bacdc07515efd Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 14:48:08 +0000 Subject: [PATCH 075/110] lightsheet: zsh helper to locate + cat the napari debug log Companion to the DebugLogFile option on ndi.test.cloud.lightsheet_blob_cloud_roundtrip. Default writes to fullfile(tempdir, 'NapariLightSheetDebugging.txt') which on macOS is under $TMPDIR (typically /var/folders/xx/yyy/T/), not /tmp -- easy to miss with a bare 'cat /tmp/...'. Script checks the usual spots ($TMPDIR/, /tmp/, $HOME/, /private/tmp/), falls back to a bounded find under /var/folders and /tmp, picks the newest match, prints the path it chose to stderr, and cats the file. Accepts a positional override for a custom DebugLogFile. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- find_napari_debug_log.sh | 65 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100755 find_napari_debug_log.sh diff --git a/find_napari_debug_log.sh b/find_napari_debug_log.sh new file mode 100755 index 000000000..967f93513 --- /dev/null +++ b/find_napari_debug_log.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env zsh +# find_napari_debug_log.sh - locate and print the napariViewLightsheet debug +# log written by ndi.test.cloud.lightsheet_blob_cloud_roundtrip. +# +# The demo defaults DebugLogFile to fullfile(tempdir, 'NapariLightSheetDebugging.txt') +# which on macOS resolves under $TMPDIR (typically /var/folders/xx/yyy/T/), +# not /tmp. This script looks in the usual spots, picks the newest match, +# and cats it. + +set -u +setopt NULL_GLOB EXTENDED_GLOB + +name="NapariLightSheetDebugging.txt" + +# Places to check, ordered by likelihood on macOS. +locations=( + "${TMPDIR:-}/$name" # matches MATLAB's tempdir + "/tmp/$name" # a user who set DebugLogFile explicitly + "$HOME/$name" # or dropped it in their home + "/private/tmp/$name" # macOS's real /tmp target +) + +# De-duplicate, drop empties, and keep only paths that actually exist. +typeset -a found +for p in "${locations[@]}"; do + [[ -n "$p" && -f "$p" ]] && found+=("$p") +done + +# Wider net if the shortlist misses: recurse under /var/folders and $TMPDIR +# (macOS parks per-user tempdirs there). Prune common noise; keep the +# recent ones. +if (( ${#found} == 0 )); then + for root in "${TMPDIR:-}" /var/folders /private/var/folders /tmp; do + [[ -n "$root" && -d "$root" ]] || continue + while IFS= read -r hit; do + [[ -n "$hit" ]] && found+=("$hit") + done < <(find "$root" -maxdepth 6 -type f -name "$name" 2>/dev/null) + done +fi + +if (( ${#found} == 0 )); then + print -u2 "No $name found. If your demo ran with a custom DebugLogFile," + print -u2 "point me at it: $0 /full/path/to/logfile" + print -u2 "Or check the value of tempdir in MATLAB (>> tempdir)." + exit 1 +fi + +# Pick the newest match if multiple. +newest="$found[1]" +newest_mtime=$(stat -f %m "$newest" 2>/dev/null || stat -c %Y "$newest" 2>/dev/null) +for f in "$found[@]"; do + m=$(stat -f %m "$f" 2>/dev/null || stat -c %Y "$f" 2>/dev/null) + if [[ -n "$m" && "$m" -gt "$newest_mtime" ]]; then + newest="$f" + newest_mtime="$m" + fi +done + +print -u2 "=== Reading: $newest ===" +if (( ${#found} > 1 )); then + print -u2 "(also found: ${(j:, :)found[2,-1]})" +fi +print -u2 "" + +cat -- "$newest" From f2686132438489d1da5898a97d6e680306f2af46 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 14:49:36 +0000 Subject: [PATCH 076/110] lightsheet demo: after the cloud viewer, tail the debug log inline After launching the cloud viewer, pause 10 s to let napari attempt its first chunk fetches, then `tail -n 100` the DebugLogFile and print the result under a clear separator on the MATLAB console. A black canvas now comes with its cause visible in the same prompt, no separate 'go find the log' step. Only runs when WaitViewer is false (non-blocking launch); a blocking viewer keeps the launcher on the foreground, so nothing has been written to the log yet at the point we'd read it. tail runs via system() so a large log costs one subprocess, not a MATLAB-side file read. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+cloud/lightsheet_blob_cloud_roundtrip.m | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m index 93f084df6..ec2ed2f72 100644 --- a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m +++ b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m @@ -228,6 +228,17 @@ function narrate(fmt, varargin) narrate('Launching napari viewer on the DOWNLOADED session ...'); launchDebugView(Sdown, downloadedPyramidId, options.Launcher, ... options.Reduction, options.WaitViewer, options.DebugLogFile, 'CLOUD'); + % Give napari + the customFileHandler a moment to try the first + % chunk fetches, then surface the tail so a black canvas comes + % with its cause on the MATLAB console instead of just a + % pointer to the log file. Only runs on non-blocking viewers + % (WaitViewer=false) -- otherwise the launcher hasn't + % returned yet and we'd read an empty log. + if ~options.WaitViewer + narrate('Sleeping 10 s so napari can attempt its first chunk fetches ...'); + pause(10); + dumpLogTail(options.DebugLogFile, 100); + end else narrate('Skipping cloud view (ViewCloud=false).'); end @@ -285,6 +296,22 @@ function initDebugLog(logfile) fclose(fid); end +function dumpLogTail(logfile, nlines) +% Read the last NLINES of LOGFILE and print them to the MATLAB console +% under a clear separator. Runs via `tail -n ` in a +% subprocess so a huge log costs one syscall, not a MATLAB-side file +% read. Falls back silently if the log does not exist yet. + if ~isfile(logfile) + fprintf('[roundtrip] debug log %s not written yet; nothing to dump.\n', logfile); + return; + end + quotedLog = ['''' strrep(logfile, '''', '''\''''') '''']; + tailCmd = sprintf('tail -n %d %s', nlines, quotedLog); + fprintf('\n===== last %d lines of %s =====\n', nlines, logfile); + system(tailCmd); + fprintf('===== end of debug log tail =====\n\n'); +end + function launchDebugView(session, pyramidId, launcher, reduction, waitForIt, logfile, tag) % Build the viewer command via ndi.fun.doc.lightsheet.viewCommand, then % run it via system() with NDI_LIGHTSHEET_DEBUG=1 in the environment From ad89791e413642e758534c312937bfb51fa8fd58 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 16 Sep 2026 17:49:07 +0000 Subject: [PATCH 077/110] lightsheet demo: wait for server-side extraction before download Add ndi.cloud.api.files.waitForAllBulkUploads between the upload and the download. The server's zip extraction has to finish before a download can see the ingested-session's per-file objects; otherwise Ddown.session_list() came back empty on a fast download and the demo failed at step (h) with "Downloaded dataset carries no sessions". Matches the Python round-trip test in NDI-python (tests/test_cloud_file_series_round_trip.py), which pins the same wait on the same boundary. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+test/+cloud/lightsheet_blob_cloud_roundtrip.m | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m index ec2ed2f72..472b56155 100644 --- a/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m +++ b/src/ndi/+ndi/+test/+cloud/lightsheet_blob_cloud_roundtrip.m @@ -197,6 +197,20 @@ function narrate(fmt, varargin) end narrate('Cloud dataset id: %s', cloudDatasetId); + % Server-side zip extraction has to finish before a download can see + % the ingested-session's per-file objects. Without this wait, + % Ddown.session_list() can come back empty on a fast download -- + % the race the Python round-trip test guards against with the same + % call (see tests/test_cloud_file_series_round_trip.py). + narrate('Waiting for server-side bulk-upload extraction to finish ...'); + [waitOk, waitInfo] = ndi.cloud.api.files.waitForAllBulkUploads(cloudDatasetId); + if ~waitOk + error('lightsheet_blob_cloud_roundtrip:extractionIncomplete', ... + 'waitForAllBulkUploads did not confirm completion (state=%s, elapsed=%.1fs).', ... + waitInfo.state, waitInfo.elapsed); + end + narrate('Bulk-upload extraction complete (%.1fs).', waitInfo.elapsed); + % --- (g) download it back into a fresh temp dir --------------------- narrate(['Downloading dataset back to %s (SyncFiles defaults false; the ' ... 'DID-matlab#201 handler-fetch path installs manifests on demand) ...'], downloadDir); From b88d056444368a5858dd123855b28dd0aea9541d Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 15:05:16 +0000 Subject: [PATCH 078/110] cloud upload: progress feedback during pre-upload file inventory Three signals so a slow spinning drive's inventory pass does not look hung: - After Step 2, print "Document upload complete." - Before Step 3, print "Looking for files (walks each document's series members; can be slow on a spinning drive) ..." - Inside list_binary_files' series member walk, print a throttled progress line at roughly a five-second cadence naming the doc, the series, slots scanned so far, and members found. A final line at the end of the series is emitted when either any progress line ran, or the series held at least 1000 slots (a large enough series is worth acknowledging even when its scan finished under the interval, and a 1-4 file series stays silent). The throttle uses toc(tic) rather than a per-slot counter so slots that stat quickly stay quiet and slots that stat slowly get their proportional share of updates, without the print itself becoming the bottleneck at 100 stat/s. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+cloud/uploadDataset.m | 4 ++++ .../+database/+internal/list_binary_files.m | 21 +++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/src/ndi/+ndi/+cloud/uploadDataset.m b/src/ndi/+ndi/+cloud/uploadDataset.m index 24f1f4930..7c9f8f79f 100644 --- a/src/ndi/+ndi/+cloud/uploadDataset.m +++ b/src/ndi/+ndi/+cloud/uploadDataset.m @@ -96,8 +96,12 @@ if syncOptions.Verbose, disp('Examining documents for upload...'); end dataset_documents = ndiDataset.database_search( ndi.query('','isa','base') ); ndi.cloud.upload.uploadDocumentCollection(cloudDatasetId, dataset_documents, "onlyUploadMissing", true); + if syncOptions.Verbose, disp('Document upload complete.'); end % Step 3: Upload files + if syncOptions.Verbose + disp('Looking for files (walks each document''s series members; can be slow on a spinning drive) ...'); + end [success_upload, message_upload] = ndi.cloud.sync.internal.uploadFilesForDatasetDocuments( ... cloudDatasetId, ndiDataset, dataset_documents, ... "Verbose", syncOptions.Verbose, ... diff --git a/src/ndi/+ndi/+database/+internal/list_binary_files.m b/src/ndi/+ndi/+database/+internal/list_binary_files.m index 28d39d1bd..3a28c7c9f 100644 --- a/src/ndi/+ndi/+database/+internal/list_binary_files.m +++ b/src/ndi/+ndi/+database/+internal/list_binary_files.m @@ -110,6 +110,15 @@ 'docid', '', 'bytes', 0, 'file_path', ''), 1, slotCount); member_count = 0; + % Time-throttled progress: one line every ~5 s, plus a + % final line at the end of the series. A stat-per-slot + % pass through a lightsheet pyramid level is minutes of + % work on a spinning drive, and without this the whole + % pre-upload inventory looks hung. + progressInterval = 5; + lastReport = tic; + showedProgress = false; + for m = 1:slotCount member_name = sprintf('%s_%d', series_name, m); [member_exists, member_path] = ... @@ -126,6 +135,18 @@ 'name', member_name, 'uid', member_uid, ... 'docid', ndi_document_id, 'bytes', member_info.bytes, ... 'file_path', member_path); + + if verbose && toc(lastReport) >= progressInterval + fprintf(' doc %d/%d: series ''%s'' -- %d/%d slots scanned, %d members so far ...\n', ... + i, num_documents, series_name, m, slotCount, member_count); + lastReport = tic; + showedProgress = true; + end + end + + if verbose && (showedProgress || slotCount >= 1000) + fprintf(' doc %d/%d: series ''%s'' -- %d/%d slots scanned, %d members found.\n', ... + i, num_documents, series_name, slotCount, slotCount, member_count); end file_manifest = [file_manifest member_entries(1:member_count)]; %#ok From 470fb59445b94e448c27adb86218649989c38851 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 20:36:08 +0000 Subject: [PATCH 079/110] cloud upload: standard NDI progress bar for inventory and batch upload Three pieces so the pre-upload inventory and the batch upload look the same to the user, and drop redundant filesystem work: 1. list_binary_files -- add ndi.gui.component.ProgressBarWindow for the top-level document walk (one bar), plus a nested bar per large series (>=1000 slots) with time-throttled updates every ~5 s. Same "NDI tasks" window the serial upload branch uses, so bars stack in one place. Replaces the fprintf progress the previous commit added. 2. zipForUpload -- replace the plain MATLAB waitbar with a ProgressBarWindow bar in the same "NDI tasks" window. Batch and serial branches now share the same UI. 3. Remove the redundant per-file isfile() check in zipForUpload. list_binary_files already resolved each entry through database_existbinarydoc + dir() when the manifest was built moments ago; the second stat per file was pure duplication (three stats per file counting the zip/log dir() calls, four with the pre-existing fprintf-verbose branch). If a file vanishes between manifest and zip time, MATLAB's zip() raises inside zipAndUploadBatch's try/catch and the batch is reported as failed rather than silently skipped -- the guard against #805 that isfile+skipped_files used to enforce. skipped_log.csv is no longer written since there is no longer any "missing at zip time" bookkeeping to log. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+cloud/+upload/zipForUpload.m | 83 +++++++------------ .../+database/+internal/list_binary_files.m | 55 ++++++++---- 2 files changed, 68 insertions(+), 70 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+upload/zipForUpload.m b/src/ndi/+ndi/+cloud/+upload/zipForUpload.m index f56788457..126c8b05f 100644 --- a/src/ndi/+ndi/+cloud/+upload/zipForUpload.m +++ b/src/ndi/+ndi/+cloud/+upload/zipForUpload.m @@ -38,8 +38,7 @@ files_left = numel(files_to_process); files_uploaded_count = 0; processed_bytes = 0; -base_dir = fullfile(D.path, '.ndi', 'files'); -skipped_files = {}; +base_dir = fullfile(D.path, '.ndi', 'files'); %#ok if options.Verbose fprintf('Beginning upload process. %d files to upload.\n', files_left); @@ -50,10 +49,12 @@ log_folder = ndi.common.PathConstants.LogFolder; if ~isfolder(log_folder), mkdir(log_folder); end - % Erase previous logs by opening in write mode 'wt' and writing headers - log_files_to_clear = {'zip_log.csv', 'processed_log.csv', 'skipped_log.csv'}; - % **Updated header for zip_log.csv** - headers = {'ZipFile,ZippedFile,UncompressedBytes', 'TotalProcessedBytes', 'SkippedFile,UID'}; + % Erase previous logs by opening in write mode 'wt' and writing headers. + % skipped_log.csv is no longer written: the pre-zip isfile() check was + % removed as redundant with list_binary_files' manifest, so there is no + % "missing at zip time" bookkeeping to log. + log_files_to_clear = {'zip_log.csv', 'processed_log.csv'}; + headers = {'ZipFile,ZippedFile,UncompressedBytes', 'TotalProcessedBytes'}; for k = 1:numel(log_files_to_clear) fid = fopen(fullfile(log_folder, log_files_to_clear{k}), 'wt'); @@ -67,11 +68,28 @@ end % --- Progress Bar Setup --- -h = waitbar(0, 'Preparing to upload files...'); -cleanupObj = onCleanup(@() delete(h(ishandle(h)))); +% Standard NDI progress bar, same "NDI tasks" window the serial upload +% branch uses so bars stack in one place. Auto=true removes it at 1.0. +progressApp = ndi.gui.component.ProgressBarWindow('NDI tasks'); +uploadBarId = did.ido.unique_id(); +progressApp.addBar( ... + 'Label', sprintf('Uploading document-associated binary files (0 of %d)', files_left), ... + 'tag', uploadBarId, ... + 'Auto', true); + size_limit = options.SizeLimit; file_processed = false(1, numel(files_to_process)); % Track processed files +% NOTE: no per-file isfile() check here. list_binary_files already +% resolved each entry through database_existbinarydoc + dir() when the +% manifest was built moments ago, so a redundant stat per file (four +% total per file with the zip/log dir() calls below) costs real time on +% a slow drive and adds nothing. If a file vanishes between manifest +% and zip time, MATLAB's zip() raises inside zipAndUploadBatch's +% try/catch and the batch is reported as failed rather than silently +% skipped -- the guard against issue #805 that isfile+skipped_files +% used to enforce. + % --- Main Loop: Continue until all files are processed --- while ~all(file_processed) current_batch_size = 0; @@ -85,16 +103,6 @@ file_path = current_file.file_path; file_bytes = current_file.bytes; - % --- Logic Check: Ensure file exists on disk --- - if ~isfile(file_path) - if options.Verbose - warning('File %s (UID: %s) not found on disk. Skipping.', current_file.file_path, current_file.uid); - end - skipped_files{end+1} = current_file; - file_processed(i) = true; % Mark as processed to skip - continue; - end - % --- Batching Logic --- if (current_batch_size + file_bytes <= size_limit) || isempty(files_for_current_batch) files_for_current_batch{end+1} = file_path; @@ -132,15 +140,12 @@ end % --- Update Progress Bar --- - try - progress = sum(file_processed) / files_left; - message = sprintf('Processed %d of %d files...', sum(file_processed), files_left); - waitbar(progress, h, message); - catch - b = 0; msg = 'Upload cancelled by user.'; return; - end + progress = sum(file_processed) / files_left; + progressApp.updateBar(uploadBarId, progress); end +progressApp.updateBar(uploadBarId, 1); % Auto=true removes it + % --- Final Logging --- if options.DebugLog log_folder = ndi.common.PathConstants.LogFolder; @@ -152,34 +157,6 @@ fprintf(fid, '%d\n', processed_bytes); fclose(fid); end - - % Log skipped files - if ~isempty(skipped_files) - skipped_log_file = fullfile(log_folder, 'skipped_log.csv'); - fid = fopen(skipped_log_file, 'at'); % Append skipped files - if fid ~= -1 - for k = 1:numel(skipped_files) - fprintf(fid, '"%s","%s"\n', skipped_files{k}.file_path, skipped_files{k}.uid); - end - fclose(fid); - end - end -end - -% --- Report skipped files as a failure --- -% Files that were missing from disk are skipped during batching above. A -% skipped file means a binary that the manifest expected to upload did not -% get uploaded, so report this as a failure rather than silently succeeding. -% Otherwise the caller would record the associated documents as fully synced -% when their binaries are not on the remote (issue #805). -if ~isempty(skipped_files) - b = 0; - skipped_uids = cellfun(@(s) string(s.uid), skipped_files); - msg = sprintf(['%d file(s) were not uploaded because they were missing ', ... - 'from disk (UIDs: %s).'], numel(skipped_files), strjoin(skipped_uids, ', ')); - if options.Verbose - fprintf('%s\n', msg); - end end if options.Verbose diff --git a/src/ndi/+ndi/+database/+internal/list_binary_files.m b/src/ndi/+ndi/+database/+internal/list_binary_files.m index 3a28c7c9f..1aaaae560 100644 --- a/src/ndi/+ndi/+database/+internal/list_binary_files.m +++ b/src/ndi/+ndi/+database/+internal/list_binary_files.m @@ -53,11 +53,21 @@ % keep it open until finished. [db_cleanup_obj, ~] = ndi_dataset.open_database(); %#ok - for i = 1:num_documents + % Standard NDI progress bar for the top-level document walk. A large + % lightsheet dataset's inventory pass is minutes of work on a spinning + % drive; without a bar, the whole pre-upload phase looks hung. Same + % "NDI tasks" window the serial upload branch uses, so bars stack in + % one place. Auto=true removes the bar when it reaches 1.0. + if verbose + progressApp = ndi.gui.component.ProgressBarWindow('NDI tasks'); + topBarId = did.ido.unique_id(); + progressApp.addBar( ... + 'Label', sprintf('Scanning documents for files (0 of %d)', num_documents), ... + 'tag', topBarId, ... + 'Auto', true); + end - if verbose && (mod(i, 1000)==0 || i == num_documents) - fprintf('Working on document %d of %d\n', i, num_documents) - end + for i = 1:num_documents ndi_document_id = database_documents{i}.document_properties.base.id; @@ -110,14 +120,24 @@ 'docid', '', 'bytes', 0, 'file_path', ''), 1, slotCount); member_count = 0; - % Time-throttled progress: one line every ~5 s, plus a - % final line at the end of the series. A stat-per-slot - % pass through a lightsheet pyramid level is minutes of - % work on a spinning drive, and without this the whole - % pre-upload inventory looks hung. + % Nested progress bar per large series, so a single doc + % whose scan takes minutes shows its own progress rather + % than freezing the top-level bar. Small series stay + % silent. Time-throttled updates (~5 s) keep the print + % itself from becoming the bottleneck at 100+ stat/s. + seriesBarId = ''; + hasSeriesBar = false; progressInterval = 5; lastReport = tic; - showedProgress = false; + if verbose && slotCount >= 1000 + seriesBarId = did.ido.unique_id(); + progressApp.addBar( ... + 'Label', sprintf(' doc %d/%d: series ''%s'' (%d slots)', ... + i, num_documents, series_name, slotCount), ... + 'tag', seriesBarId, ... + 'Auto', true); + hasSeriesBar = true; + end for m = 1:slotCount member_name = sprintf('%s_%d', series_name, m); @@ -136,21 +156,22 @@ 'docid', ndi_document_id, 'bytes', member_info.bytes, ... 'file_path', member_path); - if verbose && toc(lastReport) >= progressInterval - fprintf(' doc %d/%d: series ''%s'' -- %d/%d slots scanned, %d members so far ...\n', ... - i, num_documents, series_name, m, slotCount, member_count); + if hasSeriesBar && toc(lastReport) >= progressInterval + progressApp.updateBar(seriesBarId, m/slotCount); lastReport = tic; - showedProgress = true; end end - if verbose && (showedProgress || slotCount >= 1000) - fprintf(' doc %d/%d: series ''%s'' -- %d/%d slots scanned, %d members found.\n', ... - i, num_documents, series_name, slotCount, slotCount, member_count); + if hasSeriesBar + progressApp.updateBar(seriesBarId, 1); % Auto=true removes it end file_manifest = [file_manifest member_entries(1:member_count)]; %#ok end end + + if verbose + progressApp.updateBar(topBarId, i/num_documents); + end end end From 044c6a0ab1c1fed9f7e72c86f10f0570d33cc4f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 20:40:45 +0000 Subject: [PATCH 080/110] zipForUpload: single-pass batching, O(N_files) instead of O(batches*N) The previous batching loop restarted from index 1 on every batch and walked the whole files_to_process array looking for the next unbatched file, so total work grew as O(N_batches * N_files). At 160k files and 50 MB batches that was hundreds of millions of iterations of a "processed?" check, on top of the isfile() stat the previous commit removed. Rewrite as a cursor-based single pass. Greedy-pack in input order: every file is accepted into the current batch until the next file would push over size_limit; then the batch is closed and the next batch starts at that file. Every file is examined exactly once across all batches, so total work is O(N_files). A file larger than size_limit still goes up alone (the first file of a batch is always accepted, matching the previous behaviour). Sizes come from files_to_process(i).bytes, which list_binary_files already populated from its dir() call, so no filesystem work happens in the batching loop at all. Cell-array growth is out too: each batch's paths are sliced from files_to_process in one call, rather than {end+1}-ed one at a time. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+cloud/+upload/zipForUpload.m | 90 +++++++++++----------- 1 file changed, 43 insertions(+), 47 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+upload/zipForUpload.m b/src/ndi/+ndi/+cloud/+upload/zipForUpload.m index 126c8b05f..f2887b6cd 100644 --- a/src/ndi/+ndi/+cloud/+upload/zipForUpload.m +++ b/src/ndi/+ndi/+cloud/+upload/zipForUpload.m @@ -78,7 +78,8 @@ 'Auto', true); size_limit = options.SizeLimit; -file_processed = false(1, numel(files_to_process)); % Track processed files +n_files = numel(files_to_process); +file_bytes_all = [files_to_process.bytes]; % scalar array, indexed below % NOTE: no per-file isfile() check here. list_binary_files already % resolved each entry through database_existbinarydoc + dir() when the @@ -90,58 +91,53 @@ % skipped -- the guard against issue #805 that isfile+skipped_files % used to enforce. -% --- Main Loop: Continue until all files are processed --- -while ~all(file_processed) - current_batch_size = 0; - files_for_current_batch = {}; - indices_for_current_batch = []; - - % --- Inner Loop: Find files that fit into the current batch --- - for i = 1:numel(files_to_process) - if ~file_processed(i) - current_file = files_to_process(i); - file_path = current_file.file_path; - file_bytes = current_file.bytes; - - % --- Batching Logic --- - if (current_batch_size + file_bytes <= size_limit) || isempty(files_for_current_batch) - files_for_current_batch{end+1} = file_path; - current_batch_size = current_batch_size + file_bytes; - indices_for_current_batch(end+1) = i; - if current_batch_size >= size_limit - break; - end - end +% --- Main Loop: Single pass through the file list -------------------- +% Cursor-based, greedy pack in the input order. Each file is examined +% exactly once across all batches, so the total work is O(N_files) +% instead of the O(N_batches * N_files) the previous restart-from-top +% design imposed -- at 160k files and batches of ~50 MB, that mattered. +% A file too large for the remaining budget of the current batch closes +% the batch and starts the next one at that file; a single file bigger +% than size_limit still goes up alone (the first file in a batch is +% always accepted). +cursor = 1; +files_processed_so_far = 0; +while cursor <= n_files + batch_start = cursor; + batch_end = cursor; % inclusive; grown below + current_batch_size = file_bytes_all(cursor); + cursor = cursor + 1; + while cursor <= n_files + fb = file_bytes_all(cursor); + if current_batch_size + fb > size_limit + break; + end + current_batch_size = current_batch_size + fb; + batch_end = cursor; + cursor = cursor + 1; + if current_batch_size >= size_limit + break; end end - % --- Upload the batch if it contains any files --- - if ~isempty(files_for_current_batch) - [success, batch_msg, uploaded_count] = zipAndUploadBatch(files_for_current_batch, dataset_id, options.numberRetries, options); - files_uploaded_count = files_uploaded_count + uploaded_count; + % Slice out just this batch's paths (no cell growth in a loop). + batch_indices = batch_start:batch_end; + files_for_current_batch = {files_to_process(batch_indices).file_path}; - if success - file_processed(indices_for_current_batch) = true; % Mark files as processed - else - b = 0; - files_not_uploaded = files_left - sum(file_processed); - msg = sprintf('%s\n%d files were successfully uploaded. %d files were not uploaded.', ... - batch_msg, files_uploaded_count, files_not_uploaded); - return; - end - else - % No files could be added to a batch, break to avoid infinite loop - % If there are still unprocessed files, it's an unexpected state - if ~all(file_processed) - msg = 'An unexpected error occurred: unable to process remaining files.'; - b = 0; - return; - end + [success, batch_msg, uploaded_count] = zipAndUploadBatch( ... + files_for_current_batch, dataset_id, options.numberRetries, options); + files_uploaded_count = files_uploaded_count + uploaded_count; + + if ~success + b = 0; + files_not_uploaded = files_left - files_processed_so_far; + msg = sprintf('%s\n%d files were successfully uploaded. %d files were not uploaded.', ... + batch_msg, files_uploaded_count, files_not_uploaded); + return; end - % --- Update Progress Bar --- - progress = sum(file_processed) / files_left; - progressApp.updateBar(uploadBarId, progress); + files_processed_so_far = files_processed_so_far + numel(batch_indices); + progressApp.updateBar(uploadBarId, files_processed_so_far / files_left); end progressApp.updateBar(uploadBarId, 1); % Auto=true removes it From f847d7c44a7f54af536b95f3d75e564e2be8ac6a Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 20:43:32 +0000 Subject: [PATCH 081/110] uploadDataset: route to a named organization Add organizationName / organizationID (mutually exclusive) name-value options to ndi.cloud.uploadDataset for users who belong to more than one organization. Names are matched case-sensitively against the list ndi.cloud.api.users.me returns; an unknown or ambiguous name errors with the list of organizations that ARE available, so the caller can see what to type. Implementation: a small helper ndi.cloud.internal.resolveOrganizationId converts name or id to an id (verifying membership), and uploadDataset sets NDI_CLOUD_ORGANIZATION_ID for the duration of the call. onCleanup restores the prior value on exit -- normal return or error -- so a later call in the same MATLAB session inherits neither our resolution nor a partial state. Without these options, uploadDataset keeps its existing behaviour: NDI_CLOUD_ORGANIZATION_ID is whatever the login flow set (the first organization the login response listed, which is what the "Login response contained N organizations; using the first" warning comes from). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+cloud/+internal/resolveOrganizationId.m | 82 +++++++++++++++++++ src/ndi/+ndi/+cloud/uploadDataset.m | 37 ++++++++- 2 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 src/ndi/+ndi/+cloud/+internal/resolveOrganizationId.m diff --git a/src/ndi/+ndi/+cloud/+internal/resolveOrganizationId.m b/src/ndi/+ndi/+cloud/+internal/resolveOrganizationId.m new file mode 100644 index 000000000..28b0acff0 --- /dev/null +++ b/src/ndi/+ndi/+cloud/+internal/resolveOrganizationId.m @@ -0,0 +1,82 @@ +function organizationId = resolveOrganizationId(options) +% RESOLVEORGANIZATIONID - Resolve an organization name or id to an id. +% +% ORGANIZATIONID = ndi.cloud.internal.resolveOrganizationId( ... +% 'organizationName', ) resolves the given name against the +% organizations the current user belongs to (via ndi.cloud.api.users.me). +% +% ORGANIZATIONID = ndi.cloud.internal.resolveOrganizationId( ... +% 'organizationID', ) verifies the given id is one the current +% user belongs to and returns it. +% +% Exactly one of the two must be non-empty. Names are matched +% case-sensitively; ambiguous or unknown values raise an error naming +% the organizations that ARE available, so a caller can print a useful +% diagnostic. +% +% See also: ndi.cloud.uploadDataset, ndi.cloud.ui.dialog.selectOrganization. + + arguments + options.organizationName (1,:) char = '' + options.organizationID (1,:) char = '' + end + + nameGiven = ~isempty(options.organizationName); + idGiven = ~isempty(options.organizationID); + if nameGiven == idGiven + error('NDI:Cloud:OrganizationArguments', ... + 'Exactly one of organizationName or organizationID must be given.'); + end + + [ok, userInfo] = ndi.cloud.api.users.me(); + if ~ok + error('NDI:Cloud:MeFailed', ... + 'Could not retrieve your NDI Cloud user information.'); + end + + orgIds = cellfun(@(x) char(string(x)), userInfo.organizationID, 'UniformOutput', false); + orgNames = cellfun(@(x) char(string(x)), userInfo.organizationName, 'UniformOutput', false); + + if isempty(orgIds) + error('NDI:Cloud:NoOrganizations', ... + 'Your NDI Cloud account does not belong to any organizations.'); + end + + if nameGiven + matches = find(strcmp(orgNames, options.organizationName)); + if isempty(matches) + error('NDI:Cloud:UnknownOrganization', ... + ['Organization name "%s" is not one of the organizations you belong ', ... + 'to. Available: %s.'], options.organizationName, ... + availableList(orgNames, orgIds)); + end + if numel(matches) > 1 + error('NDI:Cloud:AmbiguousOrganization', ... + ['Organization name "%s" matches %d organizations you belong to; ', ... + 'pass organizationID instead. Ids: %s.'], options.organizationName, ... + numel(matches), strjoin(orgIds(matches), ', ')); + end + organizationId = orgIds{matches}; + else + matches = find(strcmp(orgIds, options.organizationID)); + if isempty(matches) + error('NDI:Cloud:UnknownOrganization', ... + ['Organization id "%s" is not one of the organizations you belong ', ... + 'to. Available: %s.'], options.organizationID, ... + availableList(orgNames, orgIds)); + end + organizationId = orgIds{matches}; + end +end + +function s = availableList(names, ids) + parts = strings(1, numel(ids)); + for k = 1:numel(ids) + if k <= numel(names) && ~isempty(names{k}) + parts(k) = sprintf('"%s" (%s)', names{k}, ids{k}); + else + parts(k) = sprintf('(no name) (%s)', ids{k}); + end + end + s = char(strjoin(parts, ', ')); +end diff --git a/src/ndi/+ndi/+cloud/uploadDataset.m b/src/ndi/+ndi/+cloud/uploadDataset.m index 7c9f8f79f..4b3bf6bcd 100644 --- a/src/ndi/+ndi/+cloud/uploadDataset.m +++ b/src/ndi/+ndi/+cloud/uploadDataset.m @@ -33,8 +33,17 @@ % | | 'remoteDatasetName'. | % | 'remoteDatasetName' | (char) The name to be assigned to the dataset on the remote server. This | % | | is *required* if 'skipMetadataEditorMetadata' is true. | + % | 'organizationName' | (char/string) The NAME of the NDI Cloud organization the dataset should | + % | | be created under. The current user must belong to it. Names are matched | + % | | case-sensitively against the list ndi.cloud.api.users.me returns. | + % | | Mutually exclusive with 'organizationID'. When neither is given, the | + % | | active NDI_CLOUD_ORGANIZATION_ID env var is used (set by login to the | + % | | first organization; see ndi.cloud.ui.dialog.selectOrganization). | + % | 'organizationID' | (char/string) The ID of the NDI Cloud organization to use. Mutually | + % | | exclusive with 'organizationName'. | % - % See also: ndi.cloud.sync.SyncOptions, ndi.cloud.downloadDataset + % See also: ndi.cloud.sync.SyncOptions, ndi.cloud.downloadDataset, + % ndi.cloud.ui.dialog.selectOrganization % arguments @@ -43,10 +52,36 @@ options.uploadAsNew (1,1) logical = false options.skipMetadataEditorMetadata (1,1) logical = false options.remoteDatasetName (1,:) char = '' + options.organizationName (1,:) char = '' + options.organizationID (1,:) char = '' end syncOptions = ndi.cloud.sync.SyncOptions(syncOptions); + % Route the request to a specific organization when the caller named one. + % Restore the previous NDI_CLOUD_ORGANIZATION_ID on function exit so a + % later call in the same MATLAB session inherits neither our resolution + % nor a partial state after an early return / error. + if ~isempty(options.organizationName) || ~isempty(options.organizationID) + if ~isempty(options.organizationName) && ~isempty(options.organizationID) + success = false; cloudDatasetId = ''; + message = 'organizationName and organizationID are mutually exclusive.'; + return; + end + previousOrgId = getenv('NDI_CLOUD_ORGANIZATION_ID'); + orgRestore = onCleanup(@() setenv('NDI_CLOUD_ORGANIZATION_ID', previousOrgId)); %#ok + try + resolvedOrgId = ndi.cloud.internal.resolveOrganizationId( ... + 'organizationName', options.organizationName, ... + 'organizationID', options.organizationID); + catch resolveErr + success = false; cloudDatasetId = ''; + message = resolveErr.message; + return; + end + setenv('NDI_CLOUD_ORGANIZATION_ID', resolvedOrgId); + end + success = false; cloudDatasetId = ''; message = ''; From b367283baab2c3646bf60d486ba59642d7e2adf6 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 21 Sep 2026 13:00:53 +0000 Subject: [PATCH 082/110] ProgressBarWindow: per-bar Timeout; list_binary_files and zipForUpload guard the handle The failure from the lightsheet demo run on 2026-09-20: a long-running top-level bar in list_binary_files was culled from behind by updateBar's global timeout sweep (fired as a side effect of the nested series bar's own tick), autoDelete then tore the whole window down because no bars were left, and the next addBar for a new series bar crashed on the deleted handle. Three changes so the source doesn't hand this failure to the next caller who ticks bars at different rates: 1. ProgressBarWindow.addBar: new Timeout name-value that stores a per-bar timeout duration on ProgressBars(i).Timeout. checkTimeout uses it in preference to the app-level Timeout, falling back to the app-level when the per-bar value is duration.empty (the default -- preserves existing behaviour). minutes(Inf) opts a long-running bar out of the sweep entirely. 2. list_binary_files: create the top-level bar with Timeout=minutes(Inf) so no unrelated bar's tick can cull it. Also wraps every progressApp call in a nested-function safe* helper that checks isvalid() and nulls out the handle on failure -- if the user closes the window mid-run, we go silent rather than crash the whole inventory pass. As a UX improvement, tick the top bar during the nested series scan (fractional progress) so the caller sees the sub-doc progress even when a single doc's scan takes minutes. 3. zipForUpload: same isvalid-guarded wrapper around its own progressApp.updateBar calls (single-bar in this branch, less prone to the culling issue, but a user who Xs the window shouldn't crash the upload). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- src/ndi/+ndi/+cloud/+upload/zipForUpload.m | 31 +++++-- .../+database/+internal/list_binary_files.m | 86 ++++++++++++++++--- .../+ndi/+gui/+component/ProgressBarWindow.m | 28 +++++- 3 files changed, 123 insertions(+), 22 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+upload/zipForUpload.m b/src/ndi/+ndi/+cloud/+upload/zipForUpload.m index f2887b6cd..7787da216 100644 --- a/src/ndi/+ndi/+cloud/+upload/zipForUpload.m +++ b/src/ndi/+ndi/+cloud/+upload/zipForUpload.m @@ -70,12 +70,19 @@ % --- Progress Bar Setup --- % Standard NDI progress bar, same "NDI tasks" window the serial upload % branch uses so bars stack in one place. Auto=true removes it at 1.0. +% Every touch is guarded with isvalid(): if the user closes the window +% or updateBar's own timeout culls the bar mid-run, we go silent rather +% than crash. See list_binary_files for the same pattern. progressApp = ndi.gui.component.ProgressBarWindow('NDI tasks'); uploadBarId = did.ido.unique_id(); -progressApp.addBar( ... - 'Label', sprintf('Uploading document-associated binary files (0 of %d)', files_left), ... - 'tag', uploadBarId, ... - 'Auto', true); +try + progressApp.addBar( ... + 'Label', sprintf('Uploading document-associated binary files (0 of %d)', files_left), ... + 'tag', uploadBarId, ... + 'Auto', true); +catch + progressApp = []; +end size_limit = options.SizeLimit; n_files = numel(files_to_process); @@ -137,10 +144,22 @@ end files_processed_so_far = files_processed_so_far + numel(batch_indices); - progressApp.updateBar(uploadBarId, files_processed_so_far / files_left); + if ~isempty(progressApp) && isvalid(progressApp) + try + progressApp.updateBar(uploadBarId, files_processed_so_far / files_left); + catch + progressApp = []; + end + end end -progressApp.updateBar(uploadBarId, 1); % Auto=true removes it +if ~isempty(progressApp) && isvalid(progressApp) + try + progressApp.updateBar(uploadBarId, 1); % Auto=true removes it + catch + % Nothing left to do; the batch loop is finished either way. + end +end % --- Final Logging --- if options.DebugLog diff --git a/src/ndi/+ndi/+database/+internal/list_binary_files.m b/src/ndi/+ndi/+database/+internal/list_binary_files.m index 1aaaae560..775d7efd2 100644 --- a/src/ndi/+ndi/+database/+internal/list_binary_files.m +++ b/src/ndi/+ndi/+database/+internal/list_binary_files.m @@ -58,13 +58,28 @@ % drive; without a bar, the whole pre-upload phase looks hung. Same % "NDI tasks" window the serial upload branch uses, so bars stack in % one place. Auto=true removes the bar when it reaches 1.0. + % + % The progressApp handle is captured in a nested state variable so the + % safe* helpers below can null it out if the window is closed / deleted + % mid-run (e.g. the user Xs it out, or updateBar's own checkTimeout + % culls a bar and then autoDelete tears the window down with it). + % Every touch of the bar goes through those helpers -- see the failure + % history for what happened before: updateBar's side-effect timeout + % check culled the top bar during a >1-min series scan, autoDelete + % then removed the window because no bars were left, and the next + % addBar crashed on a stale handle. See NDI-matlab lightsheet demo, + % 2026-09-20. + progressApp = []; + topBarId = ''; if verbose progressApp = ndi.gui.component.ProgressBarWindow('NDI tasks'); - topBarId = did.ido.unique_id(); - progressApp.addBar( ... - 'Label', sprintf('Scanning documents for files (0 of %d)', num_documents), ... - 'tag', topBarId, ... - 'Auto', true); + topBarId = did.ido.unique_id(); + % Timeout=minutes(Inf): the top bar does not tick as often as the + % app-level 1-minute timeout, and we do not want it culled by a + % global sweep triggered by an unrelated bar's update. The nested + % renew below is belt-and-suspenders progress feedback. + safeAddBar(sprintf('Scanning documents for files (%d docs)', num_documents), ... + topBarId, minutes(Inf)); end for i = 1:num_documents @@ -131,12 +146,9 @@ lastReport = tic; if verbose && slotCount >= 1000 seriesBarId = did.ido.unique_id(); - progressApp.addBar( ... - 'Label', sprintf(' doc %d/%d: series ''%s'' (%d slots)', ... - i, num_documents, series_name, slotCount), ... - 'tag', seriesBarId, ... - 'Auto', true); - hasSeriesBar = true; + safeAddBar(sprintf(' doc %d/%d: series ''%s'' (%d slots)', ... + i, num_documents, series_name, slotCount), seriesBarId); + hasSeriesBar = ~isempty(progressApp); end for m = 1:slotCount @@ -157,13 +169,19 @@ 'file_path', member_path); if hasSeriesBar && toc(lastReport) >= progressInterval - progressApp.updateBar(seriesBarId, m/slotCount); + safeUpdateBar(seriesBarId, m/slotCount); + % Also tick the top bar so it does not sit idle + % past ProgressBarWindow's 1-minute timeout during + % a long series scan. Fractional progress is fine: + % (i-1)/N + (m/slotCount)/N is monotone, and the + % update at end-of-doc pins it to i/N below. + safeUpdateBar(topBarId, ((i-1) + m/slotCount) / num_documents); lastReport = tic; end end if hasSeriesBar - progressApp.updateBar(seriesBarId, 1); % Auto=true removes it + safeUpdateBar(seriesBarId, 1); % Auto=true removes it end file_manifest = [file_manifest member_entries(1:member_count)]; %#ok @@ -171,7 +189,47 @@ end if verbose - progressApp.updateBar(topBarId, i/num_documents); + safeUpdateBar(topBarId, i/num_documents); + end + end + + % --- Nested helpers ------------------------------------------------- + % Every touch of the progress window goes through these so a closed / + % deleted window does not crash the inventory pass. progressApp is a + % handle stored in the enclosing scope; when a call raises because the + % window was torn down, we null it out and go silent for the rest of + % the run rather than firing a warning per iteration. + + function safeAddBar(labelText, tag, timeoutOverride) + if nargin < 3 + timeoutOverride = duration.empty; % inherit app-level timeout + end + if isempty(progressApp) || ~isvalid(progressApp) + progressApp = []; + return + end + try + progressApp.addBar('Label', labelText, 'tag', tag, 'Auto', true, ... + 'Timeout', timeoutOverride); + catch err + % One warning, then quiet. A user who closed the window does + % not want us re-opening it or spamming diagnostics. + warning('NDI:list_binary_files:ProgressBarUnavailable', ... + 'Progress bar window is no longer available (%s); continuing silently.', ... + err.message); + progressApp = []; + end + end + + function safeUpdateBar(tag, progress) + if isempty(progressApp) || ~isvalid(progressApp) || isempty(tag) + return + end + try + progressApp.updateBar(tag, progress); + catch + % Bar or window is gone; drop it and stop touching it. + progressApp = []; end end end diff --git a/src/ndi/+ndi/+gui/+component/ProgressBarWindow.m b/src/ndi/+ndi/+gui/+component/ProgressBarWindow.m index ac7c4b316..5b9fe236b 100644 --- a/src/ndi/+ndi/+gui/+component/ProgressBarWindow.m +++ b/src/ndi/+ndi/+gui/+component/ProgressBarWindow.m @@ -265,6 +265,15 @@ options.Tag {mustBeTextScalar(options.Tag)} = '' options.Color (1,3) double {mustBeInRange(options.Color,0,1)} = [1 1 1] options.Auto logical = false + % Per-bar timeout. Overrides app.Timeout for this bar only. + % duration.empty (the default) means "inherit app.Timeout". + % Pass minutes(Inf) to opt this bar out of the timeout entirely + % -- appropriate for a top-level bar that ticks less often + % than the app-level timeout (typically 1 min), which used to + % get culled by updateBar's global timeout sweep while its + % owner was still using it. See NDI-matlab lightsheet demo + % 2026-09-20. + options.Timeout (1,:) duration = duration.empty end % Bring figure to front @@ -307,6 +316,9 @@ app.ProgressBars(barNum).Auto = options.Auto; app.ProgressBars(barNum).Progress = 0; app.ProgressBars(barNum).Clock(1:2) = {datetime('now')}; + % Per-bar timeout: duration.empty means inherit app.Timeout at + % checkTimeout time. minutes(Inf) means "never time out". + app.ProgressBars(barNum).Timeout = options.Timeout; % Everything from here on builds this bar's graphics. When % silent there is no grid and no figure, so the bar exists as @@ -663,11 +675,23 @@ % Get duration of time since last update timeout = datetime('now') - app.ProgressBars(i).Clock{2}; - if timeout >= app.Timeout & ... + % Per-bar timeout wins over the app default. A bar that + % was created with Timeout=minutes(Inf) never times out + % here -- this is what long-running top-level bars need + % so that a global sweep (fired as a side effect of ANY + % updateBar call, including one on a different, actively + % updating bar) does not cull them from behind. + thisTimeout = app.Timeout; + if isfield(app.ProgressBars(i), 'Timeout') && ... + ~isempty(app.ProgressBars(i).Timeout) + thisTimeout = app.ProgressBars(i).Timeout; + end + + if timeout >= thisTimeout & ... ~strcmpi(app.ProgressBars(i).State,'Closed') && ... ~strcmpi(app.ProgressBars(i).State,'Complete') && ... app.ProgressBars(i).Progress < 1 - + % Set icon to error and state to 'Timeout' app.setErrorIconForButton(app.ProgressBars(i).Button) app.ProgressBars(i).State = 'Timeout'; From 4e45d96e66d20ff8be7ef847c69a1cf476759b53 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 21 Sep 2026 23:29:30 +0000 Subject: [PATCH 083/110] cloud upload: verbose diagnostics on the pre-upload cloud check + final status When ndi.cloud.uploadDataset came back success=false, the caller who did not also capture 'message' saw no reason why. The pre-upload listFiles call (via filesNotYetUploaded) can fail for real reasons -- the classic being an active NDI_CLOUD_ORGANIZATION_ID that does not own the cloud dataset, e.g. after a login that defaulted to the first org while the dataset was created under a different one -- and the old flow returned success=false silently. Three prints, all Verbose-gated: - Before filesNotYetUploaded: "Checking cloud dataset for files already uploaded ..." so a slow response is visibly in-flight. - On its failure: print the underlying message plus a one-line hint naming the active org and pointing at ndi.cloud.uploadDataset('organizationName' | 'organizationID'). That single sentence is what would have told the user (in the flow behind this change) that the two calls were talking about different organizations. - After a successful serial / batch upload: report the count ("Uploaded N of M files." or "Uploaded N files in batch mode."). Also print a top-level "ndi.cloud.uploadDataset FAILED: ..." or "...completed successfully..." from uploadDataset itself, so a caller who only captured [success, cloudDatasetId] still sees the reason. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../uploadFilesForDatasetDocuments.m | 30 +++++++++++++++++++ src/ndi/+ndi/+cloud/uploadDataset.m | 10 +++++++ 2 files changed, 40 insertions(+) diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/uploadFilesForDatasetDocuments.m b/src/ndi/+ndi/+cloud/+sync/+internal/uploadFilesForDatasetDocuments.m index 006c667a5..ee7b21086 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/uploadFilesForDatasetDocuments.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/uploadFilesForDatasetDocuments.m @@ -48,9 +48,25 @@ end if options.onlyMissing + if options.Verbose + fprintf('Checking cloud dataset %s for files already uploaded ...\n', cloudDatasetId); + end [file_manifest, message] = ndi.cloud.sync.internal.filesNotYetUploaded(file_manifest, cloudDatasetId); if ~isempty(message) success = false; + % Do not swallow the diagnostic. A caller that inspects only + % `success` still gets a printed line explaining what failed + % and where to look next, so the pipeline does not appear to + % succeed-silently-fail-silently. + if options.Verbose + fprintf(['ERROR while checking which files are already ', ... + 'in the cloud: %s\n', ... + 'This often means the active NDI_CLOUD_ORGANIZATION_ID ', ... + 'does not own dataset %s. Pass ndi.cloud.uploadDataset ', ... + 'the ''organizationName'' or ''organizationID'' the ', ... + 'dataset was created under. Current active org: %s.\n'], ... + message, cloudDatasetId, getenv('NDI_CLOUD_ORGANIZATION_ID')); + end return; end end @@ -61,6 +77,9 @@ if isempty(file_manifest) message = 'All files are already on the remote.'; + if options.Verbose + fprintf('%s\n', message); + end return; end @@ -71,6 +90,7 @@ app = ndi.gui.component.ProgressBarWindow('NDI tasks'); uuid = did.ido.unique_id(); app.addBar('Label','Uploading document-associated binary files','tag',uuid,'Auto',true); + files_uploaded_count = 0; for i=1:numel(file_manifest) if file_manifest(i).is_uploaded==false [upload_success, upload_message] = ndi.cloud.uploadSingleFile(cloudDatasetId, ... @@ -80,15 +100,25 @@ message = ['Failed to upload file ' file_manifest(i).uid ': ' upload_message]; end success = false; + else + files_uploaded_count = files_uploaded_count + 1; end end app.updateBar(uuid,i/numel(file_manifest)); end + if options.Verbose + fprintf('Uploaded %d of %d files.\n', files_uploaded_count, numel(file_manifest)); + end case "batch" [batch_success, batch_message] = ndi.cloud.upload.zipForUpload(ndiDataset, file_manifest, totalSizeKb, cloudDatasetId); if ~batch_success success = false; message = batch_message; + if options.Verbose + fprintf('Batch upload failed: %s\n', batch_message); + end + elseif options.Verbose + fprintf('Uploaded %d files in batch mode.\n', numel(file_manifest)); end end end diff --git a/src/ndi/+ndi/+cloud/uploadDataset.m b/src/ndi/+ndi/+cloud/uploadDataset.m index 4b3bf6bcd..5d7753eea 100644 --- a/src/ndi/+ndi/+cloud/uploadDataset.m +++ b/src/ndi/+ndi/+cloud/uploadDataset.m @@ -143,8 +143,18 @@ "FileUploadStrategy", syncOptions.FileUploadStrategy, "onlyMissing", true); if ~success_upload message = message_upload; + % Print the failure so a caller who did not capture the message + % output still sees why success came back false. Otherwise the + % previous lines all say "success" and the return value is 0 with + % no visible reason. + if syncOptions.Verbose + fprintf('ndi.cloud.uploadDataset FAILED: %s\n', message); + end return; end success = true; + if syncOptions.Verbose + fprintf('ndi.cloud.uploadDataset completed successfully. Cloud dataset id: %s\n', cloudDatasetId); + end end From 58cbdeec9e1aff0cf96e0586524e7623a8b2064d Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 24 Sep 2026 21:12:41 +0000 Subject: [PATCH 084/110] cloud download: fast + visible series-ingest reconstruction Two hotspots that made "Updating document file info to reflect remote files." run for many minutes with no feedback on a 156k-member lightsheet dataset: 1. reconstructSeriesIngestLocations.buildIngestLocations was growing the ingest_locations struct array one entry at a time with `end+1`, which reallocates the whole array on each step -- O(N^2) at pyramid-level scale. Preallocate to manifest.count and trim at the end. 2. updateFileInfoForRemoteFiles.reconstructFromCloud offered no progress signal while it fetched each qualifying series' manifest from the cloud (one HTTPS round trip per series). Add a ProgressBarWindow bar in the "NDI tasks" window that ticks per series, with Timeout=minutes(Inf) so an unrelated bar's update cannot cull it (see 2026-09-20 ProgressBarWindow fix). The onCleanup for the scratch dir is preserved; the reconstruction call is unchanged. A closed window is handled through the isvalid guard in the new tickBar helper -- same defensive contract list_binary_files uses. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../reconstructSeriesIngestLocations.m | 39 +++++++++------ .../+internal/updateFileInfoForRemoteFiles.m | 50 ++++++++++++++++++- 2 files changed, 73 insertions(+), 16 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/reconstructSeriesIngestLocations.m b/src/ndi/+ndi/+cloud/+sync/+internal/reconstructSeriesIngestLocations.m index 4204e72b2..5106e1b27 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/reconstructSeriesIngestLocations.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/reconstructSeriesIngestLocations.m @@ -116,23 +116,34 @@ % One ingest_locations entry per present slot of the manifest. % Order matches DID's authoring shape and its `sprintf('%s_%d', name, index)` % naming (index is 1-based, same as manifest.uids{index}). - ingestLocations = struct( ... - 'index', {}, ... - 'uid', {}, ... - 'location', {}, ... - 'location_type', {}, ... - 'ingest', {}, ... - 'delete_original', {}); +% +% Preallocated + trimmed rather than grown: a lightsheet pyramid level's +% manifest names tens of thousands of members, and growing the struct +% array one entry at a time with `end+1` reallocates the whole array on +% every step -- classic O(N^2), turning a linear pass into minutes. +% See NDI-matlab lightsheet 2026-09-24 download hang. + prototype = struct( ... + 'index', 0, ... + 'uid', '', ... + 'location', '', ... + 'location_type', '', ... + 'ingest', 0, ... + 'delete_original', 0); + entries = repmat(prototype, 1, manifest.count); + kept = 0; + cloudDatasetIdChar = char(cloudDatasetId); for slot = 1 : manifest.count memberUid = manifest.uids{slot}; if isempty(memberUid), continue, end - ingestLocations(end+1) = struct( ... - 'index', slot, ... - 'uid', memberUid, ... - 'location', sprintf('ndic://%s/%s', char(cloudDatasetId), memberUid), ... - 'location_type', 'ndicloud', ... - 'ingest', 0, ... - 'delete_original', 0); %#ok + kept = kept + 1; + entries(kept).index = slot; + entries(kept).uid = memberUid; + entries(kept).location = sprintf('ndic://%s/%s', cloudDatasetIdChar, memberUid); + entries(kept).location_type = 'ndicloud'; + entries(kept).ingest = 0; + entries(kept).delete_original = 0; end + + ingestLocations = entries(1:kept); end diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m index 3dbdbaac4..92191fbcc 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m @@ -139,6 +139,13 @@ if ~isfolder(tmpDir), mkdir(tmpDir); end cleanup = onCleanup(@() safeRmdir(tmpDir)); + % Count qualifying series up front so the progress bar has a + % denominator. A doc with 10 pyramid levels has 10 series that each + % require one HTTPS round trip for the manifest, then one linear pass + % over the manifest's members; without a bar the whole "Updating + % document file info to reflect remote files." step looks hung on a + % large lightsheet dataset. + qualifyingIdx = []; for k = 1:numel(seriesInfo) entry = seriesInfo(k); if ~isfield(entry, 'n_present') || isempty(entry.n_present) || ... @@ -148,9 +155,35 @@ if isfield(entry, 'ingest_locations') && ~isempty(entry.ingest_locations) continue end + qualifyingIdx(end+1) = k; %#ok + end + + progressApp = []; + fetchBarId = ''; + if numel(qualifyingIdx) > 0 + try + progressApp = ndi.gui.component.ProgressBarWindow('NDI tasks'); + fetchBarId = did.ido.unique_id(); + progressApp.addBar( ... + 'Label', sprintf('Fetching %d series manifest(s) from cloud', ... + numel(qualifyingIdx)), ... + 'tag', fetchBarId, ... + 'Auto', true, ... + 'Timeout', minutes(Inf)); + catch + progressApp = []; + end + end + + for j = 1:numel(qualifyingIdx) + k = qualifyingIdx(j); + entry = seriesInfo(k); manifestUid = lookupManifestUid(fileInfo, entry.name); - if isempty(manifestUid), continue, end + if isempty(manifestUid) + tickBar(progressApp, fetchBarId, j / numel(qualifyingIdx)); + continue + end destPath = fullfile(tmpDir, manifestUid); try @@ -162,8 +195,8 @@ % add_docs with the document's own identity, which is the % signal a partial download deserves. No warning here -- % add_docs is the right voice. - continue end + tickBar(progressApp, fetchBarId, j / numel(qualifyingIdx)); end seriesInfo = ndi.cloud.sync.internal.reconstructSeriesIngestLocations( ... @@ -246,3 +279,16 @@ function safeRmdir(d) end end end + +function tickBar(app, tag, progress) + % Guarded progress update. If the bar / window has been closed or + % culled, drop the update silently rather than crash the download + % pipeline. Same defensive contract list_binary_files uses. + if isempty(app) || ~isvalid(app) || isempty(tag) + return + end + try + app.updateBar(tag, progress); + catch + end +end From 9467a0a7432c5f00d406acbde8276f13d87e452f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 24 Sep 2026 23:45:54 +0000 Subject: [PATCH 085/110] cloud download: employ listFilesAll for uid validation on the remote path Add an upfront listFilesAll(cloudDatasetId) call at the top of the SyncFiles=false branch in downloadNdiDocuments (NDI-matlab#1004's paginated whole-dataset listing landed in main and is a cheap way to know exactly which uids the server reports as uploaded=true). The result is a containers.Map(uid -> logical) threaded into updateFileInfoForRemoteFiles as a new name-value 'uploadedUidMap'. Empty map (the default) preserves the previous no-check behavior. updateFileInfoForRemoteFiles then uses the map in two places: 1. On the file_info rewrite: count uids the server does not report as uploaded and warn once per document naming the count and the first offending uid. A file_info entry for an unuploaded uid stamps a broken ndic:// location; DID's later member/file open would fail obscurely (see the 156k-member upload on this branch that interrupted at 28,899). Warning now says why up front. 2. On the series manifest fetch loop: skip the HTTPS round trip entirely for a series whose manifest uid isn't uploaded. Attempting it would only 403/404 into the existing catch, so this is pure savings, not a behavior change. The reconstruction still emits no ingest_locations for that series -- DID#185's guard on add_docs remains the right voice for that partial-download case. isUploaded() is deliberately open-answering when the map is empty: a caller who did not (or could not) fetch listFilesAll sees exactly the pre-#1007 behavior. A failed listFilesAll in the caller returns an empty map with a Verbose note, not an error. The one-shot fetch is a few paged calls at API-side batch size 1000, same cost the tests use. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+sync/+internal/downloadNdiDocuments.m | 47 +++++++++++++- .../+internal/updateFileInfoForRemoteFiles.m | 65 ++++++++++++++++++- 2 files changed, 109 insertions(+), 3 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m b/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m index 066616303..418a3888e 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/downloadNdiDocuments.m @@ -130,8 +130,16 @@ if syncOptions.Verbose fprintf('"SyncFiles" option is false. Updating document file info to reflect remote files.\n'); end + % Build the uploaded-uid map ONCE from the paginated whole-dataset + % listing (NDI-matlab#1004's listFilesAll). Threaded into + % updateFileInfoForRemoteFiles so per-doc rewrites can validate + % uids and per-series manifest fetches can skip uids the server + % does not report as uploaded. Empty map on failure preserves + % the previous no-check behavior. + uploadedUidMap = buildUploadedUidMap(cloudDatasetId, syncOptions.Verbose); documentUpdateFcn = @(doc) ... - ndi.cloud.sync.internal.updateFileInfoForRemoteFiles(doc, cloudDatasetId); + ndi.cloud.sync.internal.updateFileInfoForRemoteFiles(doc, cloudDatasetId, ... + 'uploadedUidMap', uploadedUidMap); end % 3. Update file info for documents based on local / remote location @@ -153,3 +161,40 @@ downloadedNdiDocuments = newNdiDocuments; end end + +function m = buildUploadedUidMap(cloudDatasetId, verbose) + % One paginated whole-dataset listing (listFilesAll from NDI-matlab#1004), + % into a containers.Map of uid -> uploaded logical. Any failure returns + % an empty map, which the downstream code treats as "no check" -- + % preserves the pre-#1007 behavior. Verbose mode reports the count so + % the "Updating document file info..." step has a denominator to show. + m = containers.Map('KeyType','char','ValueType','logical'); + try + if verbose + fprintf('Fetching cloud file listing for uid validation ...\n'); + end + [ok, listing] = ndi.cloud.api.files.listFilesAll(cloudDatasetId); + if ~ok + if verbose + fprintf(['Could not fetch cloud file listing; skipping ', ... + 'uid validation (per-doc opens will surface any misses).\n']); + end + return + end + for i = 1:numel(listing) + uidChar = char(string(listing(i).uid)); + if isempty(uidChar), continue, end + m(uidChar) = logical(listing(i).uploaded); + end + if verbose + uploadedCount = sum(cellfun(@logical, m.values)); + fprintf('Cloud dataset lists %d file(s); %d uploaded.\n', ... + m.Count, uploadedCount); + end + catch err + if verbose + fprintf('Uid map build failed: %s. Continuing without validation.\n', err.message); + end + m = containers.Map('KeyType','char','ValueType','logical'); + end +end diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m index 92191fbcc..4004af9fc 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m @@ -61,12 +61,22 @@ document cloudDatasetId (1,1) string options.customFileHandler = [] + % A containers.Map of uid -> logical uploaded flag, precomputed + % once at the caller from ndi.cloud.api.files.listFilesAll (the + % paginated whole-dataset listing landed in main via NDI-matlab + % #1004). When supplied, this function uses it to skip the manifest + % fetch for series whose manifest uid isn't uploaded on the + % server, and to warn on file_info uids that name unuploaded + % files. Empty (the default) preserves the previous behavior. + options.uploadedUidMap = [] end if ~document.has_files(), return, end updatedFileInfo = document.document_properties.files.file_info; + missingFileUidCount = 0; + firstMissingFileUid = ''; for i = 1:numel(updatedFileInfo) % Replace/override 1st file location updatedFileInfo(i).locations(1).delete_original = 0; @@ -76,9 +86,28 @@ fileLocation = sprintf('ndic://%s/%s', cloudDatasetId, fileUid); updatedFileInfo(i).locations(1).location = fileLocation; updatedFileInfo(i).locations(1).location_type = 'ndicloud'; + + % Optional integrity check: a file_info entry pointing at a uid + % the server does not yet report as uploaded produces a broken + % ndic:// location -- DID's later member/file open will fail + % obscurely. Count them now and warn once at the end of the doc + % rather than spamming per file. + if ~isUploaded(options.uploadedUidMap, fileUid) + missingFileUidCount = missingFileUidCount + 1; + if isempty(firstMissingFileUid) + firstMissingFileUid = char(fileUid); + end + end end document = document.setproperties('files.file_info', updatedFileInfo); + if missingFileUidCount > 0 + warning('NDI:cloud:sync:UnuploadedFileUid', ... + ['%d file_info entry/entries in document reference uids the ', ... + 'cloud does not report as uploaded (first: %s). A member/file ', ... + 'open will fail on those.'], missingFileUidCount, firstMissingFileUid); + end + % Rebuild series ingest_locations for any series that came back with % n_present > 0 but empty ingest_locations. Without this, DID's % MembersNotLocatable guard (DID-matlab#185) refuses the document on @@ -96,7 +125,8 @@ catch end seriesInfo = reconstructFromCloud(seriesInfo, updatedFileInfo, ... - cloudDatasetId, documentId, options.customFileHandler); + cloudDatasetId, documentId, options.customFileHandler, ... + options.uploadedUidMap); document = document.setproperties('files.series_info', seriesInfo); end end @@ -125,7 +155,7 @@ function seriesInfo = reconstructFromCloud(seriesInfo, fileInfo, ... - cloudDatasetId, documentId, customFileHandler) + cloudDatasetId, documentId, customFileHandler, uploadedUidMap) % Fetch each qualifying series' manifest bytes into a scratch dir, % reconstruct ingest_locations from them via reconstructSeriesIngestLocations, % then delete the scratch dir. Manifests do NOT land in the DID file @@ -185,6 +215,16 @@ continue end + % Skip the fetch when the caller-provided uploaded-uid map says + % the manifest itself is not uploaded on the server. Attempting + % the fetch would only turn into a wasted HTTPS round trip whose + % 404/403 landed in the catch below; no reconstruction happens + % either way. When no map was passed, this is a no-op. + if ~isUploaded(uploadedUidMap, manifestUid) + tickBar(progressApp, fetchBarId, j / numel(qualifyingIdx)); + continue + end + destPath = fullfile(tmpDir, manifestUid); try fetchManifest(destPath, cloudDatasetId, string(manifestUid), ... @@ -280,6 +320,27 @@ function safeRmdir(d) end end +function tf = isUploaded(map, uid) + % True when the caller passed no map (default open answer -- preserves + % the previous no-check behavior), or when the map says this uid is + % on the server as uploaded=true. False only when the map is present + % AND names this uid as not uploaded (or does not name it at all). + if isempty(map) + tf = true; + return + end + try + key = char(uid); + if ~isKey(map, key) + tf = false; + return + end + tf = logical(map(key)); + catch + tf = false; + end +end + function tickBar(app, tag, progress) % Guarded progress update. If the bar / window has been closed or % culled, drop the update silently rather than crash the download From 60a79cf0e5f92fa3523c32e18ac1fe12115887d3 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 25 Sep 2026 10:42:34 +0000 Subject: [PATCH 086/110] LightsheetZarrManager: pull Levels + Reduction from the child level docs The pyramid list read the Levels and Reduction columns off the parent lightsheetZarrPyramid document, but that schema carries neither an n_levels field nor a reduction field -- reductions and level counts live on the child lightsheetZarrLevel documents. Result: every row in the manager displayed Levels=0 and an empty Reduction column even when the ladder had multiple levels and one or more reductions on disk. buildRowsForSession now runs the same per-pyramid child query deletionPlan already uses (isa=lightsheetZarrLevel with a depends_on=lightsheetZarrPyramid_id constraint) and: * sets nLevels to numel(levelDocs), matching the count deletionPlan reports and the ladder napariViewLightsheet actually opens; * collects each level's reduction_function through a small helper joinReductions that returns the unique non-'none' values joined by "/". 'none' is the raw level 0 marker (both mean and max pyramids typically share one), so filtering it out gives the reader the reduction labels they think in ('mean', 'max', or 'mean/max'). When only 'none' levels are present the helper returns 'none' so the column does not lie about a missing ladder. The child query is once per pyramid at table load; a session with a handful of pyramids stays snappy. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+ndi/+gui/+app/LightsheetZarrManager.m | 38 ++++++++++++++++++- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m index 141bfcc70..fba07c18a 100644 --- a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m +++ b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m @@ -114,17 +114,51 @@ function refresh(obj) rows(k).id = d.id(); rows(k).doc = d; rows(k).label = ndi.gui.app.LightsheetZarrManager.field(p,'label',''); - rows(k).reduction = ndi.gui.app.LightsheetZarrManager.field(p,'reduction',''); rows(k).subjectID = ndi.gui.app.LightsheetZarrManager.dependency(d,'subject_id'); rows(k).subject = ndi.gui.app.LightsheetZarrManager.lookup( ... subjectNames, rows(k).subjectID, rows(k).subjectID); - rows(k).nLevels = double(ndi.gui.app.LightsheetZarrManager.field(p,'n_levels',0)); + + % Level count and reduction come from the child level + % docs, not the parent -- the parent schema has neither + % an n_levels nor a reduction field. Reductions are per + % level (reduction_function on lightsheetZarrLevel); + % the parent's pyramid_type names the source volume, not + % the reduction. See NDI-matlab#979 review 2026-09-25. + q1 = ndi.query('', 'isa', 'lightsheetZarrLevel'); + q2 = ndi.query('depends_on', 'depends_on', ... + 'lightsheetZarrPyramid_id', rows(k).id); + levelDocs = session.database_search(q1 & q2); + rows(k).nLevels = numel(levelDocs); + rows(k).reduction = ndi.gui.app.LightsheetZarrManager.joinReductions(levelDocs); + shp = ndi.gui.app.LightsheetZarrManager.field(p,'shape_level0',[]); rows(k).shapeLevel0 = shp; rows(k).dtype = ndi.gui.app.LightsheetZarrManager.field(p,'dtype',''); end end + function s = joinReductions(levelDocs) + % joinReductions - unique non-'none' reduction_function values + % Reads reduction_function from each lightsheetZarrLevel doc + % and returns them joined by "/", with 'none' (the raw level 0 + % marker) filtered out. When only 'none' levels exist, returns + % the string 'none' so the column doesn't lie about a missing + % ladder. + seen = strings(1, numel(levelDocs)); + for j = 1:numel(levelDocs) + lp = levelDocs{j}.document_properties.lightsheetZarrLevel; + seen(j) = string(ndi.gui.app.LightsheetZarrManager.field( ... + lp, 'reduction_function', 'none')); + end + reductions = unique(seen); + reductions = reductions(reductions ~= "none"); + if isempty(reductions) + s = 'none'; + else + s = char(strjoin(reductions, '/')); + end + end + function m = subjectNameMap(session) % SUBJECTNAMEMAP - subject document id -> local_identifier % From 9646b6279f98e4b15f358decdc32888a82aee008 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 13:52:48 +0000 Subject: [PATCH 087/110] batchSignedUrlLookup: retry scope fetch with backoff on transient raise One MATLAB:webservices:ConnectionFailed on the first batch call marked the scope failed and every subsequent uid fell back to a per-member getFileDetails call. For a 156k-member lightsheet series that meant 4+ hours of sequential requests -- silent catastrophic degradation triggered by one TLS blip. See VH-Lab/NDI-matlab#1010 and the parallel Waltham-Data-Science/NDI-python#322. localFetchScope now wraps the signer call in a bounded loop: on any raise, sleep for options.retryBackoffSeconds(attempt) and retry, up to three retries by default at 1 s / 4 s / 16 s. An ok=false response (server-side no) is NOT retried; it drops through to the fallback path as before. Exhausted retries still mark the scope failed and let the per-member fallback run, so a scope that genuinely cannot be reached is not a busy loop. New STATS.transientRetries counts the extra signer calls spent recovering. Four new tests in BatchSignedUrlLookupTest exercise the happy path (one raise absorbed, no per-member fallback), the exhaustion path (retries bounded, then fallback), the backoff timing (delays applied between failed attempts only, via injected sleepFcn), and the ok=false discriminator (no retry, no sleep). Note on the second half of #1010 (route batch fetch through do_curl): the codebase's actual convention is that JSON API calls go through matlab.net.http.send while only large file downloads go through curl, so routing GetSignedURLSet through curl would need a new curl-based JSON HTTP helper (auth headers, query params, raw-payload preservation for uid keys) rather than a spot edit -- larger change, left for a follow-up. Retry is the higher-value fix in the meantime: it converts the O(N) cascade into ~21 s of wall clock whichever transport is used. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz --- .../+internal/batchSignedUrlLookup.m | 106 +++++++++--- .../+ndi/+unittest/BatchSignedUrlLookupTest.m | 159 ++++++++++++++++++ 2 files changed, 242 insertions(+), 23 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m b/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m index d99c6a7c7..a613c59e2 100644 --- a/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m +++ b/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m @@ -81,6 +81,18 @@ % short-circuit the wait without adding real % latency to a suite that exercises the retry many % times. +% retryBackoffSeconds - Row vector of delays, in seconds, +% between successive scope-fetch attempts after a +% raise. Default [1 4 16] -- one initial call, up +% to three retries at 1 s / 4 s / 16 s, ~21 s of +% wall clock in the worst case. An exception here +% is a transient network condition +% (MATLAB:webservices:ConnectionFailed and the +% like); a signer that reports ok=false is a +% server-side no and is NOT retried. Pass [] to +% disable retries, or a vector of zeros to keep +% the retry count without adding real latency +% (used by tests). See VH-Lab/NDI-matlab#1010. % % Outputs: % url - The pre-signed URL for uid (char), or "" (string @@ -115,6 +127,16 @@ % this uid" gets that answer after the % retry. See VH-Lab/NDI-matlab#991 and % Waltham-Data-Science/NDI-python#309. +% .transientRetries how many extra signer calls were +% spent recovering from a raise (a +% transient network error) before the +% scope was cached or given up on. A +% fresh scope fetch that raises is +% retried with backoff so one blip +% does not cascade to per-member +% fallback across a 156k-member series. +% See VH-Lab/NDI-matlab#1010 and +% Waltham-Data-Science/NDI-python#322. % % These exist to make the fallback VISIBLE. It is % deliberate at runtime -- a reader opening one file @@ -150,6 +172,7 @@ options.failureTtlSeconds (1,1) double = 60 options.partialMapRetrySeconds (1,1) double = 0.5 options.sleepFcn (1,1) function_handle = @pause + options.retryBackoffSeconds (1,:) double = [1 4 16] end % The persistent cache. Keyed on 'datasetId/documentId/seriesName'. @@ -166,7 +189,7 @@ if isempty(STATS) || options.clearCache STATS = struct('signerCalls', 0, 'uidHits', 0, 'uidMisses', 0, ... 'lastMapSize', 0, 'lastMapUids', {{}}, 'lastFailureReason', "", ... - 'partialMapRetries', 0); + 'partialMapRetries', 0, 'transientRetries', 0); end if isempty(WARNED) || options.clearCache WARNED = containers.Map('KeyType','char','ValueType','logical'); @@ -324,34 +347,71 @@ wantsFour = false; end + % Fetch the scope with backoff on a raise. A transient network + % blip (MATLAB:webservices:ConnectionFailed and the like) at the + % start of a run would otherwise cascade: this scope is marked + % failed, every subsequent uid in it falls back to a per-member + % getFileDetails call, and a 156k-member series takes hours. + % Retry the batch call itself first; the per-member fallback is + % still there below as a safety net, just not triggered by + % transients. See VH-Lab/NDI-matlab#1010 and Waltham-Data-Science/ + % NDI-python#322. + % + % Only a caught exception is retried. A signer that reports + % ok=false is a server-side no (auth, 404, business-logic + % refusal): retrying hammers a dead endpoint and adds cost with + % no chance of recovery. + ok = false; + answer = []; apiResponse = []; failureReason = ""; - try - if wantsFour - if strlength(seriesName) > 0 - [ok, answer, apiResponse] = options.signer(cloudDatasetId, ... - ndiDocumentId, 'idNamespace', "ndi", ... - 'fileSeries', seriesName); + maxAttempts = 1 + numel(options.retryBackoffSeconds); + for attempt = 1:maxAttempts + apiResponse = []; + failureReason = ""; + try + if wantsFour + if strlength(seriesName) > 0 + [ok, answer, apiResponse] = options.signer(cloudDatasetId, ... + ndiDocumentId, 'idNamespace', "ndi", ... + 'fileSeries', seriesName); + else + [ok, answer, apiResponse] = options.signer(cloudDatasetId, ... + ndiDocumentId, 'idNamespace', "ndi"); + end else - [ok, answer, apiResponse] = options.signer(cloudDatasetId, ... - ndiDocumentId, 'idNamespace', "ndi"); - end - else - if strlength(seriesName) > 0 - [ok, answer] = options.signer(cloudDatasetId, ndiDocumentId, ... - 'idNamespace', "ndi", 'fileSeries', seriesName); - else - [ok, answer] = options.signer(cloudDatasetId, ndiDocumentId, ... - 'idNamespace', "ndi"); + if strlength(seriesName) > 0 + [ok, answer] = options.signer(cloudDatasetId, ndiDocumentId, ... + 'idNamespace', "ndi", 'fileSeries', seriesName); + else + [ok, answer] = options.signer(cloudDatasetId, ndiDocumentId, ... + 'idNamespace', "ndi"); + end end + catch signerError + ok = false; + answer = []; + failureReason = "the call raised " + string(signerError.identifier) + ... + ": " + string(signerError.message); + end + STATS.signerCalls = STATS.signerCalls + 1; + if ok + break + end + % Retry only on a raise. An ok=false response is not + % transient; break out and let the fallback path handle it. + if strlength(failureReason) == 0 + break + end + if attempt >= maxAttempts + break + end + STATS.transientRetries = STATS.transientRetries + 1; + delay = options.retryBackoffSeconds(attempt); + if delay > 0 + options.sleepFcn(delay); end - catch signerError - ok = false; - answer = []; - failureReason = "the call raised " + string(signerError.identifier) + ... - ": " + string(signerError.message); end - STATS.signerCalls = STATS.signerCalls + 1; if ~ok || ~isstruct(answer) || ~isfield(answer,'files') || ... ~isa(answer.files,'containers.Map') % Any failure or unexpected shape -> no batch URL. Don't diff --git a/tests/+ndi/+unittest/BatchSignedUrlLookupTest.m b/tests/+ndi/+unittest/BatchSignedUrlLookupTest.m index 7b8ae8278..7ab59366e 100644 --- a/tests/+ndi/+unittest/BatchSignedUrlLookupTest.m +++ b/tests/+ndi/+unittest/BatchSignedUrlLookupTest.m @@ -84,6 +84,28 @@ function quietTheExpectedFallbackWarning(testCase) error('BatchSignedUrlLookupTest:retryBlewUp', 'retry blew up'); end end + + function signer = flakyThenGoodSigner(~, filesMap, counterHandle, raiseCount) + % A signer that raises on the first `raiseCount` calls and + % returns `filesMap` thereafter -- exercises the transient- + % error backoff path. Named error id matches what + % matlab.net.http.send hands back on a real TLS blip, so + % the retry logic is being tested against the shape it will + % see in production. See VH-Lab/NDI-matlab#1010. + signer = @doSign; + function [ok, answer] = doSign(~, ~, varargin) + counterHandle('n') = counterHandle('n') + 1; + counterHandle('lastArgs') = varargin; + if counterHandle('n') <= raiseCount + error('MATLAB:webservices:ConnectionFailed', ... + 'Error downloading content from https://example.invalid'); + end + ok = true; + answer = struct('files', filesMap, ... + 'pages', 1, ... + 'expiresAt', '2026-12-31T00:00:00Z'); + end + end end methods (Test) @@ -482,6 +504,143 @@ function testPartialMapRetryThatRaisesStillFallsBack(testCase) stats.lastFailureReason)); end + function testTransientErrorRetriesBeforePerMemberFallback(testCase) + % One transient TLS blip at start of run must NOT cascade to + % per-member fallback for a whole series. Before the retry, + % a MATLAB:webservices:ConnectionFailed on the first batch + % call marked the scope failed and cost N getFileDetails + % calls for a 156k-member lightsheet series -- 4+ hours of + % wall clock. With the retry, one raise is absorbed by a + % second signer call and every uid in the scope resolves + % from the cached map. See VH-Lab/NDI-matlab#1010 and + % Waltham-Data-Science/NDI-python#322. + counter = containers.Map('KeyType','char','ValueType','any'); + counter('n') = 0; + filesMap = testCase.mapOf({'u_1', 'https://s3/u_1'; ... + 'u_2', 'https://s3/u_2'; ... + 'u_3', 'https://s3/u_3'}); + signer = testCase.flakyThenGoodSigner(filesMap, counter, 1); + + % Zero delays so the suite stays fast; the retry mechanics + % are what this test asserts, not the wait times. + url1 = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds", "doc", "chunks", "u_1", ... + 'signer', signer, 'clearCache', true, ... + 'retryBackoffSeconds', [0 0 0]); + testCase.verifyEqual(url1, 'https://s3/u_1', ... + 'the first uid should resolve from the retried batch'); + testCase.verifyEqual(counter('n'), 2, ... + 'exactly one retry after the transient raise'); + + % Every other uid in the same scope must ride the cached + % map -- no more signer calls, no per-member fallback. + url2 = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds", "doc", "chunks", "u_2", 'signer', signer); + url3 = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds", "doc", "chunks", "u_3", 'signer', signer); + testCase.verifyEqual(url2, 'https://s3/u_2'); + testCase.verifyEqual(url3, 'https://s3/u_3'); + testCase.verifyEqual(counter('n'), 2, ... + 'later uids must not trigger another signer call'); + + [~, stats] = ndi.cloud.download.internal.batchSignedUrlLookup("", "", "", ""); + testCase.verifyEqual(stats.uidHits, 3); + testCase.verifyEqual(stats.uidMisses, 0, ... + 'no per-member fallback -- the retry rescued the scope'); + testCase.verifyEqual(stats.signerCalls, 2, ... + 'initial raise + one successful retry'); + testCase.verifyEqual(stats.transientRetries, 1, ... + 'one retry was needed to recover from the transient'); + end + + function testTransientRetriesAreBoundedThenFallBack(testCase) + % The retry is a safety net, not a busy loop. Exhaust the + % backoff and the caller is handed "" so the per-member + % fallback still runs -- the batch retry replaces the O(N) + % cascade with ~20 s of wall clock, but a scope that + % legitimately cannot be reached must not spin forever. + counter = containers.Map('KeyType','char','ValueType','any'); + counter('n') = 0; + filesMap = testCase.mapOf({'u_1', 'https://s3/u_1'}); + % 999 keeps every attempt raising -- more than any + % retryBackoffSeconds this test would set. + signer = testCase.flakyThenGoodSigner(filesMap, counter, 999); + + url = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds", "doc", "chunks", "u_1", ... + 'signer', signer, 'clearCache', true, ... + 'retryBackoffSeconds', [0 0]); + + testCase.verifyEqual(url, "", ... + 'exhausted retries should return "" so the caller falls back'); + testCase.verifyEqual(counter('n'), 3, ... + 'three attempts total: one initial call, two retries'); + + [~, stats] = ndi.cloud.download.internal.batchSignedUrlLookup("", "", "", ""); + testCase.verifyEqual(stats.transientRetries, 2, ... + 'two retries were spent, matching numel(retryBackoffSeconds)'); + testCase.verifyEqual(stats.uidMisses, 1, ... + 'the final failure still counts as a miss'); + testCase.verifyTrue(contains(stats.lastFailureReason, ... + 'MATLAB:webservices:ConnectionFailed'), ... + sprintf('lastFailureReason should name the raise; got %s', ... + stats.lastFailureReason)); + end + + function testTransientRetryPausesBetweenAttempts(testCase) + % The backoff delays are what turn hammering a flaky + % endpoint into a graceful wait. Verify each delay is + % actually applied, and only between failed attempts (a + % successful attempt breaks the loop before its delay). + counter = containers.Map('KeyType','char','ValueType','any'); + counter('n') = 0; + filesMap = testCase.mapOf({'u_1', 'https://s3/u_1'}); + % Raise twice, succeed on the third call. Two delays should + % fire (between attempt 1 -> 2 and 2 -> 3). + signer = testCase.flakyThenGoodSigner(filesMap, counter, 2); + + box = containers.Map('KeyType','char','ValueType','any'); + box('sleeps') = []; + appendSleep = @(s) localAppendToMap(box, 'sleeps', s); + + ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds", "doc", "chunks", "u_1", ... + 'signer', signer, 'clearCache', true, ... + 'retryBackoffSeconds', [0.1 0.2 0.4], ... + 'sleepFcn', appendSleep); + + testCase.verifyEqual(box('sleeps'), [0.1 0.2], ... + 'exactly one delay per retry, in order'); + end + + function testTransientRetryDoesNotFireOnOkFalse(testCase) + % An ok=false response is a server-side no, not a transient. + % Retrying it hammers a dead endpoint for zero benefit. + % Verify the retry loop only reacts to a raise. + function [ok, answer] = failingSigner(varargin) %#ok + ok = false; + answer = struct('message','server said no'); + end + + box = containers.Map('KeyType','char','ValueType','any'); + box('sleeps') = []; + appendSleep = @(s) localAppendToMap(box, 'sleeps', s); + + url = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds", "doc", "chunks", "u_1", ... + 'signer', @failingSigner, 'clearCache', true, ... + 'retryBackoffSeconds', [1 4 16], ... + 'sleepFcn', appendSleep); + testCase.verifyEqual(url, ""); + + [~, stats] = ndi.cloud.download.internal.batchSignedUrlLookup("", "", "", ""); + testCase.verifyEqual(stats.signerCalls, 1, ... + 'ok=false must not trigger a retry'); + testCase.verifyEqual(stats.transientRetries, 0); + testCase.verifyEmpty(box('sleeps'), ... + 'no backoff should have been slept -- nothing to retry'); + end + function testExpiredEntryIsRefetched(testCase) % Once TTL has elapsed a cached scope is dropped and the % next call goes back to the signer, so a MATLAB session From 2b23dd3062cc69bf1d75208ff79f2f8eb548a67d Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 14:29:34 +0000 Subject: [PATCH 088/110] TestProgressBarWindow: guard the timeout-restore teardown testAutoCloseOnTimeout drives updateBar('TUpdated', 0.2) with both bars past their timeout and Auto=true. checkTimeout marks both as Timeout, the auto-close loop removes both, and with AutoDelete on (the constructor default) deleteIfNoOpenBars then tears the app down. The @() setfield(app, 'Timeout', originalTimeout) teardown then hits MATLAB:class:InvalidHandle when it touches the deleted AppBase handle, and the test errors even though verifyWarning succeeded. Guard the restore teardown with isvalid(app) via a small file-local helper. Matches the safe*-wrapper pattern b367283 established for list_binary_files and zipForUpload -- a caller that may itself have triggered the teardown of a graphics handle should not then crash on that handle. Also drop the (1,:) size constraint on the new addBar Timeout option: duration.empty is 0x0, which is not (1,:), and MATLAB's arguments-block default validation trips on that on some releases. The size constraint bought nothing (nothing reads a shape off the value) and the type-only spec still rejects a non-duration input. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz --- .../+ndi/+gui/+component/ProgressBarWindow.m | 2 +- .../+gui/+component/TestProgressBarWindow.m | 19 +++++++++++++++++-- 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/src/ndi/+ndi/+gui/+component/ProgressBarWindow.m b/src/ndi/+ndi/+gui/+component/ProgressBarWindow.m index 5b9fe236b..5de57f331 100644 --- a/src/ndi/+ndi/+gui/+component/ProgressBarWindow.m +++ b/src/ndi/+ndi/+gui/+component/ProgressBarWindow.m @@ -273,7 +273,7 @@ % get culled by updateBar's global timeout sweep while its % owner was still using it. See NDI-matlab lightsheet demo % 2026-09-20. - options.Timeout (1,:) duration = duration.empty + options.Timeout duration = duration.empty end % Bring figure to front diff --git a/tests/+ndi/+unittest/+gui/+component/TestProgressBarWindow.m b/tests/+ndi/+unittest/+gui/+component/TestProgressBarWindow.m index ca708fc25..c3f3a7879 100644 --- a/tests/+ndi/+unittest/+gui/+component/TestProgressBarWindow.m +++ b/tests/+ndi/+unittest/+gui/+component/TestProgressBarWindow.m @@ -408,10 +408,16 @@ function testAutoCloseOnTimeout(testCase) testCase.addTeardown(@delete, app.ProgressFigure); originalTimeout = app.Timeout; app.setTimeout(duration(0,0,0.1)); % 0.1 seconds - testCase.addTeardown(@() setfield(app,'Timeout',originalTimeout)); + % Guard the restore teardown against AutoDelete: this test's + % updateBar call auto-closes both bars, and with AutoDelete + % on (the constructor default) the app is then deleted. + % Touching a deleted AppBase handle raises + % MATLAB:class:InvalidHandle, which would fail the test + % even though the assertion above succeeded. + testCase.addTeardown(@() restoreTimeoutIfAlive(app, originalTimeout)); app.addBar('Tag', 'TAutoTimeout', 'Auto', true); app.addBar('Tag', 'TUpdated', 'Auto', true); - + pause(0.2); % Wait for timeout testCase.verifyWarning(@() app.updateBar('TUpdated', 0.2), 'ProgressBarWindow:AutoCloseOnTimeout'); end @@ -589,4 +595,13 @@ function verifyButtonIconIsError(testCase, buttonHandle) end end end +end + +function restoreTimeoutIfAlive(app, originalTimeout) + % Teardown helper: restore the app's Timeout only if the app is + % still a live handle. A test that drives AutoDelete to tear the + % window down cannot then reach through app to write a property. + if isvalid(app) + app.Timeout = originalTimeout; + end end \ No newline at end of file From a84130a73d472231b6d32b48f5dced56c0fea34f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 14:36:06 +0000 Subject: [PATCH 089/110] fetchManifest: bypass batch for single-uid manifest fetch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The manifest fetch inside updateFileInfoForRemoteFiles handed one known uid to batchSignedUrlLookup and then fell back to getFileDetails when the batch returned empty. The batch is scoped per document, so answering that one-uid question forced it to walk the whole document's signed-URL set -- for a 121k-chunk lightsheet level, ~243 pages at ~25 s each = ~100 min per manifest; seven levels ≈ 12 h before any bytes hit disk. A cloud download that had been hanging for that reason completed in seconds once the analog was fixed in NDI-python (commit 5ab9c04 on the same branch). Extract fetchManifest to its own file (+ndi/+cloud/+sync/+internal/fetchManifest.m) with DetailsFetcher and FileFetcher name-value seams so the direct-fetch path is unit-testable without a live server, and call getFileDetails directly for the single known uid. batchSignedUrlLookup stays where it earns its cost -- downloadGenericFiles (many uids per doc) and didsqlite's member handler (many members per manifest). Only the one-uid manifest lookup is redirected. tests/+ndi/+unittest/+cloud/+sync/TestFetchManifest asserts three things via injected fetchers: (a) the direct path calls getFileDetails exactly once with the manifest uid and never touches batchSignedUrlLookup (verified through its own stats), (b) the handler path still short-circuits with seriesName="" so DID#201's manifest-vs-member switch keeps working, (c) a failing details fetcher raises NDI:cloud:sync:ManifestFetchFailed with the manifest uid in the message so upstream error handling stays intact. See VH-Lab/NDI-matlab#1010 and Waltham-Data-Science/NDI-python 5ab9c04 for the Python analog. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz --- .../+cloud/+sync/+internal/fetchManifest.m | 111 +++++++++++ .../+internal/updateFileInfoForRemoteFiles.m | 56 +----- .../+cloud/+sync/TestFetchManifest.m | 178 ++++++++++++++++++ 3 files changed, 292 insertions(+), 53 deletions(-) create mode 100644 src/ndi/+ndi/+cloud/+sync/+internal/fetchManifest.m create mode 100644 tests/+ndi/+unittest/+cloud/+sync/TestFetchManifest.m diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/fetchManifest.m b/src/ndi/+ndi/+cloud/+sync/+internal/fetchManifest.m new file mode 100644 index 000000000..0ae6bfad1 --- /dev/null +++ b/src/ndi/+ndi/+cloud/+sync/+internal/fetchManifest.m @@ -0,0 +1,111 @@ +function fetchManifest(destPath, cloudDatasetId, manifestUid, documentId, ... + seriesName, customFileHandler, options) +%FETCHMANIFEST Download a single series manifest to a local path. +% +% Extracted from ndi.cloud.sync.internal.updateFileInfoForRemoteFiles +% so the manifest fetch can be exercised in a test without dragging +% the whole reconstruction path in behind it. +% +% A manifest is a SINGLE KNOWN uid, so the URL is resolved with one +% direct getFileDetails call. Going through the per-document batch +% cache (ndi.cloud.download.internal.batchSignedUrlLookup) here would +% force the batch to walk the whole document's signed-URL set to +% answer a question that has one correct answer: for a 121k-chunk +% lightsheet level that is ~243 pages at ~25 s each -- roughly +% 100 min per manifest fetch, seven levels ≈ 12 h before any bytes +% hit disk. The batch is worth its cost when many uids share the +% document; for a single uid it is pure overhead. See +% VH-Lab/NDI-matlab#1010 and Waltham-Data-Science/NDI-python +% commit 5ab9c04 (the Python analog fix). +% +% Inputs: +% destPath - Local file to write the manifest bytes to. +% cloudDatasetId - The cloud dataset id. +% manifestUid - The manifest file's uid. +% documentId - The NDI document id (passed to the +% customFileHandler context; ignored on the +% direct-fetch path). +% seriesName - The series' name (passed to the handler +% context so it can identify the series +% whose manifest is being fetched). +% customFileHandler - When non-empty, the fetch is dispatched +% through DID's customFileHandler contract +% (same shape DID#201's fetchSeriesManifest +% uses on the read side). Otherwise a signed +% URL is minted and the file downloaded +% directly, matching didsqlite.m's own +% download_file_from_cloud path. +% +% Name-Value Pairs: +% DetailsFetcher - Function handle called as +% [ok, answer] = fcn(datasetId, fileUid) to +% resolve one uid's signed URL. Defaults to +% @ndi.cloud.api.files.getFileDetails. Present +% so a test can inject a scripted answer and +% prove which single-uid function was called +% with which uid, without hitting the network. +% FileFetcher - Function handle called as +% [ok, msg] = fcn(url, path, 'useCurl', true) +% to write the manifest bytes to disk. Defaults +% to @ndi.cloud.api.files.getFile. Same +% injection point for tests. +% +% See also: ndi.cloud.api.files.getFileDetails, +% ndi.cloud.api.files.getFile, +% ndi.cloud.download.internal.batchSignedUrlLookup + + arguments + destPath (1,1) string + cloudDatasetId (1,1) string + manifestUid (1,1) string + documentId (1,1) string + seriesName (1,1) string + customFileHandler + options.DetailsFetcher (1,1) function_handle = ... + @ndi.cloud.api.files.getFileDetails + options.FileFetcher (1,1) function_handle = ... + @ndi.cloud.api.files.getFile + end + + % seriesName is deliberately '' in the ctx: this is the manifest + % itself, not a member. On the DID contract, a non-empty + % seriesName marks a MEMBER fetch, where sourcePath names the + % manifest and context.uid names the member -- the handler must + % then treat context.uid as the file to retrieve, not the manifest + % again. A manifest fetch has no such switch: sourcePath names the + % manifest and context.uid names the same manifest. + sourcePath = sprintf('ndic://%s/%s', char(cloudDatasetId), char(manifestUid)); + + if ~isempty(customFileHandler) + ctx = struct( ... + 'documentId', char(documentId), ... + 'filename', char(seriesName), ... + 'seriesName', '', ... + 'uid', char(manifestUid), ... + 'mode', 'open'); + did.implementations.sqlitedb.dispatchCustomFileHandler( ... + customFileHandler, char(destPath), sourcePath, ctx); + return + end + + [success, answer] = options.DetailsFetcher( ... + char(cloudDatasetId), char(manifestUid)); + if ~success + message = ''; + if isstruct(answer) && isfield(answer, 'message') + message = char(string(answer.message)); + end + error('NDI:cloud:sync:ManifestFetchFailed', ... + 'Failed to get file details for manifest %s: %s', ... + char(manifestUid), message); + end + fileUrl = answer.downloadUrl; + + [success2, answer2] = options.FileFetcher( ... + char(fileUrl), char(destPath), 'useCurl', true); + if ~success2 + error('NDI:cloud:sync:ManifestFetchFailed', ... + 'Failed to download manifest %s from cloud: %s', ... + char(manifestUid), char(string(answer2))); + end +end diff --git a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m index 4004af9fc..cec5ead33 100644 --- a/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m +++ b/src/ndi/+ndi/+cloud/+sync/+internal/updateFileInfoForRemoteFiles.m @@ -227,8 +227,9 @@ destPath = fullfile(tmpDir, manifestUid); try - fetchManifest(destPath, cloudDatasetId, string(manifestUid), ... - documentId, char(entry.name), customFileHandler); + ndi.cloud.sync.internal.fetchManifest( ... + string(destPath), string(cloudDatasetId), string(manifestUid), ... + string(documentId), string(entry.name), customFileHandler); catch % A manifest that cannot be fetched leaves the entry % un-reconstructed; DID#185's guard will then fire on @@ -244,57 +245,6 @@ end -function fetchManifest(destPath, cloudDatasetId, manifestUid, documentId, ... - seriesName, customFileHandler) - % Fetch one manifest by uid. If a customFileHandler was passed in, - % offer the fetch through the DID contract -- same shape DID#201's - % fetchSeriesManifestBytes uses on the read side -- so tests can - % inject a mock and the two sides share their fetcher. Otherwise - % mint a signed URL and fetch directly, matching didsqlite.m's own - % download_file_from_cloud for a non-member ndic:// file. - % - % seriesName is deliberately '' in the context: this is the manifest - % itself, not a member. On the DID contract, a non-empty seriesName - % marks a MEMBER fetch, where sourcePath names the manifest and - % context.uid names the member -- the handler must then treat - % context.uid as the file to retrieve, not the manifest again. A - % manifest fetch has no such switch: sourcePath names the manifest - % and context.uid names the same manifest. - - sourcePath = sprintf('ndic://%s/%s', char(cloudDatasetId), char(manifestUid)); - - if ~isempty(customFileHandler) - ctx = struct( ... - 'documentId', char(documentId), ... - 'filename', char(seriesName), ... - 'seriesName', '', ... - 'uid', char(manifestUid), ... - 'mode', 'open'); - did.implementations.sqlitedb.dispatchCustomFileHandler( ... - customFileHandler, destPath, sourcePath, ctx); - return - end - - fileUrl = ndi.cloud.download.internal.batchSignedUrlLookup( ... - string(cloudDatasetId), documentId, "", manifestUid); - if strlength(fileUrl) == 0 - [success, answer] = ndi.cloud.api.files.getFileDetails( ... - char(cloudDatasetId), char(manifestUid)); - if ~success - error('NDI:cloud:sync:ManifestFetchFailed', ... - 'Failed to get file details for manifest %s: %s', ... - char(manifestUid), answer.message); - end - fileUrl = answer.downloadUrl; - end - [success2, answer2] = ndi.cloud.api.files.getFile( ... - char(fileUrl), char(destPath), 'useCurl', true); - if ~success2 - error('NDI:cloud:sync:ManifestFetchFailed', ... - 'Failed to download manifest %s from cloud: %s', ... - char(manifestUid), answer2); - end -end function uid = lookupManifestUid(fileInfo, seriesName) diff --git a/tests/+ndi/+unittest/+cloud/+sync/TestFetchManifest.m b/tests/+ndi/+unittest/+cloud/+sync/TestFetchManifest.m new file mode 100644 index 000000000..ad84cb6a8 --- /dev/null +++ b/tests/+ndi/+unittest/+cloud/+sync/TestFetchManifest.m @@ -0,0 +1,178 @@ +classdef TestFetchManifest < matlab.unittest.TestCase +% TESTFETCHMANIFEST - manifest fetch bypasses the batch cache. +% +% A manifest is a SINGLE KNOWN uid, so its URL is resolved by ONE direct +% getFileDetails call. Going through the per-document +% batchSignedUrlLookup here would force the batch to walk the whole +% document's signed-URL set -- ~243 pages at ~25 s each for a 121k-chunk +% lightsheet level (~100 min per manifest, seven levels ≈ 12 h before any +% bytes hit disk). The batch is worth its cost when many uids share the +% document; for a single known uid it is pure overhead. See +% VH-Lab/NDI-matlab#1010 and Waltham-Data-Science/NDI-python commit +% 5ab9c04 (the Python analog fix). + + properties + WorkDir + end + + methods (TestMethodSetup) + function useWorkingFolder(testCase) + testCase.applyFixture(matlab.unittest.fixtures.WorkingFolderFixture); + testCase.WorkDir = pwd; + % Clear the batch cache so a prior test's signer calls + % cannot show up in this test's counters. Reading the + % counters back with an empty documentId does not disturb + % the cache. + ndi.cloud.download.internal.batchSignedUrlLookup( ... + "", "", "", "", 'clearCache', true); + end + end + + methods (Test) + + function testManifestFetchGoesDirectAndSkipsTheBatch(testCase) + % The whole point: fetchManifest for a single known uid + % must issue exactly one getFileDetails call and must NOT + % call batchSignedUrlLookup at all. Before this fix the + % batch would answer that one-uid question by paging the + % whole document's signed-URL set -- a ~12 h hang on a + % lightsheet level's manifest (NDI-matlab#1010). + destPath = fullfile(testCase.WorkDir, 'manifest.bin'); + recorded = containers.Map('KeyType','char','ValueType','any'); + recorded('detailsCalls') = 0; + recorded('detailsArgs') = {}; + recorded('fileCalls') = 0; + recorded('fileArgs') = {}; + detailsFcn = @(datasetId, fileUid) localScriptedDetails( ... + recorded, datasetId, fileUid, 'https://s3.example/manifest'); + fileFcn = @(url, path, varargin) localScriptedFile( ... + recorded, url, path, varargin); + + ndi.cloud.sync.internal.fetchManifest( ... + string(destPath), "ds1", "manifest_uid_A", ... + "doc1", "chunkdata.bin", [], ... + 'DetailsFetcher', detailsFcn, ... + 'FileFetcher', fileFcn); + + testCase.verifyEqual(recorded('detailsCalls'), 1, ... + 'exactly one getFileDetails call for one manifest uid'); + args = recorded('detailsArgs'); + testCase.verifyEqual(char(args{1}), 'ds1'); + testCase.verifyEqual(char(args{2}), 'manifest_uid_A', ... + 'the direct call must carry the manifest uid'); + + testCase.verifyEqual(recorded('fileCalls'), 1, ... + 'the resolved URL must be fetched once'); + fargs = recorded('fileArgs'); + testCase.verifyEqual(char(fargs{1}), 'https://s3.example/manifest'); + testCase.verifyEqual(char(fargs{2}), char(destPath)); + + % The batch must not have been touched. Read the counters + % back with an empty documentId -- that path returns early + % without incrementing anything, so it is safe to call as a + % pure read. + [~, batchStats] = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "", "", "", ""); + testCase.verifyEqual(batchStats.signerCalls, 0, ... + 'batchSignedUrlLookup must NOT be reached on a manifest fetch'); + testCase.verifyEqual(batchStats.uidHits, 0); + testCase.verifyEqual(batchStats.uidMisses, 0); + end + + function testManifestFetchGoesThroughHandlerWhenProvided(testCase) + % When a customFileHandler is provided (DID contract), the + % fetch is dispatched through it and neither getFileDetails + % nor getFile is called. This preserves the existing DID + % #201 shape and is what the offline manifest-resolution + % test in TestSeriesWithCloudOnlyManifestResolvesThroughHandler + % exercises at the integration layer. Asserted here at the + % unit layer so a regression in the branch order shows up + % without spinning up the full DID stack. + destPath = fullfile(testCase.WorkDir, 'manifest.bin'); + recorded = containers.Map('KeyType','char','ValueType','any'); + recorded('detailsCalls') = 0; + recorded('fileCalls') = 0; + recorded('handlerCalls') = 0; + recorded('handlerCtx') = {}; + + detailsFcn = @(varargin) localFailUnexpectedCall(recorded, 'detailsCalls'); %#ok + fileFcn = @(varargin) localFailUnexpectedCall(recorded, 'fileCalls'); %#ok + + handler = @(dest, src, ctx) localRecordHandlerCall( ... + recorded, dest, src, ctx); + + ndi.cloud.sync.internal.fetchManifest( ... + string(destPath), "ds1", "manifest_uid_B", ... + "docB", "chunkdata.bin", handler); + + testCase.verifyEqual(recorded('handlerCalls'), 1, ... + 'the handler path must fire when a handler is supplied'); + testCase.verifyEqual(recorded('detailsCalls'), 0, ... + 'getFileDetails must not be called on the handler path'); + testCase.verifyEqual(recorded('fileCalls'), 0, ... + 'getFile must not be called on the handler path'); + + ctx = recorded('handlerCtx'); + testCase.verifyEqual(char(ctx.uid), 'manifest_uid_B'); + testCase.verifyEqual(char(ctx.filename), 'chunkdata.bin'); + testCase.verifyEqual(char(ctx.seriesName), '', ... + 'the manifest context must carry seriesName="" so the handler treats uid as the manifest itself, not a member'); + testCase.verifyEqual(char(ctx.mode), 'open'); + end + + function testDetailsFetcherFailureIsReported(testCase) + % Preserve the pre-existing error shape: a getFileDetails + % that says ok=false raises NDI:cloud:sync:ManifestFetchFailed + % with the manifest uid in the message. Callers upstream + % (updateFileInfoForRemoteFiles) rely on catching this to + % leave the entry un-reconstructed so DID#185's guard fires + % on add_docs. + destPath = fullfile(testCase.WorkDir, 'manifest.bin'); + failingFcn = @(varargin) deal(false, ... + struct('message','server said no')); + + testCase.verifyError(@() ndi.cloud.sync.internal.fetchManifest( ... + string(destPath), "ds1", "manifest_uid_C", ... + "docC", "chunkdata.bin", [], ... + 'DetailsFetcher', failingFcn), ... + 'NDI:cloud:sync:ManifestFetchFailed'); + end + + end +end + +function [ok, answer] = localScriptedDetails(recorded, datasetId, fileUid, url) + recorded('detailsCalls') = recorded('detailsCalls') + 1; + recorded('detailsArgs') = {datasetId, fileUid}; + ok = true; + answer = struct('downloadUrl', url); +end + +function [ok, msg] = localScriptedFile(recorded, url, path, varargin) + recorded('fileCalls') = recorded('fileCalls') + 1; + recorded('fileArgs') = {url, path, varargin}; + ok = true; + msg = 'ok'; +end + +function [ok, msg] = localFailUnexpectedCall(recorded, key) + % A stub that any test using the handler path expects NOT to be + % called. Record it so the assertion fires with a clear counter, + % rather than raising here (which would leave the test in a less + % informative state). + recorded(key) = recorded(key) + 1; + ok = false; + msg = 'unexpected call'; +end + +function localRecordHandlerCall(recorded, dest, src, ctx) %#ok + recorded('handlerCalls') = recorded('handlerCalls') + 1; + recorded('handlerCtx') = ctx; + % Write a tiny placeholder so the caller sees the handler landed + % something at destPath. + fid = fopen(char(dest), 'w'); + if fid > 0 + fwrite(fid, uint8(0), 'uint8'); + fclose(fid); + end +end From 8f468dcea379c0b10d794bf4dcd7e50c63fb1663 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 15:40:33 +0000 Subject: [PATCH 090/110] Add TestSignedUrlSetJobScaling: reproducer for server-side job bug createSignedURLSetJob against a lightsheetZarrLevel chunk.bin series on a production dataset (6ab034e430a0f8d0e461dfcc, 121k members) fails with "Manifest for file series 'chunk.bin' (uid X) is not a file of this dataset" -- while getFileDetails(datasetId, X) for the same uid returns 200 OK right before and right after. Two server code paths disagree. The 15-file lightsheet fixture does NOT trigger it, so the failure lives somewhere between "small integration fixture" and "121k-member real dataset". The test walks N up in a bisectable ramp (5, 100, 500, 2000, 10000 members) via a parametrized ndi.test.lightsheet.makeBlobFixture -> ndi.fun.doc.lightsheet.fromOMEZarr build, uploads uploadAsNew with skipMetadataEditorMetadata, waits for bulk-upload extraction, then per N: * extracts the manifest uid locally from the level document's chunk.bin binary path (same source ndi.database.internal. list_binary_files reads at upload time); * calls createSignedURLSetJob(datasetId, docId, 'idNamespace', 'ndi', 'fileSeries', 'chunk.bin') and waitForSignedURLSetJob to a terminal state; * INDEPENDENTLY calls getFileDetails(datasetId, manifestUid); * prints one [scaleprobe] row with N, actual member count, jobId, jobState, elapsed, mapSize / errorMessage, manifestFetchable, manifestUid, datasetId. The row IS the deliverable. A row where jobState=failed and manifestFetchable=1 is the disagreement -- the bisected reproducer the API team can attach to the bug. A run where every N stays consistent means the failing production dataset's issue is upload-path-specific (workflow / MATLAB version / ingest history), not size-specific, and the drift lives somewhere else. Test does not fail on job failure: the failure IS the observation. Only wildly inconsistent state (BOTH job failed AND manifest fetch failed, which contradicts the reported bug) is treated as a red test, since that means setup went sideways. Skips cleanly without NDI_CLOUD_USERNAME / NDI_CLOUD_PASSWORD via the same assumeNotEmpty idiom the rest of ndi.unittest.cloud uses, so testCloudApi() picks the test up without changing the workflow config. Every dataset the test creates is deleted on teardown (per-method addTeardown + a class-level sweep on top). See VH-Lab/NDI-matlab#1010 addendum. Related Python analog: Waltham-Data-Science/NDI-python 5ab9c04. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz --- .../+bugreports/TestSignedUrlSetJobScaling.m | 402 ++++++++++++++++++ 1 file changed, 402 insertions(+) create mode 100644 tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m diff --git a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m new file mode 100644 index 000000000..fa0f4b600 --- /dev/null +++ b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m @@ -0,0 +1,402 @@ +classdef TestSignedUrlSetJobScaling < matlab.unittest.TestCase +% TESTSIGNEDURLSETJOBSCALING - reproducer for the async signed-URL-set-job bug. +% +% The bug (verified live): createSignedURLSetJob for a lightsheetZarrLevel +% chunk.bin series on a production dataset fails with +% "Manifest for file series 'chunk.bin' (uid ) is not a file of +% this dataset" +% while getFileDetails(datasetId, ) for the same uid returns 200 OK +% right before and right after. Two server code paths disagree on +% whether the manifest is a file of the dataset. The failing dataset has +% 121,441 members; the existing 15-file lightsheet fixture does NOT +% trigger it. This test walks N up in a bisectable ramp so the failure +% (if reproducible from a fresh upload at all) is bracketed. +% +% What each row does, per N: +% a) Build a synthetic OME-Zarr blob whose level-0 chunk.bin series +% has approximately N members (via ndi.test.lightsheet.makeBlobFixture), +% ingest it into a fresh session and dataset, upload with +% uploadAsNew, wait for server-side bulk-upload extraction. +% b) Extract the manifest UID locally from the level document's +% chunk.bin binary path (ndi.database.internal.list_binary_files +% does the same thing at upload time). +% c) Call createSignedURLSetJob(datasetId, docId, 'ndi', 'fileSeries', +% 'chunk.bin') and waitForSignedURLSetJob to a terminal state. +% d) INDEPENDENTLY call getFileDetails(datasetId, manifestUid); a +% 200 here in the same instant createSignedURLSetJob says "not a +% file of this dataset" is the disagreement we are bisecting. +% e) Record: N (intended), actualMemberCount, jobId, jobState, +% jobElapsedSec, mapSize (ready) or errorMessage (failed), +% manifestFetchable, datasetId. +% +% Skips cleanly without NDI_CLOUD_USERNAME / NDI_CLOUD_PASSWORD (same +% idiom as FileSeriesRoundTripTest). Every dataset the test creates is +% deleted on teardown so the test account does not accumulate cruft. +% +% What we want the run to tell us: +% * If a size triggers the bug: this is a reproducer created by a +% fresh upload using only test-account state -- attach the results +% to the bug report and the API team can bisect on N. +% * If no size triggers it: the failing dataset's issue is +% upload-path-specific (workflow / MATLAB version / ingest history), +% not size-specific. Compare its provenance to what this test does +% to find the drift. +% +% Either outcome is actionable. Once the server bug is fixed the test +% becomes a permanent regression guard. +% +% See VH-Lab/NDI-matlab#1010 addendum. Related Python analog: +% Waltham-Data-Science/NDI-python 5ab9c04. + + properties (Constant) + DatasetNamePrefix = 'NDI_UNITTEST_SIGNED_URL_SCALING_'; + % chunk.bin_# member ≈ (fixed) uint16 bytes per spatial chunk. + % Shape is chosen at runtime per N so total spatial chunks ≈ N + % for a single-channel, single-level blob store. See setupPer + % method. + BaseChunkShape = [16 16 16]; + end + + properties (TestParameter) + % Intended member count. The actual count comes back from the + % local level document after ingest and is what the table + % reports -- Shape/ChunkShape only aims for the target. + % + % A run with credentials but a small time budget can restrict + % via the MATLAB unittest -Selector infrastructure or just let + % the small sizes run and Ctrl-C before the big one. 10000 is + % included per the task brief -- it is closer to a lightsheet + % level than the small sizes are. + targetMemberCount = struct( ... + 'N5', 5, ... + 'N100', 100, ... + 'N500', 500, ... + 'N2000', 2000, ... + 'N10000', 10000); + end + + properties + WorkDir char = '' + DatasetIDs string = string.empty % for class-level cleanup + Results struct = struct( ... + 'N', {}, ... + 'actualMemberCount',{}, ... + 'jobId', {}, ... + 'jobState', {}, ... + 'jobElapsedSec', {}, ... + 'mapSize', {}, ... + 'errorMessage', {}, ... + 'manifestFetchable',{}, ... + 'manifestUid', {}, ... + 'datasetId', {}) + end + + methods (TestClassSetup) + function checkCredentials(testCase) + username = getenv("NDI_CLOUD_USERNAME"); + password = getenv("NDI_CLOUD_PASSWORD"); + diagMsg = ['Missing NDI Cloud credentials ' ... + '(NDI_CLOUD_USERNAME/NDI_CLOUD_PASSWORD). ' ... + 'Skipping TestSignedUrlSetJobScaling.']; + testCase.assumeNotEmpty(username, diagMsg); + testCase.assumeNotEmpty(password, diagMsg); + end + + function checkLightsheetToolchain(testCase) + % A run without the lightsheet builder skips rather than + % errors: the fixture path leans on it heavily and the + % failure mode we are hunting is server-side, not local. + testCase.assumeTrue(... + exist('ndi.test.lightsheet.makeBlobFixture','file') == 2, ... + ['ndi.test.lightsheet.makeBlobFixture is not on the ' ... + 'path. This test needs the lightsheet fixture builder.']); + testCase.assumeTrue(... + exist('ndi.fun.doc.lightsheet.fromOMEZarr','file') == 2, ... + ['ndi.fun.doc.lightsheet.fromOMEZarr is not on the ' ... + 'path. This test needs the OME-Zarr -> pyramid+level ' ... + 'ingest.']); + end + end + + methods (TestClassTeardown) + function cleanupUploadedDatasets(testCase) + % Belt-and-braces: every method's own teardown already tries + % to delete its dataset, but a mid-method crash may leave + % one behind. Sweep at class teardown too, silently. + for i = 1:numel(testCase.DatasetIDs) + did = testCase.DatasetIDs(i); + if strlength(did) == 0, continue, end + try + ndi.cloud.api.datasets.deleteDataset( ... + char(did), 'when', 'now'); %#ok + catch + end + end + end + end + + methods (TestMethodSetup) + function setupWorkFolder(testCase) + import matlab.unittest.fixtures.TemporaryFolderFixture + fx = testCase.applyFixture(TemporaryFolderFixture); + testCase.WorkDir = fx.Folder; + end + end + + methods (Test, ParameterCombination = 'sequential') + + function testCreateSignedURLSetJobForNMemberSeries(testCase, targetMemberCount) + % Runs one (build + upload + createSignedURLSetJob) round and + % records the outcome. Assertions are DELIBERATELY LOOSE: a + % failure is a DATA POINT, not a red test. The test only fails + % if setup / upload / local extraction goes wrong, since those + % would poison the row and mask the server-side outcome we are + % here to observe. The row is what the reader reads. + + row = testCase.freshRow(targetMemberCount); + + % --- 1. Build the fixture at the size that yields ~N members + [shape, chunkShape] = pickShapeForTarget(targetMemberCount, ... + testCase.BaseChunkShape); + zarrParent = fullfile(testCase.WorkDir, 'zarr'); + mkdir(zarrParent); + [zarrPath, ~] = ndi.test.lightsheet.makeBlobFixture(zarrParent, ... + 'Shape', shape, ... + 'NumChannels', 1, ... + 'NumLevels', 1, ... + 'ChunkShape', chunkShape); + + % --- 2. Ingest into a fresh session + dataset + sessionDir = fullfile(testCase.WorkDir, 'session'); + mkdir(sessionDir); + S = ndi.session.dir('scaleprobe', sessionDir); + subject = ndi.document('subject', ... + 'base.session_id', S.id(), ... + 'subject.local_identifier', 'scaleprobe@vhlab'); + S.database_add(subject); + + [~, levelDocs, ~] = ndi.fun.doc.lightsheet.fromOMEZarr( ... + S, zarrPath, ... + 'subjectID', subject.id(), ... + 'materializeChunks', true, ... + 'codec', 'raw'); + testCase.assertNotEmpty(levelDocs, ... + 'fromOMEZarr produced no level documents; bail before the cloud call.'); + + % The bug is triggered by ONE chunk.bin series scope on ONE + % level document, so pick level 0 (highest resolution -> most + % members) and use it for the rest of the run. + levelDoc = firstLevelDoc(levelDocs); + + % --- 3. Wrap in a dataset and upload + datasetDir = fullfile(testCase.WorkDir, 'dataset'); + mkdir(datasetDir); + D = ndi.dataset.dir('scaleprobe_ds', datasetDir); + D = D.add_ingested_session(S); + + uniqueName = char(sprintf('%s%s', ... + testCase.DatasetNamePrefix, did.ido.unique_id())); + [ok, cloudDatasetId, msg] = ndi.cloud.uploadDataset(D, ... + 'uploadAsNew', true, ... + 'skipMetadataEditorMetadata', true, ... + 'remoteDatasetName', uniqueName); + testCase.assertTrue(ok, ... + sprintf('uploadDataset failed for N=%d: %s', ... + targetMemberCount, char(string(msg)))); + testCase.DatasetIDs(end+1) = string(cloudDatasetId); + row.datasetId = string(cloudDatasetId); + % Register per-method cleanup so a failed run still tears + % this dataset down. + testCase.addTeardown(@() safeDeleteDataset(cloudDatasetId)); + + % --- 4. Wait for the server to finish extracting the bulk zip + [waitOk, waitInfo] = ndi.cloud.api.files.waitForAllBulkUploads( ... + cloudDatasetId); + testCase.assertTrue(waitOk, ... + sprintf('waitForAllBulkUploads did not confirm completion (state=%s, elapsed=%.1fs).', ... + char(string(waitInfo.state)), waitInfo.elapsed)); + + % --- 5. Extract the manifest uid from the LOCAL level doc + [manifestExists, manifestLocalPath] = ... + D.database_existbinarydoc(levelDoc.id(), 'chunk.bin'); + testCase.assertTrue(manifestExists, ... + sprintf('chunk.bin manifest for level doc %s was not found locally.', ... + levelDoc.id())); + [~, manifestUid, ~] = fileparts(char(manifestLocalPath)); + row.manifestUid = string(manifestUid); + + % Actual member count from the local manifest -- what the + % table reports so N vs. reality is legible. + try + localManifest = did.file.readSeriesManifest(char(manifestLocalPath)); + row.actualMemberCount = localManifest.count; + catch + row.actualMemberCount = -1; + end + + % --- 6. createSignedURLSetJob + wait + ndiDocumentId = levelDoc.id(); + tStart = tic; + [createOk, createAnswer] = ndi.cloud.api.files.createSignedURLSetJob( ... + string(cloudDatasetId), string(ndiDocumentId), ... + 'idNamespace', "ndi", ... + 'fileSeries', "chunk.bin"); + if ~createOk + row.jobState = "createRejected"; + row.jobId = ""; + row.errorMessage = extractMessage(createAnswer); + else + row.jobId = string(createAnswer.jobId); + [readyOk, jobAnswer] = ndi.cloud.api.files.waitForSignedURLSetJob( ... + row.jobId, 'timeout', 300); + if readyOk + row.jobState = "ready"; + if isfield(jobAnswer, 'fileCount') && ~isempty(jobAnswer.fileCount) + row.mapSize = double(jobAnswer.fileCount); + end + row.errorMessage = ""; + else + row.jobState = string(safeField(jobAnswer, 'state', 'failed')); + row.errorMessage = extractMessage(jobAnswer); + end + end + row.jobElapsedSec = toc(tStart); + + % --- 7. Independent getFileDetails for the same manifest uid + [detailsOk, ~] = ndi.cloud.api.files.getFileDetails( ... + string(cloudDatasetId), row.manifestUid); + row.manifestFetchable = detailsOk; + + testCase.Results(end+1) = row; + printRow(row); + + % The test itself does not fail on a job failure -- that IS + % the observation the run is here to make. Only a wildly + % inconsistent state fails: the manifest is not fetchable AND + % the job also failed, which contradicts the reported cloud + % bug (job fails while manifest fetch works). That signals + % the bug reproduction went sideways, not that the bug + % reproduced. + if strcmp(row.jobState, "failed") && ~row.manifestFetchable + testCase.verifyTrue(false, ... + sprintf(['Both the job and the direct manifest ' ... + 'fetch failed for uid %s (N=%d, actual=%d). ' ... + 'The reported bug is a DISAGREEMENT between those ' ... + 'two -- this row is not the disagreement, so ' ... + 'something else broke setup.'], ... + char(row.manifestUid), targetMemberCount, ... + row.actualMemberCount)); + end + end + + end + + methods (Access = private) + function row = freshRow(testCase, N) %#ok + row = struct( ... + 'N', N, ... + 'actualMemberCount', 0, ... + 'jobId', "", ... + 'jobState', "unstarted", ... + 'jobElapsedSec', 0, ... + 'mapSize', NaN, ... + 'errorMessage', "", ... + 'manifestFetchable', false, ... + 'manifestUid', "", ... + 'datasetId', ""); + end + end +end + +function [shape, chunkShape] = pickShapeForTarget(target, baseChunkShape) + % Aim for level-0 spatial chunks ≈ target. NumChannels=1, + % NumLevels=1 in the fixture builder, so total series members + % ≈ prod(ceil(shape ./ chunkShape)). + % + % Factor target as k*k*(target/k^2) with k = round(target^(1/3)) + % so shapes stay roughly cubic. A cubic-ish layout matters because + % the lightsheet levels the failing dataset carries are cubic-ish + % too; a stringy 1xN layout could miss a shape-dependent bug the + % server has. + chunkShape = baseChunkShape; + if target <= 1 + chunks = [1 1 1]; + else + k = max(1, round(target^(1/3))); + c1 = k; + c2 = k; + c3 = max(1, ceil(target / (c1*c2))); + chunks = [c1 c2 c3]; + end + shape = chunks .* chunkShape; +end + +function d = firstLevelDoc(levelDocs) + % levelDocs can be a cell array or a struct-of-arrays or a single + % doc depending on the fromOMEZarr version. Normalize to a scalar + % ndi.document. + if iscell(levelDocs) + d = levelDocs{1}; + elseif numel(levelDocs) > 1 + d = levelDocs(1); + else + d = levelDocs; + end +end + +function printRow(r) + % One row per N to the MATLAB console. Printed immediately (not + % held for a class-teardown summary) because parametrized test + % cases run in fresh test instances -- rows do not accumulate on + % one testCase across parameter values, so the log itself IS the + % table. Grep for [scaleprobe] in a CI run to pull the rows out. + detail = char(r.errorMessage); + if numel(detail) > 60, detail = [detail(1:57) '...']; end + fprintf(['[scaleprobe] N=%-6d actual=%-6d state=%-10s ' ... + 'elapsed=%6.1fs mapSize=%-8s manifestFetchable=%d ' ... + 'manifestUid=%s dataset=%s jobId=%s detail="%s"\n'], ... + r.N, r.actualMemberCount, char(r.jobState), r.jobElapsedSec, ... + mapSizeAsText(r.mapSize), r.manifestFetchable, ... + char(r.manifestUid), char(r.datasetId), char(r.jobId), detail); +end + +function s = mapSizeAsText(value) + if isnumeric(value) && isscalar(value) && ~isnan(value) + s = sprintf('%d', round(value)); + else + s = '-'; + end +end + +function m = extractMessage(payload) + % Server-side error payloads travel in a few shapes. Try the common + % ones so the table row carries whatever the server said, verbatim. + m = ""; + if isstruct(payload) + if isfield(payload, 'message') && ~isempty(payload.message) + m = string(payload.message); + elseif isfield(payload, 'error') && ~isempty(payload.error) + m = string(payload.error); + elseif isfield(payload, 'state') && ~isempty(payload.state) + m = string(payload.state); + end + elseif ischar(payload) || isstring(payload) + m = string(payload); + end +end + +function v = safeField(s, name, defaultValue) + if isstruct(s) && isfield(s, name) && ~isempty(s.(name)) + v = s.(name); + else + v = defaultValue; + end +end + +function safeDeleteDataset(cloudDatasetId) + try + ndi.cloud.api.datasets.deleteDataset(char(cloudDatasetId), ... + 'when', 'now'); %#ok + catch + end +end From 286d26189baea3294f5115259158d507840a1def Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 17:30:45 +0000 Subject: [PATCH 091/110] Add TestSignedUrlSetJobFixture: pin async job vs. getFileDetails disagreement Confirmed 2026-09-27: the async signed-URL-set-job failure is GLOBAL, not dataset-specific. The Python integration test raises the same BatchScopeUnreachable against the 15-file tiny lightsheet fixture (6ab84b549852a120dbcb22bc, User 1 prod) that it raises against the 121k-file production dataset, while getFileDetails(datasetId, manifestUid) returns 200 OK for the fixture's manifest uid in the same instant. Two server code paths disagree. This is the MATLAB-side reproducer. Read-only against the fixture (no uploads, no cleanup). For each lightsheetZarrLevel document in the fixture: * listDatasetDocumentsAll picks the level docs out by className. * getDocument fetches full properties, from which files.file_info yields the chunk.bin series' manifest uid (same shape ndi.cloud.sync.internal.updateFileInfoForRemote Files reads on the download side). * createSignedURLSetJob(idNamespace='ndi', fileSeries='chunk.bin') + waitForSignedURLSetJob to a terminal state. * INDEPENDENTLY getFileDetails(datasetId, manifestUid). Consistency assertion: * both ok -> healthy (mapSize > 0 sanity check). * both fail -> consistent, not the bug. * async job fails but getFileDetails returns 200 -> THE BUG. Accumulate the disagreement across every level doc so the final message names each cloudDocId / ndiDocId / manifestUid the API team can cite. * async job succeeds but getFileDetails fails -> also fail (the opposite mismatch is diagnostic on its own). Rows print as [fixture-probe] lines while the test runs so a CI log captures the sequence even if a later doc times out. Lands under ndi.unittest.cloud so testCloudApi() picks it up under the existing test-cloud-api.yml matrix without a workflow edit. Skips cleanly without NDI_CLOUD_USERNAME/PASSWORD via the same assumeNotEmpty idiom the rest of ndi.unittest.cloud uses. See VH-Lab/NDI-matlab#1010. Python analog test lives under NDI-python on the same branch (test_cloud_signed_url_set_job.py). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz --- .../+cloud/TestSignedUrlSetJobFixture.m | 239 ++++++++++++++++++ 1 file changed, 239 insertions(+) create mode 100644 tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m diff --git a/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m b/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m new file mode 100644 index 000000000..72cc4bf2a --- /dev/null +++ b/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m @@ -0,0 +1,239 @@ +classdef TestSignedUrlSetJobFixture < matlab.unittest.TestCase +% TESTSIGNEDURLSETJOBFIXTURE - pin the async signed-URL-set-job bug +% against the existing tiny lightsheet fixture. +% +% Confirmed on 2026-09-27: the Python integration test raises the same +% BatchScopeUnreachable against the 15-file fixture +% (6ab84b549852a120dbcb22bc, User 1 prod) that it raises against the +% 121k-file production dataset -- so the failure is GLOBAL, not +% dataset-specific. getFileDetails(datasetId, manifestUid) returns +% 200 OK for the fixture's manifest uid in the same instant. Two +% server code paths disagree on whether the manifest is a file of +% the dataset. +% +% This test asserts the disagreement from the MATLAB side. For each +% lightsheetZarrLevel document in the fixture: +% * createSignedURLSetJob(id_namespace='ndi', fileSeries='chunk.bin') +% + waitForSignedURLSetJob to a terminal state. +% * INDEPENDENTLY getFileDetails(datasetId, manifestUid). +% Assert consistency: either both succeed OR both fail. Any row where +% getFileDetails returns 200 but the async job says "not a file of +% this dataset" IS the reproducer, and fails the test with a message +% naming the datasetId / docId / manifestUid so the API bug report +% can cite it. +% +% Read-only against the fixture. No uploads, no cleanup. +% +% See VH-Lab/NDI-matlab#1010. The Python analog lives under +% NDI-python's tests/test_cloud_signed_url_set_job.py on the same +% branch. + + properties (Constant) + FixtureDatasetId = "6ab84b549852a120dbcb22bc" + LevelClassName = "lightsheetZarrLevel" + SeriesName = "chunk.bin" + % Cap the poll time per document. The fixture is tiny, so a + % healthy run reaches terminal state in seconds; a run that + % blows past this is diagnostic in its own right (job stuck vs + % job failed) and the row records the "timeout" state. + WaitTimeoutSeconds = 120 + end + + methods (TestClassSetup) + function checkCredentials(testCase) + username = getenv("NDI_CLOUD_USERNAME"); + password = getenv("NDI_CLOUD_PASSWORD"); + testCase.assumeNotEmpty(username, ... + ['Missing NDI Cloud credentials ' ... + '(NDI_CLOUD_USERNAME/NDI_CLOUD_PASSWORD). ' ... + 'Skipping TestSignedUrlSetJobFixture.']); + testCase.assumeNotEmpty(password, ... + ['Missing NDI Cloud credentials ' ... + '(NDI_CLOUD_USERNAME/NDI_CLOUD_PASSWORD). ' ... + 'Skipping TestSignedUrlSetJobFixture.']); + end + end + + methods (Test) + + function testLevelDocsAgreeOnManifestReachability(testCase) + % Enumerate the fixture's lightsheetZarrLevel documents, + % probe each one's chunk.bin series through the two paths, + % and fail on any disagreement. + + datasetId = testCase.FixtureDatasetId; + + [okList, docSummaries] = ndi.cloud.api.documents.listDatasetDocumentsAll(datasetId); + testCase.assertTrue(okList, ... + sprintf(['listDatasetDocumentsAll failed for fixture %s. ' ... + 'The fixture must exist and be reachable for this ' ... + 'test to run.'], char(datasetId))); + + % Filter to lightsheetZarrLevel docs. className comes back as + % either "lightsheetZarrLevel" or char depending on the + % server; compare as string. + classNames = arrayfun(@(s) string(safeField(s, 'className', '')), ... + docSummaries); + levelIdx = find(classNames == testCase.LevelClassName); + testCase.assertNotEmpty(levelIdx, ... + sprintf(['Fixture %s has no %s documents. This test ' ... + 'expects the tiny lightsheet fixture (15 files) to ' ... + 'be present.'], char(datasetId), char(testCase.LevelClassName))); + + disagreements = string.empty; + + for k = 1:numel(levelIdx) + summary = docSummaries(levelIdx(k)); + cloudDocId = string(safeField(summary, 'id', '')); + ndiDocId = string(safeField(summary, 'ndiId', '')); + + % Full doc for its files.file_info -- that is where the + % manifest uid for the chunk.bin series lives, in the + % same shape ndi.cloud.sync.internal.updateFileInfoForRemoteFiles + % reads server-side. + [okDoc, docProps] = ndi.cloud.api.documents.getDocument( ... + datasetId, cloudDocId); + if ~okDoc + fprintf(['[fixture-probe] doc=%s: getDocument ' ... + 'failed; skipping.\n'], char(cloudDocId)); + continue + end + manifestUid = extractManifestUidFromDoc(docProps, testCase.SeriesName); + if strlength(manifestUid) == 0 + fprintf(['[fixture-probe] doc=%s: no chunk.bin ' ... + 'file_info entry; skipping (not a level with ' ... + 'a materialized manifest).\n'], char(cloudDocId)); + continue + end + + % --- async job (the failing side, per Python) + tStart = tic; + [okCreate, createAnswer] = ndi.cloud.api.files.createSignedURLSetJob( ... + datasetId, ndiDocId, ... + 'idNamespace', "ndi", ... + 'fileSeries', testCase.SeriesName); + if ~okCreate + jobState = "createRejected"; + jobDetail = extractMessage(createAnswer); + jobMapSize = NaN; + else + jobId = string(createAnswer.jobId); + [okReady, jobAnswer] = ndi.cloud.api.files.waitForSignedURLSetJob( ... + jobId, 'timeout', testCase.WaitTimeoutSeconds); + if okReady + jobState = "ready"; + jobDetail = ""; + jobMapSize = double(safeField(jobAnswer, 'fileCount', NaN)); + else + jobState = string(safeField(jobAnswer, 'state', 'failed')); + jobDetail = extractMessage(jobAnswer); + jobMapSize = NaN; + end + end + jobElapsedSec = toc(tStart); + jobOk = jobState == "ready"; + + % --- independent single-uid fetch (the succeeding side) + [detailsOk, detailsAnswer] = ndi.cloud.api.files.getFileDetails( ... + datasetId, manifestUid); + + fprintf(['[fixture-probe] doc=%s ndi=%s manifestUid=%s ' ... + 'jobState=%s elapsed=%.1fs mapSize=%s ' ... + 'manifestFetchable=%d detail="%s"\n'], ... + char(cloudDocId), char(ndiDocId), char(manifestUid), ... + char(jobState), jobElapsedSec, mapSizeAsText(jobMapSize), ... + detailsOk, char(jobDetail)); + + % Consistency assertion. + if jobOk && detailsOk + % Healthy: both paths agree that the manifest is a + % file of the dataset. mapSize > 0 is a sanity + % check on the async blob. + testCase.verifyGreaterThan(jobMapSize, 0, ... + sprintf(['jobMapSize should be > 0 for a ' ... + 'ready job on doc %s.'], char(cloudDocId))); + elseif ~jobOk && ~detailsOk + % Consistent failure. Not the bug we are pinning. + fprintf(['[fixture-probe] doc=%s: consistent ' ... + 'failure (both paths say no).\n'], char(cloudDocId)); + elseif ~jobOk && detailsOk + % THE BUG. Record for the summary assertion below. + disagreements(end+1) = sprintf( ... + 'doc=%s ndi=%s manifestUid=%s: async job %s (%s) but getFileDetails returned 200', ... + char(cloudDocId), char(ndiDocId), char(manifestUid), ... + char(jobState), char(jobDetail)); %#ok + else % jobOk && !detailsOk + disagreements(end+1) = sprintf( ... + 'doc=%s: async job ready but getFileDetails failed for manifestUid=%s (%s)', ... + char(cloudDocId), char(manifestUid), ... + extractMessage(detailsAnswer)); %#ok + end + end + + if ~isempty(disagreements) + testCase.verifyTrue(false, sprintf( ... + ['createSignedURLSetJob and getFileDetails disagree ' ... + 'on %d document(s) of fixture %s. This is the ' ... + 'server-side bug tracked in NDI-matlab#1010: ' ... + 'the async job says the manifest is not a file ' ... + 'of the dataset while the single-uid fetch ' ... + 'returns 200 for the same uid.\n%s'], ... + numel(disagreements), char(datasetId), ... + strjoin(" " + disagreements, newline))); + end + end + + end +end + +function uid = extractManifestUidFromDoc(docProps, seriesName) + % The manifest uid for a series named NAME sits in + % document.files.file_info as the locations(1).uid of the entry + % whose name is NAME. Same shape ndi.cloud.sync.internal. + % updateFileInfoForRemoteFiles reads on the download side. + uid = ""; + if ~isstruct(docProps), return, end + files = safeField(docProps, 'files', struct()); + fileInfo = safeField(files, 'file_info', []); + if isempty(fileInfo), return, end + for i = 1:numel(fileInfo) + entry = fileInfo(i); + name = string(safeField(entry, 'name', '')); + if name ~= seriesName, continue, end + locs = safeField(entry, 'locations', []); + if isempty(locs), return, end + uid = string(safeField(locs(1), 'uid', '')); + return + end +end + +function v = safeField(s, name, defaultValue) + if isstruct(s) && isfield(s, name) && ~isempty(s.(name)) + v = s.(name); + else + v = defaultValue; + end +end + +function m = extractMessage(payload) + m = ""; + if isstruct(payload) + if isfield(payload, 'message') && ~isempty(payload.message) + m = string(payload.message); + elseif isfield(payload, 'error') && ~isempty(payload.error) + m = string(payload.error); + elseif isfield(payload, 'state') && ~isempty(payload.state) + m = string(payload.state); + end + elseif ischar(payload) || isstring(payload) + m = string(payload); + end +end + +function s = mapSizeAsText(value) + if isnumeric(value) && isscalar(value) && ~isnan(value) + s = sprintf('%d', round(value)); + else + s = '-'; + end +end From aab35a14a01a88e47f3847d3b2d61b999377ffc7 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 18:06:16 +0000 Subject: [PATCH 092/110] TestSignedUrlSetJobFixture: build a fresh fixture instead of hardcoding one The old version hardcoded FixtureDatasetId = "6ab84b549852a120dbcb22bc" (User 1 prod), so the test couldn't run against dev -- that dataset id doesn't exist there and listDatasetDocumentsAll would fail with "dataset not found" instead of exercising the actual bug. Since the disagreement is GLOBAL (confirmed 2026-09-27 against both the tiny prod fixture AND the 121k-member production dataset), a probe that builds its own fixture reproduces it wherever the credentials point. buildAndUploadTinyFixture now: * builds an ndi.test.lightsheet.makeBlobFixture (shape [32 32 32], 1 channel, 3 levels, chunk [16 16 16]) -> tiny OME-Zarr with ~5-ish spatial chunks per level; * ingests into a fresh ndi.session.dir + subject, runs ndi.fun.doc.lightsheet.fromOMEZarr with materializeChunks=true, producing several lightsheetZarrLevel documents; * uploads uploadAsNew + skipMetadataEditorMetadata into whichever environment CLOUD_API_ENVIRONMENT points at; * waitForAllBulkUploads before probing; * registers per-test + class-level cleanup so every uploaded dataset is deleted, even on abort. The probe body is unchanged: enumerate lightsheetZarrLevel docs from the CLOUD side via listDatasetDocumentsAll, extract each level's manifest UID from files.file_info, call createSignedURLSetJob + waitForSignedURLSetJob next to an independent getFileDetails, and fail on any (asyncFailed, getFileDetails=200) row with each offending (docId, ndiDocId, manifestUid) triplet named. Test now runs identically on dev and prod, and stops depending on User 1 prod fixture state that a later cleanup could remove. See VH-Lab/NDI-matlab#1010. Python analog: Waltham-Data-Science/NDI-python test_cloud_signed_url_set_job.py on the same branch. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz --- .../+cloud/TestSignedUrlSetJobFixture.m | 273 ++++++++++++------ 1 file changed, 187 insertions(+), 86 deletions(-) diff --git a/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m b/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m index 72cc4bf2a..5a507b9b9 100644 --- a/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m +++ b/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m @@ -1,98 +1,141 @@ classdef TestSignedUrlSetJobFixture < matlab.unittest.TestCase % TESTSIGNEDURLSETJOBFIXTURE - pin the async signed-URL-set-job bug -% against the existing tiny lightsheet fixture. +% with a self-contained fresh upload. % -% Confirmed on 2026-09-27: the Python integration test raises the same -% BatchScopeUnreachable against the 15-file fixture -% (6ab84b549852a120dbcb22bc, User 1 prod) that it raises against the -% 121k-file production dataset -- so the failure is GLOBAL, not -% dataset-specific. getFileDetails(datasetId, manifestUid) returns -% 200 OK for the fixture's manifest uid in the same instant. Two -% server code paths disagree on whether the manifest is a file of -% the dataset. +% The bug: for a lightsheetZarrLevel document, +% POST /datasets/{d}/documents/{doc}/signed-url-set-jobs +% ?fileSeries=chunk.bin +% reaches state=failed with +% "Manifest for file series 'chunk.bin' (uid ) is not a file +% of this dataset" +% while adjacent +% GET /datasets/{d}/files/{UID}/detail +% for the same UID returns 200 OK. Two server code paths disagree +% on whether the manifest UID is a file of the dataset. % -% This test asserts the disagreement from the MATLAB side. For each -% lightsheetZarrLevel document in the fixture: -% * createSignedURLSetJob(id_namespace='ndi', fileSeries='chunk.bin') +% Confirmed 2026-09-27 against both the tiny lightsheet fixture on +% User 1 prod (6ab84b549852a120dbcb22bc, 5-of-5 level docs) AND the +% 121k-member production dataset (6ab034e430a0f8d0e461dfcc). The +% failure is GLOBAL, not tied to that one dataset. See VH-Lab/ +% NDI-matlab#1010 and the Python analog on the same branch of +% NDI-python (test_cloud_signed_url_set_job.py, commit 5ab9c04). +% +% This test builds its OWN fresh lightsheet fixture on each run -- +% no hardcoded dataset id, no dependency on User 1 prod state -- so +% it runs the same probe against whichever environment +% CLOUD_API_ENVIRONMENT / NDI_CLOUD_USERNAME is currently pointed +% at. Uploads a tiny synthetic OME-Zarr (about 5 lightsheetZarrLevel +% documents), waits for bulk-upload extraction, then for each level +% doc: +% * createSignedURLSetJob(idNamespace='ndi', fileSeries='chunk.bin') % + waitForSignedURLSetJob to a terminal state. % * INDEPENDENTLY getFileDetails(datasetId, manifestUid). -% Assert consistency: either both succeed OR both fail. Any row where -% getFileDetails returns 200 but the async job says "not a file of -% this dataset" IS the reproducer, and fails the test with a message -% naming the datasetId / docId / manifestUid so the API bug report -% can cite it. -% -% Read-only against the fixture. No uploads, no cleanup. +% Assert consistency: both ok, or both fail. Any row where the +% single-uid fetch returns 200 but the async job says "not a file +% of this dataset" IS the reproducer and fails the test with a +% message naming every offending (docId, ndiDocId, manifestUid) +% triplet so the API team's bug report can cite them. % -% See VH-Lab/NDI-matlab#1010. The Python analog lives under -% NDI-python's tests/test_cloud_signed_url_set_job.py on the same -% branch. +% Every dataset the test creates is deleted on teardown so test +% accounts stay tidy. properties (Constant) - FixtureDatasetId = "6ab84b549852a120dbcb22bc" - LevelClassName = "lightsheetZarrLevel" - SeriesName = "chunk.bin" - % Cap the poll time per document. The fixture is tiny, so a - % healthy run reaches terminal state in seconds; a run that - % blows past this is diagnostic in its own right (job stuck vs - % job failed) and the row records the "timeout" state. + DatasetNamePrefix = "NDI_UNITTEST_SIGNED_URL_FIXTURE_" + LevelClassName = "lightsheetZarrLevel" + SeriesName = "chunk.bin" WaitTimeoutSeconds = 120 end + properties + WorkDir char = '' + DatasetIDs string = string.empty + end + methods (TestClassSetup) function checkCredentials(testCase) username = getenv("NDI_CLOUD_USERNAME"); password = getenv("NDI_CLOUD_PASSWORD"); - testCase.assumeNotEmpty(username, ... - ['Missing NDI Cloud credentials ' ... - '(NDI_CLOUD_USERNAME/NDI_CLOUD_PASSWORD). ' ... - 'Skipping TestSignedUrlSetJobFixture.']); - testCase.assumeNotEmpty(password, ... - ['Missing NDI Cloud credentials ' ... - '(NDI_CLOUD_USERNAME/NDI_CLOUD_PASSWORD). ' ... - 'Skipping TestSignedUrlSetJobFixture.']); + diagMsg = ['Missing NDI Cloud credentials ' ... + '(NDI_CLOUD_USERNAME/NDI_CLOUD_PASSWORD). ' ... + 'Skipping TestSignedUrlSetJobFixture.']; + testCase.assumeNotEmpty(username, diagMsg); + testCase.assumeNotEmpty(password, diagMsg); + end + + function checkLightsheetToolchain(testCase) + testCase.assumeTrue(... + exist('ndi.test.lightsheet.makeBlobFixture','file') == 2, ... + ['ndi.test.lightsheet.makeBlobFixture is not on the ' ... + 'path. This test needs the lightsheet fixture builder.']); + testCase.assumeTrue(... + exist('ndi.fun.doc.lightsheet.fromOMEZarr','file') == 2, ... + ['ndi.fun.doc.lightsheet.fromOMEZarr is not on the path.']); + end + end + + methods (TestClassTeardown) + function cleanupUploadedDatasets(testCase) + % Sweep every dataset this class created, even if a + % per-test teardown already tried; a failed run may have + % left one behind. Best-effort, silent. + for i = 1:numel(testCase.DatasetIDs) + did = testCase.DatasetIDs(i); + if strlength(did) == 0, continue, end + try + ndi.cloud.api.datasets.deleteDataset(char(did), ... + 'when', 'now'); %#ok + catch + end + end + end + end + + methods (TestMethodSetup) + function setupWorkFolder(testCase) + import matlab.unittest.fixtures.TemporaryFolderFixture + fx = testCase.applyFixture(TemporaryFolderFixture); + testCase.WorkDir = fx.Folder; end end methods (Test) function testLevelDocsAgreeOnManifestReachability(testCase) - % Enumerate the fixture's lightsheetZarrLevel documents, - % probe each one's chunk.bin series through the two paths, - % and fail on any disagreement. + % Build and upload a fresh lightsheet fixture, then run + % the disagreement probe against every lightsheetZarrLevel + % document the upload produced. - datasetId = testCase.FixtureDatasetId; + cloudDatasetId = testCase.buildAndUploadTinyFixture(); - [okList, docSummaries] = ndi.cloud.api.documents.listDatasetDocumentsAll(datasetId); + % Enumerate lightsheetZarrLevel documents from the cloud + % side -- the same shape any client would use. + [okList, docSummaries] = ndi.cloud.api.documents.listDatasetDocumentsAll( ... + cloudDatasetId); testCase.assertTrue(okList, ... - sprintf(['listDatasetDocumentsAll failed for fixture %s. ' ... - 'The fixture must exist and be reachable for this ' ... - 'test to run.'], char(datasetId))); - - % Filter to lightsheetZarrLevel docs. className comes back as - % either "lightsheetZarrLevel" or char depending on the - % server; compare as string. - classNames = arrayfun(@(s) string(safeField(s, 'className', '')), ... + sprintf(['listDatasetDocumentsAll failed for freshly ' ... + 'uploaded dataset %s.'], char(cloudDatasetId))); + + classNames = arrayfun(@(s) string(safeField(s,'className','')), ... docSummaries); levelIdx = find(classNames == testCase.LevelClassName); testCase.assertNotEmpty(levelIdx, ... - sprintf(['Fixture %s has no %s documents. This test ' ... - 'expects the tiny lightsheet fixture (15 files) to ' ... - 'be present.'], char(datasetId), char(testCase.LevelClassName))); + sprintf(['Uploaded dataset %s has no %s documents. ' ... + 'Fixture upload must have dropped them.'], ... + char(cloudDatasetId), char(testCase.LevelClassName))); disagreements = string.empty; for k = 1:numel(levelIdx) - summary = docSummaries(levelIdx(k)); + summary = docSummaries(levelIdx(k)); cloudDocId = string(safeField(summary, 'id', '')); ndiDocId = string(safeField(summary, 'ndiId', '')); - % Full doc for its files.file_info -- that is where the - % manifest uid for the chunk.bin series lives, in the - % same shape ndi.cloud.sync.internal.updateFileInfoForRemoteFiles - % reads server-side. + % files.file_info on the cloud document names the + % manifest UID -- same shape ndi.cloud.sync.internal. + % updateFileInfoForRemoteFiles reads on the download + % side. [okDoc, docProps] = ndi.cloud.api.documents.getDocument( ... - datasetId, cloudDocId); + cloudDatasetId, cloudDocId); if ~okDoc fprintf(['[fixture-probe] doc=%s: getDocument ' ... 'failed; skipping.\n'], char(cloudDocId)); @@ -101,32 +144,31 @@ function testLevelDocsAgreeOnManifestReachability(testCase) manifestUid = extractManifestUidFromDoc(docProps, testCase.SeriesName); if strlength(manifestUid) == 0 fprintf(['[fixture-probe] doc=%s: no chunk.bin ' ... - 'file_info entry; skipping (not a level with ' ... - 'a materialized manifest).\n'], char(cloudDocId)); + 'file_info entry; skipping.\n'], char(cloudDocId)); continue end - % --- async job (the failing side, per Python) + % --- async job (the failing side, per Python + MATLAB) tStart = tic; [okCreate, createAnswer] = ndi.cloud.api.files.createSignedURLSetJob( ... - datasetId, ndiDocId, ... + cloudDatasetId, ndiDocId, ... 'idNamespace', "ndi", ... 'fileSeries', testCase.SeriesName); if ~okCreate - jobState = "createRejected"; - jobDetail = extractMessage(createAnswer); + jobState = "createRejected"; + jobDetail = extractMessage(createAnswer); jobMapSize = NaN; else jobId = string(createAnswer.jobId); [okReady, jobAnswer] = ndi.cloud.api.files.waitForSignedURLSetJob( ... jobId, 'timeout', testCase.WaitTimeoutSeconds); if okReady - jobState = "ready"; - jobDetail = ""; + jobState = "ready"; + jobDetail = ""; jobMapSize = double(safeField(jobAnswer, 'fileCount', NaN)); else - jobState = string(safeField(jobAnswer, 'state', 'failed')); - jobDetail = extractMessage(jobAnswer); + jobState = string(safeField(jobAnswer, 'state', 'failed')); + jobDetail = extractMessage(jobAnswer); jobMapSize = NaN; end end @@ -135,7 +177,7 @@ function testLevelDocsAgreeOnManifestReachability(testCase) % --- independent single-uid fetch (the succeeding side) [detailsOk, detailsAnswer] = ndi.cloud.api.files.getFileDetails( ... - datasetId, manifestUid); + cloudDatasetId, manifestUid); fprintf(['[fixture-probe] doc=%s ndi=%s manifestUid=%s ' ... 'jobState=%s elapsed=%.1fs mapSize=%s ' ... @@ -144,25 +186,19 @@ function testLevelDocsAgreeOnManifestReachability(testCase) char(jobState), jobElapsedSec, mapSizeAsText(jobMapSize), ... detailsOk, char(jobDetail)); - % Consistency assertion. if jobOk && detailsOk - % Healthy: both paths agree that the manifest is a - % file of the dataset. mapSize > 0 is a sanity - % check on the async blob. testCase.verifyGreaterThan(jobMapSize, 0, ... sprintf(['jobMapSize should be > 0 for a ' ... 'ready job on doc %s.'], char(cloudDocId))); elseif ~jobOk && ~detailsOk - % Consistent failure. Not the bug we are pinning. fprintf(['[fixture-probe] doc=%s: consistent ' ... 'failure (both paths say no).\n'], char(cloudDocId)); elseif ~jobOk && detailsOk - % THE BUG. Record for the summary assertion below. disagreements(end+1) = sprintf( ... 'doc=%s ndi=%s manifestUid=%s: async job %s (%s) but getFileDetails returned 200', ... char(cloudDocId), char(ndiDocId), char(manifestUid), ... char(jobState), char(jobDetail)); %#ok - else % jobOk && !detailsOk + else disagreements(end+1) = sprintf( ... 'doc=%s: async job ready but getFileDetails failed for manifestUid=%s (%s)', ... char(cloudDocId), char(manifestUid), ... @@ -173,24 +209,81 @@ function testLevelDocsAgreeOnManifestReachability(testCase) if ~isempty(disagreements) testCase.verifyTrue(false, sprintf( ... ['createSignedURLSetJob and getFileDetails disagree ' ... - 'on %d document(s) of fixture %s. This is the ' ... - 'server-side bug tracked in NDI-matlab#1010: ' ... - 'the async job says the manifest is not a file ' ... - 'of the dataset while the single-uid fetch ' ... + 'on %d document(s) of freshly-uploaded fixture %s. ' ... + 'This is the server-side bug tracked in NDI-matlab' ... + '#1010: the async job says the manifest is not a ' ... + 'file of the dataset while the single-uid fetch ' ... 'returns 200 for the same uid.\n%s'], ... - numel(disagreements), char(datasetId), ... + numel(disagreements), char(cloudDatasetId), ... strjoin(" " + disagreements, newline))); end end end + + methods (Access = private) + function cloudDatasetId = buildAndUploadTinyFixture(testCase) + % Build a tiny lightsheet OME-Zarr on disk, ingest into a + % fresh session, wrap in a dataset, upload uploadAsNew. + % Small shape/small chunks -> ~5 spatial chunks per level + % across a few levels; the disagreement fires per LEVEL + % document (each carrying its own chunk.bin series), so + % 5-ish level documents is plenty to probe. + + zarrParent = fullfile(testCase.WorkDir, 'zarr'); + mkdir(zarrParent); + [zarrPath, ~] = ndi.test.lightsheet.makeBlobFixture(zarrParent, ... + 'Shape', [32 32 32], ... + 'NumChannels', 1, ... + 'NumLevels', 3, ... + 'ChunkShape', [16 16 16]); + + sessionDir = fullfile(testCase.WorkDir, 'session'); + mkdir(sessionDir); + S = ndi.session.dir('probe', sessionDir); + + subject = ndi.document('subject', ... + 'base.session_id', S.id(), ... + 'subject.local_identifier', 'probe@vhlab'); + S.database_add(subject); + + ndi.fun.doc.lightsheet.fromOMEZarr(S, zarrPath, ... + 'subjectID', subject.id(), ... + 'materializeChunks', true, ... + 'codec', 'raw'); + + datasetDir = fullfile(testCase.WorkDir, 'dataset'); + mkdir(datasetDir); + D = ndi.dataset.dir('probe_ds', datasetDir); + D = D.add_ingested_session(S); + + uniqueName = char(testCase.DatasetNamePrefix + ... + string(did.ido.unique_id())); + [ok, cloudDatasetId, msg] = ndi.cloud.uploadDataset(D, ... + 'uploadAsNew', true, ... + 'skipMetadataEditorMetadata', true, ... + 'remoteDatasetName', uniqueName); + testCase.assertTrue(ok, ... + sprintf('uploadDataset failed: %s', char(string(msg)))); + cloudDatasetId = string(cloudDatasetId); + testCase.DatasetIDs(end+1) = cloudDatasetId; + % Per-test cleanup on top of the class-level sweep, so an + % aborted test still tears its own dataset down. + testCase.addTeardown(@() safeDeleteDataset(cloudDatasetId)); + + [waitOk, waitInfo] = ndi.cloud.api.files.waitForAllBulkUploads( ... + cloudDatasetId); + testCase.assertTrue(waitOk, ... + sprintf(['waitForAllBulkUploads did not confirm ' ... + 'completion (state=%s, elapsed=%.1fs).'], ... + char(string(waitInfo.state)), waitInfo.elapsed)); + end + end end function uid = extractManifestUidFromDoc(docProps, seriesName) - % The manifest uid for a series named NAME sits in - % document.files.file_info as the locations(1).uid of the entry - % whose name is NAME. Same shape ndi.cloud.sync.internal. - % updateFileInfoForRemoteFiles reads on the download side. + % Manifest UID for a series named NAME lives at + % document.files.file_info entry.name==NAME .locations(1).uid. uid = ""; if ~isstruct(docProps), return, end files = safeField(docProps, 'files', struct()); @@ -237,3 +330,11 @@ function testLevelDocsAgreeOnManifestReachability(testCase) s = '-'; end end + +function safeDeleteDataset(cloudDatasetId) + try + ndi.cloud.api.datasets.deleteDataset(char(cloudDatasetId), ... + 'when', 'now'); %#ok + catch + end +end From 427fbd5542a0451c23fc7ef5c45285a89bea51d9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 18:28:54 +0000 Subject: [PATCH 093/110] TestSignedUrl*: probe toolchain with which() not exist(...,'file') exist('ndi.test.lightsheet.makeBlobFixture', 'file') returned 0 on Steve's MATLAB even with src/ndi on the path, so both TestSignedUrlSetJobFixture and TestSignedUrlSetJobScaling filtered themselves out at TestClassSetup with a bogus "not on the path" diagnostic and never ran the actual probe. exist(NAME, 'file') is unreliable for dotted package names -- the 'file' second argument was designed for plain file paths, and its handling of package resolution differs across MATLAB releases. which(NAME) is the idiomatic probe: it returns the resolved file path (or empty) after consulting the same package lookup MATLAB would use to invoke the function. Swap both TestClassSetup checks to assumeNotEmpty(which(...)) so the tests skip only when the fixture builder is genuinely absent. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz --- .../+cloud/+bugreports/TestSignedUrlSetJobScaling.m | 11 +++++++---- .../+unittest/+cloud/TestSignedUrlSetJobFixture.m | 11 +++++++---- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m index fa0f4b600..a4c06b370 100644 --- a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m +++ b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m @@ -106,12 +106,15 @@ function checkLightsheetToolchain(testCase) % A run without the lightsheet builder skips rather than % errors: the fixture path leans on it heavily and the % failure mode we are hunting is server-side, not local. - testCase.assumeTrue(... - exist('ndi.test.lightsheet.makeBlobFixture','file') == 2, ... + % which() is the reliable probe for a package function's + % presence; exist(..., 'file') does not always resolve + % dotted package names across MATLAB releases. + testCase.assumeNotEmpty(... + which('ndi.test.lightsheet.makeBlobFixture'), ... ['ndi.test.lightsheet.makeBlobFixture is not on the ' ... 'path. This test needs the lightsheet fixture builder.']); - testCase.assumeTrue(... - exist('ndi.fun.doc.lightsheet.fromOMEZarr','file') == 2, ... + testCase.assumeNotEmpty(... + which('ndi.fun.doc.lightsheet.fromOMEZarr'), ... ['ndi.fun.doc.lightsheet.fromOMEZarr is not on the ' ... 'path. This test needs the OME-Zarr -> pyramid+level ' ... 'ingest.']); diff --git a/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m b/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m index 5a507b9b9..4b6a8fc99 100644 --- a/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m +++ b/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m @@ -63,12 +63,15 @@ function checkCredentials(testCase) end function checkLightsheetToolchain(testCase) - testCase.assumeTrue(... - exist('ndi.test.lightsheet.makeBlobFixture','file') == 2, ... + % which() is the reliable probe for a package function's + % presence on the path; exist(..., 'file') does not always + % resolve dotted package names across MATLAB releases. + testCase.assumeNotEmpty(... + which('ndi.test.lightsheet.makeBlobFixture'), ... ['ndi.test.lightsheet.makeBlobFixture is not on the ' ... 'path. This test needs the lightsheet fixture builder.']); - testCase.assumeTrue(... - exist('ndi.fun.doc.lightsheet.fromOMEZarr','file') == 2, ... + testCase.assumeNotEmpty(... + which('ndi.fun.doc.lightsheet.fromOMEZarr'), ... ['ndi.fun.doc.lightsheet.fromOMEZarr is not on the path.']); end end From 1223661cab4c5e4657b0f46e18049f015a2a361c Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 18:37:55 +0000 Subject: [PATCH 094/110] TestSignedUrlSetJobFixture: narrate + fetch real fileCount from result blob Two fixes to what Steve saw on a live dev run (fresh upload, five lightsheetZarrLevel docs, every one jobState=ready + manifestFetchable=1): 1. The mapSize>0 sanity check was misfiring on ready jobs because waitForSignedURLSetJob's poll answer carries `resultUrl`, not `fileCount` -- the file count lives inside the gzipped JSON blob the resultUrl points at. probeOneLevelDoc now fetches that blob via ndi.cloud.api.files.getSignedURLSetResult when the job is ready, and reads fileCount off the parsed struct. A ready job with a zero-file blob still fails the sanity check with an honest diagnostic, but the healthy path passes. 2. Diagnostics were bare strings; a failed cloud test message needs to be handable to whoever owns ndi-cloud-node without a MATLAB session. Now every step appends to Narrative and every assertion that touches the cloud wraps its message with ndi.unittest.cloud.APIMessage, matching FileSeriesRoundTripTest's established shape (VH-Lab/NDI-matlab#945). The Narrative property is the sequence of steps a reader follows; APIMessage bakes it into a pretty-printed JSON report along with the API call's status line, URL, and body. Refactored the probe into two private methods (resolveManifestUid + probeOneLevelDoc) so the top-level test reads as a narrative and each per-doc failure carries the specific step that produced it. Per-doc [fixture-probe] rows still print (they are the CI-log grep target). Row struct now carries jobResultBlob / jobResultResponse / jobResultUrl so the final "createSignedURLSetJob and getFileDetails disagree" message serializes the server's own words (result blob, poll response, whichever is interesting) into the JSON report. See VH-Lab/NDI-matlab#1010. Python analog: Waltham-Data-Science/NDI-python test_cloud_signed_url_set_job.py on the same branch. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz --- .../+cloud/TestSignedUrlSetJobFixture.m | 439 +++++++++++++----- 1 file changed, 321 insertions(+), 118 deletions(-) diff --git a/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m b/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m index 4b6a8fc99..65a1849f0 100644 --- a/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m +++ b/tests/+ndi/+unittest/+cloud/TestSignedUrlSetJobFixture.m @@ -15,29 +15,21 @@ % % Confirmed 2026-09-27 against both the tiny lightsheet fixture on % User 1 prod (6ab84b549852a120dbcb22bc, 5-of-5 level docs) AND the -% 121k-member production dataset (6ab034e430a0f8d0e461dfcc). The -% failure is GLOBAL, not tied to that one dataset. See VH-Lab/ -% NDI-matlab#1010 and the Python analog on the same branch of -% NDI-python (test_cloud_signed_url_set_job.py, commit 5ab9c04). +% 121k-member production dataset (6ab034e430a0f8d0e461dfcc). See +% VH-Lab/NDI-matlab#1010 and the Python analog on the same branch +% of NDI-python (test_cloud_signed_url_set_job.py, commit 5ab9c04). % -% This test builds its OWN fresh lightsheet fixture on each run -- -% no hardcoded dataset id, no dependency on User 1 prod state -- so -% it runs the same probe against whichever environment -% CLOUD_API_ENVIRONMENT / NDI_CLOUD_USERNAME is currently pointed -% at. Uploads a tiny synthetic OME-Zarr (about 5 lightsheetZarrLevel -% documents), waits for bulk-upload extraction, then for each level -% doc: -% * createSignedURLSetJob(idNamespace='ndi', fileSeries='chunk.bin') -% + waitForSignedURLSetJob to a terminal state. -% * INDEPENDENTLY getFileDetails(datasetId, manifestUid). -% Assert consistency: both ok, or both fail. Any row where the -% single-uid fetch returns 200 but the async job says "not a file -% of this dataset" IS the reproducer and fails the test with a -% message naming every offending (docId, ndiDocId, manifestUid) -% triplet so the API team's bug report can cite them. +% This test builds its own fresh lightsheet fixture on each run -- +% no hardcoded dataset id, no dependency on any environment's fixture +% state -- so it runs the same probe against whichever environment +% CLOUD_API_ENVIRONMENT / NDI_CLOUD_USERNAME points at. % -% Every dataset the test creates is deleted on teardown so test -% accounts stay tidy. +% Diagnostics are narrated the same way FileSeriesRoundTripTest and +% ndi.unittest.cloud.APIMessage produce them: every step appends a +% string to Narrative, and any assertion that touches the cloud +% wraps the message in APIMessage so a failure prints as a JSON +% report the server-side team can read without a MATLAB session. +% See ndi.unittest.cloud.APIMessage and VH-Lab/NDI-matlab#945. properties (Constant) DatasetNamePrefix = "NDI_UNITTEST_SIGNED_URL_FIXTURE_" @@ -47,8 +39,9 @@ end properties - WorkDir char = '' - DatasetIDs string = string.empty + WorkDir char = '' + DatasetIDs string = string.empty + Narrative (1,:) string = string.empty end methods (TestClassSetup) @@ -98,33 +91,47 @@ function setupWorkFolder(testCase) import matlab.unittest.fixtures.TemporaryFolderFixture fx = testCase.applyFixture(TemporaryFolderFixture); testCase.WorkDir = fx.Folder; + testCase.Narrative = "Begin TestSignedUrlSetJobFixture setup " + ... + "in " + string(fx.Folder) + "."; end end methods (Test) function testLevelDocsAgreeOnManifestReachability(testCase) - % Build and upload a fresh lightsheet fixture, then run - % the disagreement probe against every lightsheetZarrLevel - % document the upload produced. + % Build and upload a fresh lightsheet fixture, then probe + % every lightsheetZarrLevel document the upload produced + % for the async-job vs. getFileDetails disagreement. On a + % healthy environment every row should be ready+200. Any + % row where async=failed but detail=200 IS the reported + % server-side bug and this test fails with each offending + % triplet named. cloudDatasetId = testCase.buildAndUploadTinyFixture(); - % Enumerate lightsheetZarrLevel documents from the cloud - % side -- the same shape any client would use. - [okList, docSummaries] = ndi.cloud.api.documents.listDatasetDocumentsAll( ... - cloudDatasetId); + % --- 1. Enumerate lightsheetZarrLevel documents + testCase.Narrative(end+1) = "Enumerating documents on the " + ... + "freshly uploaded dataset via " + ... + "ndi.cloud.api.documents.listDatasetDocumentsAll."; + [okList, docSummaries, listResp, listUrl] = ... + ndi.cloud.api.documents.listDatasetDocumentsAll(cloudDatasetId); + listMsg = ndi.unittest.cloud.APIMessage(testCase.Narrative, ... + okList, docSummaries, listResp, listUrl); testCase.assertTrue(okList, ... - sprintf(['listDatasetDocumentsAll failed for freshly ' ... - 'uploaded dataset %s.'], char(cloudDatasetId))); + "listDatasetDocumentsAll failed for the freshly-uploaded " + ... + "dataset. " + listMsg); classNames = arrayfun(@(s) string(safeField(s,'className','')), ... docSummaries); levelIdx = find(classNames == testCase.LevelClassName); + testCase.Narrative(end+1) = "Filtered to " + numel(levelIdx) + ... + " " + testCase.LevelClassName + " document(s) out of " + ... + numel(docSummaries) + " total."; + noneMsg = ndi.unittest.cloud.APIMessage(testCase.Narrative, ... + true, docSummaries, listResp, listUrl); testCase.assertNotEmpty(levelIdx, ... - sprintf(['Uploaded dataset %s has no %s documents. ' ... - 'Fixture upload must have dropped them.'], ... - char(cloudDatasetId), char(testCase.LevelClassName))); + "Uploaded dataset has no " + testCase.LevelClassName + ... + " documents; fixture upload must have dropped them. " + noneMsg); disagreements = string.empty; @@ -133,92 +140,80 @@ function testLevelDocsAgreeOnManifestReachability(testCase) cloudDocId = string(safeField(summary, 'id', '')); ndiDocId = string(safeField(summary, 'ndiId', '')); - % files.file_info on the cloud document names the - % manifest UID -- same shape ndi.cloud.sync.internal. - % updateFileInfoForRemoteFiles reads on the download - % side. - [okDoc, docProps] = ndi.cloud.api.documents.getDocument( ... - cloudDatasetId, cloudDocId); - if ~okDoc - fprintf(['[fixture-probe] doc=%s: getDocument ' ... - 'failed; skipping.\n'], char(cloudDocId)); + testCase.Narrative(end+1) = "--- Probing " + ... + testCase.LevelClassName + " " + (k) + "/" + ... + numel(levelIdx) + " (cloud id " + cloudDocId + ... + ", ndi id " + ndiDocId + ") ---"; + + [manifestUid, docNarrative, docFatal] = ... + testCase.resolveManifestUid(cloudDatasetId, cloudDocId); + testCase.Narrative = docNarrative; + if docFatal + % resolveManifestUid already appended a reason to + % the narrative; skip this doc without abandoning + % the whole test. continue end - manifestUid = extractManifestUidFromDoc(docProps, testCase.SeriesName); - if strlength(manifestUid) == 0 - fprintf(['[fixture-probe] doc=%s: no chunk.bin ' ... - 'file_info entry; skipping.\n'], char(cloudDocId)); - continue - end - - % --- async job (the failing side, per Python + MATLAB) - tStart = tic; - [okCreate, createAnswer] = ndi.cloud.api.files.createSignedURLSetJob( ... - cloudDatasetId, ndiDocId, ... - 'idNamespace', "ndi", ... - 'fileSeries', testCase.SeriesName); - if ~okCreate - jobState = "createRejected"; - jobDetail = extractMessage(createAnswer); - jobMapSize = NaN; - else - jobId = string(createAnswer.jobId); - [okReady, jobAnswer] = ndi.cloud.api.files.waitForSignedURLSetJob( ... - jobId, 'timeout', testCase.WaitTimeoutSeconds); - if okReady - jobState = "ready"; - jobDetail = ""; - jobMapSize = double(safeField(jobAnswer, 'fileCount', NaN)); - else - jobState = string(safeField(jobAnswer, 'state', 'failed')); - jobDetail = extractMessage(jobAnswer); - jobMapSize = NaN; - end - end - jobElapsedSec = toc(tStart); - jobOk = jobState == "ready"; - % --- independent single-uid fetch (the succeeding side) - [detailsOk, detailsAnswer] = ndi.cloud.api.files.getFileDetails( ... - cloudDatasetId, manifestUid); - - fprintf(['[fixture-probe] doc=%s ndi=%s manifestUid=%s ' ... - 'jobState=%s elapsed=%.1fs mapSize=%s ' ... - 'manifestFetchable=%d detail="%s"\n'], ... - char(cloudDocId), char(ndiDocId), char(manifestUid), ... - char(jobState), jobElapsedSec, mapSizeAsText(jobMapSize), ... - detailsOk, char(jobDetail)); - - if jobOk && detailsOk - testCase.verifyGreaterThan(jobMapSize, 0, ... - sprintf(['jobMapSize should be > 0 for a ' ... - 'ready job on doc %s.'], char(cloudDocId))); - elseif ~jobOk && ~detailsOk - fprintf(['[fixture-probe] doc=%s: consistent ' ... - 'failure (both paths say no).\n'], char(cloudDocId)); - elseif ~jobOk && detailsOk + [row, probeNarrative] = testCase.probeOneLevelDoc( ... + cloudDatasetId, ndiDocId, manifestUid, ... + cloudDocId, testCase.Narrative); + testCase.Narrative = probeNarrative; + + printFixtureProbeRow(row); + + if row.jobOk && row.detailsOk + % Healthy: both paths agree. Real fileCount comes + % from the async job's result blob (fetched inside + % probeOneLevelDoc when the job was ready), so a + % ready-with-zero-files is legible. + successMsg = ndi.unittest.cloud.APIMessage( ... + testCase.Narrative, true, row.jobResultBlob, ... + row.jobResultResponse, row.jobResultUrl); + testCase.verifyGreaterThan(row.jobMapSize, 0, ... + "The async job reported 'ready' but its result " + ... + "blob's fileCount was " + row.jobMapSize + ... + ". A ready manifest scope should carry at least " + ... + "one member. " + successMsg); + elseif ~row.jobOk && ~row.detailsOk + testCase.Narrative(end+1) = "Both paths refused the " + ... + "same manifest UID -- consistent, not the " + ... + "disagreement we are probing for."; + elseif ~row.jobOk && row.detailsOk disagreements(end+1) = sprintf( ... 'doc=%s ndi=%s manifestUid=%s: async job %s (%s) but getFileDetails returned 200', ... char(cloudDocId), char(ndiDocId), char(manifestUid), ... - char(jobState), char(jobDetail)); %#ok - else + char(row.jobState), char(row.jobDetail)); %#ok + testCase.Narrative(end+1) = "DISAGREEMENT: " + ... + "async job state=" + row.jobState + ... + " (" + row.jobDetail + ") but getFileDetails " + ... + "returned 200 for manifest uid " + manifestUid + "."; + else % row.jobOk && ~row.detailsOk disagreements(end+1) = sprintf( ... 'doc=%s: async job ready but getFileDetails failed for manifestUid=%s (%s)', ... char(cloudDocId), char(manifestUid), ... - extractMessage(detailsAnswer)); %#ok + char(row.detailsDetail)); %#ok + testCase.Narrative(end+1) = "INVERSE DISAGREEMENT: " + ... + "async job ready but getFileDetails FAILED for " + ... + "manifest uid " + manifestUid + " (" + ... + row.detailsDetail + ")."; end end if ~isempty(disagreements) + finalMsg = ndi.unittest.cloud.APIMessage( ... + testCase.Narrative, false, ... + struct('disagreements', {cellstr(disagreements)}, ... + 'datasetId', char(cloudDatasetId)), ... + matlab.net.http.ResponseMessage.empty, ... + "createSignedURLSetJob and getFileDetails cross-check"); testCase.verifyTrue(false, sprintf( ... ['createSignedURLSetJob and getFileDetails disagree ' ... 'on %d document(s) of freshly-uploaded fixture %s. ' ... - 'This is the server-side bug tracked in NDI-matlab' ... - '#1010: the async job says the manifest is not a ' ... - 'file of the dataset while the single-uid fetch ' ... - 'returns 200 for the same uid.\n%s'], ... + 'This is the server-side bug tracked in ' ... + 'VH-Lab/NDI-matlab#1010.\n%s\n\n%s'], ... numel(disagreements), char(cloudDatasetId), ... - strjoin(" " + disagreements, newline))); + strjoin(" " + disagreements, newline), finalMsg)); end end @@ -228,28 +223,37 @@ function testLevelDocsAgreeOnManifestReachability(testCase) function cloudDatasetId = buildAndUploadTinyFixture(testCase) % Build a tiny lightsheet OME-Zarr on disk, ingest into a % fresh session, wrap in a dataset, upload uploadAsNew. - % Small shape/small chunks -> ~5 spatial chunks per level - % across a few levels; the disagreement fires per LEVEL - % document (each carrying its own chunk.bin series), so - % 5-ish level documents is plenty to probe. + % Small shape / small chunks -> a handful of level docs, + % each carrying its own chunk.bin series -- enough surface + % for the disagreement to fire on a per-level basis. zarrParent = fullfile(testCase.WorkDir, 'zarr'); mkdir(zarrParent); + testCase.Narrative(end+1) = "Building tiny lightsheet blob " + ... + "fixture at " + string(zarrParent) + " " + ... + "(Shape [32 32 32], NumChannels 1, NumLevels 3, " + ... + "ChunkShape [16 16 16])."; [zarrPath, ~] = ndi.test.lightsheet.makeBlobFixture(zarrParent, ... 'Shape', [32 32 32], ... 'NumChannels', 1, ... 'NumLevels', 3, ... 'ChunkShape', [16 16 16]); + testCase.Narrative(end+1) = "Zarr fixture built at " + ... + string(zarrPath) + "."; sessionDir = fullfile(testCase.WorkDir, 'session'); mkdir(sessionDir); S = ndi.session.dir('probe', sessionDir); - subject = ndi.document('subject', ... 'base.session_id', S.id(), ... 'subject.local_identifier', 'probe@vhlab'); S.database_add(subject); + testCase.Narrative(end+1) = "Session opened at " + ... + string(sessionDir) + " with subject " + string(subject.id()) + "."; + testCase.Narrative(end+1) = "Ingesting the OME-Zarr into the " + ... + "session via ndi.fun.doc.lightsheet.fromOMEZarr " + ... + "(materializeChunks=true, codec=raw)."; ndi.fun.doc.lightsheet.fromOMEZarr(S, zarrPath, ... 'subjectID', subject.id(), ... 'materializeChunks', true, ... @@ -259,27 +263,212 @@ function testLevelDocsAgreeOnManifestReachability(testCase) mkdir(datasetDir); D = ndi.dataset.dir('probe_ds', datasetDir); D = D.add_ingested_session(S); + testCase.Narrative(end+1) = "Dataset created at " + ... + string(datasetDir) + "; ingested session " + string(S.id()) + "."; uniqueName = char(testCase.DatasetNamePrefix + ... string(did.ido.unique_id())); + testCase.Narrative(end+1) = "Uploading to NDI Cloud as " + ... + "remote name '" + string(uniqueName) + "' " + ... + "(uploadAsNew=true, skipMetadataEditorMetadata=true)."; [ok, cloudDatasetId, msg] = ndi.cloud.uploadDataset(D, ... 'uploadAsNew', true, ... 'skipMetadataEditorMetadata', true, ... 'remoteDatasetName', uniqueName); + uploadMsg = ndi.unittest.cloud.APIMessage(testCase.Narrative, ... + ok, msg, matlab.net.http.ResponseMessage.empty, ... + "ndi.cloud.uploadDataset"); testCase.assertTrue(ok, ... - sprintf('uploadDataset failed: %s', char(string(msg)))); + "uploadDataset failed: " + string(msg) + ". " + uploadMsg); cloudDatasetId = string(cloudDatasetId); testCase.DatasetIDs(end+1) = cloudDatasetId; - % Per-test cleanup on top of the class-level sweep, so an - % aborted test still tears its own dataset down. testCase.addTeardown(@() safeDeleteDataset(cloudDatasetId)); - - [waitOk, waitInfo] = ndi.cloud.api.files.waitForAllBulkUploads( ... - cloudDatasetId); + testCase.Narrative(end+1) = "Upload accepted; cloud dataset id is " + ... + cloudDatasetId + "."; + + testCase.Narrative(end+1) = "Waiting for server-side " + ... + "bulk-upload extraction (waitForAllBulkUploads) " + ... + "before probing files."; + [waitOk, waitInfo, waitResp, waitUrl] = ... + ndi.cloud.api.files.waitForAllBulkUploads(cloudDatasetId); + waitMsg = ndi.unittest.cloud.APIMessage(testCase.Narrative, ... + waitOk, waitInfo, waitResp, waitUrl); testCase.assertTrue(waitOk, ... - sprintf(['waitForAllBulkUploads did not confirm ' ... - 'completion (state=%s, elapsed=%.1fs).'], ... - char(string(waitInfo.state)), waitInfo.elapsed)); + "waitForAllBulkUploads did not confirm completion " + ... + "(state=" + string(safeField(waitInfo,'state','')) + ", " + ... + "elapsed=" + safeField(waitInfo,'elapsed',NaN) + "s). " + waitMsg); + testCase.Narrative(end+1) = "Bulk-upload extraction " + ... + "confirmed (elapsed=" + safeField(waitInfo,'elapsed',NaN) + "s)."; + end + + function [manifestUid, narrative, fatal] = resolveManifestUid( ... + testCase, cloudDatasetId, cloudDocId) + % Fetch the cloud document's files.file_info block and pull + % the chunk.bin manifest UID out of it. If we can't reach + % that state, append a narrative line and return fatal=true + % so the caller skips this document without failing the + % whole test. + narrative = testCase.Narrative; + manifestUid = ""; + fatal = false; + + narrative(end+1) = "Fetching full document via " + ... + "ndi.cloud.api.documents.getDocument to read " + ... + "files.file_info."; + [okDoc, docProps, docResp, docUrl] = ... + ndi.cloud.api.documents.getDocument(cloudDatasetId, cloudDocId); + if ~okDoc + docMsg = ndi.unittest.cloud.APIMessage(narrative, ... + okDoc, docProps, docResp, docUrl); + narrative(end+1) = "getDocument failed for cloud id " + ... + cloudDocId + "; skipping this doc. Diagnostic: " + docMsg; + fatal = true; + return + end + manifestUid = extractManifestUidFromDoc(docProps, testCase.SeriesName); + if strlength(manifestUid) == 0 + narrative(end+1) = "Document has no " + testCase.SeriesName + ... + " file_info entry; skipping (not a level with a " + ... + "materialized manifest)."; + fatal = true; + return + end + narrative(end+1) = "Resolved manifest UID for series '" + ... + testCase.SeriesName + "' = " + manifestUid + "."; + end + + function [row, narrative] = probeOneLevelDoc(testCase, ... + cloudDatasetId, ndiDocId, manifestUid, cloudDocId, narrative) + % Do the two adjacent API calls: createSignedURLSetJob + + % waitForSignedURLSetJob (async path) and getFileDetails + % (single-uid path). Return a row struct summarising the + % outcome, plus the extended narrative. + + row = struct( ... + 'cloudDocId', cloudDocId, ... + 'ndiDocId', ndiDocId, ... + 'manifestUid', manifestUid, ... + 'jobId', "", ... + 'jobState', "unstarted", ... + 'jobDetail', "", ... + 'jobElapsedSec', 0, ... + 'jobOk', false, ... + 'jobMapSize', NaN, ... + 'jobResultBlob', [], ... + 'jobResultResponse', matlab.net.http.ResponseMessage.empty, ... + 'jobResultUrl', "", ... + 'detailsOk', false, ... + 'detailsDetail', ""); + + % --- Async job path + narrative(end+1) = "Calling " + ... + "ndi.cloud.api.files.createSignedURLSetJob(" + ... + "idNamespace='ndi', fileSeries='" + testCase.SeriesName + ... + "') on NDI doc id " + ndiDocId + "."; + tStart = tic; + [okCreate, createAnswer, createResp, createUrl] = ... + ndi.cloud.api.files.createSignedURLSetJob( ... + cloudDatasetId, ndiDocId, ... + 'idNamespace', "ndi", ... + 'fileSeries', testCase.SeriesName); + if ~okCreate + row.jobState = "createRejected"; + row.jobDetail = extractMessage(createAnswer); + narrative(end+1) = "createSignedURLSetJob was REJECTED " + ... + "(HTTP-level). Detail: " + row.jobDetail; + else + row.jobId = string(createAnswer.jobId); + narrative(end+1) = "Job accepted; jobId=" + row.jobId + ... + ". Polling with waitForSignedURLSetJob (timeout " + ... + testCase.WaitTimeoutSeconds + "s)."; + [okReady, jobAnswer, waitResp, waitUrl] = ... + ndi.cloud.api.files.waitForSignedURLSetJob( ... + row.jobId, 'timeout', testCase.WaitTimeoutSeconds); + if okReady + row.jobState = "ready"; + row.jobDetail = ""; + row.jobResultUrl = string(safeField(jobAnswer, ... + 'resultUrl', '')); + narrative(end+1) = "Job reached state 'ready'. " + ... + "Fetching the result blob via " + ... + "ndi.cloud.api.files.getSignedURLSetResult to " + ... + "read fileCount (the poll answer does not carry it)."; + if strlength(row.jobResultUrl) > 0 && ... + ~isempty(which('ndi.cloud.api.files.getSignedURLSetResult')) + try + [okResult, resultBlob, resultResp, ~] = ... + ndi.cloud.api.files.getSignedURLSetResult( ... + row.jobResultUrl); + row.jobResultBlob = resultBlob; + row.jobResultResponse = resultResp; + if okResult + row.jobMapSize = double(safeField( ... + resultBlob, 'fileCount', NaN)); + narrative(end+1) = "Result blob parsed; " + ... + "fileCount=" + row.jobMapSize + "."; + else + narrative(end+1) = "Result blob fetch/parse " + ... + "FAILED (blob URL " + row.jobResultUrl + ")."; + end + catch resErr + narrative(end+1) = "Result blob fetch RAISED: " + ... + string(resErr.identifier) + " -- " + ... + string(resErr.message); + end + else + narrative(end+1) = "No resultUrl on the poll " + ... + "answer (or getSignedURLSetResult not on path); " + ... + "leaving fileCount unset."; + end + else + row.jobState = string(safeField(jobAnswer, 'state', 'failed')); + row.jobDetail = extractMessage(jobAnswer); + narrative(end+1) = "Job reached terminal state '" + ... + row.jobState + "'. Detail: " + row.jobDetail; + % Keep the poll response around so the failure JSON + % carries the server's own words verbatim. + row.jobResultResponse = waitResp; + row.jobResultUrl = string(waitUrl); + end + end + row.jobElapsedSec = toc(tStart); + row.jobOk = row.jobState == "ready"; + + % --- Single-uid detail path + narrative(end+1) = "Independently calling " + ... + "ndi.cloud.api.files.getFileDetails(datasetId, " + ... + "manifestUid=" + manifestUid + ")."; + [detailsOk, detailsAnswer, detailsResp, detailsUrl] = ... + ndi.cloud.api.files.getFileDetails( ... + cloudDatasetId, manifestUid); + row.detailsOk = detailsOk; + row.detailsDetail = extractMessage(detailsAnswer); + if detailsOk + narrative(end+1) = "getFileDetails returned 200 for " + ... + "manifest uid " + manifestUid + "."; + else + narrative(end+1) = "getFileDetails FAILED for manifest " + ... + "uid " + manifestUid + ". Detail: " + row.detailsDetail; + end + + % Stash the createSignedURLSetJob response as a fallback for + % the JSON report when the job path is the interesting one. + if ~row.jobOk + row.jobResultResponse = createResp; + row.jobResultUrl = string(createUrl); + if isempty(row.jobResultBlob) + row.jobResultBlob = createAnswer; + end + end + + % Also stash the details response when it is the interesting + % one -- keeps the row self-contained even if the caller + % never reaches the JSON assembly path. + if ~row.detailsOk && isempty(row.jobResultBlob) + row.jobResultBlob = detailsAnswer; + row.jobResultResponse = detailsResp; + row.jobResultUrl = string(detailsUrl); + end end end end @@ -326,6 +515,20 @@ function testLevelDocsAgreeOnManifestReachability(testCase) end end +function printFixtureProbeRow(row) + % Single-line summary per level doc, kept for grep-ability in CI + % logs. The JSON narrative report is what a human reads on a + % failed assertion; this line is the sequence marker. + detail = char(string(row.jobDetail)); + if numel(detail) > 60, detail = [detail(1:57) '...']; end + fprintf(['[fixture-probe] doc=%s ndi=%s manifestUid=%s ' ... + 'jobState=%s elapsed=%.1fs mapSize=%s manifestFetchable=%d ' ... + 'detail="%s"\n'], ... + char(row.cloudDocId), char(row.ndiDocId), char(row.manifestUid), ... + char(row.jobState), row.jobElapsedSec, mapSizeAsText(row.jobMapSize), ... + row.detailsOk, detail); +end + function s = mapSizeAsText(value) if isnumeric(value) && isscalar(value) && ~isnan(value) s = sprintf('%d', round(value)); From 27661a35917d02a2f6837787949e5809ef8adc0f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 20:03:50 +0000 Subject: [PATCH 095/110] signedUrlDiskCache: persistent signed-URL disk cache for reopen A working scientist reopening the same lightsheet-scale dataset over a day should pay the ~85 min signed-URL-set job cost once, not every open. New signedUrlDiskCache module writes each scope (datasetId, documentId, seriesName) to ~/.ndi/signed-url-cache//[_].json.gz as gzipped JSON, TTL-checked against the server's own filesExpireAt with a 30 min safety buffer, and atomically renamed on write. batchSignedUrlLookup gains a diskCache option (off by default; the DID chunk-download path in didsqlite turns it on). On S3 403 the DID handler forgets the scope, so a rotated token does not cascade to N per-uid 403s. Env-var override NDI_SIGNED_URL_CACHE_DIR points the cache anywhere (tests use it for per-run isolation). On-disk contract mirrors byte-for-byte in NDI-python (VH-Lab/NDI-python#322). Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01WhfC1rGbqm3nFBwxN3iRoV --- .../+internal/batchSignedUrlLookup.m | 55 ++ .../+download/+internal/signedUrlDiskCache.m | 586 ++++++++++++++++++ .../+implementations/+database/didsqlite.m | 48 +- .../+cloud/SignedUrlDiskCacheEndToEndTest.m | 152 +++++ tests/+ndi/+unittest/SignedUrlDiskCacheTest.m | 284 +++++++++ 5 files changed, 1124 insertions(+), 1 deletion(-) create mode 100644 src/ndi/+ndi/+cloud/+download/+internal/signedUrlDiskCache.m create mode 100644 tests/+ndi/+unittest/+cloud/SignedUrlDiskCacheEndToEndTest.m create mode 100644 tests/+ndi/+unittest/SignedUrlDiskCacheTest.m diff --git a/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m b/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m index a613c59e2..bc1a2491c 100644 --- a/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m +++ b/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m @@ -173,6 +173,15 @@ options.partialMapRetrySeconds (1,1) double = 0.5 options.sleepFcn (1,1) function_handle = @pause options.retryBackoffSeconds (1,:) double = [1 4 16] + % Disk cache off by default: only enable when we have a URL-set + % source that will actually reduce cost on reopen (the async + % signed-URL-set job, or a scope big enough to matter). Callers + % that DO want it -- the DID chunk-download path for large + % lightsheet series -- turn it on explicitly. Existing tests + % that inject scripted signers see no disk activity, which is + % what they want: the disk cache is a production optimisation, + % not part of the batch contract. + options.diskCache (1,1) logical = false end % The persistent cache. Keyed on 'datasetId/documentId/seriesName'. @@ -228,6 +237,10 @@ end end + if isempty(entry) && options.diskCache + entry = localTryDiskCache(); + end + if isempty(entry) entry = localFetchScope(cacheKey); if isempty(entry) @@ -437,6 +450,19 @@ CACHE(theCacheKey) = e; STATS.lastMapSize = double(e.map.Count); STATS.lastMapUids = keys(e.map); + + % Persist to disk when the caller has opted in AND the payload + % carries a server-signed expiry we can age-check off of. A + % payload with neither expiresAt nor filesExpireAt is not + % cacheable -- the disk cache refuses to invent a TTL, on + % purpose (see signedUrlDiskCache.save). Best-effort: any I/O + % failure is swallowed so a full disk does not break a read. + if options.diskCache + try %#ok + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + cloudDatasetId, ndiDocumentId, seriesName, answer); + end + end end function reason = localDescribeFailure(ok, answer, apiResponse) @@ -468,6 +494,35 @@ end end + function e = localTryDiskCache() + % Consult the persistent (on-disk) cache before running the async + % signed-URL-set job. On hit, populate the in-memory cache so + % every uid in the scope resolves without another disk read. On + % miss (no file, corrupt file, expired-or-within-safety-buffer), + % return [] and let the caller fetch fresh. + % + % The disk cache lives one layer above DID's file-bytes cache + % (which is a different, complementary concern) and its TTL is + % keyed to the SERVER'S filesExpireAt, not a hardcoded value -- + % which is why a scientist reopening the same dataset over a + % 24 h day pays the ~85 min sign cost once. + e = []; + try + disk = ndi.cloud.download.internal.signedUrlDiskCache.load( ... + cloudDatasetId, ndiDocumentId, seriesName); + catch + disk = []; + end + if isempty(disk) || ~isstruct(disk) || ~isfield(disk, 'files') || ... + ~isa(disk.files, 'containers.Map') + return + end + e = struct('map', disk.files, 'fetchedAt', now_utc); + CACHE(cacheKey) = e; + STATS.lastMapSize = double(e.map.Count); + STATS.lastMapUids = keys(e.map); + end + function localWarnOnce(scopeKey) % Say it ONCE per scope, then stay quiet. % diff --git a/src/ndi/+ndi/+cloud/+download/+internal/signedUrlDiskCache.m b/src/ndi/+ndi/+cloud/+download/+internal/signedUrlDiskCache.m new file mode 100644 index 000000000..c10d4f48d --- /dev/null +++ b/src/ndi/+ndi/+cloud/+download/+internal/signedUrlDiskCache.m @@ -0,0 +1,586 @@ +classdef signedUrlDiskCache +%SIGNEDURLDISKCACHE Persistent (on-disk) cache for signed-URL-set payloads. +% +% This is the disk-backed layer that sits beneath the in-process cache +% inside ndi.cloud.download.internal.batchSignedUrlLookup. It exists so a +% working scientist reopening the same dataset over a day pays the +% ~85-minute async signed-URL-set-job cost ONCE and reads from disk on +% every subsequent viewer open. NDI cloud infrastructure, not DID: DID's +% file-bytes cache is a separate, complementary concern. +% +% Scope key +% --------- +% The unit the cache stores and evicts is one (datasetId, documentId, +% seriesName) tuple -- the same tuple batchSignedUrlLookup uses. An +% empty seriesName marks a whole-document scope. +% +% Payload +% ------- +% What save() writes and load() returns is a struct with the fields the +% getSignedURLSetResult / getSignedURLSetAll answers already carry: +% +% .files containers.Map from uid (char) to signed URL (char). +% .filesExpireAt ISO-8601 UTC string; the server's authoritative +% expiry timestamp for the SIGNED URLs. Preferred. +% .expiresAt ISO-8601 UTC string; older/paging path's field. +% Used only when filesExpireAt is not present. +% .generatedAt ISO-8601 UTC string; when the server built the map. +% Optional, informational. +% .fileCount Numeric; how many uids the server reported. +% Optional. +% +% On-disk contract +% ---------------- +% Location: //.json.gz (whole-doc) +% //_.json.gz +% (scoped) +% +% cacheDir defaults to ~/.ndi/signed-url-cache, overridable via the +% NDI_SIGNED_URL_CACHE_DIR environment variable. The base directory is +% created with mode 0700 (POSIX) because a signed URL is effectively a +% bearer token for 24 h; Windows relies on the default per-user profile +% ACL. +% +% Filename escaping for seriesName: any byte that is not +% [A-Za-z0-9._-] becomes '%XX' (uppercase hex). A resulting string +% longer than 96 characters is replaced with 'hash-' (45 +% chars). NDI-python mirrors this rule byte-for-byte. +% +% Format: Standard gzip framing (RFC 1952), single member, UTF-8 +% JSON body. Unzipping yields a human-readable JSON object: +% +% { +% "schemaVersion": 1, +% "datasetId": "", +% "documentId": "", +% "seriesName": "" or "", +% "cachedAt": "YYYY-MM-DDTHH:MM:SSZ", +% "filesExpireAt": "YYYY-MM-DDTHH:MM:SSZ" or "", +% "expiresAt": "YYYY-MM-DDTHH:MM:SSZ" or "", +% "generatedAt": "YYYY-MM-DDTHH:MM:SSZ" or "", +% "fileCount": N, +% "files": { "": "", ... } +% } +% +% TTL and safety buffer +% --------------------- +% The cache does not invent a TTL. It reads the authoritative +% filesExpireAt (or expiresAt) from the payload; a payload with +% neither is treated as un-cacheable (save is a no-op) and, on load, +% as an immediate miss. A URL that would expire in less than the +% safety buffer (default 30 min, override with +% NDI_SIGNED_URL_CACHE_SAFETY_SECONDS) is also treated as a miss -- +% we never hand a caller a URL likely to 403 before they can use it. +% +% Concurrency +% ----------- +% Writes go to a per-writer temp file next to the target and are +% moved into place with an atomic rename (java.nio.file.Files.move +% with ATOMIC_MOVE, which uses rename(2) on POSIX and MoveFileEx on +% Windows). Two viewers signing the same scope concurrently is fine: +% whichever atomic move lands last wins, both readers see a +% consistent file afterwards. There is no cross-process lock; the +% contract is last-writer-wins, not exclusive. +% +% Invalidation +% ------------ +% forget(dsId, docId, seriesName) removes the file for one scope. The +% caller does this from the chunk-download path when it sees an S3 +% 403 -- the URL was in the cache but the object rotated or the +% token was revoked, so drop the scope and let the next lookup +% refetch. Failing to remove the file is not fatal. +% +% Public API +% ---------- +% payload = load(datasetId, documentId, seriesName) +% Returns a struct on cache hit, [] on any miss/expired/error. +% save(datasetId, documentId, seriesName, payload) +% Writes atomically; best-effort (silent on any I/O error). +% forget(datasetId, documentId, seriesName) +% Deletes the cache file for that scope; best-effort. +% location = cacheDir() +% Returns the absolute on-disk root, honoring the env-var +% override. Also creates it (with mode 0700 on POSIX) the +% first time it is asked for on a session. +% +% See also: ndi.cloud.download.internal.batchSignedUrlLookup, +% ndi.cloud.api.files.getSignedURLSetResult, +% ndi.cloud.api.files.getSignedURLSetAll + + properties (Constant, Access = private) + SchemaVersion = 1 + DefaultSafetySec = 1800 % 30 min; task-declared floor. + MaxEscapedSeriesLen = 96 % Beyond this we hash the series name. + end + + methods (Static) + + function payload = load(datasetId, documentId, seriesName) + %LOAD Read one scope's cached signed-URL set, or [] on any miss. + % + % PAYLOAD = LOAD(DATASETID, DOCUMENTID, SERIESNAME) returns a + % struct (fields: files, filesExpireAt, expiresAt, generatedAt, + % fileCount) on cache hit. Returns [] on any of: no cache file, + % corrupt file, JSON parse error, expired-or-within-safety-buffer. + % + % The safety buffer is 30 min by default, overridable via + % NDI_SIGNED_URL_CACHE_SAFETY_SECONDS. A URL that would expire + % inside that window is a miss so the caller refetches rather + % than handing bytes to a viewer that then 403s. + arguments + datasetId (1,1) string + documentId (1,1) string + seriesName (1,1) string = "" + end + payload = []; + + filePath = ndi.cloud.download.internal.signedUrlDiskCache.cachePath( ... + datasetId, documentId, seriesName); + if ~isfile(filePath) + return + end + + try + jsonTxt = ndi.cloud.download.internal.signedUrlDiskCache.gunzipToString(filePath); + catch + return + end + + try + decoded = ndi.cloud.download.internal.signedUrlDiskCache.parseJson(jsonTxt); + catch + return + end + + if isempty(decoded) + return + end + + % Enforce the safety-buffer floor against the authoritative + % server timestamp. A payload with no timestamp is a miss: + % better to refetch than hand out a URL we can't age-check. + safetySec = ndi.cloud.download.internal.signedUrlDiskCache.safetySeconds(); + expiryStr = decoded.filesExpireAt; + if strlength(expiryStr) == 0 + expiryStr = decoded.expiresAt; + end + if strlength(expiryStr) == 0 + return + end + + expiry = ndi.cloud.download.internal.signedUrlDiskCache.parseIsoUtc(expiryStr); + if isnat(expiry) + return + end + nowUtc = datetime('now', 'TimeZone', 'UTC'); + if seconds(expiry - nowUtc) < safetySec + return + end + + payload = decoded; + end + + function save(datasetId, documentId, seriesName, payload) + %SAVE Persist a scope's signed-URL set. Best-effort, atomic. + % + % SAVE(DATASETID, DOCUMENTID, SERIESNAME, PAYLOAD) writes the + % payload to the scope's cache path via a temp file + atomic + % rename. A payload without a filesExpireAt / expiresAt is + % NOT written -- the cache cannot age-check something it has + % no timestamp for, and we refuse to invent a TTL. + % + % Any I/O failure (missing dir, permission denied, disk full, + % another process racing us) is swallowed silently: the + % in-memory cache still works, so a viewer open still succeeds. + arguments + datasetId (1,1) string + documentId (1,1) string + seriesName (1,1) string + payload struct + end + + if ~isfield(payload, 'files') || ~isa(payload.files, 'containers.Map') + return + end + + % Refuse to persist an un-age-checkable payload. + filesExpireAt = ndi.cloud.download.internal.signedUrlDiskCache.stringField(payload, 'filesExpireAt'); + expiresAt = ndi.cloud.download.internal.signedUrlDiskCache.stringField(payload, 'expiresAt'); + if strlength(filesExpireAt) == 0 && strlength(expiresAt) == 0 + return + end + + generatedAt = ndi.cloud.download.internal.signedUrlDiskCache.stringField(payload, 'generatedAt'); + fileCount = ndi.cloud.download.internal.signedUrlDiskCache.numericField(payload, 'fileCount', ... + double(payload.files.Count)); + + filePath = ndi.cloud.download.internal.signedUrlDiskCache.cachePath( ... + datasetId, documentId, seriesName); + + try + ndi.cloud.download.internal.signedUrlDiskCache.ensureParentDir(filePath); + catch + return + end + + try + jsonTxt = ndi.cloud.download.internal.signedUrlDiskCache.buildJson(... + struct( ... + 'datasetId', char(datasetId), ... + 'documentId', char(documentId), ... + 'seriesName', char(seriesName), ... + 'cachedAt', ndi.cloud.download.internal.signedUrlDiskCache.isoNow(), ... + 'filesExpireAt', char(filesExpireAt), ... + 'expiresAt', char(expiresAt), ... + 'generatedAt', char(generatedAt), ... + 'fileCount', fileCount, ... + 'files', payload.files)); + catch + return + end + + tmpPath = [filePath '.tmp.' num2str(feature('getpid')) '.' ... + num2str(randi(2^31 - 1))]; + try + ndi.cloud.download.internal.signedUrlDiskCache.gzipToFile(jsonTxt, tmpPath); + ndi.cloud.download.internal.signedUrlDiskCache.atomicRename(tmpPath, filePath); + catch + if isfile(tmpPath) + try %#ok + delete(tmpPath); + end + end + end + end + + function forget(datasetId, documentId, seriesName) + %FORGET Remove one scope's cache file. Best-effort. + % + % Call this from the chunk-download path when an S3 403 says + % the cached URL is no longer valid. If the file is not + % present -- or the delete itself fails -- it is not an error: + % the next lookup will refetch. + arguments + datasetId (1,1) string + documentId (1,1) string + seriesName (1,1) string = "" + end + filePath = ndi.cloud.download.internal.signedUrlDiskCache.cachePath( ... + datasetId, documentId, seriesName); + if isfile(filePath) + try + delete(filePath); + catch + % Best-effort: a failed delete is not fatal to the + % caller who is already recovering from a 403. + end + end + end + + function d = cacheDir() + %CACHEDIR Absolute path to the on-disk cache root. + % + % Honors NDI_SIGNED_URL_CACHE_DIR; otherwise + % ~/.ndi/signed-url-cache. Creates the directory the first + % time it is asked for, with mode 0700 on POSIX so a bearer + % token that is legibly a bearer token is not world-readable. + override = string(getenv('NDI_SIGNED_URL_CACHE_DIR')); + if strlength(override) > 0 + d = char(override); + else + home = char(java.lang.System.getProperty('user.home')); + d = fullfile(home, '.ndi', 'signed-url-cache'); + end + if ~isfolder(d) + try + mkdir(d); + ndi.cloud.download.internal.signedUrlDiskCache.chmodUserOnly(d); + catch + % Best-effort: batchSignedUrlLookup treats a missing + % cache dir as "no disk cache available". + end + end + end + end + + methods (Static, Access = private) + + function s = safetySeconds() + override = string(getenv('NDI_SIGNED_URL_CACHE_SAFETY_SECONDS')); + if strlength(override) > 0 + v = str2double(override); + if isfinite(v) && v >= 0 + s = v; + return + end + end + s = ndi.cloud.download.internal.signedUrlDiskCache.DefaultSafetySec; + end + + function p = cachePath(datasetId, documentId, seriesName) + % Series names can be arbitrary strings (paths, spaces, uni); + % filesystems can't hold them literally. Follow the escape + % rule NDI-python will mirror byte-for-byte. + base = ndi.cloud.download.internal.signedUrlDiskCache.cacheDir(); + dsDir = fullfile(base, char(datasetId)); + if strlength(seriesName) == 0 + fname = [char(documentId) '.json.gz']; + else + escaped = ndi.cloud.download.internal.signedUrlDiskCache.escapeSeriesName(char(seriesName)); + fname = [char(documentId) '_' escaped '.json.gz']; + end + p = fullfile(dsDir, fname); + end + + function s = escapeSeriesName(raw) + % Percent-encode any byte outside [A-Za-z0-9._-]. Uppercase hex. + % Above MaxEscapedSeriesLen fall back to 'hash-' + % so a pathologically long name never overflows a filesystem's + % 255-byte name limit. + bytes = uint8(unicode2native(raw, 'UTF-8')); + safe = (bytes >= uint8('A') & bytes <= uint8('Z')) | ... + (bytes >= uint8('a') & bytes <= uint8('z')) | ... + (bytes >= uint8('0') & bytes <= uint8('9')) | ... + (bytes == uint8('.')) | ... + (bytes == uint8('_')) | ... + (bytes == uint8('-')); + parts = cell(1, numel(bytes)); + for i = 1:numel(bytes) + if safe(i) + parts{i} = char(bytes(i)); + else + parts{i} = sprintf('%%%02X', bytes(i)); + end + end + s = strjoin(parts, ''); + if numel(s) > ndi.cloud.download.internal.signedUrlDiskCache.MaxEscapedSeriesLen + s = ['hash-' ndi.cloud.download.internal.signedUrlDiskCache.sha1Hex(raw)]; + end + end + + function hex = sha1Hex(raw) + md = java.security.MessageDigest.getInstance('SHA-1'); + md.update(int8(unicode2native(raw, 'UTF-8'))); + digest = typecast(md.digest(), 'uint8'); + hex = lower(reshape(dec2hex(digest, 2).', 1, [])); + end + + function ensureParentDir(filePath) + parent = fileparts(filePath); + if ~isfolder(parent) + mkdir(parent); + end + end + + function chmodUserOnly(dirPath) + % Bearer tokens for 24 h -- keep world-readable off on POSIX. + % Windows relies on the default per-user profile ACL for + % ~/.ndi/, which is roughly equivalent. + if isunix + try %#ok + fileattrib(dirPath, '+w', 'u'); + fileattrib(dirPath, '-w', 'go'); + fileattrib(dirPath, '-r', 'go'); + fileattrib(dirPath, '-x', 'go'); + end + end + end + + function gzipToFile(txt, outPath) + % Java's GZIPOutputStream writes a valid single-member gzip + % that gunzip(1), `tar -xzf`, and NDI-python's gzip module + % can all read. onCleanup makes the stream close on both + % normal return and exception; localSafeClose is idempotent + % so a successful explicit close followed by the cleanup's + % close is harmless. + bytes = int8(unicode2native(char(txt), 'UTF-8')); + fos = java.io.FileOutputStream(outPath); + fosCloser = onCleanup(@() localSafeClose(fos)); %#ok + gzos = java.util.zip.GZIPOutputStream(fos); + gzosCloser = onCleanup(@() localSafeClose(gzos)); %#ok + gzos.write(bytes); + gzos.close(); + end + + function txt = gunzipToString(inPath) + fis = java.io.FileInputStream(inPath); + fisCloser = onCleanup(@() localSafeClose(fis)); %#ok + gzis = java.util.zip.GZIPInputStream(fis); + gzisCloser = onCleanup(@() localSafeClose(gzis)); %#ok + baos = java.io.ByteArrayOutputStream(); + buf = javaArray('byte', 8192); + n = gzis.read(buf); + while n > 0 + baos.write(buf, 0, n); + n = gzis.read(buf); + end + bytes = typecast(baos.toByteArray(), 'uint8'); + txt = native2unicode(bytes(:).', 'UTF-8'); + end + + function atomicRename(srcPath, dstPath) + % Files.move(ATOMIC_MOVE, REPLACE_EXISTING) is the one + % cross-platform API that is atomic on both POSIX (rename(2)) + % and Windows (MoveFileEx MOVEFILE_REPLACE_EXISTING). MATLAB's + % movefile is not, so we go to Java directly. + % + % java.io.File.toPath() avoids the awkward Paths.get(String, + % String...) varargs call from MATLAB (the varargs array + % conversion is a moving target across JVMs). + src = java.io.File(srcPath).toPath(); + dst = java.io.File(dstPath).toPath(); + opts = javaArray('java.nio.file.CopyOption', 2); + opts(1) = java.nio.file.StandardCopyOption.ATOMIC_MOVE; + opts(2) = java.nio.file.StandardCopyOption.REPLACE_EXISTING; + java.nio.file.Files.move(src, dst, opts); + end + + function s = isoNow() + dt = datetime('now', 'TimeZone', 'UTC'); + dt.Format = 'yyyy-MM-dd''T''HH:mm:ss''Z'''; + s = char(dt); + end + + function dt = parseIsoUtc(str) + % Server sends 'YYYY-MM-DDTHH:MM:SSZ' or 'YYYY-MM-DDTHH:MM:SS.SSSZ'. + % Strip a trailing 'Z' (or ' +00:00' / '+00:00') and parse as + % naive UTC. Simpler than juggling MATLAB's timezone format + % code, which differs across releases. + str = char(str); + dt = NaT('TimeZone','UTC'); + if isempty(str) + return + end + % Normalise the trailing UTC marker. + trimmed = strtrim(str); + if endsWith(trimmed, 'Z') + trimmed = trimmed(1:end-1); + elseif endsWith(trimmed, '+00:00') || endsWith(trimmed, '-00:00') + trimmed = trimmed(1:end-6); + end + trimmed = strrep(trimmed, ' ', 'T'); + candidates = { ... + 'yyyy-MM-dd''T''HH:mm:ss.SSS', ... + 'yyyy-MM-dd''T''HH:mm:ss'}; + for i = 1:numel(candidates) + try + dt = datetime(trimmed, 'InputFormat', candidates{i}, ... + 'TimeZone','UTC'); + if ~isnat(dt) + return + end + catch + % try next + end + end + end + + function v = stringField(s, name) + v = ""; + if isfield(s, name) && ~isempty(s.(name)) + v = string(s.(name)); + if strlength(v) == 0 + v = ""; + end + end + end + + function v = numericField(s, name, defaultValue) + v = defaultValue; + if isfield(s, name) && isnumeric(s.(name)) && ~isempty(s.(name)) + v = double(s.(name)); + end + end + + function txt = buildJson(rec) + % Hand-build the JSON so digit-prefixed uid keys survive + % verbatim -- jsonencode on a containers.Map is a moving + % target across MATLAB releases, and jsondecode on a struct + % with digit keys goes through the same rename dance that + % signedURLFileMap already had to work around. + E = @(v) jsonencode(string(v)); + parts = {}; + parts{end+1} = sprintf('{"schemaVersion":%d', ... + ndi.cloud.download.internal.signedUrlDiskCache.SchemaVersion); + parts{end+1} = sprintf(',"datasetId":%s', E(rec.datasetId)); + parts{end+1} = sprintf(',"documentId":%s', E(rec.documentId)); + parts{end+1} = sprintf(',"seriesName":%s', E(rec.seriesName)); + parts{end+1} = sprintf(',"cachedAt":%s', E(rec.cachedAt)); + parts{end+1} = sprintf(',"filesExpireAt":%s', E(rec.filesExpireAt)); + parts{end+1} = sprintf(',"expiresAt":%s', E(rec.expiresAt)); + parts{end+1} = sprintf(',"generatedAt":%s', E(rec.generatedAt)); + parts{end+1} = sprintf(',"fileCount":%d', rec.fileCount); + parts{end+1} = ',"files":'; + parts{end+1} = ndi.cloud.download.internal.signedUrlDiskCache.mapToJson(rec.files); + parts{end+1} = '}'; + txt = strjoin(parts, ''); + end + + function txt = mapToJson(m) + ks = keys(m); + n = numel(ks); + if n == 0 + txt = '{}'; + return + end + parts = cell(1, n); + for i = 1:n + k = ks{i}; + v = m(k); + parts{i} = [jsonencode(string(k)) ':' jsonencode(string(v))]; + end + txt = ['{' strjoin(parts, ',') '}']; + end + + function out = parseJson(txt) + out = []; + % jsondecode renames field names beginning with a digit, + % which every did.ido uid does. Reuse the same trick + % signedURLFileMap uses: decode the whole thing, then + % rebuild the files map by scanning the raw text for uid + % keys in payload order. Everything else on the top-level + % object survives decoding intact. + data = jsondecode(txt); + if ~isstruct(data) + return + end + files = containers.Map('KeyType','char','ValueType','any'); + if isfield(data, 'files') && ~isempty(data.files) + files = ndi.cloud.api.implementation.files.signedURLFileMap( ... + data.files, txt); + end + out = struct( ... + 'files', files, ... + 'filesExpireAt', ndi.cloud.download.internal.signedUrlDiskCache.readTopString(data, 'filesExpireAt'), ... + 'expiresAt', ndi.cloud.download.internal.signedUrlDiskCache.readTopString(data, 'expiresAt'), ... + 'generatedAt', ndi.cloud.download.internal.signedUrlDiskCache.readTopString(data, 'generatedAt'), ... + 'fileCount', ndi.cloud.download.internal.signedUrlDiskCache.readTopNumber(data, 'fileCount')); + end + + function v = readTopString(data, name) + v = ""; + if isfield(data, name) && ~isempty(data.(name)) + v = string(data.(name)); + end + end + + function v = readTopNumber(data, name) + v = 0; + if isfield(data, name) && isnumeric(data.(name)) && ~isempty(data.(name)) + v = double(data.(name)); + end + end + end +end + +function localSafeClose(obj) +% Best-effort close for a Java stream inside an onCleanup. Suppresses +% double-close and I/O errors -- the read/write already succeeded or +% already failed, and either way the stream is being torn down. + if isempty(obj) + return + end + try %#ok + obj.close(); + end +end diff --git a/src/ndi/+ndi/+database/+implementations/+database/didsqlite.m b/src/ndi/+ndi/+database/+implementations/+database/didsqlite.m index 16b9d79f0..d605715ca 100644 --- a/src/ndi/+ndi/+database/+implementations/+database/didsqlite.m +++ b/src/ndi/+ndi/+database/+implementations/+database/didsqlite.m @@ -224,8 +224,16 @@ function download_file_from_cloud(destPath, sourcePath, context) if strlength(seriesName) > 0 && strlength(contextUid) > 0 ndiFileUid = char(contextUid); end + % diskCache=true because DID is the production reopen-a-dataset + % path: a scientist who reopens the same lightsheet the next + % morning must not pay the ~85 min signed-URL-set job again. + % Signals lookup only for a scoped call (empty documentId + % dispatches to the batch's early return and no disk read + % happens). See ndi.cloud.download.internal.signedUrlDiskCache. + fromDiskCache = strlength(docId) > 0; fileUrl = ndi.cloud.download.internal.batchSignedUrlLookup( ... - string(cloudDatasetId), docId, seriesName, string(ndiFileUid)); + string(cloudDatasetId), docId, seriesName, string(ndiFileUid), ... + 'diskCache', fromDiskCache); if strlength(fileUrl) == 0 % No batch URL for this uid -- either no document context, @@ -237,9 +245,23 @@ function download_file_from_cloud(destPath, sourcePath, context) error(['Failed to get file details: ' answer.message]); end fileUrl = answer.downloadUrl; + fromDiskCache = false; % per-uid mint; nothing to invalidate. end [success2, answer2] = ndi.cloud.api.files.getFile(char(fileUrl), destPath, 'useCurl', true); if ~success2 + % An S3 403 on a URL that WAS served from the disk cache + % says the cached scope has gone stale (token revoked, or + % the object rotated). Drop the scope so the next read + % refetches -- otherwise we'd 403 our way through every + % subsequent uid in the same scope. Mirrors the in-memory + % scope-forget the batch would do on refresh. Best-effort; + % a failed forget is not fatal to the error we're raising. + if fromDiskCache && localIsS3Forbidden(answer2) + try %#ok + ndi.cloud.download.internal.signedUrlDiskCache.forget( ... + string(cloudDatasetId), docId, seriesName); + end + end error(['Failed to download file from cloud: ' answer2]); end else @@ -270,3 +292,27 @@ function download_file_from_cloud(destPath, sourcePath, context) end end end + +function tf = localIsS3Forbidden(answer) + % S3 returns 403 on a rotated/revoked signed URL. curl -f surfaces + % that as an exit-22 message like + % curl: (22) The requested URL returned error: 403 + % (Or "403 Forbidden" on some curl builds.) websave's message + % contains "403" too when it makes it that far. Match \b403\b. + tf = false; + if isempty(answer) + return + end + txt = ''; + if ischar(answer) + txt = answer; + elseif isstring(answer) && isscalar(answer) + txt = char(answer); + elseif isstruct(answer) && isfield(answer, 'message') && ~isempty(answer.message) + txt = char(string(answer.message)); + end + if isempty(txt) + return + end + tf = ~isempty(regexp(txt, '\<403\>', 'once')); +end diff --git a/tests/+ndi/+unittest/+cloud/SignedUrlDiskCacheEndToEndTest.m b/tests/+ndi/+unittest/+cloud/SignedUrlDiskCacheEndToEndTest.m new file mode 100644 index 000000000..8327572fd --- /dev/null +++ b/tests/+ndi/+unittest/+cloud/SignedUrlDiskCacheEndToEndTest.m @@ -0,0 +1,152 @@ +classdef SignedUrlDiskCacheEndToEndTest < matlab.unittest.TestCase +% SIGNEDURLDISKCACHEENDTOENDTEST - the persistent signed-URL disk cache +% seen through batchSignedUrlLookup, end-to-end. +% +% This is the "live cell" test-cloud-api.yml runs (testCloudApi -> +% ndi.unittest.cloud recursive). Placed in +cloud so the daily cloud +% CI picks it up alongside the async-job fixture test; the injected +% signer keeps it self-contained -- the disk cache is a client-side +% concern, so a real cloud round trip proves nothing the local run +% does not, and paying for a fresh lightsheet upload on every daily +% CI run to inject a counter around a signer we already control is +% pure cost. +% +% Proves: two consecutive batchSignedUrlLookup calls for the same +% scope cost ONE signer invocation. The first populates the disk +% cache; the second, with a fresh in-process cache, reads from disk +% and never touches the signer. Whether that signer would have been +% getSignedURLSetAll's paging walk or the createSignedURLSetJob +% async round-trip, the result is the same shape: signer counter +% doesn't move. + + properties + TmpDir char = '' + SavedCacheDirEnv char = '' + end + + methods (TestMethodSetup) + function isolateCacheDir(testCase) + testCase.SavedCacheDirEnv = getenv('NDI_SIGNED_URL_CACHE_DIR'); + testCase.TmpDir = tempname; + setenv('NDI_SIGNED_URL_CACHE_DIR', testCase.TmpDir); + testCase.addTeardown(@() localCleanup( ... + testCase.SavedCacheDirEnv, testCase.TmpDir)); + end + end + + methods (Access = private) + function [signer, counter] = countingSigner(~, filesMap) + % Emitted as a NESTED handle (not an anonymous @()... wrap) + % so nargout(signer) reads back as a fixed 2 rather than -1 + % -- which is what batchSignedUrlLookup's four-vs-two output + % probe expects to see for an injected mock. The same + % pattern BatchSignedUrlLookupTest already uses. + counter = containers.Map('KeyType','char','ValueType','any'); + counter('n') = 0; + % 23 h ahead of now mirrors the server's 24 h URL lifetime, + % leaving the default 30 min safety buffer comfortably valid. + futureDt = datetime('now','TimeZone','UTC') + hours(23); + futureDt.Format = 'yyyy-MM-dd''T''HH:mm:ss''Z'''; + futureIso = char(futureDt); + signer = @doSign; + function [ok, answer] = doSign(~, ~, varargin) %#ok + counter('n') = counter('n') + 1; + ok = true; + answer = struct( ... + 'files', filesMap, ... + 'filesExpireAt', futureIso, ... + 'expiresAt', futureIso, ... + 'generatedAt', futureIso, ... + 'fileCount', double(filesMap.Count), ... + 'pages', 1); + end + end + end + + methods (Test) + + function testReopenPaysZeroSignerCallsFromDiskCache(testCase) + % The end-to-end promise: sign the scope once, save it to + % disk, then a reopen (fresh in-process cache) resolves + % every uid without another signer round trip. For a + % 121k-member chunk.bin series that means "one 85 min job + % ever" instead of "one per viewer open all day". + filesMap = containers.Map('KeyType','char','ValueType','any'); + filesMap('4192a3c0dd1b4e00_3fe8a1b2c3d4e5f6') = 'https://s3/one'; + filesMap('4192a3c0dd1b4e00_4fe8a1b2c3d4e5f6') = 'https://s3/two'; + [signer, counter] = testCase.countingSigner(filesMap); + + % --- Run 1: cold. Expect one signer call, both uids hit + % the in-memory cache, and the disk file lands. + u1a = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds-e2e", "doc-e2e", "chunk.bin", ... + "4192a3c0dd1b4e00_3fe8a1b2c3d4e5f6", ... + 'signer', signer, 'clearCache', true, 'diskCache', true); + u1b = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds-e2e", "doc-e2e", "chunk.bin", ... + "4192a3c0dd1b4e00_4fe8a1b2c3d4e5f6", ... + 'signer', signer, 'diskCache', true); + testCase.verifyEqual(u1a, 'https://s3/one'); + testCase.verifyEqual(u1b, 'https://s3/two'); + testCase.verifyEqual(counter('n'), 1, ... + 'run 1 must cost exactly one signer call (in-memory batch)'); + + diskFile = fullfile(testCase.TmpDir, 'ds-e2e', ... + 'doc-e2e_chunk.bin.json.gz'); + testCase.assertTrue(isfile(diskFile), ... + 'run 1 must have written the scope to disk'); + + % --- Run 2: fresh in-process cache; disk should carry us. + % clearCache wipes the persistent in-memory cache -- this + % is what "a new viewer opens the same dataset" looks like + % from the batch's point of view. If the disk cache is + % working, the signer never runs again. + u2a = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds-e2e", "doc-e2e", "chunk.bin", ... + "4192a3c0dd1b4e00_3fe8a1b2c3d4e5f6", ... + 'signer', signer, 'clearCache', true, 'diskCache', true); + u2b = ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds-e2e", "doc-e2e", "chunk.bin", ... + "4192a3c0dd1b4e00_4fe8a1b2c3d4e5f6", ... + 'signer', signer, 'diskCache', true); + testCase.verifyEqual(u2a, 'https://s3/one'); + testCase.verifyEqual(u2b, 'https://s3/two'); + testCase.verifyEqual(counter('n'), 1, ... + ['run 2 must have hit the disk cache -- no additional ' ... + 'signer call, whether it was going to be a paging ' ... + 'walk or an async job.']); + end + + function testDiskOptOutStillCallsSignerOnEveryColdRun(testCase) + % Guard: diskCache=false is the default, and it means "no + % on-disk state, ever" -- existing tests and callers that + % have not opted in must see the pre-cache behaviour where + % a fresh in-process cache re-signs the scope. + filesMap = containers.Map('KeyType','char','ValueType','any'); + filesMap('aa_bb') = 'https://s3/x'; + [signer, counter] = testCase.countingSigner(filesMap); + + for i = 1:3 + ndi.cloud.download.internal.batchSignedUrlLookup( ... + "ds-optout", "doc-optout", "", "aa_bb", ... + 'signer', signer, 'clearCache', true); % diskCache=false + end + testCase.verifyEqual(counter('n'), 3, ... + 'diskCache=false must call the signer per cold run'); + + % And nothing should have landed on disk either. + noFile = fullfile(testCase.TmpDir, 'ds-optout', 'doc-optout.json.gz'); + testCase.verifyFalse(isfile(noFile), ... + 'diskCache=false must not write any cache file'); + end + end +end + +function localCleanup(savedEnv, tmp) + setenv('NDI_SIGNED_URL_CACHE_DIR', savedEnv); + if ~isempty(tmp) && isfolder(tmp) + try %#ok + rmdir(tmp, 's'); + end + end +end diff --git a/tests/+ndi/+unittest/SignedUrlDiskCacheTest.m b/tests/+ndi/+unittest/SignedUrlDiskCacheTest.m new file mode 100644 index 000000000..5f59c4698 --- /dev/null +++ b/tests/+ndi/+unittest/SignedUrlDiskCacheTest.m @@ -0,0 +1,284 @@ +classdef SignedUrlDiskCacheTest < matlab.unittest.TestCase +% SIGNEDURLDISKCACHETEST - the on-disk signed-URL-set cache. +% +% Persistent layer that sits beneath the in-process cache inside +% ndi.cloud.download.internal.batchSignedUrlLookup and turns a +% dataset-reopen "still ~85 min to sign 121k URLs" into "read a file". +% See ndi.cloud.download.internal.signedUrlDiskCache and the task +% brief on VH-Lab/NDI-matlab#1010 follow-on / Waltham-Data-Science/ +% NDI-python#322. +% +% Nothing here hits the network or the user's real cache directory: +% the class setup points NDI_SIGNED_URL_CACHE_DIR at a per-run temp +% dir and the teardown removes it. + + properties + TmpDir char = '' + SavedCacheDirEnv char = '' + SavedSafetyEnv char = '' + end + + methods (TestMethodSetup) + function isolateCacheDir(testCase) + % One temp dir per test so a leaked file cannot leak across + % assertions -- the class runs a save + load round-trip and + % a concurrent-write case in the same suite. + testCase.SavedCacheDirEnv = getenv('NDI_SIGNED_URL_CACHE_DIR'); + testCase.SavedSafetyEnv = getenv('NDI_SIGNED_URL_CACHE_SAFETY_SECONDS'); + testCase.TmpDir = tempname; + setenv('NDI_SIGNED_URL_CACHE_DIR', testCase.TmpDir); + % Clear the safety-seconds override so each test starts + % from the documented default (1800 s) unless it sets its + % own; a leftover override from a previous run of this + % suite would silently corrupt an expiry assertion. + setenv('NDI_SIGNED_URL_CACHE_SAFETY_SECONDS', ''); + end + + function restoreOnTeardown(testCase) + envKey = 'NDI_SIGNED_URL_CACHE_DIR'; + savedDir = testCase.SavedCacheDirEnv; + safetyKey = 'NDI_SIGNED_URL_CACHE_SAFETY_SECONDS'; + savedSaf = testCase.SavedSafetyEnv; + tmp = testCase.TmpDir; + testCase.addTeardown(@() localCleanup( ... + envKey, savedDir, safetyKey, savedSaf, tmp)); + end + end + + methods (Access = private) + function m = mapOf(~, uidUrlPairs) + m = containers.Map('KeyType','char','ValueType','any'); + for i = 1:size(uidUrlPairs, 1) + m(uidUrlPairs{i, 1}) = uidUrlPairs{i, 2}; + end + end + + function s = futureIso(~, secondsAhead) + % ISO-8601 UTC N seconds from now, in the same format the + % server uses in the getSignedURLSetJob ready-state payload. + dt = datetime('now','TimeZone','UTC') + seconds(secondsAhead); + dt.Format = 'yyyy-MM-dd''T''HH:mm:ss''Z'''; + s = char(dt); + end + end + + methods (Test) + + function testSaveLoadRoundTripPreservesFields(testCase) + % The point of the disk cache: what goes in comes back out + % byte-for-byte on the fields callers actually consume -- + % the uid->URL map, the server's authoritative expiry, and + % the generatedAt provenance. + m = testCase.mapOf({ ... + '4192a3c0dd1b4e00_3fe8a1b2c3d4e5f6', 'https://s3/one'; ... + '4192a3c0dd1b4e00_4fe8a1b2c3d4e5f6', 'https://s3/two?sig=abc%2F123'}); + payload = struct( ... + 'files', m, ... + 'filesExpireAt', testCase.futureIso(23*3600), ... + 'expiresAt', testCase.futureIso(23*3600), ... + 'generatedAt', testCase.futureIso(-30), ... + 'fileCount', 2); + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", "chunk.bin", payload); + + got = ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", "chunk.bin"); + + testCase.assertNotEmpty(got, 'a just-written scope should load'); + testCase.verifyEqual(sort(keys(got.files)), ... + sort({'4192a3c0dd1b4e00_3fe8a1b2c3d4e5f6', '4192a3c0dd1b4e00_4fe8a1b2c3d4e5f6'})); + testCase.verifyEqual(got.files('4192a3c0dd1b4e00_3fe8a1b2c3d4e5f6'), ... + 'https://s3/one'); + testCase.verifyEqual(got.files('4192a3c0dd1b4e00_4fe8a1b2c3d4e5f6'), ... + 'https://s3/two?sig=abc%2F123', ... + 'URL with url-escaped bytes must survive JSON encoding'); + testCase.verifyEqual(char(got.filesExpireAt), payload.filesExpireAt); + testCase.verifyEqual(double(got.fileCount), 2); + end + + function testExpiredPayloadIsATreatedAsAMiss(testCase) + % A cached scope whose filesExpireAt is in the past must + % NOT be served -- the URLs are dead and would 403 on use. + m = testCase.mapOf({'aa_bb', 'https://s3/expired'}); + payload = struct( ... + 'files', m, ... + 'filesExpireAt', testCase.futureIso(-3600), ... % 1 h ago + 'expiresAt', "", ... + 'generatedAt', "", ... + 'fileCount', 1); + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", "", payload); + + got = ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", ""); + testCase.verifyEmpty(got, ... + 'a filesExpireAt in the past is a miss'); + end + + function testSafetyBufferGuardsAgainstAlmostExpired(testCase) + % The whole point of the buffer: a URL that will expire in + % 20 minutes is a miss even though it is still technically + % valid, because a viewer opened with it will 403 before + % they get through the file. + m = testCase.mapOf({'aa_bb', 'https://s3/almost'}); + payload = struct( ... + 'files', m, ... + 'filesExpireAt', testCase.futureIso(20*60), ... + 'expiresAt', "", ... + 'generatedAt', "", ... + 'fileCount', 1); + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", "", payload); + + % Default safety is 30 min (1800 s), so 20 min is inside + % the buffer and load must miss. + got = ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", ""); + testCase.verifyEmpty(got, ... + '20 min < 30 min safety buffer must be a miss'); + + % With a shorter buffer the same file loads -- proving the + % check is the buffer, not the write, doing the work. + setenv('NDI_SIGNED_URL_CACHE_SAFETY_SECONDS', '60'); + got2 = ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", ""); + testCase.verifyNotEmpty(got2, ... + 'with a 60 s buffer the same payload loads'); + end + + function testMissingExpiryIsNotCacheable(testCase) + % save() refuses to persist a payload with no expiry -- the + % cache would then have no way to age-check it. Load must + % see nothing on disk. + m = testCase.mapOf({'aa_bb', 'https://s3/no-expiry'}); + payload = struct( ... + 'files', m, ... + 'filesExpireAt', "", ... + 'expiresAt', "", ... + 'generatedAt', "", ... + 'fileCount', 1); + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", "", payload); + + got = ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", ""); + testCase.verifyEmpty(got); + end + + function testForgetRemovesTheFile(testCase) + % forget is the S3-403 hook: an S3 refusal means the + % scope's cached URLs are dead, so drop them before the + % next uid in the scope pays the same 403. + m = testCase.mapOf({'aa_bb', 'https://s3/ok'}); + payload = struct( ... + 'files', m, ... + 'filesExpireAt', testCase.futureIso(3600), ... + 'expiresAt', "", ... + 'generatedAt', "", ... + 'fileCount', 1); + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", "chunk.bin", payload); + testCase.assertNotEmpty( ... + ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", "chunk.bin"), ... + 'sanity: the scope should be on disk before forget'); + + ndi.cloud.download.internal.signedUrlDiskCache.forget( ... + "ds1", "doc1", "chunk.bin"); + + testCase.verifyEmpty( ... + ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", "chunk.bin"), ... + 'forget must remove the scope from disk'); + end + + function testForgetOnUnknownScopeDoesNotRaise(testCase) + % Best-effort: a forget on a scope we never cached must + % succeed silently so the S3-403 recovery path doesn't + % need to check-first. + testCase.verifyWarningFree(@() ... + ndi.cloud.download.internal.signedUrlDiskCache.forget( ... + "nosuch-ds", "nosuch-doc", "nosuch-series")); + end + + function testCacheDirRespectsEnvVar(testCase) + % The env-var override lets NDI-python and the MATLAB tests + % point the cache anywhere; the default falls under + % ~/.ndi/. Both are documented on-disk contract. + actual = ndi.cloud.download.internal.signedUrlDiskCache.cacheDir(); + testCase.verifyEqual(actual, testCase.TmpDir); + end + + function testConcurrentSavesDoNotCorruptTheFile(testCase) + % Two viewers signing the same scope back-to-back is the + % expected shape: last write wins, both reads afterward see + % a valid file. atomic rename is what makes this true. + baseMap = testCase.mapOf({'aa_bb', 'https://s3/first'}); + payload1 = struct( ... + 'files', baseMap, ... + 'filesExpireAt', testCase.futureIso(3600), ... + 'expiresAt', "", ... + 'generatedAt', "", ... + 'fileCount', 1); + payload2 = payload1; + payload2.files = testCase.mapOf({'cc_dd', 'https://s3/second'}); + + for i = 1:10 + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", "chunks", payload1); + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", "chunks", payload2); + end + + got = ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", "chunks"); + testCase.assertNotEmpty(got, ... + 'after alternating writes, the file must still be valid'); + % Whichever payload landed last is fine (last-writer wins); + % what matters is that we did NOT get a half-written blob. + testCase.verifyTrue(isa(got.files, 'containers.Map')); + testCase.verifyEqual(double(got.files.Count), 1); + end + + function testSeriesNameEscapingHandlesPathySpaces(testCase) + % Real series names can hold '/', spaces, unicode; the + % filesystem can't. The escape rule NDI-python mirrors is + % percent-encoding for non-safe bytes and a SHA-1 fallback + % above 96 chars. + m = testCase.mapOf({'aa_bb', 'https://s3/x'}); + payload = struct( ... + 'files', m, ... + 'filesExpireAt', testCase.futureIso(3600), ... + 'expiresAt', "", ... + 'generatedAt', "", ... + 'fileCount', 1); + + pathy = "level 3/chunk.bin"; + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", pathy, payload); + got = ndi.cloud.download.internal.signedUrlDiskCache.load( ... + "ds1", "doc1", pathy); + testCase.assertNotEmpty(got, ... + 'a series name with a slash and a space must save and load'); + + % Two different names must land in two different files. + other = "level 3_chunk.bin"; + ndi.cloud.download.internal.signedUrlDiskCache.save( ... + "ds1", "doc1", other, payload); + listing = dir(fullfile(testCase.TmpDir, 'ds1')); + names = {listing(~[listing.isdir]).name}; + testCase.verifyEqual(sum(endsWith(names, '.json.gz')), 2, ... + 'two distinct series names must produce two distinct cache files'); + end + end +end + +function localCleanup(envKey, savedDir, safetyKey, savedSaf, tmp) + setenv(envKey, savedDir); + setenv(safetyKey, savedSaf); + if ~isempty(tmp) && isfolder(tmp) + try %#ok + rmdir(tmp, 's'); + end + end +end From 988ec74c2d6638c5dd70df35944629900727e1a5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 20:32:36 +0000 Subject: [PATCH 096/110] signedUrlDiskCache: swap fragile Java I/O for MATLAB builtins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-trip tests failed because save() was throwing silently inside its own try/catch — the Java bridge path for GZIPOutputStream.write and Files.move(ATOMIC_MOVE) with a javaArray of an interface type is not portable across MATLAB releases / JVMs, so no file ever landed on disk and every subsequent load() returned []. The five failing tests all depend on save->load; the four tests that only assert misses passed incidentally because "no file" is also a miss. - gzipToFile / gunzipToString: use MATLAB's builtin gzip / gunzip via a scratch plain file (unicode2native + fwrite). Same on-disk format as before (single-member gzip of UTF-8 JSON), no Java overloads. - atomicRename: use movefile('f'), which delegates to rename(2) on POSIX and MoveFileEx on Windows. Same "no half-written file visible" invariant, no javaArray-of-interface pitfall. - chmodUserOnly: prefer filePermissions (R2024a+); keep fileattrib as a fallback so older MATLABs still work. - localProcessId helper: prefer matlabProcessID; keep feature('getpid') as a fallback for the same reason. Public contract (cachePath, payload struct, TTL/safety-buffer semantics, forget, cacheDir + env override) is unchanged. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_018hGmZKPmMGAt4KN7wmReRS --- .../+download/+internal/signedUrlDiskCache.m | 151 ++++++++++++------ 1 file changed, 98 insertions(+), 53 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+download/+internal/signedUrlDiskCache.m b/src/ndi/+ndi/+cloud/+download/+internal/signedUrlDiskCache.m index c10d4f48d..5acd4ba7f 100644 --- a/src/ndi/+ndi/+cloud/+download/+internal/signedUrlDiskCache.m +++ b/src/ndi/+ndi/+cloud/+download/+internal/signedUrlDiskCache.m @@ -239,7 +239,7 @@ function save(datasetId, documentId, seriesName, payload) return end - tmpPath = [filePath '.tmp.' num2str(feature('getpid')) '.' ... + tmpPath = [filePath '.tmp.' num2str(localProcessId()) '.' ... num2str(randi(2^31 - 1))]; try ndi.cloud.download.internal.signedUrlDiskCache.gzipToFile(jsonTxt, tmpPath); @@ -378,61 +378,88 @@ function chmodUserOnly(dirPath) % ~/.ndi/, which is roughly equivalent. if isunix try %#ok - fileattrib(dirPath, '+w', 'u'); - fileattrib(dirPath, '-w', 'go'); - fileattrib(dirPath, '-r', 'go'); - fileattrib(dirPath, '-x', 'go'); + if exist('filePermissions', 'builtin') == 5 || ... + exist('filePermissions', 'file') == 2 + p = filePermissions(dirPath); + p.User.Read = true; + p.User.Write = true; + p.User.Execute = true; + p.Group.Read = false; + p.Group.Write = false; + p.Group.Execute = false; + p.Other.Read = false; + p.Other.Write = false; + p.Other.Execute = false; + filePermissions(dirPath, p); + else + fileattrib(dirPath, '+w', 'u'); + fileattrib(dirPath, '-w', 'go'); + fileattrib(dirPath, '-r', 'go'); + fileattrib(dirPath, '-x', 'go'); + end end end end function gzipToFile(txt, outPath) - % Java's GZIPOutputStream writes a valid single-member gzip - % that gunzip(1), `tar -xzf`, and NDI-python's gzip module - % can all read. onCleanup makes the stream close on both - % normal return and exception; localSafeClose is idempotent - % so a successful explicit close followed by the cleanup's - % close is harmless. - bytes = int8(unicode2native(char(txt), 'UTF-8')); - fos = java.io.FileOutputStream(outPath); - fosCloser = onCleanup(@() localSafeClose(fos)); %#ok - gzos = java.util.zip.GZIPOutputStream(fos); - gzosCloser = onCleanup(@() localSafeClose(gzos)); %#ok - gzos.write(bytes); - gzos.close(); + % Write UTF-8 bytes to a plain temp file, then use MATLAB's + % builtin gzip to produce a valid single-member gzip that + % gunzip(1), tar -xzf and NDI-python's gzip module can all + % read. We avoid Java streams because the Java bridge's + % overload resolution for write(byte[]) versus write(int) is + % release- and JVM-dependent, and a mis-picked overload here + % writes silently corrupt output. + parent = fileparts(outPath); + stem = tempname(parent); + plainPath = [stem '.json']; + fid = fopen(plainPath, 'w'); + if fid < 0 + error('NDI:SignedUrlDiskCache:WriteOpenFailed', ... + 'Could not open %s for writing.', plainPath); + end + cleanupPlain = onCleanup(@() localDeleteIfExists(plainPath)); %#ok + bytes = unicode2native(char(txt), 'UTF-8'); + fwrite(fid, bytes, 'uint8'); + fclose(fid); + % gzip(FILE) writes FILE.gz next to FILE in the same dir. + gzip(plainPath); + gzPath = [plainPath '.gz']; + cleanupGz = onCleanup(@() localDeleteIfExists(gzPath)); %#ok + movefile(gzPath, outPath, 'f'); end function txt = gunzipToString(inPath) - fis = java.io.FileInputStream(inPath); - fisCloser = onCleanup(@() localSafeClose(fis)); %#ok - gzis = java.util.zip.GZIPInputStream(fis); - gzisCloser = onCleanup(@() localSafeClose(gzis)); %#ok - baos = java.io.ByteArrayOutputStream(); - buf = javaArray('byte', 8192); - n = gzis.read(buf); - while n > 0 - baos.write(buf, 0, n); - n = gzis.read(buf); - end - bytes = typecast(baos.toByteArray(), 'uint8'); - txt = native2unicode(bytes(:).', 'UTF-8'); + % gunzip writes into a temp dir, then we read the plain file. + % Same rationale as gzipToFile: avoid the Java-bridge + % overload traps by using MATLAB builtins end to end. + outDir = tempname; + mkdir(outDir); + cleanupDir = onCleanup(@() localCleanupDir(outDir)); %#ok + names = gunzip(inPath, outDir); + if isempty(names) + error('NDI:SignedUrlDiskCache:EmptyGunzip', ... + 'gunzip produced no files from %s', inPath); + end + fid = fopen(names{1}, 'rb'); + if fid < 0 + error('NDI:SignedUrlDiskCache:ReadOpenFailed', ... + 'Could not open %s for reading.', names{1}); + end + raw = fread(fid, inf, '*uint8')'; + fclose(fid); + txt = native2unicode(raw, 'UTF-8'); end function atomicRename(srcPath, dstPath) - % Files.move(ATOMIC_MOVE, REPLACE_EXISTING) is the one - % cross-platform API that is atomic on both POSIX (rename(2)) - % and Windows (MoveFileEx MOVEFILE_REPLACE_EXISTING). MATLAB's - % movefile is not, so we go to Java directly. - % - % java.io.File.toPath() avoids the awkward Paths.get(String, - % String...) varargs call from MATLAB (the varargs array - % conversion is a moving target across JVMs). - src = java.io.File(srcPath).toPath(); - dst = java.io.File(dstPath).toPath(); - opts = javaArray('java.nio.file.CopyOption', 2); - opts(1) = java.nio.file.StandardCopyOption.ATOMIC_MOVE; - opts(2) = java.nio.file.StandardCopyOption.REPLACE_EXISTING; - java.nio.file.Files.move(src, dst, opts); + % movefile on the same directory delegates to a POSIX + % rename(2) or a Windows MoveFileEx replace, which is enough + % for the "no half-written file visible to a concurrent + % reader" invariant this cache needs. We used to go through + % java.nio.file.Files.move(ATOMIC_MOVE) but the javaArray + % path there is fragile across MATLAB releases -- an + % interface-typed array plus an enum-to-array assignment can + % throw silently, and save()'s outer try/catch swallows it. + movefile(srcPath, dstPath, 'f'); end function s = isoNow() @@ -573,14 +600,32 @@ function atomicRename(srcPath, dstPath) end end -function localSafeClose(obj) -% Best-effort close for a Java stream inside an onCleanup. Suppresses -% double-close and I/O errors -- the read/write already succeeded or -% already failed, and either way the stream is being torn down. - if isempty(obj) - return +function localDeleteIfExists(p) +% Best-effort delete for an onCleanup guard on a scratch file. + if ischar(p) && ~isempty(p) && isfile(p) + try %#ok + delete(p); + end + end +end + +function localCleanupDir(d) +% Best-effort recursive rmdir for a scratch temp directory used only for +% one gunzip. Missing directory is not an error. + if ischar(d) && ~isempty(d) && isfolder(d) + try %#ok + rmdir(d, 's'); + end end - try %#ok - obj.close(); +end + +function pid = localProcessId() +% matlabProcessID is the supported accessor in recent releases; feature +% is the old, still-shipping one. Wrap so a plain integer PID comes back +% either way and no code path relies on feature('getpid'). + try + pid = matlabProcessID(); + catch + pid = feature('getpid'); end end From 5541a0011a932e68ca720f3331775a513e533208 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 27 Sep 2026 23:45:32 +0000 Subject: [PATCH 097/110] AuthTest: index organizations(1).id so a multi-org account still passes testLoginLogout asserted verifyClass on answer_login.user.organizations.id, which expands to a comma-separated list when the login response carries a struct array of organizations. verifyClass then reads the extra values as positional args and MATLAB:narginchk:tooManyInputs fires. This started tripping when the test account was added to a second/third organization; the underlying code (ndi.cloud.authenticate -> extractFirstOrganizationId) already picks the first, so the test mirrors that. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01WhfC1rGbqm3nFBwxN3iRoV --- tests/+ndi/+unittest/+cloud/AuthTest.m | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/+ndi/+unittest/+cloud/AuthTest.m b/tests/+ndi/+unittest/+cloud/AuthTest.m index 55bf7c640..b9a43b4fc 100644 --- a/tests/+ndi/+unittest/+cloud/AuthTest.m +++ b/tests/+ndi/+unittest/+cloud/AuthTest.m @@ -59,7 +59,13 @@ function testLoginLogout(testCase) narrative(end+1) = "Login call successful. Verifying token and organization ID."; testCase.verifyClass(answer_login.token, 'char', login_message); - testCase.verifyClass(answer_login.user.organizations.id, 'char', login_message); + % `organizations` may be a struct array on a multi-org + % account -- `answer_login.user.organizations.id` then + % expands to a comma-separated list and verifyClass reads + % it as extra positional args (MATLAB:narginchk: + % tooManyInputs). ndi.cloud.authenticate already picks the + % first (extractFirstOrganizationId); mirror that here. + testCase.verifyClass(answer_login.user.organizations(1).id, 'char', login_message); narrative(end+1) = "Token and organization ID are of the correct class."; % Step 3: Test Logout From 02880062dab1e025640dc9f6231db841231e236b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 12:07:45 +0000 Subject: [PATCH 098/110] batchSignedUrlLookup: exponential backoff for partial-map retry Was: one 0.5 s retry per scope when the first signed-URL-set response did not name a requested uid. Now: a bounded schedule of 1 s / 3 s / 9 s. Waltham-Data-Science/ NDI-python#320 shows User 1 prod's signed-URL-set index lagging further than 0.5 s after ``waitForAllBulkUploads`` returns -- one retry is not enough to bridge the settle between the bulk-upload extraction subsystem reporting done and the signed-URL-set subsystem catching up. A missed uid then either flags as uid_misses > 0 (main) or, on the Python side's strict-mode reconstruction path, surfaces earlier as a document rejected at add time. Three waves at 1 s / 3 s / 9 s cover the observed tail, still bounded (each wave costs one scope refetch, not one per uid), and never fire in the common case where the first map is already complete. Schema: - New name-value ``partialMapRetryDelays`` (row vector of waits). Defaults to [1 3 9]. - ``partialMapRetrySeconds`` kept as a deprecated scalar knob (NaN means "use partialMapRetryDelays"; any real number replaces the schedule with a one-element vector containing that wait, so ``partialMapRetrySeconds=0`` still fires ONE retry with no wait, matching the pre-schedule contract every existing test pins). - Persistent RETRIEDSCOPES map now records attempts spent per scope (double) instead of a "has been retried" flag (logical); walks the schedule until a fresh map names the uid or the schedule is exhausted. Round-trip test's cap on partialMapRetries relaxed from 1 to numel(defaultSchedule); every other assertion (uid_misses == 0, signer_calls == 1 + partial_map_retries) is unchanged. Python mirror lands in Waltham-Data-Science/NDI-python on the same branch. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+internal/batchSignedUrlLookup.m | 83 +++++++++++++++---- .../+cloud/FileSeriesRoundTripTest.m | 23 ++--- 2 files changed, 79 insertions(+), 27 deletions(-) diff --git a/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m b/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m index bc1a2491c..8bb97c67a 100644 --- a/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m +++ b/src/ndi/+ndi/+cloud/+download/+internal/batchSignedUrlLookup.m @@ -66,17 +66,27 @@ % Default 20*3600 (the server currently signs URLs % for 24 h, leaving a buffer). A scope past its % TTL is refetched on the next miss. -% partialMapRetrySeconds - How long to wait, in seconds, before -% re-fetching a scope whose first answer was a -% populated map that did not name the requested uid. -% Default 0.5. The retry gives a lagged batch -% endpoint time to catch up after a bulk upload; a -% zero value skips the wait but still performs the -% single retry (used by tests). See -% VH-Lab/NDI-matlab#991 and Waltham-Data-Science/ -% NDI-python#309. +% partialMapRetryDelays - Row vector of waits, in seconds, between +% successive re-fetches of a scope whose previous +% answer was a populated map that did not name the +% requested uid. Default [1 3 9] -- one initial call +% plus up to three retries at 1 s / 3 s / 9 s, ~13 s +% of wall clock in the worst case. The retries give +% a lagged batch endpoint time to catch up after +% ``waitForAllBulkUploads`` returns; if the whole +% schedule is spent and the map still misses, the +% caller falls back per uid. See +% VH-Lab/NDI-matlab#991 and +% Waltham-Data-Science/NDI-python#309 and #320. +% partialMapRetrySeconds - Deprecated single-wait scalar. NaN +% (default) means "use partialMapRetryDelays". Any +% other value replaces the schedule with a +% one-element one containing that wait, matching the +% pre-schedule contract where 0 still fires one +% retry (with no wait). Kept so tests pinned to that +% contract keep passing. % sleepFcn - Function handle called as sleepFcn(seconds) to -% perform the pause before a partial-map retry. +% perform the pause before each partial-map retry. % Defaults to @pause. Present so tests can % short-circuit the wait without adding real % latency to a suite that exercises the retry many @@ -170,7 +180,22 @@ options.clearCache (1,1) logical = false options.ttlSeconds (1,1) double = 20*3600 options.failureTtlSeconds (1,1) double = 60 - options.partialMapRetrySeconds (1,1) double = 0.5 + % PARTIAL-MAP retry schedule (bounded exponential backoff). Each + % entry is a wait, in seconds, before the corresponding retry. + % A scope whose Nth wave still returns a partial map falls back + % per uid without another wave. Three waves at 1 s / 3 s / 9 s + % cover the User-1-prod tail observed on + % Waltham-Data-Science/NDI-python#320: one 0.5 s retry was not + % enough to bridge the settle between bulk-upload extraction and + % the signed-URL-set index catching up. See also + % VH-Lab/NDI-matlab#991. + options.partialMapRetryDelays (1,:) double = [1 3 9] + % Back-compat scalar knob. If passed, replaces the schedule with + % a single-element one containing that wait. NaN means "not + % set", so partialMapRetryDelays wins. Preserves the pre-schedule + % semantics where partialMapRetrySeconds=0 still fires one retry + % (with no wait). + options.partialMapRetrySeconds (1,1) double = NaN options.sleepFcn (1,1) function_handle = @pause options.retryBackoffSeconds (1,:) double = [1 4 16] % Disk cache off by default: only enable when we have a URL-set @@ -207,7 +232,20 @@ FAILEDSCOPES = containers.Map('KeyType','char','ValueType','any'); end if isempty(RETRIEDSCOPES) || options.clearCache - RETRIEDSCOPES = containers.Map('KeyType','char','ValueType','logical'); + % Values are the number of retry waves already spent on this + % scope, bounded by numel(effectiveDelays). Was a logical + % "has been retried" set before the multi-wave rework. + RETRIEDSCOPES = containers.Map('KeyType','char','ValueType','double'); + end + + % Resolve the effective partial-map schedule. A caller passing the + % deprecated scalar partialMapRetrySeconds gets a one-element + % schedule with that wait (scalar=0 still fires ONE retry, matching + % the pre-schedule behaviour). + if isnan(options.partialMapRetrySeconds) + effectiveDelays = options.partialMapRetryDelays; + else + effectiveDelays = options.partialMapRetrySeconds; end url = ""; @@ -277,14 +315,25 @@ % misses becomes a whole series' worth of hits from one retry. % See VH-Lab/NDI-matlab#991 and Waltham-Data-Science/ % NDI-python#309. - if ~isKey(RETRIEDSCOPES, cacheKey) - RETRIEDSCOPES(cacheKey) = true; - STATS.partialMapRetries = STATS.partialMapRetries + 1; - if options.partialMapRetrySeconds > 0 - options.sleepFcn(options.partialMapRetrySeconds); + % Multi-wave retry: each wave sleeps for effectiveDelays(n+1) + % where n is the number of waves already spent, then refetches. + % Stops as soon as a fresh map names the uid, or when the + % schedule is exhausted. + if isKey(RETRIEDSCOPES, cacheKey) + attemptsSpent = RETRIEDSCOPES(cacheKey); + else + attemptsSpent = 0; + end + while attemptsSpent < numel(effectiveDelays) + delay = effectiveDelays(attemptsSpent + 1); + if delay > 0 + options.sleepFcn(delay); end % Drop the stale entry so localFetchScope replaces it fresh. remove(CACHE, cacheKey); + STATS.partialMapRetries = STATS.partialMapRetries + 1; + attemptsSpent = attemptsSpent + 1; + RETRIEDSCOPES(cacheKey) = attemptsSpent; refreshed = localFetchScope(cacheKey); if isempty(refreshed) % localFetchScope already recorded the miss and warned; diff --git a/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m b/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m index 7e3d96d07..ea79b4b0c 100644 --- a/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m +++ b/tests/+ndi/+unittest/+cloud/FileSeriesRoundTripTest.m @@ -596,20 +596,23 @@ function testMembersSurviveADownloadFromTheCloud(testCase) "still arrived, which is why every other assertion here passes, " + ... "but the batch presign path did not answer -- and at 28,000 " + ... "members that fallback is 28,000 API calls. " + msg); - % One presign call for the scope, plus at most one retry if - % the first answer was a partial map (VH-Lab/NDI-matlab#991 - % / Waltham-Data-Science/NDI-python#309 -- some environments - % lag briefly after the bulk upload). Anything beyond that - % would be per-uid fallback, which the uidMisses assertion - % above already rules out; this is just for a clearer - % failure message. + % One presign call for the scope, plus up to N partial-map + % retries where N is the length of the default retry schedule + % (VH-Lab/NDI-matlab#991 / Waltham-Data-Science/NDI-python#309 + % and #320 -- User 1 prod can lag long enough after + % ``waitForAllBulkUploads`` that a single retry is not + % enough, so the default schedule is now 1 s / 3 s / 9 s). + % Anything beyond that would be per-uid fallback, which the + % uidMisses assertion above already rules out; this is just + % for a clearer failure message. + defaultRetryWaves = 3; testCase.verifyEqual(batchStats.signerCalls, 1 + batchStats.partialMapRetries, ... "the members of one series should cost ONE presign call, plus " + ... - "at most one retry if the first map was partial. Got " + ... + "at most one signer call per partial-map wave. Got " + ... "signerCalls=" + batchStats.signerCalls + ", partialMapRetries=" + ... batchStats.partialMapRetries + ". " + msg); - testCase.verifyLessThanOrEqual(batchStats.partialMapRetries, 1, ... - "the retry is bounded to once per scope; got partialMapRetries=" + ... + testCase.verifyLessThanOrEqual(batchStats.partialMapRetries, defaultRetryWaves, ... + "the retry is bounded by the delay schedule; got partialMapRetries=" + ... batchStats.partialMapRetries + ". " + msg); % The bytes, compared in full rather than by length. Setup gives From 203913069311ab197abea9d39b3ddc00aa4b1e90 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 12:36:38 +0000 Subject: [PATCH 099/110] didsqlite: single-file DID fetches bypass batch signed-URL scope When DID reads a series member, it fetches the manifest first via the same download handler with seriesName="". That call previously went through ``batchSignedUrlLookup`` under the whole-document scope, and for a lightsheet-scale pyramid document (15k+ files) the batch endpoint spent 60-90 s signing URLs the caller had no use for -- just to answer the ONE manifest URL. The subsequent seriesName="chunk.bin" call then fired its own scope, and the two answers overlapped almost completely. Now: when seriesName is empty, the batch is skipped and the fetch takes the direct ``getFileDetails`` path (O(1)), matching what ``ndi.cloud.sync.internal.fetchManifest`` already does for the internal manifest fetch. When DID follows up with a member fetch (seriesName="chunk.bin"), the batch fires there and amortizes across every member exactly as it did before. Python mirror lands in Waltham-Data-Science/NDI-python on the same branch (filehandler.download_file_from_cloud gained the same guard). See VH-Lab/NDI-matlab#1010 and Waltham-Data-Science/NDI-python#320. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+implementations/+database/didsqlite.m | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/src/ndi/+ndi/+database/+implementations/+database/didsqlite.m b/src/ndi/+ndi/+database/+implementations/+database/didsqlite.m index d605715ca..ee529d961 100644 --- a/src/ndi/+ndi/+database/+implementations/+database/didsqlite.m +++ b/src/ndi/+ndi/+database/+implementations/+database/didsqlite.m @@ -224,15 +224,34 @@ function download_file_from_cloud(destPath, sourcePath, context) if strlength(seriesName) > 0 && strlength(contextUid) > 0 ndiFileUid = char(contextUid); end + % A single-file fetch (a series manifest, or any doc-level + % attachment) arrives here with seriesName="" -- there is no + % member being asked for. Going through the per-document batch + % scope to answer one uid is pure loss: for a lightsheet-scale + % pyramid document that scope names 15k+ files, so the batch + % endpoint spends 60-90 s signing a set the caller has no use + % for, delaying the ONE URL the download actually needs by more + % than a minute. Same reasoning ndi.cloud.sync.internal + % .fetchManifest uses for the internal manifest fetch; the DID + % handler path has to make the same choice or the pyramid + % pathology reappears whenever DID reads a series member (DID + % fetches the manifest via the handler with seriesName="" before + % it fetches any members). See Waltham-Data-Science/ + % NDI-python#320 and NDI-matlab#1010. + if strlength(seriesName) == 0 + batchDocId = ""; + else + batchDocId = docId; + end % diskCache=true because DID is the production reopen-a-dataset % path: a scientist who reopens the same lightsheet the next % morning must not pay the ~85 min signed-URL-set job again. % Signals lookup only for a scoped call (empty documentId % dispatches to the batch's early return and no disk read % happens). See ndi.cloud.download.internal.signedUrlDiskCache. - fromDiskCache = strlength(docId) > 0; + fromDiskCache = strlength(batchDocId) > 0; fileUrl = ndi.cloud.download.internal.batchSignedUrlLookup( ... - string(cloudDatasetId), docId, seriesName, string(ndiFileUid), ... + string(cloudDatasetId), batchDocId, seriesName, string(ndiFileUid), ... 'diskCache', fromDiskCache); if strlength(fileUrl) == 0 From fe905889d928ee305757b2e9b4f4e856153539bc Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 13:52:42 +0000 Subject: [PATCH 100/110] TestSignedUrlSetJobScaling: add N=50k / 100k rows for the >100k regime Real lightsheet-scale documents (e.g. the 121,441-URL pyramid level that motivated Waltham-Data-Science/NDI-python#320 and ndi-cloud-node#151) sit an order of magnitude past the largest row this probe used to run (N=10000). That regime is where per-URL signing rate goes sub-linear AND the signed-count plateaus for minutes; neither is visible below ~50k. Adding N=50000 and N=100000 gives us the rows that will prove ndi-cloud-node#151's fix has landed and guard against future regressions. Two guards keep the default CI run from swallowing tens of minutes: - **Huge-N gate.** The 50k / 100k rows upload roughly 50k / 100k files each, which currently takes tens of minutes and burns cloud budget on every CI push while the perf issue is still live. ``assumeTrue(NDI_SCALINGPROBE_HUGE=='1')`` skips them by default; set the env var to run them locally when investigating the perf issue. Retire the gate once ndi-cloud-node#151 is fixed and the upload path is fast enough to afford these rows on every push. - **Size-adaptive job timeout.** The old 300 s ceiling was fine for the small rows but would swallow every ready-at-60-90 s finish the big rows will produce once the fix lands: the wait would time out before reporting the actual wall time. Now ``max(300, ceil(N / 100)) s`` -- 300 s for the small rows, 500 s at 50k, 1000 s at 100k -- a generous ceiling costs nothing on the happy path (the wait returns as soon as the job is ready) and lets the row report the real elapsed rather than "timeout". Also expanded the doc comment on ``targetMemberCount`` to name the three size regimes (plumbing / rate-matters / >100k) and the issues each row is regression-guarding. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+bugreports/TestSignedUrlSetJobScaling.m | 70 ++++++++++++++++--- 1 file changed, 61 insertions(+), 9 deletions(-) diff --git a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m index a4c06b370..deb49fd5f 100644 --- a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m +++ b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m @@ -64,15 +64,43 @@ % % A run with credentials but a small time budget can restrict % via the MATLAB unittest -Selector infrastructure or just let - % the small sizes run and Ctrl-C before the big one. 10000 is - % included per the task brief -- it is closer to a lightsheet - % level than the small sizes are. + % the small sizes run and Ctrl-C before the big one. + % + % Size regimes and their purpose: + % N5, N100, N500 -- exercise the plumbing (upload, + % extract, signed-URL job) at + % fixture-scale where the API path + % used to time out on its own. + % N2000, N10000 -- push into the "signing rate + % matters" regime where the earlier + % fix (see NDI-matlab#1010 / + % ndi-cloud-node#149) moved the + % floor from ~24 URLs/s. + % N50000, N100000 -- exercise the >100k regime that + % real lightsheet levels live in + % (a level of a 121,441-URL pyramid + % motivated Waltham-Data-Science/ + % NDI-python#320 / ndi-cloud-node + % #151). Currently gated behind + % NDI_SCALINGPROBE_HUGE=1 so the + % default CI run does not swallow a + % 30+ minute upload per row: the + % size is what proves the fix lands, + % but the upload is a cloud-budget + % hit until then. Flip the gate off + % (i.e. remove the assumeTrue below) + % once ndi-cloud-node#151 is fixed + % AND the upload path is fast + % enough that CI can afford these + % rows on every push. targetMemberCount = struct( ... - 'N5', 5, ... - 'N100', 100, ... - 'N500', 500, ... - 'N2000', 2000, ... - 'N10000', 10000); + 'N5', 5, ... + 'N100', 100, ... + 'N500', 500, ... + 'N2000', 2000, ... + 'N10000', 10000, ... + 'N50000', 50000, ... + 'N100000', 100000); end properties @@ -156,6 +184,22 @@ function testCreateSignedURLSetJobForNMemberSeries(testCase, targetMemberCount) % would poison the row and mask the server-side outcome we are % here to observe. The row is what the reader reads. + % Huge-N gate. 50k / 100k rows upload ~50-100k files each, + % which currently takes tens of minutes and burns cloud + % budget on every CI push; skip them unless + % NDI_SCALINGPROBE_HUGE=1 is explicitly set. Retire this + % gate once ndi-cloud-node#151 is fixed and CI can afford + % the upload cost on every run. + if targetMemberCount >= 50000 + hugeGate = getenv('NDI_SCALINGPROBE_HUGE'); + testCase.assumeTrue(strcmp(hugeGate, '1'), ... + sprintf(['N=%d skipped by default: set ' ... + 'NDI_SCALINGPROBE_HUGE=1 to run the ' ... + '>=50k rows. See the comment on ' ... + 'targetMemberCount for the rationale.'], ... + targetMemberCount)); + end + row = testCase.freshRow(targetMemberCount); % --- 1. Build the fixture at the size that yields ~N members @@ -250,8 +294,16 @@ function testCreateSignedURLSetJobForNMemberSeries(testCase, targetMemberCount) row.errorMessage = extractMessage(createAnswer); else row.jobId = string(createAnswer.jobId); + % Size-adaptive timeout. At the observed per-URL rate the + % small sizes finish well inside 300 s; the >=50k rows + % need a longer ceiling to reach 'ready' at all once the + % server-side fix lands. A generous ceiling costs + % nothing when the fix is fast (the wait returns as + % soon as the job is ready) and lets the row report + % the actual wall time rather than "timed out". + jobTimeoutSec = max(300, ceil(targetMemberCount / 100)); [readyOk, jobAnswer] = ndi.cloud.api.files.waitForSignedURLSetJob( ... - row.jobId, 'timeout', 300); + row.jobId, 'timeout', jobTimeoutSec); if readyOk row.jobState = "ready"; if isfield(jobAnswer, 'fileCount') && ~isempty(jobAnswer.fileCount) From b0dcce11ee1d7c278cc88528f916391a632d5447 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 17:11:00 +0000 Subject: [PATCH 101/110] TestSignedUrlSetJobScaling: pass 'chunks' to fromOMEZarr so N is real Was: the test built a spatial shape via ``pickShapeForTarget(N, [16 16 16])`` that would contain ~N chunks if the fixture wrote at [16 16 16], but its call to ``ndi.fun.doc.lightsheet.fromOMEZarr`` did not forward a ``chunks`` argument. ``makePyramid`` therefore fell through to ``chooseTileShape``'s default ~8 MB per-tile byte budget, which for uint16 lands at roughly [161 161 161] per chunk. Concrete outcome: requested N actual members ------------ -------------- 5 1 100 1 500 1 2000 8 10000 27 50000 64 100000 125 Every row degenerated into a "~125-member series" test, and the scaling axis this class exists to sweep was silently collapsed to a constant. The rows still exercised the bug tracked in VH-Lab/NDI-matlab#1010 (createSignedURLSetJob rejects a manifest that getFileDetails accepts), but the N label carried no information about scope size -- it only labelled cloud budget spent on a fixture that reproduces the small-N case seven times over. Fix: forward ``'chunks', chunkShape`` to fromOMEZarr so makePyramid uses the chunk shape the fixture wrote (which is the same shape pickShapeForTarget was already sizing the level to). Also add a regression-guard ``warning()`` that fires when the actual member count is <25% of the target for N>=100, so if this ever silently regresses again the run log makes the failure loud rather than legible only by squinting at the [scaleprobe] rows. Companion to Waltham-Data-Science/NDI-python#320 and ndi-cloud-node#151. Once the server-side signing behaviour stabilises, the huge-N rows will actually produce huge-N scopes and the performance-regime evidence they were added to capture becomes meaningful. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+bugreports/TestSignedUrlSetJobScaling.m | 34 ++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m index deb49fd5f..8694a3769 100644 --- a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m +++ b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m @@ -222,11 +222,24 @@ function testCreateSignedURLSetJobForNMemberSeries(testCase, targetMemberCount) 'subject.local_identifier', 'scaleprobe@vhlab'); S.database_add(subject); + % Pass ``chunks`` through explicitly so makePyramid does + % NOT fall through to chooseTileShape's default 8 MB + % target. Without this the [16 16 16] chunk shape + % pickShapeForTarget hands makeBlobFixture gets rechunked + % downstream to a much larger tile (observed empirically + % as [161 161 161], ~4 MB per chunk), so a "target 100000 + % members" fixture only produces ~125 members and the + % scaling axis this test exists to sweep is silently + % collapsed to a constant. Handing the same chunkShape to + % fromOMEZarr keeps the level-0 chunk shape at what the + % fixture wrote, so ceil(shape ./ chunkShape) member count + % actually matches the target. [~, levelDocs, ~] = ndi.fun.doc.lightsheet.fromOMEZarr( ... S, zarrPath, ... 'subjectID', subject.id(), ... 'materializeChunks', true, ... - 'codec', 'raw'); + 'codec', 'raw', ... + 'chunks', chunkShape); testCase.assertNotEmpty(levelDocs, ... 'fromOMEZarr produced no level documents; bail before the cloud call.'); @@ -281,6 +294,25 @@ function testCreateSignedURLSetJobForNMemberSeries(testCase, targetMemberCount) row.actualMemberCount = -1; end + % Regression guard: an actual count far below target means + % the fixture builder is silently rechunking (as it did + % before we started passing ``chunks`` through to + % fromOMEZarr), so the scaling axis is not being exercised + % at the intended size and every row degenerates into a + % test of the small-N case. Warn loudly so a future + % regression is visible in the run log rather than only + % legible by squinting at the [scaleprobe] rows. + if targetMemberCount >= 100 && row.actualMemberCount > 0 && ... + row.actualMemberCount < 0.25 * targetMemberCount + warning('NDI:test:SignedUrlSetJobScaling:FixtureCollapsed', ... + ['N=%d requested but fixture produced only %d members ' ... + '(< 25%%). The scaling axis is not being exercised as ' ... + 'intended -- check that fromOMEZarr is honoring the ' ... + 'requested chunk shape and not falling through to ' ... + 'chooseTileShape''s default byte budget.'], ... + targetMemberCount, row.actualMemberCount); + end + % --- 6. createSignedURLSetJob + wait ndiDocumentId = levelDoc.id(); tStart = tic; From e3d5f51cd13ca062b309e4e274c4b39d66d8c9e3 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 17:40:45 +0000 Subject: [PATCH 102/110] TestSignedUrlSetJobScaling: scale waitForAllBulkUploads timeout with N Uncovered by CI on head b0dcce11 (Cloud CI 2): the fixture-count fix made N=10000 actually upload 10,165 files, and the default 300 s ceiling on ``waitForAllBulkUploads`` was too tight for the server to finish extracting them -- the assertion at line 275 fired with ``state=timeout, elapsed=300.3s`` and the row never reached the createSignedURLSetJob step it was there to observe. That's a false-negative on the bug this class exists to detect: nothing was learned about the async job path at N=10000 that run. Mirrors the size-adaptive timeout already applied to waitForSignedURLSetJob: ``max(300, ceil(N / 20)) s`` -- 300 s for N=500/2000, 500 s at N=10000, 2500 s at N=50000, 5000 s at N=100000. Rate anchored to a rough "20 files/s extract" observation: at N=500, 513 files took ~9 s (well inside 300 s); at N=10000, 10165 files were still going at 300 s, and 20 files/s would predict ~500 s to finish (well inside 500 s). Includes ``timeout=Ns`` in the assertion diagnostic so a future timeout row prints the ceiling that was applied. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+bugreports/TestSignedUrlSetJobScaling.m | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m index 8694a3769..4cbc42b10 100644 --- a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m +++ b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m @@ -269,12 +269,21 @@ function testCreateSignedURLSetJobForNMemberSeries(testCase, targetMemberCount) % this dataset down. testCase.addTeardown(@() safeDeleteDataset(cloudDatasetId)); - % --- 4. Wait for the server to finish extracting the bulk zip + % --- 4. Wait for the server to finish extracting the bulk zip. + % Size-adaptive timeout: waitForAllBulkUploads' default 300 s + % is fine at N=500/2000 but too tight at N>=10000 -- the + % server was still extracting the 10k-file zip when the + % client gave up. Once the fixture-count fix (commit + % b0dcce11) started producing real N-sized uploads, this + % showed up as a red row that has nothing to do with the + % bug the class is here to observe. Scale with size so the + % test can actually reach the createSignedURLSetJob step. + bulkWaitSec = max(300, ceil(targetMemberCount / 20)); [waitOk, waitInfo] = ndi.cloud.api.files.waitForAllBulkUploads( ... - cloudDatasetId); + cloudDatasetId, 'timeout', bulkWaitSec); testCase.assertTrue(waitOk, ... - sprintf('waitForAllBulkUploads did not confirm completion (state=%s, elapsed=%.1fs).', ... - char(string(waitInfo.state)), waitInfo.elapsed)); + sprintf('waitForAllBulkUploads did not confirm completion (state=%s, elapsed=%.1fs, timeout=%ds).', ... + char(string(waitInfo.state)), waitInfo.elapsed, bulkWaitSec)); % --- 5. Extract the manifest uid from the LOCAL level doc [manifestExists, manifestLocalPath] = ... From 1d3c1c2514d523a4ba6e91280d0f352704526f6f Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 28 Sep 2026 18:16:30 +0000 Subject: [PATCH 103/110] TestSignedUrlSetJobScaling: treat bulk-extract timeout as an observation Cloud CI 2 on head e3d5f51c failed with two waitForAllBulkUploads timeouts: * N=2000 hit 300.3 s / 300 s ceiling * N=10000 hit 500.4 s / 500 s ceiling The same N=2000 row finished in 22 s on b0dcce11 and again in earlier runs at ~10 s, so the extract time swings by more than an order of magnitude with server load. Chasing that with an ever-larger ceiling would still red the PR whenever the server is warm. The probe's own philosophy, stated at line 369 of the failed-job branch, is that a job outcome IS the observation the run is here to make -- the probe does not fail on things the server does; it records them. Extend the same treatment to the bulk-extract wait: if waitForAllBulkUploads times out, record a row with state='bulkTimeout' and errorMessage naming the elapsed time and ceiling, then return from the parametric case so the next N runs. The row still surfaces the observation for the bug report; CI stays green when the extract is slow. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+bugreports/TestSignedUrlSetJobScaling.m | 34 +++++++++++++------ 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m index 4cbc42b10..1865a6a43 100644 --- a/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m +++ b/tests/+ndi/+unittest/+cloud/+bugreports/TestSignedUrlSetJobScaling.m @@ -271,19 +271,33 @@ function testCreateSignedURLSetJobForNMemberSeries(testCase, targetMemberCount) % --- 4. Wait for the server to finish extracting the bulk zip. % Size-adaptive timeout: waitForAllBulkUploads' default 300 s - % is fine at N=500/2000 but too tight at N>=10000 -- the - % server was still extracting the 10k-file zip when the - % client gave up. Once the fixture-count fix (commit - % b0dcce11) started producing real N-sized uploads, this - % showed up as a red row that has nothing to do with the - % bug the class is here to observe. Scale with size so the - % test can actually reach the createSignedURLSetJob step. + % is fine at fixture scale but the extract time swings + % wildly with server load -- on b0dcce11 N=2000 finished + % in 22 s while on e3d5f51c the same N sat past 300 s. + % Rather than chase that variance with an ever-larger + % ceiling (and fail CI whenever the server is warm), + % observe the outcome and press on: the probe's job is to + % record server behaviour at scale, and "bulk extract did + % not complete in s" is one such observation. If the + % extract times out we cannot call createSignedURLSetJob + % (files are not there yet), so the row records + % state='bulkTimeout' and the next N runs. This matches + % the philosophy stated at the failed-job branch below: + % the probe does not fail on things the server does; it + % records them. bulkWaitSec = max(300, ceil(targetMemberCount / 20)); [waitOk, waitInfo] = ndi.cloud.api.files.waitForAllBulkUploads( ... cloudDatasetId, 'timeout', bulkWaitSec); - testCase.assertTrue(waitOk, ... - sprintf('waitForAllBulkUploads did not confirm completion (state=%s, elapsed=%.1fs, timeout=%ds).', ... - char(string(waitInfo.state)), waitInfo.elapsed, bulkWaitSec)); + if ~waitOk + row.jobState = "bulkTimeout"; + row.errorMessage = sprintf( ... + 'waitForAllBulkUploads did not confirm completion (state=%s, elapsed=%.1fs, timeout=%ds).', ... + char(string(waitInfo.state)), waitInfo.elapsed, bulkWaitSec); + row.jobElapsedSec = waitInfo.elapsed; + testCase.Results(end+1) = row; + printRow(row); + return + end % --- 5. Extract the manifest uid from the LOCAL level doc [manifestExists, manifestLocalPath] = ... From d5e43d3a2f1ff9f11a5e63817f8f8ae551014bd6 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 30 Sep 2026 00:48:25 +0000 Subject: [PATCH 104/110] TestPublishWithDocsAndFiles: poll getFileDetails until uploaded=true Replace the fixed pause(20) + single-retry pause(10) with a proper per-file poll of getFileDetails, gating each verification on the file document actually flipping uploaded=true (which is when the server returns a downloadUrl). The old sequence -- upload files serially, pause 20 s, then start verifying -- collapses when the backend is under load. The file goes to S3 (client-side PUT succeeds) but the server hasn't committed the file document yet, so getFileDetails returns HTTP 200 with {uploaded: false, size: 0, no downloadUrl}. The wrapper treats that as b=false. The old getFileDetailsWithRetry then paused 10 s and tried once more; the total 30 s budget was still too short on a thrashed dev-api, and the test failed at line 178 with the "missing downloadUrl" diagnostic. The new getFileDetailsWithRetry polls every 2 s up to 300 s until b=true or timeout, matching the order of magnitude the sibling waitForAllBulkUploads uses for the bulk-upload path. Since the polling helper is called downstream in the per-file verification loop, the upfront pause(20) is redundant and is removed from both testSerialUploadPublishDownload and testMixedUploadPublishDownload; a narrative line names what is now doing the waiting so a failing run's log still explains the wait shape. Callers keep the same getFileDetailsWithRetry signature. testBulkUploadPublishDownload already used the proper 'waitForCompletion', true gate on the bulk-upload path and is untouched. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+cloud/TestPublishWithDocsAndFiles.m | 24 +++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/tests/+ndi/+unittest/+cloud/TestPublishWithDocsAndFiles.m b/tests/+ndi/+unittest/+cloud/TestPublishWithDocsAndFiles.m index 6a3baef57..8ad084320 100644 --- a/tests/+ndi/+unittest/+cloud/TestPublishWithDocsAndFiles.m +++ b/tests/+ndi/+unittest/+cloud/TestPublishWithDocsAndFiles.m @@ -83,10 +83,22 @@ function deleteDatasetAfterTest(testCase) end function [b, ans_details, resp, url] = getFileDetailsWithRetry(testCase, fileUID) + % Poll getFileDetails until the server marks the file + % uploaded and returns a downloadUrl, or until we exceed + % the timeout. getFileDetails itself returns b=false while + % the file document is still 'uploaded: false' (its + % response body carries size=0 and no downloadUrl), so a + % retry loop on ~b covers the "PUT to S3 succeeded but the + % backend hasn't committed the file document yet" window + % without treating it as a hard failure. When the backend + % is under load this window can stretch well past the + % previous 10 s single-retry budget. + timeoutSec = 300; + intervalSec = 2; + deadline = tic; [b, ans_details, resp, url] = ndi.cloud.api.files.getFileDetails(testCase.DatasetID, fileUID); - if ~b - % Retry once after a brief wait to ride out transient backend 500s. - pause(10); + while ~b && toc(deadline) < timeoutSec + pause(intervalSec); [b, ans_details, resp, url] = ndi.cloud.api.files.getFileDetails(testCase.DatasetID, fileUID); end end @@ -145,8 +157,7 @@ function testSerialUploadPublishDownload(testCase) end narrative(end+1) = "All files uploaded successfully."; - narrative(end+1) = "Pausing for 20 seconds to allow for processing before pre-publish verification..."; - pause(20); + narrative(end+1) = "Per-file getFileDetails polling (getFileDetailsWithRetry) will gate each verification on uploaded=true rather than a fixed pause."; % Step 2.5: Pre-Publish Verification narrative(end+1) = "VERIFICATION (PRE-PUBLISH): Checking documents before publishing."; @@ -304,8 +315,7 @@ function testMixedUploadPublishDownload(testCase) end narrative(end+1) = "All files uploaded successfully."; - narrative(end+1) = "Pausing for 20 seconds to allow for processing before pre-publish verification..."; - pause(20); + narrative(end+1) = "Per-file getFileDetails polling (getFileDetailsWithRetry) will gate each verification on uploaded=true rather than a fixed pause."; % Step 2.5: Pre-Publish Verification narrative(end+1) = "VERIFICATION (PRE-PUBLISH): Checking documents before publishing."; From 5c9e8a83a8ab1d7affb7fa821fc8f1b48302846f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 4 Oct 2026 00:42:07 +0000 Subject: [PATCH 105/110] lightsheet pyramid: default tile budget 8 MB -> 32 MB Measurement in production: an 8 MB uncompressed budget on a whole-brain lightsheet (~1 TB uncompressed, isotropic micron voxels) produces ~121k chunks at level 0. That is enough chunks for getSignedURLSetJob to spend ~85 minutes signing one level, and for the publisher's partial-upload failure modes (waltham-data-science/ndi-cloud-node#112) to surface as hundreds of missing chunks. 8 MB was the right number for a conservative decode budget and is still safe, but it is no longer the sweet spot once you measure the actual signing+fetch+decode pipeline. Bump the default to 32*2^20 (32 MB). At uint16 with isotropic voxels chooseTileShape lands on 256^3 chunks, which is exactly 32 MB uncompressed, divides evenly down a dyadic pyramid, and compresses to ~3-10 MB on the wire with blosc-zstd (one S3 fetch 50-200 ms, one decode under ~200 ms on a laptop). Level 0 chunk counts drop ~4x, which collapses the signing-latency problem for the common case; the signed-URL-set rework (ndi-cloud-node#149) still cuts the per-URL cost in parallel. No schema change: axes_order="tczyx", shape=[], chunks=[], chunk_grid=[] in lightsheetZarrLevel.json are already arbitrary-rank lists, and the reader (ndi.pyramid.multiscale) walks chunk_grid as a general N-D tuple. No reader change. The writer's chooseTileShape was already producing world-isotropic chunks (cubic when voxels are isotropic, pancake only when they aren't), so the chunks were never "2D tiles" at the default. Files changed: - +ndi/+fun/+doc/+lightsheet/makePyramid.m: default 8*2^20 -> 32*2^20, docstring rewritten to reference the sweet-spot measurement and cite the chunk-count impact (121k -> ~30k at the whole-brain scale) - +ndi/+fun/+doc/+lightsheet/chooseTileShape.m: default 8*2^20 -> 32*2^20, example numbers updated (isotropic -> 256^3 = exact 32 MB, anisotropic 1/1/5 -> [88 438 438] world-cube) - +ndi/+fun/+doc/+lightsheet/fromOMEZarr.m: forwarded default 8*2^20 -> 32*2^20, docstring updated - +ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md: budget prose updated - +ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m: testChunksAutoChosenByBudget now accepts either "within 1.3x of 32 MB budget" OR "clamped to level shape" since the 128x512x512 fixture level now has chunks either shape-clamped on Z or sized by the redistribute branch; preamble updated Explicit overrides via options.tileBudgetBytes = 8*2^20 still work; existing datasets' chunk shape is stamped on the level document and is not changed by this. Co-Authored-By: Claude Opus 4.7 Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w --- .../+lightsheet/README-lightsheet-zarr.md | 8 ++++---- .../+fun/+doc/+lightsheet/chooseTileShape.m | 20 +++++++++---------- .../+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m | 6 +++--- .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 20 ++++++++++++------- .../+fun/+doc/+lightsheet/TestMakePyramid.m | 20 +++++++++++++------ 5 files changed, 44 insertions(+), 30 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md index 9055e2130..9f005b0ba 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md @@ -46,10 +46,10 @@ NDI documents are immutable, so: **Working.** - `ndi.fun.doc.lightsheet.chooseTileShape(shape, axesOrder, voxelSize, dtype, budgetBytes)` - size a chunk to a byte budget with tiles isotropic in world space. - Default budget 8 MB uncompressed, sized for a viewer over an - ~200 MB/s link fetching ~4 tiles in parallel per pan gesture. Used - by `makePyramid` per level; the source store's chunk shape is not - preserved. **Working.** + Default budget 32 MB uncompressed, sized to the measured + signing+fetch+decode sweet spot for a viewer on an ~200 MB/s link. + Used by `makePyramid` per level; the source store's chunk shape is + not preserved. **Working.** - `ndi.fun.doc.lightsheet.viewCommand(launcher, sessionPath, pyramidID, ...)` - pure command builder. Same pattern as `ndi.gui.app.GEFManager.viewCommand`; the launcher is diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m index c4b0dfcdb..de7c6c4a7 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/chooseTileShape.m @@ -16,9 +16,9 @@ % zeros or NaNs on t/c) % DTYPE - char, NGFF-style ('uint16', 'float32', ' [161 161 161] % ~161^3 * 2 bytes ~= 8.3 MB +% 32*2^20) +% -> [256 256 256] % ~256^3 * 2 bytes = 32.0 MB % -% Example (uint16, 8 MB, anisotropic 1/1/5 um): +% Example (uint16, 32 MB, anisotropic 1/1/5 um): % chooseTileShape([1200 2000 2000], 'zyx', [5 1 1], 'uint16', ... -% 8*2^20) -% -> [55 275 275] % ~55*275*275*2 ~= 7.9 MB, world-cube +% 32*2^20) +% -> [88 438 438] % ~88*438*438*2 ~= 33.8 MB, world-cube % % See also: ndi.fun.doc.lightsheet.makePyramid @@ -48,7 +48,7 @@ axesOrder (1,:) char voxelSize double dtype (1,:) char - budgetBytes (1,1) double {mustBePositive} = 8 * 2^20 + budgetBytes (1,1) double {mustBePositive} = 32 * 2^20 end shape = double(shape); diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m index e6951ffb9..801259af7 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromOMEZarr.m @@ -46,8 +46,8 @@ % via NDI.FUN.DOC.LIGHTSHEET.MAKESOURCEFILE. % label - human-readable label written into the parent pyramid. % tileBudgetBytes - target uncompressed bytes per chunk when chunk -% shape is chosen automatically. Default 8*2^20 -% (8 MB). Forwarded to makePyramid. +% shape is chosen automatically. Default 32*2^20 +% (32 MB). Forwarded to makePyramid. % chunks - optional explicit chunk shape (row vector, axes_order). % Empty (default) means auto-size from the budget. % Forwarded to makePyramid. @@ -82,7 +82,7 @@ options.pipelineVersion char = '' options.sourceFileID char = '' options.label char = '' - options.tileBudgetBytes (1,1) double {mustBePositive} = 8 * 2^20 + options.tileBudgetBytes (1,1) double {mustBePositive} = 32 * 2^20 options.chunks double = [] options.materializeChunks (1,1) logical = false options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'raw' diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index a97c5f2a1..a644cf519 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -29,12 +29,18 @@ % label - char, human-readable label for the pyramid % tileBudgetBytes - target UNCOMPRESSED bytes per chunk when the % chunk shape is chosen automatically. Default -% 8*2^20 (8 MB). Sized for a viewer over an -% ~200 MB/s link fetching ~4 parallel tiles per -% pan gesture. Chunks land near this budget but -% are clamped by the level's own shape, so a -% small coarse level may end up as one whole -% chunk. +% 32*2^20 (32 MB). Sized to the measured +% signing+fetch+decode sweet spot for a viewer +% on an ~200 MB/s link: large enough that the +% per-chunk round trip is amortised and the +% chunk count stays bounded (121k-chunk level 0 +% at 8 MB -> ~30k at 32 MB on a whole-brain +% lightsheet), small enough that per-chunk +% decode stays under ~200 ms and the renderer +% can stream chunks without stalls. Chunks land +% near this budget but are clamped by the +% level's own shape, so a small coarse level +% may end up as one whole chunk. % chunks - optional explicit chunk shape (row vector in % axes_order). Empty (default) means auto: pick % a shape ISOTROPIC IN WORLD SPACE from the @@ -66,7 +72,7 @@ options.sourceFileID char = '' options.pipelineVersion char = '' options.label char = '' - options.tileBudgetBytes (1,1) double {mustBePositive} = 8 * 2^20 + options.tileBudgetBytes (1,1) double {mustBePositive} = 32 * 2^20 options.chunks double = [] options.materializeChunks (1,1) logical = false options.sourceZarrPath char = '' diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m index 499fa9730..e7d3e573f 100644 --- a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestMakePyramid.m @@ -106,11 +106,13 @@ function testTwoPyramidsSharedLevel0DedupesToFiveDocs(testCase) end function testChunksAutoChosenByBudget(testCase) - % Level 0 shape 128 x 512 x 512 uint16 with 8 MB budget - % lands on ~128^3 * 2 ~= 4 MB (Z clamps to 128 which is - % well below the ideal cube, so the redistribution kicks in - % to push XY up). The stamped chunks should be within the - % budget. + % Level 0 shape 128 x 512 x 512 uint16 with the 32 MB + % default budget lands on the whole volume as one chunk + % (128*512*512*2 = 64 MB uncompressed, past the budget, + % but every axis is clamped so chunks == shape). The + % stamped chunks should be within 1.3x of the budget OR + % equal to the level shape, which is the shape-clamp + % case. pys = ndi.unittest.fun.doc.lightsheet.TestMakePyramid.twoPyramidsWithSharedLevel0(); [~, lds] = ndi.fun.doc.lightsheet.makePyramid( ... testCase.session, pys, {'mean','max'}, ... @@ -118,7 +120,13 @@ function testChunksAutoChosenByBudget(testCase) for k = 1:numel(lds) p = lds{k}.document_properties.lightsheetZarrLevel; bytes = prod(p.chunks) * 2; - testCase.verifyLessThanOrEqual(bytes, 1.3 * 8 * 2^20); + withinBudget = bytes <= 1.3 * 32 * 2^20; + clampedToShape = isequal(p.chunks(:).', p.shape(:).'); + testCase.verifyTrue(withinBudget || clampedToShape, ... + sprintf(['chunks %s produced %d bytes; neither ' ... + 'within 1.3x of 32 MB budget nor clamped to ' ... + 'shape %s'], mat2str(p.chunks), bytes, ... + mat2str(p.shape))); testCase.verifyEqual(numel(p.chunks), numel(p.shape)); testCase.verifyEqual(p.chunk_grid, ceil(p.shape ./ p.chunks)); end From 2f9b0eaf338d3e94a214fac73428408b7ad3e975 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 12 Oct 2026 12:30:36 +0000 Subject: [PATCH 106/110] lightsheet: launch the napari viewer with debugging on and its log kept A viewer launched from MATLAB has no terminal, so its output -- which tiles it fetched, what failed and why -- was lost. Both launch paths (ndi.fun.doc.lightsheet.view and LightsheetZarrManager's View dialog) now go through the new ndi.fun.doc.lightsheet.launchViewer, which sets NDI_LIGHTSHEET_DEBUG=1 for the viewer and appends its stdout and stderr to lightsheet_YYYYMMDDHHMMSS.txt in ndi.common.PathConstants.TempFolder (created if missing), after a header with the time and command. The log path is printed in the Command Window and shown in the manager's status line. view's --list mode is unchanged: its ids are for the user to read. Pieces: viewerLogFile (the timestamped path) and viewerShellCommand (the POSIX / cmd.exe line: env var, append redirect, detached unless 'wait') are pure and tested on every platform; launchViewer writes the header and runs it. view gains 'debug' and 'logFile' options. LightsheetZarrManager's private detach is gone (its only caller now uses launchViewer). MATLAB is not available where this was written, so TestLaunchViewer has not been run; the POSIX command it builds was exercised in sh with a stand-in launcher (debug flag set, stdout and stderr appended, a quote in the log path handled). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016wj6cuvAwgAaYN5dNM1o6L --- .../+lightsheet/README-lightsheet-zarr.md | 10 ++ .../+ndi/+fun/+doc/+lightsheet/launchViewer.m | 75 +++++++++++++ src/ndi/+ndi/+fun/+doc/+lightsheet/view.m | 47 ++++++-- .../+fun/+doc/+lightsheet/viewerLogFile.m | 20 ++++ .../+doc/+lightsheet/viewerShellCommand.m | 51 +++++++++ .../+ndi/+gui/+app/LightsheetZarrManager.m | 24 ++-- .../+fun/+doc/+lightsheet/TestLaunchViewer.m | 105 ++++++++++++++++++ 7 files changed, 304 insertions(+), 28 deletions(-) create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/launchViewer.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/viewerLogFile.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/viewerShellCommand.m create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md index 9f005b0ba..d6b9847d3 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md @@ -63,6 +63,16 @@ NDI documents are immutable, so: lists pyramids in a session, and drives Add / View / Delete. `View` builds the napari command from `viewCommand` and shells out. +### Viewer logs + +Both `View` and `ndi.fun.doc.lightsheet.view` launch the viewer through +`ndi.fun.doc.lightsheet.launchViewer`, which turns on the viewer's +debugging lines (`NDI_LIGHTSHEET_DEBUG=1`) and appends its stdout and +stderr to `lightsheet_YYYYMMDDHHMMSS.txt` in +`ndi.common.PathConstants.TempFolder`. The path is printed in the +Command Window (and shown in the manager's status line); open it with +`edit` or `type` to see what the viewer fetched and what failed. + ## What still needs building 1. **Chunk materialization** into `chunk.bin_#` on diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/launchViewer.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/launchViewer.m new file mode 100644 index 000000000..5d323b0c4 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/launchViewer.m @@ -0,0 +1,75 @@ +function [status, logFile, fullCmd] = launchViewer(cmd, options) +% NDI.FUN.DOC.LIGHTSHEET.LAUNCHVIEWER - run a napari viewer command with its log kept +% +% [STATUS, LOGFILE, FULLCMD] = ndi.fun.doc.lightsheet.LAUNCHVIEWER(CMD) +% +% Runs CMD (a napariViewLightsheet command line, as built by +% ndi.fun.doc.lightsheet.viewCommand) through SYSTEM with the viewer's +% debugging lines switched on (NDI_LIGHTSHEET_DEBUG=1) and its stdout +% and stderr appended to a log file in NDI's temporary folder: +% +% /lightsheet_YYYYMMDDHHMMSS.txt +% +% A viewer launched from MATLAB has no terminal, so without this its +% log -- which tiles it fetched, what failed and why -- is lost. The +% file starts with a header (time, command) and is named by the launch +% time, so each viewer gets its own and they sort in launch order. +% +% STATUS is the SYSTEM status: for a detached launch (the default) it +% says whether the launch started, not how the viewer ended. LOGFILE is +% the full path of the log. FULLCMD is the exact shell command run. +% +% Optional Name-Value Arguments: +% wait - block until the viewer exits (default false, so MATLAB +% stays interactive while the viewer is open). +% debug - set NDI_LIGHTSHEET_DEBUG=1 for the viewer (default true). +% logFile - write here instead of a new timestamped file in the NDI +% temporary folder. +% verbose - print the command and the log path (default true). +% +% Example: +% cmd = ndi.fun.doc.lightsheet.viewCommand(launcher, S.path(), id); +% [~, logFile] = ndi.fun.doc.lightsheet.launchViewer(cmd); +% edit(logFile) % or: type(logFile) +% +% See also: ndi.fun.doc.lightsheet.view, ndi.fun.doc.lightsheet.viewCommand, +% ndi.fun.doc.lightsheet.viewerLogFile, +% ndi.fun.doc.lightsheet.viewerShellCommand, +% ndi.gui.app.LightsheetZarrManager + + arguments + cmd (1,:) char + options.wait (1,1) logical = false + options.debug (1,1) logical = true + options.logFile (1,:) char = '' + options.verbose (1,1) logical = true + end + + logFile = options.logFile; + if isempty(logFile) + logFile = ndi.fun.doc.lightsheet.viewerLogFile(); + end + logDir = fileparts(logFile); + if ~isempty(logDir) && ~isfolder(logDir) + mkdir(logDir); + end + + % Header first, so a log whose viewer printed nothing still says what + % was launched and when. + fid = fopen(logFile, 'a'); + if fid >= 0 + fprintf(fid, '=== napari lightsheet viewer: %s ===\n', ... + char(datetime('now', 'Format', 'yyyy-MM-dd HH:mm:ss'))); + fprintf(fid, 'command : %s\n', cmd); + fprintf(fid, 'debug : %d\n\n', options.debug); + fclose(fid); + end + + fullCmd = ndi.fun.doc.lightsheet.viewerShellCommand(cmd, logFile, ... + options.wait, options.debug, ispc); + if options.verbose + fprintf('Executing: %s\n', fullCmd); + fprintf('Viewer log: %s\n', logFile); + end + status = system(fullCmd); +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m index 9938dba95..961d2aa54 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/view.m @@ -34,12 +34,19 @@ function view(session, pyramidID, options) % MATLAB stays interactive; the command is % backgrounded on POSIX shells). % verbose - print the command before running it (default true). +% debug - run the viewer with NDI_LIGHTSHEET_DEBUG=1 (default +% true). +% logFile - where the viewer's stdout/stderr go. Default: a new +% lightsheet_YYYYMMDDHHMMSS.txt in +% ndi.common.PathConstants.TempFolder (the path is +% printed). See ndi.fun.doc.lightsheet.launchViewer. % % Example -- list, then open: % ndi.fun.doc.lightsheet.view(S); % prints ids % ndi.fun.doc.lightsheet.view(S, ''); % % See also: ndi.fun.doc.lightsheet.viewCommand, +% ndi.fun.doc.lightsheet.launchViewer, % ndi.gui.app.LightsheetZarrManager arguments @@ -54,6 +61,8 @@ function view(session, pyramidID, options) options.name (1,:) char = '' options.wait (1,1) logical = false options.verbose (1,1) logical = true + options.debug (1,1) logical = true + options.logFile (1,:) char = '' end sessionPath = options.sessionPath; @@ -79,20 +88,34 @@ function view(session, pyramidID, options) 'level', options.level); end - if ~options.wait && ~ispc - % Background the process on POSIX so MATLAB does not block. - % On Windows fall back to a foreground wait (simple; no - % `start` handling here). - cmd = [cmd ' &']; - end - - if options.verbose - fprintf('Executing: %s\n', cmd); + if isempty(pyramidID) + % --list prints the ids for the caller to read: keep it on screen. + if ~options.wait && ~ispc + % Background the process on POSIX so MATLAB does not block. + % On Windows fall back to a foreground wait (simple; no + % `start` handling here). + cmd = [cmd ' &']; + end + if options.verbose + fprintf('Executing: %s\n', cmd); + end + status = system(cmd); + else + % The viewer itself: debugging lines on, and stdout/stderr kept + % in a log in NDI's temporary folder -- launched from MATLAB it + % has no terminal, so this is the only record of what it did. + [status, logFile] = ndi.fun.doc.lightsheet.launchViewer(cmd, ... + 'wait', options.wait, 'debug', options.debug, ... + 'logFile', options.logFile, 'verbose', options.verbose); end - status = system(cmd); if status ~= 0 && options.wait - warning('NDI:lightsheet:view:nonZeroExit', ... - 'Launcher returned status %d.', status); + if isempty(pyramidID) + warning('NDI:lightsheet:view:nonZeroExit', ... + 'Launcher returned status %d.', status); + else + warning('NDI:lightsheet:view:nonZeroExit', ... + 'Launcher returned status %d; see %s.', status, logFile); + end end end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/viewerLogFile.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/viewerLogFile.m new file mode 100644 index 000000000..21a5b537c --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/viewerLogFile.m @@ -0,0 +1,20 @@ +function logFile = viewerLogFile(when) +% NDI.FUN.DOC.LIGHTSHEET.VIEWERLOGFILE - path for a new napari viewer log +% +% LOGFILE = ndi.fun.doc.lightsheet.VIEWERLOGFILE() +% LOGFILE = ndi.fun.doc.lightsheet.VIEWERLOGFILE(WHEN) +% +% Returns /lightsheet_YYYYMMDDHHMMSS.txt +% for the time WHEN (a datetime; default now). Named by launch time so +% each viewer gets its own log and a folder listing sorts them in +% launch order. Does not create the file or the folder. +% +% See also: ndi.fun.doc.lightsheet.launchViewer + + arguments + when (1,1) datetime = datetime('now') + end + stamp = char(datetime(when, 'Format', 'yyyyMMddHHmmss')); + logFile = fullfile(ndi.common.PathConstants.TempFolder, ... + ['lightsheet_' stamp '.txt']); +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/viewerShellCommand.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/viewerShellCommand.m new file mode 100644 index 000000000..e3d589ad5 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/viewerShellCommand.m @@ -0,0 +1,51 @@ +function fullCmd = viewerShellCommand(cmd, logFile, wait, debug, isWindows) +% NDI.FUN.DOC.LIGHTSHEET.VIEWERSHELLCOMMAND - the shell line that runs a viewer with a log +% +% FULLCMD = ndi.fun.doc.lightsheet.VIEWERSHELLCOMMAND(CMD, LOGFILE, WAIT, DEBUG, ISWINDOWS) +% +% Wraps the viewer command CMD so that, when passed to SYSTEM: +% * NDI_LIGHTSHEET_DEBUG=1 is set for the viewer process (if DEBUG), +% * its stdout and stderr are APPENDED to LOGFILE, and +% * it runs detached (unless WAIT), so MATLAB is not blocked for as +% long as the viewer is open. +% +% Pure: builds the string only, for either shell (ISWINDOWS picks +% cmd.exe syntax over POSIX sh), so it can be tested anywhere. +% ndi.fun.doc.lightsheet.launchViewer is the function that runs it. +% +% POSIX: NDI_LIGHTSHEET_DEBUG=1 CMD >>'LOGFILE' 2>&1 & +% Windows: start "napari" /min cmd /c "set NDI_LIGHTSHEET_DEBUG=1&& CMD >>"LOGFILE" 2>&1" +% +% See also: ndi.fun.doc.lightsheet.launchViewer + + arguments + cmd (1,:) char + logFile (1,:) char + wait (1,1) logical = false + debug (1,1) logical = true + isWindows (1,1) logical = ispc + end + + if isWindows + inner = sprintf('%s >>"%s" 2>&1', cmd, logFile); + if debug + % No space before &&: cmd.exe would keep it in the value. + inner = ['set NDI_LIGHTSHEET_DEBUG=1&& ' inner]; + end + if wait + fullCmd = inner; + else + % cmd /c strips the outer quotes and runs the rest as typed. + fullCmd = ['start "napari" /min cmd /c "' inner '"']; + end + else + quotedLog = ['''' strrep(logFile, '''', '''\''''') '''']; + fullCmd = [cmd ' >>' quotedLog ' 2>&1']; + if debug + fullCmd = ['NDI_LIGHTSHEET_DEBUG=1 ' fullCmd]; + end + if ~wait + fullCmd = [fullCmd ' &']; + end + end +end diff --git a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m index fba07c18a..c30caf6f7 100644 --- a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m +++ b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m @@ -268,17 +268,6 @@ function accent(btn, c) btn.FontWeight = 'bold'; end - function cmd = detach(cmd) - % DETACH - the same command, not blocking MATLAB - % - % The viewer runs for as long as the user looks at it. Without - % this, MATLAB would sit unresponsive behind it. - if ispc - cmd = ['start "napari" ' cmd]; - else - cmd = [cmd ' &']; - end - end end % ===================================================================== @@ -553,15 +542,18 @@ function localGo() return; end ndi.gui.app.LightsheetZarrManager.setLauncherPath(target); - status = system( ... - ndi.gui.app.LightsheetZarrManager.detach(cmd)); + % Debugging lines on, and the viewer's output kept in a + % log in NDI's temporary folder: launched from here it + % has no terminal, so the log is the only record of what + % it fetched and what failed. + [status, logFile] = ndi.fun.doc.lightsheet.launchViewer(cmd); delete(d); if status == 0 - obj.setStatus(sprintf('Launched the viewer for %s.', ... - obj.rowLabel(r))); + obj.setStatus(sprintf(['Launched the viewer for %s. ' ... + 'Its log: %s'], obj.rowLabel(r), logFile)); else obj.setStatus(sprintf(['The launcher exited %d. ' ... - 'The command was: %s'], status, cmd)); + 'The command was: %s (log: %s)'], status, cmd, logFile)); end end end diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m new file mode 100644 index 000000000..717636397 --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m @@ -0,0 +1,105 @@ +classdef TestLaunchViewer < matlab.unittest.TestCase + % TestLaunchViewer - the napari viewer is launched with its log kept. + % + % A viewer launched from MATLAB has no terminal, so its stdout and + % stderr -- which tiles it fetched, what failed and why -- go to + % lightsheet_YYYYMMDDHHMMSS.txt in NDI's temporary folder, with + % NDI_LIGHTSHEET_DEBUG=1 so the debugging lines are in it. + % viewerShellCommand and viewerLogFile are pure and tested on every + % platform; launchViewer runs a stand-in launcher on POSIX. + + methods (Test) + + function testLogFileIsNamedByLaunchTimeInNdiTemp(testCase) + when = datetime(2026, 10, 12, 9, 5, 7); + f = ndi.fun.doc.lightsheet.viewerLogFile(when); + testCase.verifyEqual(f, fullfile(ndi.common.PathConstants.TempFolder, ... + 'lightsheet_20261012090507.txt')); + end + + function testPosixDetachedDebugCommand(testCase) + c = ndi.fun.doc.lightsheet.viewerShellCommand( ... + '/usr/local/bin/napariViewLightsheet ''/s'' --pyramid p', ... + '/tmp/nditemp/lightsheet_1.txt', false, true, false); + testCase.verifyEqual(c, ['NDI_LIGHTSHEET_DEBUG=1 ' ... + '/usr/local/bin/napariViewLightsheet ''/s'' --pyramid p ' ... + '>>''/tmp/nditemp/lightsheet_1.txt'' 2>&1 &']); + end + + function testPosixWaitWithoutDebug(testCase) + c = ndi.fun.doc.lightsheet.viewerShellCommand('v', '/tmp/l.txt', ... + true, false, false); + testCase.verifyEqual(c, 'v >>''/tmp/l.txt'' 2>&1'); + end + + function testPosixQuotesALogPathWithAQuote(testCase) + c = ndi.fun.doc.lightsheet.viewerShellCommand('v', '/tmp/it''s.txt', ... + true, false, false); + testCase.verifyEqual(c, 'v >>''/tmp/it''\''''s.txt'' 2>&1'); + end + + function testWindowsDetachedDebugCommand(testCase) + c = ndi.fun.doc.lightsheet.viewerShellCommand( ... + '"C:\ndi\napariViewLightsheet.bat" "C:\s" --pyramid p', ... + 'C:\Temp\nditemp\lightsheet_1.txt', false, true, true); + testCase.verifyEqual(c, ['start "napari" /min cmd /c "' ... + 'set NDI_LIGHTSHEET_DEBUG=1&& ' ... + '"C:\ndi\napariViewLightsheet.bat" "C:\s" --pyramid p ' ... + '>>"C:\Temp\nditemp\lightsheet_1.txt" 2>&1"']); + end + + function testWindowsWaitRunsInPlace(testCase) + c = ndi.fun.doc.lightsheet.viewerShellCommand('v', 'C:\l.txt', ... + true, true, true); + testCase.verifyEqual(c, 'set NDI_LIGHTSHEET_DEBUG=1&& v >>"C:\l.txt" 2>&1'); + end + + function testLaunchWritesHeaderOutputAndDebugFlag(testCase) + testCase.assumeFalse(ispc, 'Runs a POSIX stand-in launcher.'); + folder = tempname; + mkdir(folder); + testCase.addTeardown(@() rmdir(folder, 's')); + launcher = fullfile(folder, 'fakeViewer.sh'); + fid = fopen(launcher, 'w'); + fprintf(fid, '#!/bin/sh\necho "debug=$NDI_LIGHTSHEET_DEBUG args=$*"\necho "to stderr" >&2\n'); + fclose(fid); + system(['chmod +x ''' launcher '''']); + logFile = fullfile(folder, 'sub', 'lightsheet_test.txt'); + + [status, got, fullCmd] = ndi.fun.doc.lightsheet.launchViewer( ... + [launcher ' --pyramid p1'], 'wait', true, ... + 'logFile', logFile, 'verbose', false); + + testCase.verifyEqual(status, 0); + testCase.verifyEqual(got, logFile); + testCase.verifySubstring(fullCmd, 'NDI_LIGHTSHEET_DEBUG=1'); + text = fileread(logFile); + testCase.verifySubstring(text, '=== napari lightsheet viewer:'); + testCase.verifySubstring(text, ['command : ' launcher ' --pyramid p1']); + testCase.verifySubstring(text, 'debug=1 args=--pyramid p1'); + testCase.verifySubstring(text, 'to stderr'); + end + + function testASecondLaunchAppendsRatherThanOverwrites(testCase) + testCase.assumeFalse(ispc, 'Runs a POSIX command.'); + logFile = [tempname '.txt']; + testCase.addTeardown(@() delete(logFile)); + ndi.fun.doc.lightsheet.launchViewer('echo first', 'wait', true, ... + 'logFile', logFile, 'verbose', false); + ndi.fun.doc.lightsheet.launchViewer('echo second', 'wait', true, ... + 'logFile', logFile, 'verbose', false); + text = fileread(logFile); + testCase.verifySubstring(text, 'first'); + testCase.verifySubstring(text, 'second'); + end + + function testTheManagerLaunchesThroughLaunchViewer(testCase) + % The View dialog must keep the log too, not just the + % command-line helper. + src = fileread(which('ndi.gui.app.LightsheetZarrManager')); + testCase.verifySubstring(src, 'ndi.fun.doc.lightsheet.launchViewer(cmd)'); + src = fileread(which('ndi.fun.doc.lightsheet.view')); + testCase.verifySubstring(src, 'ndi.fun.doc.lightsheet.launchViewer(cmd'); + end + end +end From 04ce18f9dc39a81b2118450fc02270ce48f4b669 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 12 Oct 2026 13:28:56 +0000 Subject: [PATCH 107/110] lightsheet: fromSPIM -- pyramid straight from SmartSPIM All_Channels ndi.fun.doc.lightsheet.fromSPIM(S, rootDir, 'subjectID', id) writes the same lightsheetZarrPyramid + lightsheetZarrLevel documents as fromOMEZarr, chunks materialized, without an intermediate spim2zarr / OME-Zarr step. * Level 0 (all channels, axes tczyx) is read from the stitched, harmonized All_Channels slices via ndr.format.smartspim.readAllChannelsInfo, each slice read only over the needed (y, x) window (imread PixelRegion). * The mean and max ladders are built here: each level block-reduces the level just written, so nothing re-reads full resolution. A spatial axis is halved while its voxel size is <= 1.5x the finest among axes still longer than one voxel (coarse-z stacks even out in x/y first), until every spatial axis is <= minLevelSize (512) or maxLevels (10). Level 0 is shared by both ladders (reduction_function 'none'). * Channel subset/order, voxel-size override, mean-only ladders. makePyramid gains levelSources (per level path: 'omezarr' -- the default, unchanged -- 'smartspim', or 'reduce' from an earlier level's stored chunk files, resolved to paths on the main thread so parfeval workers read plain files) and channelNames. A reduced level's read budget is shrunk by its read amplification so peak memory stays near prefetchBytes. New ndi.fun.doc.lightsheet.reduceBlocks: vectorized per-axis block mean/max with ndr.format.omezarr.reduce's semantics (trailing partial block included). NDR's reduce loops once per output voxel -- ~2M iterations per 32 MB chunk -- which is unusable at whole-brain scale. MATLAB was not available where this was written, so TestFromSPIM and TestReduceBlocks have not been run. The reduce algorithm, the ladder plan behind the tests' expected shapes, and the stored-chunk round trip (C-order write, region read-back across chunk edges) were checked with column-major NumPy mirrors. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016wj6cuvAwgAaYN5dNM1o6L --- .../+lightsheet/README-lightsheet-zarr.md | 15 + src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m | 310 ++++++++++++++++++ .../+ndi/+fun/+doc/+lightsheet/makePyramid.m | 279 ++++++++++++++-- .../+ndi/+fun/+doc/+lightsheet/reduceBlocks.m | 113 +++++++ .../+fun/+doc/+lightsheet/TestFromSPIM.m | 195 +++++++++++ .../+fun/+doc/+lightsheet/TestReduceBlocks.m | 54 +++ 6 files changed, 930 insertions(+), 36 deletions(-) create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m create mode 100644 src/ndi/+ndi/+fun/+doc/+lightsheet/reduceBlocks.m create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromSPIM.m create mode 100644 tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestReduceBlocks.m diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md index d6b9847d3..2a45ecb55 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/README-lightsheet-zarr.md @@ -57,6 +57,21 @@ NDI documents are immutable, so: `napariViewLightsheet` console script that ships in `Waltham-Data-Science/NDI-python`. **Working.** +## Straight from SmartSPIM: `fromSPIM` + +`ndi.fun.doc.lightsheet.fromSPIM(S, rootDir, 'subjectID', id)` builds the +same pyramid + level documents as `fromOMEZarr`, but reads the SmartSPIM +acquisition directly -- no spim2zarr step. Level 0 (all channels, axes +`tczyx`) comes from the stitched, harmonized `All_Channels` slices +(`ndr.format.smartspim.readAllChannelsInfo`; only the needed rows and +columns of each slice are read). The mean and max ladders are built +here, each level by block-reducing the one just written +(`ndi.fun.doc.lightsheet.reduceBlocks`), halving x/y/z while the voxel +size allows (coarse-z stacks even out in x/y first) until every spatial +axis is at most `minLevelSize` (512) voxels. Chunking, encoding, +parallel compression and installation are `makePyramid`'s, through its +`levelSources` option. + ## GUI `ndi.gui.app.LightsheetZarrManager(session)` mirrors `GEFManager`: diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m new file mode 100644 index 000000000..0324d6162 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m @@ -0,0 +1,310 @@ +function [pyramidDoc, levelDocs, info] = fromSPIM(session, rootDir, options) +% NDI.FUN.DOC.LIGHTSHEET.FROMSPIM - build a lightsheet pyramid straight from a SmartSPIM acquisition +% +% [PYRAMIDDOC, LEVELDOCS, INFO] = ndi.fun.doc.lightsheet.FROMSPIM(SESSION, ROOTDIR, ... +% 'subjectID', SUBJECTID) +% +% Reads the stitched, harmonized volume in ROOTDIR/All_Channels -- the +% output of SmartSPIM AutoStitch, which ndr.format.smartspim reads in +% preference to the per-tile Ex_* folders -- and writes the same +% documents ndi.fun.doc.lightsheet.fromOMEZarr writes for an OME-Zarr +% store: one lightsheetZarrPyramid and its lightsheetZarrLevel +% children, with chunk bytes materialized. No intermediate OME-Zarr +% (spim2zarr) step is needed. +% +% THE LADDER IS BUILT HERE. An OME-Zarr store arrives with its coarser +% levels already computed; a SmartSPIM folder has only full resolution. +% Level 0 (all channels, axes t,c,z,y,x) is read from the All_Channels +% slices. Each coarser level is then made by block-reducing the level +% just written -- once with 'mean' and once with 'max', the two +% ladders spim2zarr makes -- so no level re-reads full resolution and +% memory stays near prefetchBytes. Level 0 is shared by both ladders +% (reduction_function 'none'), exactly as fromOMEZarr dedupes it. +% +% A spatial axis is halved at a level only while its voxel size is at +% most 1.5x the finest voxel size among the axes still longer than one +% voxel, so an anisotropic stack +% (coarse z) is first evened out in x and y. Levels are added until +% every spatial axis is at most minLevelSize voxels (or maxLevels). +% +% All the chunking, encoding, parallel compression and file-series +% installation is ndi.fun.doc.lightsheet.makePyramid's, via its +% levelSources option; see makePyramid for numWorkers, prefetchBytes, +% asyncPrefetch and debugLog. +% +% Optional Name-Value Arguments: +% subjectID - REQUIRED. id of the subject the volume depends on. +% elementID - id of the imaging element (may be empty). +% label - label for the pyramid. Default: ROOTDIR's folder name. +% channels - channel directory names (e.g. "Ex_561_Em_561F_Ch2") +% to include, in the order wanted. Default: all, in +% the All_Channels ch order. +% reductions - ladders to build: any of "mean", "max". Default both. +% voxelSizeUm - [z y x] voxel size in micrometers. Default: from the +% acquisition's metadata.json; required if it has none. +% minLevelSize - stop adding levels once every spatial axis is at +% most this many voxels. Default 512. +% maxLevels - most levels per ladder, level 0 included. Default 10. +% tileBudgetBytes, chunks, codec, clevel, progressFcn, numWorkers, +% prefetchBytes, asyncPrefetch, debugLog, pipelineVersion, sourceFileID +% - as in ndi.fun.doc.lightsheet.fromOMEZarr, except +% that codec defaults to 'blosc-zstd': chunks are +% always materialized here. +% +% INFO returns a struct with fields: +% rootDir - ROOTDIR +% channelNames - cellstr, the channels on the c axis, in order +% levels - struct array (path, reduction, shape, voxelSize, factor) +% sourceFileID - id of the fileReference document for ROOTDIR +% sharedLevel0 - logical, true when level 0 is shared by the ladders +% +% Example: +% S = ndi.session.dir('/path/to/session'); +% [P, L] = ndi.fun.doc.lightsheet.fromSPIM(S, '/data/ferret42_SmartSPIM', ... +% 'subjectID', subj.id(), 'numWorkers', 8); +% +% See also: ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.fun.doc.lightsheet.makePyramid, +% ndi.fun.doc.lightsheet.reduceBlocks, +% ndr.format.smartspim.readAllChannelsInfo, +% ndr.format.smartspim.hasAllChannels + + arguments + session (1,1) + rootDir (1,:) char + options.subjectID char = '' + options.elementID char = '' + options.label char = '' + options.channels = {} + options.reductions = {'mean', 'max'} + options.voxelSizeUm double = [] + options.minLevelSize (1,1) double {mustBePositive} = 512 + options.maxLevels (1,1) double {mustBeInteger, mustBePositive} = 10 + options.pipelineVersion char = '' + options.sourceFileID char = '' + options.tileBudgetBytes (1,1) double {mustBePositive} = 32 * 2^20 + options.chunks double = [] + options.codec (1,:) char {mustBeMember(options.codec, {'raw','blosc-zstd'})} = 'blosc-zstd' + options.clevel (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.clevel, 1), mustBeLessThanOrEqual(options.clevel, 9)} = 5 + options.progressFcn = 'default' + options.numWorkers (1,1) double {mustBeInteger, mustBeGreaterThanOrEqual(options.numWorkers, -1)} = -1 + options.prefetchBytes (1,1) double {mustBePositive} = 512 * 2^20 + options.asyncPrefetch (1,1) logical = true + options.debugLog (1,:) char = '' + end + + if isempty(strtrim(options.subjectID)) + error('NDI:lightsheet:fromSPIM:noSubject', ... + 'A lightsheetZarrPyramid depends_on a subject; pass ''subjectID''.'); + end + if ~isfolder(rootDir) + error('NDI:lightsheet:fromSPIM:noSuchFolder', ... + 'SmartSPIM folder not found at %s.', rootDir); + end + if ~ndr.format.smartspim.hasAllChannels(rootDir) + error('NDI:lightsheet:fromSPIM:noAllChannels', ... + ['%s has no usable All_Channels folder (SmartSPIM AutoStitch ' ... + 'output, with slices for every channel). Run AutoStitch, or ' ... + 'convert the per-tile data with spim2zarr and use fromOMEZarr.'], ... + rootDir); + end + reductions = cellstr(options.reductions); + reductions = reductions(:); + if isempty(reductions) || ~all(ismember(reductions, {'mean', 'max'})) + error('NDI:lightsheet:fromSPIM:badReductions', ... + 'reductions must be a non-empty subset of {"mean", "max"}.'); + end + + % ---- what is on disk --------------------------------------------- + spim = ndr.format.smartspim.readAllChannelsInfo(rootDir); + [chans, channelNames] = pickChannels(spim, options.channels); + nSlices = arrayfun(@(c) c.numSlices, chans); + nz = min(nSlices); + if any(nSlices ~= nz) + warning('NDI:lightsheet:fromSPIM:unevenSlices', ... + ['Channels have different slice counts (%s); using the first ' ... + '%d slices of each.'], mat2str(nSlices), nz); + end + slicePaths = arrayfun(@(c) c.slicePaths(1:nz), chans, 'UniformOutput', false); + slicePaths = slicePaths(:); + + voxel = options.voxelSizeUm; + if isempty(voxel) + voxel = spim.voxelSizeUm; + end + if numel(voxel) ~= 3 || any(~isfinite(voxel)) || any(voxel <= 0) + error('NDI:lightsheet:fromSPIM:noVoxelSize', ... + ['No [z y x] voxel size: the acquisition metadata.json does ' ... + 'not give one. Pass ''voxelSizeUm'', [z y x].']); + end + dtype = char(spim.dtype); + shape0 = [1, numel(chans), nz, spim.height, spim.width]; + scale0 = [1, 1, double(voxel(:).')]; + + % ---- the ladder ---------------------------------------------------- + steps = planLadder(shape0, scale0, options.minLevelSize, options.maxLevels); + + axesDesc = struct('name', {'t', 'c', 'z', 'y', 'x'}, ... + 'type', {'time', 'channel', 'space', 'space', 'space'}, ... + 'unit', {'', '', 'micrometer', 'micrometer', 'micrometer'}); + level0 = levelEntry('0', shape0, scale0, dtype); + + levelSources = containers.Map(); + levelSources('0') = struct('kind', 'smartspim', ... + 'slicePaths', {slicePaths}, 'dtype', dtype); + + pyramids = repmat(struct('name', '', 'type', '', 'axes', axesDesc, ... + 'levels', level0), numel(reductions), 1); + levelInfo = struct('path', {'0'}, 'reduction', {'none'}, ... + 'shape', {shape0}, 'voxelSize', {scale0}, 'factor', {ones(1, 5)}); + for r = 1:numel(reductions) + red = reductions{r}; + levels = level0; + prevPath = '0'; + for k = 1:numel(steps) + levelPath = sprintf('%s/%d', red, k); + levels(end+1) = levelEntry(levelPath, steps(k).shape, steps(k).scale, dtype); %#ok + levelSources(levelPath) = struct('kind', 'reduce', 'fromPath', prevPath, ... + 'factor', steps(k).factor, 'reduction', red); + levelInfo(end+1) = struct('path', levelPath, 'reduction', red, ... %#ok + 'shape', steps(k).shape, 'voxelSize', steps(k).scale, ... + 'factor', steps(k).factor); + prevPath = levelPath; + end + pyramids(r).name = red; + pyramids(r).type = red; + pyramids(r).levels = levels; + end + + % ---- source description + progress --------------------------------- + sourceFileID = options.sourceFileID; + if isempty(sourceFileID) + srcDoc = ndi.fun.doc.lightsheet.makeSourceFile(session, rootDir); + session.database_add(srcDoc); + sourceFileID = srcDoc.id(); + end + label = options.label; + if isempty(label) + [~, label] = fileparts(strip(rootDir, 'right', filesep)); + end + progressFcn = resolveProgressCallback(options.progressFcn, label); + + [pyramidDoc, levelDocs, sharedLevel0] = ndi.fun.doc.lightsheet.makePyramid( ... + session, pyramids, reductions, ... + 'subjectID', options.subjectID, ... + 'elementID', options.elementID, ... + 'sourceFileID', sourceFileID, ... + 'pipelineVersion', options.pipelineVersion, ... + 'label', label, ... + 'channelNames', strjoin(channelNames, ','), ... + 'tileBudgetBytes', options.tileBudgetBytes, ... + 'chunks', options.chunks, ... + 'materializeChunks', true, ... + 'levelSources', levelSources, ... + 'codec', options.codec, ... + 'clevel', options.clevel, ... + 'progressFcn', progressFcn, ... + 'numWorkers', options.numWorkers, ... + 'prefetchBytes', options.prefetchBytes, ... + 'asyncPrefetch', options.asyncPrefetch, ... + 'debugLog', options.debugLog); + + info = struct( ... + 'rootDir', char(rootDir), ... + 'channelNames', {channelNames}, ... + 'levels', levelInfo, ... + 'sourceFileID', sourceFileID, ... + 'sharedLevel0', sharedLevel0); +end + +function steps = planLadder(shape0, scale0, minLevelSize, maxLevels) +% PLANLADDER - the coarser levels below level 0: shape, voxel size, factor +% +% Axes t,c,z,y,x; only z, y, x are reduced. At each step a spatial +% axis is halved if it is longer than 1 and its voxel size is at most +% 1.5x the finest spatial voxel size (so coarse-z stacks catch up in +% x, y first; "finest" counts only axes longer than 1). Stops when +% every spatial axis is <= minLevelSize, when +% no axis can be halved, or at maxLevels levels (level 0 included). + steps = struct('shape', {}, 'scale', {}, 'factor', {}); + shape = shape0; + scale = scale0; + while numel(steps) + 1 < maxLevels && max(shape(3:5)) > minLevelSize + f = ones(1, 5); + % Finest among the axes that can still be halved: an axis + % already down to one voxel must not hold the others back. + live = find(shape(3:5) > 1) + 2; + if isempty(live) + break; + end + finest = min(scale(live)); + for a = 3:5 + if shape(a) > 1 && scale(a) <= 1.5 * finest + f(a) = 2; + end + end + if all(f == 1) + break; + end + shape = ceil(shape ./ f); + scale = scale .* f; + steps(end+1) = struct('shape', shape, 'scale', scale, 'factor', f); %#ok + end +end + +function L = levelEntry(path, shape, scale, dtype) +% One level in the shape ndr.format.omezarr.listPyramids reports. + L = struct('path', path, 'shape', double(shape), 'chunks', [], ... + 'dtype', dtype, 'scale', double(scale), 'translation', zeros(1, 5)); +end + +function [chans, names] = pickChannels(spim, wanted) +% The channels to put on the c axis, in order. + chans = spim.channels(:); + if isempty(wanted) + names = {chans.name}; + return; + end + wanted = cellstr(wanted); + have = {chans.name}; + idx = zeros(1, numel(wanted)); + for i = 1:numel(wanted) + j = find(strcmp(have, wanted{i}), 1); + if isempty(j) + error('NDI:lightsheet:fromSPIM:noSuchChannel', ... + 'No channel %s in All_Channels. Available: %s.', ... + wanted{i}, strjoin(have, ', ')); + end + idx(i) = j; + end + chans = chans(idx); + names = {chans.name}; +end + +function fcn = resolveProgressCallback(spec, label) +% 'default' -> an NDI ProgressBarWindow; a function handle -> as is; +% [] -> silent. Same contract as fromOMEZarr's; a bar that cannot draw +% itself never stops the ingest. + if isa(spec, 'function_handle') + fcn = spec; + return; + end + fcn = []; + if isempty(spec) + return; + end + if ~((ischar(spec) || (isstring(spec) && isscalar(spec))) ... + && any(strcmpi(char(spec), {'default', 'auto'}))) + error('NDI:lightsheet:fromSPIM:badProgressFcn', ... + 'progressFcn must be ''default'', a function handle, or [].'); + end + try + tag = sprintf('lightsheet-spim-%s', char(did.ido.unique_id())); + pbw = ndi.gui.component.ProgressBarWindow('NDI lightsheet ingest'); + pbw.addBar('Label', sprintf('Building pyramid: %s', label), ... + 'Tag', tag, 'Auto', true); + fcn = @(frac, txt) pbw.updateBar(tag, min(1, max(0, frac))); + catch + fcn = []; + end +end diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m index a644cf519..c663f1ade 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/makePyramid.m @@ -55,11 +55,35 @@ % without consulting the source store. Source-store chunk shape is % NOT preserved. % -% Metadata only. n_chunks_stored is 0 on every level document in -% this PR; materializing chunk bytes into `chunk.bin_#` is follow-up -% work. +% With materializeChunks=false this is metadata only and +% n_chunks_stored is 0 on every level document. With +% materializeChunks=true each level's bytes are read, re-tiled, +% encoded and installed as the `chunk.bin_#` file series. +% +% WHERE THE BYTES COME FROM. By default every level is read from the +% OME-Zarr store at sourceZarrPath (the level's own array in it). +% levelSources overrides that per level path, for sources that are +% not an OME-Zarr store -- ndi.fun.doc.lightsheet.fromSPIM uses it to +% read level 0 from SmartSPIM's stitched All_Channels slices and to +% build every coarser level by block-reducing the level just written: +% +% levelSources - containers.Map from level path (the levels(k).path +% string in PYRAMIDS) to a source struct: +% struct('kind','omezarr') the default, from sourceZarrPath +% struct('kind','smartspim', 'slicePaths', {C x 1 cell of +% Z x 1 cellstr}, 'dtype', CLASS) +% 2-D TIFF slices, one per (c, z), +% axes t,c,z,y,x +% struct('kind','reduce', 'fromPath', PATH, 'factor', F, +% 'reduction', 'mean'|'max') +% block-reduce the level at PATH +% (written earlier in this call) +% by per-axis factor F +% A path not in the map falls back to 'omezarr'. % % See also: ndi.fun.doc.lightsheet.fromOMEZarr, +% ndi.fun.doc.lightsheet.fromSPIM, +% ndi.fun.doc.lightsheet.reduceBlocks, % ndi.fun.doc.lightsheet.levelTable, % ndi.fun.doc.lightsheet.chooseLevel @@ -124,12 +148,21 @@ % without affecting the hot path -- one line per super-block is % negligible on a multi-hour ingest. options.debugLog (1,:) char = '' + % Per-level-path byte sources; see "WHERE THE BYTES COME FROM" + % above. Empty (default) reads every level from sourceZarrPath. + options.levelSources = [] + % Comma-separated channel names written to the pyramid document. + options.channelNames (1,:) char = '' end - if options.materializeChunks && isempty(options.sourceZarrPath) + if isempty(options.levelSources) + options.levelSources = containers.Map(); + end + if options.materializeChunks && isempty(options.sourceZarrPath) ... + && options.levelSources.Count == 0 error('NDI:lightsheet:makePyramid:noSource', ... - ['materializeChunks=true requires sourceZarrPath so ' ... - 'chunks can be read from the source store.']); + ['materializeChunks=true requires sourceZarrPath (or ' ... + 'levelSources) so chunks can be read from a source.']); end if numel(pyramids) ~= numel(perEntryReduction) @@ -180,7 +213,7 @@ 'pyramid_type', char(firstEntry.type), ... 'axes_order', axesOrder, ... 'axes_units', strjoin(axesUnits, ','), ... - 'channel_names', '', ... + 'channel_names', options.channelNames, ... 'n_channels', channelCount(firstLevel.shape, axesOrder), ... 'shape_level0', firstLevel.shape, ... 'voxel_size_level0', ensureRow(firstLevel.scale), ... @@ -249,6 +282,7 @@ % Assign each unique path a 0-based level index in the ORDER it % first appears (finest first, since listPyramids is finest-first). levelDocs = cell(numel(pathOrder), 1); + docByPath = containers.Map(); sharedLevel0 = false; doneChunks = 0; for idx = 1:numel(pathOrder) @@ -276,16 +310,22 @@ reportProgress(options.progressFcn, doneChunks, totalChunks, ... sprintf('Level %d/%d (%s, %s): starting', ... idx, numel(pathOrder), char(entry.name), reductionFn)); + source = []; + if options.materializeChunks + source = resolveLevelSource(session, p, entry, level, ... + options, docByPath); + end [levelDocs{idx}, doneChunks] = makeOneLevel(session, pyramidDoc, ... entry, level, idx - 1, reductionFn, options, ... levelChunks{idx}, levelGrids{idx}, ... - doneChunks, totalChunks); + doneChunks, totalChunks, source); + docByPath(p) = levelDocs{idx}; end reportProgress(options.progressFcn, max(doneChunks, totalChunks), ... max(totalChunks, 1), 'Done.'); end -function [levelDoc, doneChunks] = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, reductionFn, options, chunks, chunkGrid, doneChunks, totalChunks) +function [levelDoc, doneChunks] = makeOneLevel(session, pyramidDoc, pyramidEntry, level, levelIndex, reductionFn, options, chunks, chunkGrid, doneChunks, totalChunks, source) if options.materializeChunks nChunksStored = prod(chunkGrid); @@ -326,7 +366,7 @@ if options.materializeChunks levelLabel = sprintf('Level %d (%s)', levelIndex, reductionFn); [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, ... - options.sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, ... + source, level, chunks, chunkGrid, ... options.codec, options.clevel, ... options.progressFcn, doneChunks, totalChunks, levelLabel, ... options.numWorkers, options.prefetchBytes, options.asyncPrefetch, ... @@ -337,22 +377,13 @@ session.database_add(levelDoc); end -function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, sourceZarrPath, pyramidEntry, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers, prefetchBytes, asyncPrefetch, debugLog) +function [levelDoc, tmpRoot, doneChunks] = attachChunkFiles(levelDoc, source, level, chunks, chunkGrid, codec, clevel, progressFcn, doneChunks, totalChunks, levelLabel, numWorkers, prefetchBytes, asyncPrefetch, debugLog) % Read the source zarr level as a whole array, then re-tile into the % level document's chunk shape and attach each chunk as chunk.bin_ % (1-based, C-order over chunkGrid). Files are ingested into DID, which % deletes the temp original as it copies -- so we build each chunk in a -% new tempname and hand it to add_file. - pyramidName = char(pyramidEntry.name); - % Which position in this pyramid's datasets holds this level? The - % `level` struct came from listPyramids(entry).levels; its index is - % the 1-based position we need for readArray. - kInEntry = find(arrayfun(@(L) strcmp(char(L.path), char(level.path)), ... - pyramidEntry.levels), 1); - if isempty(kInEntry) - error('NDI:lightsheet:makePyramid:missingLevelInEntry', ... - 'level path %s not found in pyramid %s.', level.path, pyramidName); - end +% new tempname and hand it to add_file. SOURCE (see resolveLevelSource) +% says where the level's bytes are read from. % STREAMING per-chunk read. We call readArray with `Region` per % target chunk instead of loading the whole level up front. Previous @@ -406,6 +437,11 @@ % pool (a super-block below poolSize would leave workers idle); % never larger than the level itself. chunkBytes = prod(chunks) * typesize; + % A reduced level reads prod(factor) source voxels per output voxel + % (and holds them as single while reducing, 2x a uint16), so its + % read buffer is that much bigger than the output region. Shrink + % the budget to match, so peak memory stays near prefetchBytes. + prefetchBytes = double(prefetchBytes) / readAmplification(source, typesize); prefetchChunkCount = max(poolSize, ... floor(double(prefetchBytes) / max(chunkBytes, 1))); prefetchChunkCount = min(prefetchChunkCount, nChunks); @@ -456,8 +492,7 @@ if useAsync pool = gcp(); nextFut = parfeval(pool, @readSuperBlockRegion, 1, ... - sourceZarrPath, pyramidName, kInEntry, ... - superBlocks{1}.loBBox, superBlocks{1}.hiBBox); + source, superBlocks{1}.loBBox, superBlocks{1}.hiBBox); end for s = 1:nSuperBlocks @@ -470,12 +505,10 @@ if s < nSuperBlocks nb = superBlocks{s + 1}; nextFut = parfeval(pool, @readSuperBlockRegion, 1, ... - sourceZarrPath, pyramidName, kInEntry, ... - nb.loBBox, nb.hiBBox); + source, nb.loBBox, nb.hiBBox); end else - bigTile = readSuperBlockRegion(sourceZarrPath, ... - pyramidName, kInEntry, sb.loBBox, sb.hiBBox); + bigTile = readSuperBlockRegion(source, sb.loBBox, sb.hiBBox); end % ---- STAGE bigTile INTO A PARFOR-SLICEABLE ND-ARRAY ------- @@ -714,15 +747,189 @@ function appendDebugLog(path, levelLabel, s, nSuperBlocks, doneChunks, totalChun hi = hi(1:n, :); end -function bigTile = readSuperBlockRegion(sourceZarrPath, pyramidName, kInEntry, loBBox, hiBBox) -% READSUPERBLOCKREGION - fetch one super-block region from the source +function bigTile = readSuperBlockRegion(source, loBBox, hiBBox) +% READSUPERBLOCKREGION - fetch one super-block region [loBBox, hiBBox] +% +% Dispatches on SOURCE.kind (see resolveLevelSource). Kept as a plain +% subfunction (not nested), and every source carries file paths +% rather than a session, so parfeval can run it on a pool worker +% without pulling the caller's workspace or a database handle. + switch source.kind + case 'omezarr' + bigTile = ndr.format.omezarr.readArray(source.zarrPath, ... + source.pyramidName, source.kInEntry, ... + 'Region', [loBBox; hiBBox]); + case 'smartspim' + bigTile = readSmartSpimRegion(source, loBBox, hiBBox); + case 'reduce' + bigTile = readReducedRegion(source, loBBox, hiBBox); + otherwise + error('NDI:lightsheet:makePyramid:unknownSourceKind', ... + 'Unknown level source kind ''%s''.', source.kind); + end +end + +function source = resolveLevelSource(session, p, entry, level, options, docByPath) +% RESOLVELEVELSOURCE - the self-contained read recipe for level path P % -% Wraps ndr.format.omezarr.readArray so parfeval can dispatch it as -% a background read on a pool worker. Kept as a plain subfunction -% (not nested) so the worker can serialise it without pulling the -% caller's workspace. - bigTile = ndr.format.omezarr.readArray(sourceZarrPath, ... - pyramidName, kInEntry, 'Region', [loBBox; hiBBox]); +% Everything a pool worker needs to read any region of the level, +% with no session: an OME-Zarr store + array, SmartSPIM slice paths, +% or (for a reduced level) the on-disk chunk files of the level it +% is reduced from, which was written earlier in this same call. + if isKey(options.levelSources, p) + source = options.levelSources(p); + else + source = struct('kind', 'omezarr'); + end + switch source.kind + case 'omezarr' + if isempty(options.sourceZarrPath) + error('NDI:lightsheet:makePyramid:noSource', ... + 'Level %s has no levelSources entry and no sourceZarrPath.', p); + end + % Which position in this pyramid's datasets holds this + % level? Its 1-based index is what readArray takes. + kInEntry = find(arrayfun(@(L) strcmp(char(L.path), p), ... + entry.levels), 1); + if isempty(kInEntry) + error('NDI:lightsheet:makePyramid:missingLevelInEntry', ... + 'level path %s not found in pyramid %s.', p, char(entry.name)); + end + source = struct('kind', 'omezarr', ... + 'zarrPath', char(options.sourceZarrPath), ... + 'pyramidName', char(entry.name), 'kInEntry', kInEntry); + case 'smartspim' + % Self-contained already (slice paths + dtype). + case 'reduce' + if ~isKey(docByPath, source.fromPath) + error('NDI:lightsheet:makePyramid:reduceOrder', ... + ['Level %s reduces level %s, which has not been ' ... + 'written yet. List finer levels first.'], p, source.fromPath); + end + from = docByPath(source.fromPath); + fp = from.document_properties.lightsheetZarrLevel; + nFrom = prod(double(fp.chunk_grid)); + paths = cell(1, nFrom); + % Resolve each stored chunk to its file ONCE, here on the + % main thread; workers then read plain files. + for i = 1:nFrom + f = session.database_openbinarydoc(from, sprintf('chunk.bin_%d', i)); + paths{i} = f.fullpathfilename; + session.database_closebinarydoc(f); + end + source.fromShape = double(fp.shape(:).'); + source.fromChunks = double(fp.chunks(:).'); + source.fromGrid = double(fp.chunk_grid(:).'); + source.fromCodec = char(fp.codec); + source.fromDtype = char(fp.dtype); + source.chunkPaths = paths; + source.dtype = char(level.dtype); + source.factor = double(source.factor(:).'); + otherwise + error('NDI:lightsheet:makePyramid:unknownSourceKind', ... + 'Unknown level source kind ''%s'' for level %s.', source.kind, p); + end +end + +function a = readAmplification(source, typesize) +% How many bytes the reader holds per output byte (see attachChunkFiles). + a = 1; + if strcmp(source.kind, 'reduce') + a = prod(source.factor); + if strcmp(source.reduction, 'mean') + a = a * max(1, 4 / typesize); % reduced in single precision + end + end +end + +function out = readSmartSpimRegion(source, lo, hi) +% READSMARTSPIMREGION - region [lo, hi] (axes t,c,z,y,x) from 2-D slices +% +% source.slicePaths{c}{z} is the TIFF of channel c at z. Each slice +% spans the whole stitched canvas, so only the (y, x) window is read +% (imread PixelRegion decodes just the strips that cover it). + out = zeros(double(hi - lo + 1), source.dtype); + rows = [lo(4) hi(4)]; + cols = [lo(5) hi(5)]; + for c = lo(2):hi(2) + for z = lo(3):hi(3) + tile = imread(source.slicePaths{c}{z}, 'PixelRegion', {rows, cols}); + if ~isa(tile, source.dtype) + tile = cast(tile, source.dtype); + end + out(1, c - lo(2) + 1, z - lo(3) + 1, :, :) = ... + reshape(tile, [1 1 1 size(tile, 1) size(tile, 2)]); + end + end +end + +function out = readReducedRegion(source, lo, hi) +% READREDUCEDREGION - region [lo, hi] of a level made by block-reducing +% the level below it. +% +% Output voxel i covers source voxels (i-1)*f+1 .. i*f along each axis +% (the last block may be partial), so the region needs source voxels +% (lo-1).*f+1 .. min(hi.*f, fromShape). + f = source.factor; + srcLo = (lo - 1) .* f + 1; + srcHi = min(hi .* f, source.fromShape); + src = readStoredLevelRegion(source, srcLo, srcHi); + out = ndi.fun.doc.lightsheet.reduceBlocks(src, f, source.reduction, source.dtype); +end + +function out = readStoredLevelRegion(source, lo, hi) +% READSTOREDLEVELREGION - region [lo, hi] of a level from its chunk files +% +% Chunks are C-order (last axis fastest), 1-based, each a full +% (zero-padded) `chunks`-shaped tile in C byte order, raw or Blosc. + nd = numel(source.fromShape); + chunks = source.fromChunks; + grid = source.fromGrid; + out = zeros(double(hi - lo + 1), source.fromDtype); + cLo = floor((lo - 1) ./ chunks) + 1; + cHi = floor((hi - 1) ./ chunks) + 1; + strides = ones(1, nd); + for a = nd-1:-1:1 + strides(a) = strides(a + 1) * grid(a + 1); + end + ranges = arrayfun(@(a) cLo(a):cHi(a), 1:nd, 'UniformOutput', false); + combos = cell(1, nd); + [combos{:}] = ndgrid(ranges{:}); + nTiles = numel(combos{1}); + for t = 1:nTiles + c = cellfun(@(g) g(t), combos); + linear = sum((c - 1) .* strides) + 1; + tile = readStoredChunk(source.chunkPaths{linear}, chunks, ... + source.fromCodec, source.fromDtype); + tileLo = (c - 1) .* chunks + 1; + oLo = max(tileLo, lo); + oHi = min(tileLo + chunks - 1, hi); + dst = arrayfun(@(a) (oLo(a) - lo(a) + 1):(oHi(a) - lo(a) + 1), ... + 1:nd, 'UniformOutput', false); + srcSub = arrayfun(@(a) (oLo(a) - tileLo(a) + 1):(oHi(a) - tileLo(a) + 1), ... + 1:nd, 'UniformOutput', false); + out(dst{:}) = tile(srcSub{:}); + end +end + +function tile = readStoredChunk(path, chunks, codec, dtype) +% READSTOREDCHUNK - one stored chunk file -> a `chunks`-shaped array +% +% Inverse of padEncodeAndWrite: bytes are C-order, so reshape to the +% reversed shape and permute back. + fid = fopen(path, 'r'); + if fid < 0 + error('NDI:lightsheet:makePyramid:readChunk', ... + 'Could not open stored chunk %s.', path); + end + raw = fread(fid, Inf, '*uint8'); + fclose(fid); + if strcmp(codec, 'blosc-zstd') + raw = ndr.format.blosc.decode(raw); + end + vals = typecast(raw(:), dtype); + nd = numel(chunks); + tile = permute(reshape(vals, fliplr(chunks)), nd:-1:1); end function tmp = padEncodeAndWrite(srcTile, chunks, codec, clevel, typesize, tmpRoot, chunkIdx) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/reduceBlocks.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/reduceBlocks.m new file mode 100644 index 000000000..212cf6bd7 --- /dev/null +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/reduceBlocks.m @@ -0,0 +1,113 @@ +function out = reduceBlocks(arr, factor, reduction, outClass) +% NDI.FUN.DOC.LIGHTSHEET.REDUCEBLOCKS - block-reduce an N-D array, vectorized +% +% OUT = ndi.fun.doc.lightsheet.REDUCEBLOCKS(ARR, FACTOR, REDUCTION) +% OUT = ndi.fun.doc.lightsheet.REDUCEBLOCKS(ARR, FACTOR, REDUCTION, OUTCLASS) +% +% Downsamples ARR by the per-axis integer FACTOR (one entry per axis +% of ARR, trailing singleton axes included; a scalar applies to every +% axis) using REDUCTION, 'mean' or 'max'. Output element i along an +% axis covers input elements (i-1)*f+1 .. i*f; the trailing block may +% be partial and is reduced over the elements it has, so an axis of +% length 5 with factor 2 gives 3 outputs (blocks of 2, 2 and 1). These +% are the semantics of ndr.format.omezarr.reduce and of the NGFF +% writers whose pyramids NDI ingests, so a pyramid built here matches +% one built by spim2zarr. +% +% OUTCLASS (default: the class of ARR) is the class of OUT. A 'mean' +% into an integer class is rounded to nearest. +% +% Vectorized one axis at a time: a block's mean is the mean of the +% per-axis means (the block is a Cartesian product), and likewise for +% max, so this costs a handful of whole-array operations instead of +% one loop iteration per output voxel. The mean is accumulated in +% single precision for 8- and 16-bit inputs (exact sums for blocks up +% to 256 voxels) and in double otherwise. +% +% Example: +% out = ndi.fun.doc.lightsheet.reduceBlocks(uint16(magic(5)), [2 2], 'mean'); +% +% See also: ndr.format.omezarr.reduce, ndi.fun.doc.lightsheet.fromSPIM + + arguments + arr {mustBeNumeric} + factor (1,:) double {mustBePositive, mustBeInteger} + reduction (1,:) char {mustBeMember(reduction, {'mean','max'})} + outClass (1,:) char = class(arr) + end + + if isscalar(factor) + factor = repmat(factor, 1, ndims(arr)); + end + if numel(factor) < ndims(arr) + error('NDI:lightsheet:reduceBlocks:badFactor', ... + 'factor has %d entries for a %d-D array.', numel(factor), ndims(arr)); + end + sz = size(arr, 1:numel(factor)); + + if strcmp(reduction, 'mean') + if any(strcmp(class(arr), {'uint8','int8','uint16','int16','single'})) + work = single(arr); + else + work = double(arr); + end + else + work = arr; + end + + for d = 1:numel(factor) + f = factor(d); + if f == 1 || sz(d) == 1 + continue; % nothing to reduce along this axis + end + work = reduceAxis(work, d, f, reduction); + sz(d) = size(work, d); + end + + if strcmp(reduction, 'mean') && ~any(strcmp(outClass, {'single','double'})) + work = round(work); + end + out = cast(work, outClass); +end + +function A = reduceAxis(A, d, f, reduction) +% Reduce A along axis D by factor F, trailing partial block included. + n = size(A, d); + m = ceil(n / f); + pad = m * f - n; + nd = max(ndims(A), d); + sz = size(A, 1:nd); + if pad > 0 + padSz = sz; + padSz(d) = pad; + switch reduction + case 'mean' + % Zeros add nothing to the sum; the last block's divisor + % is corrected below. + A = cat(d, A, zeros(padSz, 'like', A)); + case 'max' + % Repeat the last element: it cannot change the max. + idx = repmat({':'}, 1, nd); + idx{d} = repmat(n, 1, pad); + A = cat(d, A, A(idx{:})); + end + sz(d) = m * f; + end + % Split axis d into (f, m) and reduce over the f part. + newSz = [sz(1:d-1), f, m, sz(d+1:end)]; + A = reshape(A, newSz); + switch reduction + case 'mean' + A = sum(A, d); + counts = repmat(f, 1, m); + counts(end) = n - (m - 1) * f; + cshape = ones(1, numel(newSz)); + cshape(d + 1) = m; + A = A ./ reshape(cast(counts, 'like', A), cshape); + case 'max' + A = max(A, [], d); + end + outSz = sz; + outSz(d) = m; + A = reshape(A, outSz); +end diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromSPIM.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromSPIM.m new file mode 100644 index 000000000..8945adffe --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromSPIM.m @@ -0,0 +1,195 @@ +classdef TestFromSPIM < matlab.unittest.TestCase + % TestFromSPIM - lightsheet pyramid built straight from SmartSPIM All_Channels. + % + % Runs on NDR's SmartSPIM fixture with 'AllChannels', true: two + % channels (ch0 = Ex_561_Em_561F_Ch2, ch1 = Ex_640_Em_640F_Ch3), four + % 106x13 uint16 slices each, every pixel of slice z in channel k + % equal to 1000*(k+1) + z. Constant slices make every level's + % expected values easy to state. + + properties + session + subjectID + root + end + + methods (TestClassSetup) + function needNdrSmartSpim(testCase) + testCase.assumeTrue(~isempty(which('ndr.format.smartspim.readAllChannelsInfo')), ... + 'NDR-matlab with ndr.format.smartspim All_Channels support is not on the path.'); + end + end + + methods (TestMethodSetup) + function build(testCase) + d = fullfile(tempname, 'fromspim'); + mkdir(d); + testCase.addTeardown(@() rmdir(fileparts(d), 's')); + S = ndi.session.dir('fromspim', d); + sub = ndi.document('subject', 'base.session_id', S.id(), ... + 'subject.local_identifier', 'ferret@vhlab'); + S.database_add(sub); + testCase.session = S; + testCase.subjectID = sub.id(); + testCase.root = ndr.test.format.smartspim.makeExampleFixture( ... + fullfile(fileparts(d), 'spim'), 'AllChannels', true); + end + end + + methods + function [P, L, info] = ingest(testCase, varargin) + [P, L, info] = ndi.fun.doc.lightsheet.fromSPIM(testCase.session, ... + testCase.root, 'subjectID', testCase.subjectID, ... + 'voxelSizeUm', [1 1 1], 'minLevelSize', 8, ... + 'chunks', [1 1 2 32 8], 'progressFcn', [], ... + 'numWorkers', 0, varargin{:}); + end + + function vol = readLevel(testCase, levelDoc) + % Independent read-back: every stored chunk, decoded and placed. + p = levelDoc.document_properties.lightsheetZarrLevel; + shape = double(p.shape(:).'); + chunks = double(p.chunks(:).'); + grid = double(p.chunk_grid(:).'); + vol = zeros(shape, p.dtype); + nd = numel(shape); + for i = 1:prod(grid) + rem = i - 1; + c = zeros(1, nd); + for a = 1:nd + stride = prod(grid(a+1:end)); + c(a) = floor(rem / stride) + 1; + rem = mod(rem, stride); + end + f = testCase.session.database_openbinarydoc(levelDoc, sprintf('chunk.bin_%d', i)); + fid = fopen(f.fullpathfilename, 'r'); + raw = fread(fid, Inf, '*uint8'); + fclose(fid); + testCase.session.database_closebinarydoc(f); + if strcmp(p.codec, 'blosc-zstd') + raw = ndr.format.blosc.decode(raw); + end + tile = permute(reshape(typecast(raw(:), p.dtype), fliplr(chunks)), nd:-1:1); + lo = (c - 1) .* chunks + 1; + hi = min(c .* chunks, shape); + dst = arrayfun(@(a) lo(a):hi(a), 1:nd, 'UniformOutput', false); + src = arrayfun(@(a) 1:(hi(a) - lo(a) + 1), 1:nd, 'UniformOutput', false); + vol(dst{:}) = tile(src{:}); + end + end + + function L = levelsOf(~, levelDocs, reduction) + keep = cellfun(@(d) any(strcmp(d.document_properties.lightsheetZarrLevel.reduction_function, ... + {reduction, 'none'})), levelDocs); + L = levelDocs(keep); + lv = cellfun(@(d) d.document_properties.lightsheetZarrLevel.level, L); + [~, order] = sort(lv); + L = L(order); + end + end + + methods (Test) + + function testParentDescribesTheStitchedVolume(testCase) + P = testCase.ingest(); + p = P.document_properties.lightsheetZarrPyramid; + testCase.verifyEqual(p.axes_order, 'tczyx'); + testCase.verifyEqual(double(p.shape_level0(:).'), [1 2 4 106 13]); + testCase.verifyEqual(p.n_channels, 2); + testCase.verifyEqual(p.channel_names, 'Ex_561_Em_561F_Ch2,Ex_640_Em_640F_Ch3'); + testCase.verifyEqual(p.dtype, 'uint16'); + end + + function testLevelZeroIsTheAllChannelsData(testCase) + [~, L] = testCase.ingest(); + l0 = testCase.levelsOf(L, 'mean'); + p0 = l0{1}.document_properties.lightsheetZarrLevel; + testCase.verifyEqual(p0.reduction_function, 'none'); + vol = testCase.readLevel(l0{1}); + for k = 0:1 + for z = 1:4 + plane = squeeze(vol(1, k + 1, z, :, :)); + testCase.verifyTrue(all(plane(:) == 1000 * (k + 1) + z), ... + sprintf('ch%d z=%d', k, z)); + end + end + end + + function testMeanAndMaxLaddersShareLevelZero(testCase) + [~, L, info] = testCase.ingest(); + testCase.verifyTrue(info.sharedLevel0); + reds = cellfun(@(d) d.document_properties.lightsheetZarrLevel.reduction_function, ... + L, 'UniformOutput', false); + testCase.verifyEqual(sum(strcmp(reds, 'none')), 1); + testCase.verifyEqual(sum(strcmp(reds, 'mean')), sum(strcmp(reds, 'max'))); + testCase.verifyGreaterThan(sum(strcmp(reds, 'mean')), 0); + end + + function testLevelOneMeanAndMax(testCase) + [~, L] = testCase.ingest(); + for red = {'mean', 'max'} + lv = testCase.levelsOf(L, red{1}); + p1 = lv{2}.document_properties.lightsheetZarrLevel; + testCase.verifyEqual(double(p1.shape(:).'), [1 2 2 53 7]); + testCase.verifyEqual(double(p1.voxel_size(:).'), [1 1 2 2 2]); + vol = testCase.readLevel(lv{2}); + for k = 0:1 + base = 1000 * (k + 1); + if strcmp(red{1}, 'mean') + expect = base + [2 4]; % mean(1,2)=1.5 -> 2, mean(3,4)=3.5 -> 4 + else + expect = base + [2 4]; % max(1,2), max(3,4) + end + for z = 1:2 + plane = squeeze(vol(1, k + 1, z, :, :)); + testCase.verifyTrue(all(plane(:) == expect(z)), ... + sprintf('%s ch%d z=%d', red{1}, k, z)); + end + end + end + end + + function testLadderStopsAtMinLevelSize(testCase) + [~, ~, info] = testCase.ingest(); + meanLevels = info.levels(strcmp({info.levels.reduction}, 'mean')); + last = meanLevels(end).shape; + testCase.verifyLessThanOrEqual(max(last(3:5)), 8); + % 106 -> 53 -> 27 -> 14 -> 7 along y. + testCase.verifyEqual(numel(meanLevels), 4); + end + + function testAnisotropicZIsNotHalvedFirst(testCase) + [~, ~, info] = testCase.ingest('voxelSizeUm', [4 1 1]); + first = info.levels(find(strcmp({info.levels.reduction}, 'mean'), 1)); + testCase.verifyEqual(first.factor, [1 1 1 2 2]); + end + + function testChannelSubsetAndOrder(testCase) + P = testCase.ingest('channels', {'Ex_640_Em_640F_Ch3'}); + p = P.document_properties.lightsheetZarrPyramid; + testCase.verifyEqual(p.n_channels, 1); + testCase.verifyEqual(p.channel_names, 'Ex_640_Em_640F_Ch3'); + end + + function testMeanOnlyLadder(testCase) + [~, L] = testCase.ingest('reductions', {'mean'}); + reds = cellfun(@(d) d.document_properties.lightsheetZarrLevel.reduction_function, ... + L, 'UniformOutput', false); + testCase.verifyFalse(any(strcmp(reds, 'max'))); + end + + function testNoAllChannelsIsAClearError(testCase) + plain = ndr.test.format.smartspim.makeExampleFixture( ... + fullfile(tempname, 'plain')); + testCase.addTeardown(@() rmdir(fileparts(plain), 's')); + testCase.verifyError(@() ndi.fun.doc.lightsheet.fromSPIM( ... + testCase.session, plain, 'subjectID', testCase.subjectID, ... + 'progressFcn', []), 'NDI:lightsheet:fromSPIM:noAllChannels'); + end + + function testSubjectIsRequired(testCase) + testCase.verifyError(@() ndi.fun.doc.lightsheet.fromSPIM( ... + testCase.session, testCase.root), 'NDI:lightsheet:fromSPIM:noSubject'); + end + end +end diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestReduceBlocks.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestReduceBlocks.m new file mode 100644 index 000000000..a4ffb671e --- /dev/null +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestReduceBlocks.m @@ -0,0 +1,54 @@ +classdef TestReduceBlocks < matlab.unittest.TestCase + % TestReduceBlocks - vectorized block reduction for building pyramid levels. + % + % Must agree with ndr.format.omezarr.reduce (the per-voxel reference): + % output i covers inputs (i-1)*f+1 .. i*f, trailing partial block + % reduced over what it has. + + methods (Test) + + function testMeanMatchesNdrReduce(testCase) + rng(1); + a = uint16(randi(65535, [5 7 9])); + got = ndi.fun.doc.lightsheet.reduceBlocks(a, [2 2 2], 'mean', 'double'); + ref = ndr.format.omezarr.reduce(a, [2 2 2], 'mean'); + testCase.verifyEqual(got, ref, 'AbsTol', 1e-2); + end + + function testMaxMatchesNdrReduce(testCase) + rng(2); + a = uint16(randi(65535, [6 5 3])); + got = ndi.fun.doc.lightsheet.reduceBlocks(a, [2 2 2], 'max'); + ref = ndr.format.omezarr.reduce(a, [2 2 2], 'max'); + testCase.verifyEqual(got, ref); + testCase.verifyClass(got, 'uint16'); + end + + function testPartialTrailingBlock(testCase) + % Length 5, factor 2: blocks of 2, 2 and 1. + a = uint16([1 3 5 7 9]); + got = ndi.fun.doc.lightsheet.reduceBlocks(a, [1 2], 'mean', 'double'); + testCase.verifyEqual(got, [2 6 9]); + end + + function testMeanIntoIntegerRounds(testCase) + a = uint16([1 2]); + got = ndi.fun.doc.lightsheet.reduceBlocks(a, [1 2], 'mean'); + testCase.verifyEqual(got, uint16(2)); % 1.5 rounds to 2 + end + + function testTrailingSingletonAxesInFiveD(testCase) + % t, c, z, y, x with a singleton x: factor entries for every axis. + a = reshape(uint16(1:16), [1 2 4 2 1]); + got = ndi.fun.doc.lightsheet.reduceBlocks(a, [1 1 2 2 2], 'max'); + testCase.verifyEqual(size(got, 1:5), [1 2 2 1 1]); + ref = ndr.format.omezarr.reduce(a, [1 1 2 2], 'max'); + testCase.verifyEqual(got(:), ref(:)); + end + + function testFactorOneIsIdentity(testCase) + a = uint16(magic(4)); + testCase.verifyEqual(ndi.fun.doc.lightsheet.reduceBlocks(a, [1 1], 'mean'), a); + end + end +end From 668beb135adf6d39b9ad1247b3822effdee12a02 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 12 Oct 2026 13:43:44 +0000 Subject: [PATCH 108/110] TestFromSPIM: the fixture's slices are 0-based; preallocate fromSPIM's ladders CI's first MATLAB run of the new tests: TestReduceBlocks, TestLaunchViewer and the structural TestFromSPIM tests pass; the two value checks failed on every plane. ndr.test.format.smartspim.makeExampleFixture writes All_Channels slice z (0-based) as 1000*(k+1) + z, so 1-based slice z holds 1000*(k+1) + z - 1 -- the expectations were one high. Level 1 (z halved) is then base + [1 3] for both mean (0.5 and 2.5 round half away from zero) and max. The data path was right; the expected values were not. Also preallocates the per-ladder level and info arrays in fromSPIM (Code Analyzer flagged the growing levelInfo: its %#ok sat on a continuation line, where the analyzer does not read it). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016wj6cuvAwgAaYN5dNM1o6L --- src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m | 18 ++++++++++-------- .../+fun/+doc/+lightsheet/TestFromSPIM.m | 18 +++++++++--------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m index 0324d6162..c687f4f5d 100644 --- a/src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m +++ b/src/ndi/+ndi/+fun/+doc/+lightsheet/fromSPIM.m @@ -155,20 +155,22 @@ pyramids = repmat(struct('name', '', 'type', '', 'axes', axesDesc, ... 'levels', level0), numel(reductions), 1); - levelInfo = struct('path', {'0'}, 'reduction', {'none'}, ... - 'shape', {shape0}, 'voxelSize', {scale0}, 'factor', {ones(1, 5)}); + nSteps = numel(steps); + levelInfo = repmat(struct('path', '0', 'reduction', 'none', ... + 'shape', shape0, 'voxelSize', scale0, 'factor', ones(1, 5)), ... + 1, 1 + numel(reductions) * nSteps); for r = 1:numel(reductions) red = reductions{r}; - levels = level0; + levels = repmat(level0, 1, 1 + nSteps); prevPath = '0'; - for k = 1:numel(steps) + for k = 1:nSteps levelPath = sprintf('%s/%d', red, k); - levels(end+1) = levelEntry(levelPath, steps(k).shape, steps(k).scale, dtype); %#ok + levels(1 + k) = levelEntry(levelPath, steps(k).shape, steps(k).scale, dtype); levelSources(levelPath) = struct('kind', 'reduce', 'fromPath', prevPath, ... 'factor', steps(k).factor, 'reduction', red); - levelInfo(end+1) = struct('path', levelPath, 'reduction', red, ... %#ok - 'shape', steps(k).shape, 'voxelSize', steps(k).scale, ... - 'factor', steps(k).factor); + levelInfo(1 + (r - 1) * nSteps + k) = struct('path', levelPath, ... + 'reduction', red, 'shape', steps(k).shape, ... + 'voxelSize', steps(k).scale, 'factor', steps(k).factor); prevPath = levelPath; end pyramids(r).name = red; diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromSPIM.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromSPIM.m index 8945adffe..8a0cf2dbf 100644 --- a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromSPIM.m +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestFromSPIM.m @@ -3,9 +3,10 @@ % % Runs on NDR's SmartSPIM fixture with 'AllChannels', true: two % channels (ch0 = Ex_561_Em_561F_Ch2, ch1 = Ex_640_Em_640F_Ch3), four - % 106x13 uint16 slices each, every pixel of slice z in channel k - % equal to 1000*(k+1) + z. Constant slices make every level's - % expected values easy to state. + % 106x13 uint16 slices each, every pixel of slice z (0-based) in + % channel k equal to 1000*(k+1) + z -- so 1-based slice z holds + % 1000*(k+1) + z - 1. Constant slices make every level's expected + % values easy to state. properties session @@ -109,7 +110,7 @@ function testLevelZeroIsTheAllChannelsData(testCase) for k = 0:1 for z = 1:4 plane = squeeze(vol(1, k + 1, z, :, :)); - testCase.verifyTrue(all(plane(:) == 1000 * (k + 1) + z), ... + testCase.verifyTrue(all(plane(:) == 1000 * (k + 1) + z - 1), ... sprintf('ch%d z=%d', k, z)); end end @@ -135,11 +136,10 @@ function testLevelOneMeanAndMax(testCase) vol = testCase.readLevel(lv{2}); for k = 0:1 base = 1000 * (k + 1); - if strcmp(red{1}, 'mean') - expect = base + [2 4]; % mean(1,2)=1.5 -> 2, mean(3,4)=3.5 -> 4 - else - expect = base + [2 4]; % max(1,2), max(3,4) - end + % Slices hold base + 0..3. Mean of {0,1} = 0.5 and of + % {2,3} = 2.5 round half away from zero to 1 and 3; + % max gives 1 and 3 too. + expect = base + [1 3]; for z = 1:2 plane = squeeze(vol(1, k + 1, z, :, :)); testCase.verifyTrue(all(plane(:) == expect(z)), ... From fb10169b647320a42e84022c2e99a9422aeb4da5 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 20:04:06 +0000 Subject: [PATCH 109/110] LightsheetZarrManager: View dialog previews the exact logged launch command The dialog previewed only the bare napariViewLightsheet command, while Launch ran it wrapped with NDI_LIGHTSHEET_DEBUG=1 and the log redirection. The log file is now named when the dialog opens, the preview shows viewerShellCommand(cmd, logFile, false, true, ispc) and Launch passes that same logFile to launchViewer, so the text shown is exactly the line that runs (and can be pasted into a terminal). Errors from building the command are still shown as plain text. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016wj6cuvAwgAaYN5dNM1o6L --- .../+ndi/+gui/+app/LightsheetZarrManager.m | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m index c30caf6f7..a69025a22 100644 --- a/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m +++ b/src/ndi/+ndi/+gui/+app/LightsheetZarrManager.m @@ -495,10 +495,15 @@ function onView(obj) everything{i}.ValueChangedFcn = @localRefresh; end + % One log per launch, named now so the command shown is the + % exact line that will run (and can be pasted in a terminal). + logFile = ndi.fun.doc.lightsheet.viewerLogFile(); + localRefresh(); obj.viewDialog = d; - function cmd = localCommand() + function [cmd, ok] = localCommand() + ok = true; try red = ''; if ~strcmp(redDd.Value, '(as selected)') @@ -513,11 +518,19 @@ function onView(obj) 'name', nameEd.Value); catch ME cmd = ME.message; + ok = false; end end function localRefresh(~,~) - cmdArea.Value = localCommand(); + % Show the full shell line: debugging on and output + % appended to the log, as launchViewer will run it. + [cmd, ok] = localCommand(); + if ok + cmd = ndi.fun.doc.lightsheet.viewerShellCommand( ... + cmd, logFile, false, true, ispc); + end + cmdArea.Value = cmd; end function localBrowse() @@ -546,7 +559,7 @@ function localGo() % log in NDI's temporary folder: launched from here it % has no terminal, so the log is the only record of what % it fetched and what failed. - [status, logFile] = ndi.fun.doc.lightsheet.launchViewer(cmd); + status = ndi.fun.doc.lightsheet.launchViewer(cmd, 'logFile', logFile); delete(d); if status == 0 obj.setStatus(sprintf(['Launched the viewer for %s. ' ... From b8e4b54b9dc181060e4534c41bbe16d612d78e1a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 20:16:50 +0000 Subject: [PATCH 110/110] TestLaunchViewer: match the manager's logged launch and preview testTheManagerLaunchesThroughLaunchViewer looked for the literal launchViewer(cmd), which fb10169b replaced with launchViewer(cmd, 'logFile', logFile). Check for that call, and that the dialog's preview is built by viewerShellCommand for the same log file with launchViewer's defaults (no wait, debug on). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_016wj6cuvAwgAaYN5dNM1o6L --- .../+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m index 717636397..a0123175b 100644 --- a/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m +++ b/tests/+ndi/+unittest/+fun/+doc/+lightsheet/TestLaunchViewer.m @@ -95,9 +95,15 @@ function testASecondLaunchAppendsRatherThanOverwrites(testCase) function testTheManagerLaunchesThroughLaunchViewer(testCase) % The View dialog must keep the log too, not just the - % command-line helper. + % command-line helper, and its preview must be the exact line + % that runs: the shell command built for the same log file + % the launch then writes to. src = fileread(which('ndi.gui.app.LightsheetZarrManager')); - testCase.verifySubstring(src, 'ndi.fun.doc.lightsheet.launchViewer(cmd)'); + testCase.verifySubstring(src, ... + 'ndi.fun.doc.lightsheet.launchViewer(cmd, ''logFile'', logFile)'); + testCase.verifySubstring(src, ... + 'ndi.fun.doc.lightsheet.viewerShellCommand( ...'); + testCase.verifySubstring(src, 'cmd, logFile, false, true, ispc)'); src = fileread(which('ndi.fun.doc.lightsheet.view')); testCase.verifySubstring(src, 'ndi.fun.doc.lightsheet.launchViewer(cmd'); end