Repository navigation
Conversation
…greement
Confirmed 2026-09-27: the async signed-URL-set-job failure is
GLOBAL, not dataset-specific. The Python integration test raises
the same BatchScopeUnreachable against the 15-file tiny lightsheet
fixture (6ab84b549852a120dbcb22bc, User 1 prod) that it raises
against the 121k-file production dataset, while
getFileDetails(datasetId, manifestUid) returns 200 OK for the
fixture's manifest uid in the same instant. Two server code paths
disagree.
This is the MATLAB-side reproducer. Read-only against the fixture
(no uploads, no cleanup). For each lightsheetZarrLevel document in
the fixture:
* listDatasetDocumentsAll picks the level docs out by className.
* getDocument fetches full properties, from which
files.file_info yields the chunk.bin series' manifest uid
(same shape ndi.cloud.sync.internal.updateFileInfoForRemote
Files reads on the download side).
* createSignedURLSetJob(idNamespace='ndi', fileSeries='chunk.bin')
+ waitForSignedURLSetJob to a terminal state.
* INDEPENDENTLY getFileDetails(datasetId, manifestUid).
Consistency assertion:
* both ok -> healthy (mapSize > 0 sanity check).
* both fail -> consistent, not the bug.
* async job fails but getFileDetails returns 200 -> THE BUG.
Accumulate the disagreement across every level doc so the
final message names each cloudDocId / ndiDocId / manifestUid
the API team can cite.
* async job succeeds but getFileDetails fails -> also fail
(the opposite mismatch is diagnostic on its own).
Rows print as [fixture-probe] lines while the test runs so a CI
log captures the sequence even if a later doc times out.
Lands under ndi.unittest.cloud so testCloudApi() picks it up under
the existing test-cloud-api.yml matrix without a workflow edit.
Skips cleanly without NDI_CLOUD_USERNAME/PASSWORD via the same
assumeNotEmpty idiom the rest of ndi.unittest.cloud uses.
See #1010. Python analog test lives under
NDI-python on the same branch (test_cloud_signed_url_set_job.py).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz
…ng one
The old version hardcoded FixtureDatasetId = "6ab84b549852a120dbcb22bc"
(User 1 prod), so the test couldn't run against dev -- that dataset
id doesn't exist there and listDatasetDocumentsAll would fail with
"dataset not found" instead of exercising the actual bug. Since the
disagreement is GLOBAL (confirmed 2026-09-27 against both the tiny
prod fixture AND the 121k-member production dataset), a probe that
builds its own fixture reproduces it wherever the credentials point.
buildAndUploadTinyFixture now:
* builds an ndi.test.lightsheet.makeBlobFixture (shape [32 32 32],
1 channel, 3 levels, chunk [16 16 16]) -> tiny OME-Zarr with
~5-ish spatial chunks per level;
* ingests into a fresh ndi.session.dir + subject, runs
ndi.fun.doc.lightsheet.fromOMEZarr with materializeChunks=true,
producing several lightsheetZarrLevel documents;
* uploads uploadAsNew + skipMetadataEditorMetadata into whichever
environment CLOUD_API_ENVIRONMENT points at;
* waitForAllBulkUploads before probing;
* registers per-test + class-level cleanup so every uploaded
dataset is deleted, even on abort.
The probe body is unchanged: enumerate lightsheetZarrLevel docs from
the CLOUD side via listDatasetDocumentsAll, extract each level's
manifest UID from files.file_info, call createSignedURLSetJob +
waitForSignedURLSetJob next to an independent getFileDetails, and
fail on any (asyncFailed, getFileDetails=200) row with each offending
(docId, ndiDocId, manifestUid) triplet named.
Test now runs identically on dev and prod, and stops depending on
User 1 prod fixture state that a later cleanup could remove.
See #1010. Python analog:
Waltham-Data-Science/NDI-python test_cloud_signed_url_set_job.py
on the same branch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz
exist('ndi.test.lightsheet.makeBlobFixture', 'file') returned 0 on
Steve's MATLAB even with src/ndi on the path, so both
TestSignedUrlSetJobFixture and TestSignedUrlSetJobScaling filtered
themselves out at TestClassSetup with a bogus "not on the path"
diagnostic and never ran the actual probe.
exist(NAME, 'file') is unreliable for dotted package names -- the
'file' second argument was designed for plain file paths, and its
handling of package resolution differs across MATLAB releases.
which(NAME) is the idiomatic probe: it returns the resolved file
path (or empty) after consulting the same package lookup MATLAB
would use to invoke the function.
Swap both TestClassSetup checks to assumeNotEmpty(which(...)) so
the tests skip only when the fixture builder is genuinely absent.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz
…t blob Two fixes to what Steve saw on a live dev run (fresh upload, five lightsheetZarrLevel docs, every one jobState=ready + manifestFetchable=1): 1. The mapSize>0 sanity check was misfiring on ready jobs because waitForSignedURLSetJob's poll answer carries `resultUrl`, not `fileCount` -- the file count lives inside the gzipped JSON blob the resultUrl points at. probeOneLevelDoc now fetches that blob via ndi.cloud.api.files.getSignedURLSetResult when the job is ready, and reads fileCount off the parsed struct. A ready job with a zero-file blob still fails the sanity check with an honest diagnostic, but the healthy path passes. 2. Diagnostics were bare strings; a failed cloud test message needs to be handable to whoever owns ndi-cloud-node without a MATLAB session. Now every step appends to Narrative and every assertion that touches the cloud wraps its message with ndi.unittest.cloud.APIMessage, matching FileSeriesRoundTripTest's established shape (#945). The Narrative property is the sequence of steps a reader follows; APIMessage bakes it into a pretty-printed JSON report along with the API call's status line, URL, and body. Refactored the probe into two private methods (resolveManifestUid + probeOneLevelDoc) so the top-level test reads as a narrative and each per-doc failure carries the specific step that produced it. Per-doc [fixture-probe] rows still print (they are the CI-log grep target). Row struct now carries jobResultBlob / jobResultResponse / jobResultUrl so the final "createSignedURLSetJob and getFileDetails disagree" message serializes the server's own words (result blob, poll response, whichever is interesting) into the JSON report. See #1010. Python analog: Waltham-Data-Science/NDI-python test_cloud_signed_url_set_job.py on the same branch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J1i76ogUkH8Aa8Y69tiPdz
A working scientist reopening the same lightsheet-scale dataset over a day should pay the ~85 min signed-URL-set job cost once, not every open. New signedUrlDiskCache module writes each scope (datasetId, documentId, seriesName) to ~/.ndi/signed-url-cache/<ds>/<doc>[_<escapedSeries>].json.gz as gzipped JSON, TTL-checked against the server's own filesExpireAt with a 30 min safety buffer, and atomically renamed on write. batchSignedUrlLookup gains a diskCache option (off by default; the DID chunk-download path in didsqlite turns it on). On S3 403 the DID handler forgets the scope, so a rotated token does not cascade to N per-uid 403s. Env-var override NDI_SIGNED_URL_CACHE_DIR points the cache anywhere (tests use it for per-run isolation). On-disk contract mirrors byte-for-byte in NDI-python (VH-Lab/NDI-python#322). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WhfC1rGbqm3nFBwxN3iRoV
Round-trip tests failed because save() was throwing silently inside its
own try/catch — the Java bridge path for GZIPOutputStream.write and
Files.move(ATOMIC_MOVE) with a javaArray of an interface type is not
portable across MATLAB releases / JVMs, so no file ever landed on disk
and every subsequent load() returned []. The five failing tests all
depend on save->load; the four tests that only assert misses passed
incidentally because "no file" is also a miss.
- gzipToFile / gunzipToString: use MATLAB's builtin gzip / gunzip via a
scratch plain file (unicode2native + fwrite). Same on-disk format as
before (single-member gzip of UTF-8 JSON), no Java overloads.
- atomicRename: use movefile('f'), which delegates to rename(2) on POSIX
and MoveFileEx on Windows. Same "no half-written file visible"
invariant, no javaArray-of-interface pitfall.
- chmodUserOnly: prefer filePermissions (R2024a+); keep fileattrib as a
fallback so older MATLABs still work.
- localProcessId helper: prefer matlabProcessID; keep feature('getpid')
as a fallback for the same reason.
Public contract (cachePath, payload struct, TTL/safety-buffer semantics,
forget, cacheDir + env override) is unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hGmZKPmMGAt4KN7wmReRS
testLoginLogout asserted verifyClass on answer_login.user.organizations.id, which expands to a comma-separated list when the login response carries a struct array of organizations. verifyClass then reads the extra values as positional args and MATLAB:narginchk:tooManyInputs fires. This started tripping when the test account was added to a second/third organization; the underlying code (ndi.cloud.authenticate -> extractFirstOrganizationId) already picks the first, so the test mirrors that. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WhfC1rGbqm3nFBwxN3iRoV
Was: one 0.5 s retry per scope when the first signed-URL-set response did not name a requested uid. Now: a bounded schedule of 1 s / 3 s / 9 s. Waltham-Data-Science/ NDI-python#320 shows User 1 prod's signed-URL-set index lagging further than 0.5 s after ``waitForAllBulkUploads`` returns -- one retry is not enough to bridge the settle between the bulk-upload extraction subsystem reporting done and the signed-URL-set subsystem catching up. A missed uid then either flags as uid_misses > 0 (main) or, on the Python side's strict-mode reconstruction path, surfaces earlier as a document rejected at add time. Three waves at 1 s / 3 s / 9 s cover the observed tail, still bounded (each wave costs one scope refetch, not one per uid), and never fire in the common case where the first map is already complete. Schema: - New name-value ``partialMapRetryDelays`` (row vector of waits). Defaults to [1 3 9]. - ``partialMapRetrySeconds`` kept as a deprecated scalar knob (NaN means "use partialMapRetryDelays"; any real number replaces the schedule with a one-element vector containing that wait, so ``partialMapRetrySeconds=0`` still fires ONE retry with no wait, matching the pre-schedule contract every existing test pins). - Persistent RETRIEDSCOPES map now records attempts spent per scope (double) instead of a "has been retried" flag (logical); walks the schedule until a fresh map names the uid or the schedule is exhausted. Round-trip test's cap on partialMapRetries relaxed from 1 to numel(defaultSchedule); every other assertion (uid_misses == 0, signer_calls == 1 + partial_map_retries) is unchanged. Python mirror lands in Waltham-Data-Science/NDI-python on the same branch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
When DID reads a series member, it fetches the manifest first via the same download handler with seriesName="". That call previously went through ``batchSignedUrlLookup`` under the whole-document scope, and for a lightsheet-scale pyramid document (15k+ files) the batch endpoint spent 60-90 s signing URLs the caller had no use for -- just to answer the ONE manifest URL. The subsequent seriesName="chunk.bin" call then fired its own scope, and the two answers overlapped almost completely. Now: when seriesName is empty, the batch is skipped and the fetch takes the direct ``getFileDetails`` path (O(1)), matching what ``ndi.cloud.sync.internal.fetchManifest`` already does for the internal manifest fetch. When DID follows up with a member fetch (seriesName="chunk.bin"), the batch fires there and amortizes across every member exactly as it did before. Python mirror lands in Waltham-Data-Science/NDI-python on the same branch (filehandler.download_file_from_cloud gained the same guard). See #1010 and Waltham-Data-Science/NDI-python#320. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Real lightsheet-scale documents (e.g. the 121,441-URL pyramid level that motivated Waltham-Data-Science/NDI-python#320 and ndi-cloud-node#151) sit an order of magnitude past the largest row this probe used to run (N=10000). That regime is where per-URL signing rate goes sub-linear AND the signed-count plateaus for minutes; neither is visible below ~50k. Adding N=50000 and N=100000 gives us the rows that will prove ndi-cloud-node#151's fix has landed and guard against future regressions. Two guards keep the default CI run from swallowing tens of minutes: - **Huge-N gate.** The 50k / 100k rows upload roughly 50k / 100k files each, which currently takes tens of minutes and burns cloud budget on every CI push while the perf issue is still live. ``assumeTrue(NDI_SCALINGPROBE_HUGE=='1')`` skips them by default; set the env var to run them locally when investigating the perf issue. Retire the gate once ndi-cloud-node#151 is fixed and the upload path is fast enough to afford these rows on every push. - **Size-adaptive job timeout.** The old 300 s ceiling was fine for the small rows but would swallow every ready-at-60-90 s finish the big rows will produce once the fix lands: the wait would time out before reporting the actual wall time. Now ``max(300, ceil(N / 100)) s`` -- 300 s for the small rows, 500 s at 50k, 1000 s at 100k -- a generous ceiling costs nothing on the happy path (the wait returns as soon as the job is ready) and lets the row report the real elapsed rather than "timeout". Also expanded the doc comment on ``targetMemberCount`` to name the three size regimes (plumbing / rate-matters / >100k) and the issues each row is regression-guarding. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Was: the test built a spatial shape via ``pickShapeForTarget(N, [16 16 16])`` that would contain ~N chunks if the fixture wrote at [16 16 16], but its call to ``ndi.fun.doc.lightsheet.fromOMEZarr`` did not forward a ``chunks`` argument. ``makePyramid`` therefore fell through to ``chooseTileShape``'s default ~8 MB per-tile byte budget, which for uint16 lands at roughly [161 161 161] per chunk. Concrete outcome: requested N actual members ------------ -------------- 5 1 100 1 500 1 2000 8 10000 27 50000 64 100000 125 Every row degenerated into a "~125-member series" test, and the scaling axis this class exists to sweep was silently collapsed to a constant. The rows still exercised the bug tracked in #1010 (createSignedURLSetJob rejects a manifest that getFileDetails accepts), but the N label carried no information about scope size -- it only labelled cloud budget spent on a fixture that reproduces the small-N case seven times over. Fix: forward ``'chunks', chunkShape`` to fromOMEZarr so makePyramid uses the chunk shape the fixture wrote (which is the same shape pickShapeForTarget was already sizing the level to). Also add a regression-guard ``warning()`` that fires when the actual member count is <25% of the target for N>=100, so if this ever silently regresses again the run log makes the failure loud rather than legible only by squinting at the [scaleprobe] rows. Companion to Waltham-Data-Science/NDI-python#320 and ndi-cloud-node#151. Once the server-side signing behaviour stabilises, the huge-N rows will actually produce huge-N scopes and the performance-regime evidence they were added to capture becomes meaningful. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Uncovered by CI on head b0dcce1 (Cloud CI 2): the fixture-count fix made N=10000 actually upload 10,165 files, and the default 300 s ceiling on ``waitForAllBulkUploads`` was too tight for the server to finish extracting them -- the assertion at line 275 fired with ``state=timeout, elapsed=300.3s`` and the row never reached the createSignedURLSetJob step it was there to observe. That's a false-negative on the bug this class exists to detect: nothing was learned about the async job path at N=10000 that run. Mirrors the size-adaptive timeout already applied to waitForSignedURLSetJob: ``max(300, ceil(N / 20)) s`` -- 300 s for N=500/2000, 500 s at N=10000, 2500 s at N=50000, 5000 s at N=100000. Rate anchored to a rough "20 files/s extract" observation: at N=500, 513 files took ~9 s (well inside 300 s); at N=10000, 10165 files were still going at 300 s, and 20 files/s would predict ~500 s to finish (well inside 500 s). Includes ``timeout=Ns`` in the assertion diagnostic so a future timeout row prints the ceiling that was applied. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Cloud CI 2 on head e3d5f51 failed with two waitForAllBulkUploads timeouts: * N=2000 hit 300.3 s / 300 s ceiling * N=10000 hit 500.4 s / 500 s ceiling The same N=2000 row finished in 22 s on b0dcce1 and again in earlier runs at ~10 s, so the extract time swings by more than an order of magnitude with server load. Chasing that with an ever-larger ceiling would still red the PR whenever the server is warm. The probe's own philosophy, stated at line 369 of the failed-job branch, is that a job outcome IS the observation the run is here to make -- the probe does not fail on things the server does; it records them. Extend the same treatment to the bulk-extract wait: if waitForAllBulkUploads times out, record a row with state='bulkTimeout' and errorMessage naming the elapsed time and ceiling, then return from the parametric case so the next N runs. The row still surfaces the observation for the bug report; CI stays green when the extract is slow. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Replace the fixed pause(20) + single-retry pause(10) with a proper
per-file poll of getFileDetails, gating each verification on the file
document actually flipping uploaded=true (which is when the server
returns a downloadUrl).
The old sequence -- upload files serially, pause 20 s, then start
verifying -- collapses when the backend is under load. The file goes
to S3 (client-side PUT succeeds) but the server hasn't committed the
file document yet, so getFileDetails returns HTTP 200 with
{uploaded: false, size: 0, no downloadUrl}. The wrapper treats that
as b=false. The old getFileDetailsWithRetry then paused 10 s and
tried once more; the total 30 s budget was still too short on a
thrashed dev-api, and the test failed at line 178 with the
"missing downloadUrl" diagnostic.
The new getFileDetailsWithRetry polls every 2 s up to 300 s until
b=true or timeout, matching the order of magnitude the sibling
waitForAllBulkUploads uses for the bulk-upload path. Since the
polling helper is called downstream in the per-file verification
loop, the upfront pause(20) is redundant and is removed from both
testSerialUploadPublishDownload and testMixedUploadPublishDownload;
a narrative line names what is now doing the waiting so a failing
run's log still explains the wait shape. Callers keep the same
getFileDetailsWithRetry signature.
testBulkUploadPublishDownload already used the proper
'waitForCompletion', true gate on the bulk-upload path and is
untouched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Records the decisions made so far for the V2 importer (plates and patches are subjects; excluded data is tagged; the tableOfContents row 0014 typo), the agreed principles and stage order, the open questions, and the source inventory taken from the file list and column profiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…g, videos as epochs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…proposal; mp4 via NDR imagestack Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…a PR #78) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…et licence, instrument product edge Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…tance_of built Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…4.0; strain product relation deferred Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…, dataset-level entities Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
- import_V2.m: the stage orchestrator. It runs discover and metadata and returns what each built. Nothing is written yet; doImport.m is unchanged. - import_V2_spec.json: the dataset-level metadata, taken from the eLife paper (key resources, methods, funding, author contributions, ORCIDs, data availability) and the NDI Cloud licence: - 23 organizations, 4 people, 5 awards, 1 publication; - 2 web resources, 5 software, 8 products, 13 strains, 4 instruments; - 7 studies and the dataset. Entries that need checking carry `_verify`. - ndi.setup.V2.discover: stage 0. It lists source files, skips earlier import output (.ndi, .ndi_, archives, OS clutter) and prints its denominator. - ndi.setup.V2.profileTable: describes a table's columns without interpreting them. - ndi.setup.V2.datasetMetadata: turns a spec into V2 documents through did2.build and resolves keys to ids. It emits the relations (suborganization_of, affiliated_with, issued_by, awarded_to, hosted_by, instance_of, has_author with sequence and roles, funded_by, cites, stored_at, documented_by, part_of). It REPORTS spec content with no V2 home instead of dropping it (today: software vendor). - TestDatasetMetadataHaley + test-import-v2.yml: 133 documents (86 entities, 47 relations), every edge resolvable, authors ordered with roles, OP50-GFP -> OP50 lineage. The job fails if tests were filtered. It runs against did-schema PR #78's branch until that merges. UNVERIFIED: no MATLAB in the authoring environment; test-import-v2.yml is the first execution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…e renames them x_) CI run 36749331306: 5 of 6 tests passed. The sixth found that jsondecode turns `_verify` into `x_verify`, so the maker reported the spec's own reviewer notes as unrepresented content. Both spellings are now skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…instance_of/awarded_to/roles signed and merged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…r study per day / dataset-wide) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…tabase ndi.session.dir can open a V2 session (V_eta.sqlite is database-hierarchy entry 2) but can only create a legacy one: only entry 1 has creation code. createSession fills that gap without changing ndi.session.dir: 1. it writes .ndi/V_eta.sqlite holding the V2 `session` document (built with did2.build) and any `part_of` study relations; 2. it then opens the directory the ordinary way, and checks that the id it reads back is the one it wrote. It refuses to overwrite an existing database of either kind unless told to. TestCreateSession covers the round trip (create, reopen, same id and reference, via the did2 backend), the study relation stored in the session, and the overwrite guard. UNVERIFIED: no MATLAB in the authoring environment; test-import-v2.yml is the first execution. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
ndi.session.dir scopes every search to its session id. On a one-argument open it takes that id from .ndi/unique_reference.txt, or makes up a provisional one, which found nothing (CI run 36752118828: 'Could not load the REFERENCE field'). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…sessions, naming, createSession Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…79) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
concentration_0015's 12 assay plates share one acclimation plate and agree on everything but its seeding time: 12:39 on six, 12:41 on six. When assay plates disagree only on the seeding time, the acclimation plate is now seeded at the span's midpoint with its tolerance widened by half the span (reported once as growthSeedSpread), instead of taking the first plate's time and reporting a disagreement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
The OD600 cross-check compared the map's values in detection order with the template list in template order, and flagged 14 Matching plates that hold exactly the same densities (neighbouring patches swapped). It now compares the sorted densities. The map stays the source, as in the analysis (analyzeEncounters.m:254). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
The acclimation plates of foragingSensory experiment 2 left the cold room on 5 Jan 2024 09:49 by the source, after their worms were picked onto them on 4 Jan. A named correction sets 4 Jan 09:49, as #70e does for Mini experiment 24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
analyzeGFP_24-04-04.mat is 119 GB and lives on a Google shared drive. ndi.setup.conv.haley.extractLawnProfiles reads it one image at a time through the existing HDF5 reader (never touching the ~170 GB of images in lawnProfiles) and writes only the curves, their distances and backgroundInfo, which the import reads through 'EcoliProfilesFile'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
The 2,061 metaData images with no TIFF include none of the 1,575 analysed images or their 89 backgrounds: they are .czi acquisitions never converted. They stay out, and recordingList's report now says how many missing images were analysed or used as a background. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
34 analysed images detected fewer than 12 patches, so their 267 per-patch values were left out. ndi.setup.conv.haley.matchGridPatches fits the staggered 3 x 4 grid (measured on plate 124) to the detected centres and keeps a placement only when it is the only consistent one; ecoliDocuments uses it when the full-image numbering cannot apply. showPatchMatch draws an image's mask with the numbers given, for checking by eye. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
#80) extractTemplates copies analyzeGFP's per-brightfield-image acetate registration (centre [y x], angle, scale; not the masks) into the small extract. patchLayout converts every full 12-patch image's patch centres into the acetate's frame (mm) and reports the median position of each patch, its spread, the outlier images and a spec line. The angle's sign convention is chosen by the data. The import's full-image numbering moves to private/matchFullImage so both use one rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…plate (#80) The grid fit mislabelled an image with only two rows (Jess, image 13), so it is replaced by the analysis's own registration: each fluorescence image's brightfield image has an acetate template (centre, angle, scale), and patchLayout learned the 12 patch positions in that frame from every full image (median deviation 0.3 mm, closest positions 5.97 mm apart). The layout is the spec's ecoli_patch_layout; matchTemplatePatches gives each detected patch the nearest position within 2 mm, and refuses when two claim one. showPatchMatch draws the template positions too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
On 2023-11-09 foragingMutants seeded acclimation plates only, so no OD600Real or CFU was read and their seeding dose was left out. Such a day now gets one formulation per nominal OD600 (the strain at that nominal OD600 in the growth diluent, no stock, no CFU), and the dose's notes say it was not measured. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…ing (#83) On concentration_0016, plates 161 and 162 are the only plates whose lawn clip and video are on different cameras; the source is kept and each of their recordings carries a note (recordingList reports such plates as cameraSwap, and a recording row's note now reaches its statement). An unnamed file on a day whose worms are all on listed plates is reported as not a missing plate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…open - A term_assertion declares no `notes` (only an interaction does), so decision #73's exclusion reason and "not filmed" note are now text statements beside their tags (`reason for <variable>`). - ndi.dataset/open_session ran mksqlite('close') after opening a session, closing every connection. The legacy backend reopens per call; a V2 database keeps its connection, so an ingested session opened through its dataset could not be searched ("database not open", testDataset since decision #71 made dataset runs ingest). It now closes only for legacy datasets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…73) did-schema now carries `notes` on every statement (moved up from interaction, 2026-10-14), so the exclusion reason and the "not filmed" note go on their tags again. With a schema whose statements have no notes (did-schema main today), they stay text statements beside the tag, so both CI legs pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
did-schema main has neither statement notes nor the `text` value kind,
so the exclusion reason's fallback (a text statement) failed there
("No schema file for class text", the current CI leg on 206a4a5).
With neither, the reason is now reported in `skipped` instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
On the 'current' leg (did-schema main) a statement block is still subject_statement; the other tests already pass documents through newNames. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
did-schema 877bbde makes data_body.format a bound ontology term (Jess, 2026-10-14, lightsheet point 4). Recordings MPEG-4 / TIFF, the E. coli images TIFF, ingested masks and maps binary. fileFormat writes the media type on a schema copy whose format is still a string (the CI current leg); schemaHasField now also returns the field's declaration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
DID-matlab's lightsheet migration (claude/ecstatic-pascal-md91fw, 000aeb8)
turns a v1 lightsheetZarrPyramid into an intensity `observation` (id kept)
and each lightsheetZarrLevel that held bytes into a `sampled_body` (id
kept, owner_id = the observation), with the chunk series renamed
chunk.bin_k -> body_data_{k-1}.
ndi.vintage gains the lightsheet reads, by v1 name, for both vintages:
lightsheetPyramidDocs / lightsheetLevelDocs (find them), lightsheetLevel /
lightsheetLevels / lightsheetPyramid / lightsheetKeys (read them; a v1
document is returned as its own block, untouched), lightsheetChunkName and
seriesMemberBase (a series' first member is NAME_1 on v1, NAME_0 on V_eta;
did.document/seriesMemberBase when DID has it, else the schema vintage).
Routed through it: levelTable, makePyramid's reduce-level chunk reads,
LightsheetZarrManager (rows, reductions, deletion plan, subject names),
and the series member walks in extract_docs_files and list_binary_files.
No ndi.vintage.map row: observation/sampled_body are shared classes.
Tests: TestLightsheetVintage (hand-built, no database),
TestLightsheetMigratedSession (a V2 session holding a migrated pyramid),
TestLightsheetMigrated (the real migrator's output; next leg only), and
the existing v1 lightsheet tests now also run in test-import-v2.yml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
did-schema 9bedceb adds developed_by (software -> organization) and hosted_by (dataset -> organization), Jess 2026-10-15. Stage 2 writes them on a schema that has them (new ndi.setup.V2.schemaHasRelation) and reports the spec facts as before on one that does not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…n superclasses; which() for package functions CI run 38674988309: did.document refuses a series not in file_list, the did2 database refuses a class repeated among its own superclasses, and exist(..., 'file') does not find a package function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…value, source_value} CI run 38676292511 (next leg): the real migrator's keys carry spacing and origin as pairs, not as flat source_spacing/source_origin fields. Read both; the hand-built fixtures now use the real shape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
did-schema 033a22d adds distributed_as (strain -> product), Jess 2026-10-15. entityDocuments wrote a merged-schema strain's product_id as derived_from, saying the strain was made from its listing; it now writes distributed_as, and derived_from only on a schema without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
On a schema without hosted_by (the CI current leg) storedHosts is empty, and
for-over-unique({}) still ran once on its empty column, indexing past it
(TestDatasetMetadataHaley setup, MATLAB:badsubscript at the relate call).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
Keep the original subject-block read for v1; ndi.v2.blockOf only for a subject with no subject block (the one-entity-class schema). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
ndi.query('', 'isa', 'subject') drops subjects of non-biological types
with '~exact_string' terms on subject.type.name. The v1 database
searches in SQL, where a negated term still needs the field to exist
(did.database: field_check AND NOT value = ...), so every v1 subject,
none of which has a subject.type, was dropped too. A v1 subject always
was a biological subject, so didsqlite puts back the plain search,
wherever the rewritten term sits in the query. V2 sessions are
did2sqlite and keep the rewrite.
This is what failed TestLightsheetZarrManager's two subject-name tests.
TestV1SubjectSearch covers the bare search, an AND and an OR.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
Two decisions of Jess, 2026-10-15 (DID-matlab f363d98 builds the migrator side): 1. A migrated lightsheet pyramid's observation carries method FBbi:00000369 'selective plane illumination fluorescence microscopy'. ndi.vintage.lightsheetPyramidDocs now finds migrated pyramids as observations with interaction.method.node == that CURIE, one query (ndi.v2.fieldQuery, so subject_interaction is asked too); v1 pyramids are found as before (isa lightsheetZarrPyramid). The term lives in the new ndi.vintage.lightsheetMethod, which replaces ndi.vintage.lightsheetBuilderName (deleted: nothing else used it; the fixture names the software entity literally). The builder software entity is still written; it no longer marks the pyramid. 2. ndi.vintage.lightsheetPyramid reads the label from the observation's statement.notes (the part before the migrator's separator, when the batch pass did not run), pyramid_name from the store body description as before, and pyramid_type is always '' (dropped). Tests: TestLightsheetMigrated pins the method against the real migrator's output (replacing the builder-name check), the label after the pass, and the reader's separator against a no-pass migration; TestLightsheetMigratedSession adds a decoy observation of the builder software with no method, which must not be listed; TestLightsheetVintage reads the label from notes, with and without the separator. Fixtures carry the method and the label on notes, and a description without label or type. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…lation 0 Jess, 2026-10-15 (DID-matlab jLightsheetKeys): a space or time axis the source gave no scale is now migrated as regular, origin 0, spacing 1, no unit, instead of `values` 0..n-1. ndi.vintage.lightsheetKeys already reads it as voxel size 1, translation 0, unit ''; its header now says so, and TestLightsheetVintage checks both forms read the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
…rded video A plate video that is not a recording (mutants_assayPlate0001's video 1, a failed start with no file, reported at stage 5 as rowWithoutFile) made every worm on the plate lose all its encounters: the check asked whether ALL of the plate's videos were recordings. It now asks whether the worm's TRACK rows are all on recorded videos, and uses only those videos for the plate's start, frame rate and window. A plate with no recorded video at all is still skipped. In the 2026-10-14 run this skipped mutants_worm0001-0004. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
reconstructSeriesIngestLocations rebuilt ingest_locations with index =
manifest slot (1-based), i.e. the v1 numbering. DID names an ingested
member sprintf('%s_%d', name, index) (sqlitedb/do_add_doc) and
addFileSeries writes index as the member NUMBER in the document's base,
so for a V_eta document (base 0, DID-schema #73 item 17, DID-matlab
1a6bd9f) every rebuilt entry named the next member up.
The helper takes 'MemberBase' (default 1, v1 unchanged); both callers
pass ndi.vintage.seriesMemberBase(document). Tests add the V_eta case
(members 0,2,4 for slots 1,3,5, same uids) beside an explicit v1 case,
and test-import-v2.yml now runs ReconstructSeriesIngestLocationsTest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things, now in one PR into
V2(this replaces #1011, whose branchclaude/haley-import-v2this one contains in full):ndi.setup.conv.haley.import_V2;Both work on did-schema
main's shape and on one entity class with statements by composition (did-schema#88).The Haley V2 importer
ndi.setup.conv.haley.import_V2imports the Haley C. elegans / E. coli foraging dataset as V2 (V_eta). It was built stage by stage with the dataset's owner and is meant to be the canonical example of a V2 import.doImport.m(the v1 import) is unchanged.import_V2_spec.jsonand the seeding suspensions;+haley/import_V2_decisions.md, with its reason: decisions Week of Jan 25 #1-fixes addToStructArray method in nsd_variable_struct #70, including today's comparison withdoImport, column by column.ndi.setup.conv.haley.import_V2(dataParentDir, 'Write', true, 'ReadVideos', true). Nothing is written without'Write'.The object layer
It began as a design document (
src/ndi/docs/NDI-matlab/manual/developer/V2_Object_Layer.md) and now carries the build.Decisions recorded (agreed with Jess Haley, 2026-10-06):
did2.buildfor now;value(), notdata();@inlocal_identifierstays a v1 rule and is not required for V2.What it does:
ndi.entity,ndi.subjectunder it, and one statement tree that mirrors the schema:ndi.statement,ndi.assertion,ndi.interaction,ndi.observation,ndi.manipulation,ndi.calculation. A statement'svalue()is decoded from its data body.ndi.subject.searchandndi.statement.search, by what is true of a subject: asserted properties, the six statement kinds, relations, time filters (at,during,before,after,duration), wildcards, andexplain.member_of(distributive statements) andpart_of/sample_of(assertions);contained_inas context (tenet T17);parents,children,ancestors,descendants.Both schemas (did-schema#88, signed 2026-10-08)
The import and the readers work on main's schema and on one entity class with statements by composition.
ndi.setup.V2.entityDocumentswrites an entity in either shape.ndi.setup.V2.curiewrites CURIE identifiers.instance_ofits strain.ndi.v2.isaQuery,blockOf,blockQuery,leafNameandkindOffind a document under either shape.ndi.v2.classChainincludes the value kind.ndi.session/database_search_with_dataset). A session folder opened on its own holds only the relation and cannot name the strain; the test pins that.test-import-v2runs two legs:Linked and ingested sessions (did-schema#88, signed 2026-10-09)
V_eta_linked_session_plan.md: a session is in a dataset when itssessionentity ispart_ofthe dataset, orpart_ofa study that is. An ingested session's documents are in the dataset's database. A linked one's are in its own folder, which alinked_sessiondocument (path,entity_id) names. On a V2 dataset (a did2 database with nosession_in_a_dataset):ndi.v2.datasetSessionslists the members, denominator first, and says what it left out and why (session_notes(), warningndi:dataset:sessionsLeftOut). A linked folder is opened and checked to hold its session.session_list,open_sessionanddatabase_searchwork for both kinds.add_linked_sessionandadd_ingested_sessiontake'PartOf'(a study; default the dataset).convertLinkedSessionToIngested, and the newconvertIngestedSessionToLinked(id, folder).unlink_sessionremoves the link and thepart_ofrelations; the folder stays.makeSelfContained('DryRun', tf)copies every by-location file into the dataset's store.ndi.v2.copyDocuments,ndi.v2.sessionDocuments).ndi.dataset.dir's constructor reads its own database for its lookups.v1 datasets are unchanged.
Checks
test-import-v2, run 37873856395 (179fd779): both legs pass, 111 tests each, 0 failed, 0 filtered. All 7 newTestLinkedSessionstests pass onnext. Oncurrent, whose schema has nolinked_sessionclass, the 4 that link return early.The full suite (
run-tests.yml, dispatched on this branch, run 37874409826) reports 1805 passed, 90 failed. None of the 90 is anndi.unittest.datasettest.DID_SCHEMA_PATH does not hold a V2 schema with study).OneEpochTest's fixture (Unrecognized function 'probemap') anddiffTest(ademoNDIfile-validation error in DID).🤖 Generated with Claude Code
https://claude.ai/code/session_01FxKLA8Tbf9cpvDe7fuVeKf
Generated by Claude Code