Upload binary files in zip batches (mirror NDI-matlab#1033)

[stevevanhooser]

Summary

Companion to VH-Lab/NDI-matlab#1033. Brings the Python cloud upload path in line with MATLAB's: binary files now go up in zip batches whose size comes from a preference, and the cloud batch-size preferences actually set the defaults they were registered for.

Batched binary upload (new)

• uploadFilesForDatasetDocuments now honours file_upload_strategy. 'batch' (the default, as in MATLAB) packs files greedily, in order, into zips of at most file_batch_size_limit bytes and sends each through the bulk endpoint (getFileCollectionUploadURL → putFiles). A single file larger than the limit goes up alone. 'serial' keeps the old one-signed-URL-per-file path. Previously the strategy was read by nothing and every file went up singly.
• ndi.cloud.upload.zipForUpload is replaced by a port of MATLAB's binary batcher. The old function zipped document JSON and had no caller in src (already noted in the bridge file).
• Deliberate difference from MATLAB: the batcher returns which documents' batch failed, including a batch whose server-side extraction job didn't reach complete, so sync keeps those documents out of the index (NDI-matlab#805). It waits on extraction jobs only after every batch is sent, so extraction overlaps the remaining uploads.
• SyncOptions gains file_batch_size_limit (None = use the preference), passed through _upload_binaries by every sync mode. uploadDataset gains file_upload_strategy and file_batch_size_limit.

Preferences now control batch sizes

• Cloud.Upload.Max_File_Batch_Size: default 250 MB (was 500 MB); now the zip batch size.
• Cloud.Upload.Max_Document_Batch_Count: default for uploadDocumentCollection(max_chunk=None).
• Cloud.Download.Max_Document_Batch_Count (10000): default for downloadDocumentCollection(chunk_size=None), which was 2000. Behaviour change: larger download chunks, matching MATLAB.

Fixed in passing

• uploadSingleFile(use_bulk_upload=True) passed the whole {"url", "jobId"} dict to putFiles as the URL, so that path could never succeed.

Bridge / docs

• Bridge entries for zipForUpload, uploadFilesForDatasetDocuments, uploadDocumentCollection, downloadDocumentCollection, uploadDataset, SyncOptions, uploadNew, mirrorToRemote, twoWaySync and preferences are updated to NDI-matlab 7180cc8 (the #1033 merge). FileBatchSizeLimit is added to the property allowlist. CHANGELOG updated.

INTERFACE UPDATE: the bridge contract for zipForUpload now reflects the MATLAB source (binary file batching), replacing the old document-JSON signature.

Tests

• New tests/test_cloud_upload_file_batches.py (30 tests, mocked server): packing and the oversize-file case, zip contents named by uid, temp-zip cleanup, per-batch failure reporting (URL, PUT, extraction failed or timed out, file missing), retries, sends-before-waits ordering, the strategy switch, SyncOptions pass-through, and the three preference defaults.
• Existing tests that exercise the per-file mocks now request file_upload_strategy="serial"; the old JSON-zip tests are removed.
• Locally: black/ruff clean; the affected files pass (229 passed, 15 skipped); the bridge status/properties/hash tests pass against NDI-matlab main. test_cloud_profile_secret_field_names.py fails the same way on unmodified main locally (environment, not this change).
• Not run locally: the live cloud tests. test_cloud_file_series_round_trip.py uploads a real dataset through uploadDataset, so it's the first end-to-end run of the batched path.

Follow-up: concurrent batch uploads in both languages are tracked in VH-Lab/NDI-matlab#1034.

---

Generated by Claude Code