fix(storage): allow mounted files beyond the callback payload limit - #1879
Open
joshuajbouw wants to merge 3 commits into
Open
fix(storage): allow mounted files beyond the callback payload limit#1879joshuajbouw wants to merge 3 commits into
joshuajbouw wants to merge 3 commits into
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Near-i64::MAX lengths can trigger effectively unbounded materialization under a global lock, and administrative projection still allocates the entire file.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Fixes mounted writes and resizing beyond the 4 MiB callback payload limit through streamed, atomic content replacement.
Changes:
- Streams range replacement, truncation, extension, and zero-filled gaps.
- Adds streaming publication across filesystem and workspace adapters.
- Adds regression, quota, and failure-path coverage.
File summaries
| File | Description |
|---|---|
crates/astrid-storage/src/content/store/workspace/tests.rs |
Tests streaming, quota refusal, and atomic failures. |
crates/astrid-storage/src/content/store/workspace/operations.rs |
Adds quota-aware streamed branch publication. |
crates/astrid-storage/src/content/store/workspace/filesystem.rs |
Exposes streaming writes through workspace filesystems. |
crates/astrid-kernel/src/storage_mount/tests.rs |
Updates native-offset and truncation tests. |
crates/astrid-kernel/src/storage_mount/filesystem/range.rs |
Implements streamed range replacement. |
crates/astrid-kernel/src/storage_mount/filesystem.rs |
Integrates streaming and native offset bounds. |
changes/1878.fixed.md |
Documents the mounted-file fix. |
Review details
Suppressed comments (2)
crates/astrid-kernel/src/storage_mount/filesystem.rs:387
- The same unbounded-gap problem applies to random writes: a one-byte payload at an offset just below
i64::MAXpasses this check, after whichReplacementemits the entire zero-filled gap while the global mutation lock is held. Since the default storage quota is effectively this same value, the 4 MiB payload bound no longer prevents a trivial disk-exhaustion/global-stall request. Add a practical resulting-length/materialization bound or sparse-gap support before accepting such offsets.
if current_length.max(end_offset) > i64::MAX as u64 {
crates/astrid-kernel/src/storage_mount/filesystem.rs:334
- For system-owner mounts this is not end-to-end streaming:
admin_projection::executecallsproject_fileafterexecute_blocking, andproject_filereads0..lengthinto oneVecbefore writing the host file (admin_projection.rs:112-135). Large writes/resizes can therefore exhaust memory after the authoritative content commit, returning an error while the live host projection still contains the old file. Stream ranges directly into the temporary projection file (and preserve publication/projection consistency) rather than materializing the whole value.
range::replace(filesystem, &path, current_length, length, length, &[])?;
- Files reviewed: 7/7 changed files
- Comments generated: 1
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
5 tasks
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
Signed-off-by: Joshua J. Bouw <[email protected]>
joshuajbouw
force-pushed
the
fix/mount-large-file-io
branch
from
September 8, 2026 10:59
9661cba to
38ed4bf
Compare
This was referenced Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked Issue
Closes #1878
Summary
Fix mounted writes and resizing beyond 4 MiB. The shared kernel callback had applied the per-request payload ceiling to the resulting whole-file length, making a real FSKit fsync fail with EINVAL at 4 MiB+1 byte.
Changes
Verification
cargo test --locked -p astrid-kernel --lib storage_mount -- --test-threads=1: 24 PASS (original candidate).cargo test --locked -p astrid-storage --lib content::store::workspace -- --test-threads=1: 24 PASS.cargo clippy --locked -p astrid-kernel -p astrid-storage --all-targets -- -D warnings: PASS.cargo fmt --all --checkandgit diff --check: PASS.Claim limits: this is correctness, not a random-write performance optimization. Replacement still rebuilds the content graph; workspace deferred records remain retained until publication. On the final 9661cba source build, mounted 32 MiB fsync took 3.63–3.93 seconds locally; all three samples and the remount regression passed. Warm OS-cache reads are not backend throughput. Linux/Windows use the shared kernel adapter, but this PR does not claim a new packaged native execution on those hosts.
Latest repair verification:
cargo test -p astrid-kernel storage_mount --lib -- --quiet: 25 PASS; focused range tests: 2 PASS;cargo clippy -p astrid-kernel --lib --tests -- -D warnings, formatting and diff checks PASS. The new regression rejects both SetLength and Write gaps beyond budget, including i64::MAX, with original bytes unchanged. Exact-budget extension and shrinking remain valid. Earlier mounted measurements are retained as earlier-candidate evidence, not a rerun of this repair. CI workflow repair #1891 is included as a prerequisite until it lands.Test Plan
Run the commands above. On an admitted writable mount, write more than 4 MiB, fsync, patch across the boundary, extend/shrink, sync and remount, then compare complete file bytes. Verify an oversized individual RPC is still rejected and a quota/source failure preserves the prior file.
AI / Tool Assistance
Assisted-by: Codex:GPT-6-Astra
Implemented and reviewed the adapter changes, source/publication ordering, integer bounds, quota preservation and regression coverage; exercised the real FSKit mount in a disposable AOS home.
Checklist
changes/1878.fixed.mdSigned-off-bytrailer.