[feature](be) Add asynchronous file cache writes by bobhan1 · Pull Request #65658 · apache/doris

bobhan1 · 2026-07-15T10:43:48Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

On a file-cache miss, CachedRemoteFileReader currently performs two different kinds of work on the query thread:

read bytes from remote storage to satisfy the query;
append and finalize those bytes into the local file cache for future queries.

The first operation is on the critical path. The second is best-effort cache population, but local filesystem latency and backpressure are still charged to foreground scan latency because the two operations are coupled.

This PR implements phase 1 of asynchronous file-cache writes. For ordinary reads, the query thread returns after the caller's buffer is complete and hands eligible cache-miss blocks to per-disk background workers. The feature is opt-in and disabled by default. Explicit cache-population operations, including warm-up and prefetch/download paths, retain synchronous completion semantics.

Simply moving append() to another thread is unsafe. The design also has to preserve cache-block ownership, avoid duplicate queued writes, survive clear/remove races, bound background work, and keep existing warm-up and dry-run behavior. This PR introduces dedicated components for those responsibilities instead of transferring existing FileBlock objects across threads.

Goals

Remove best-effort local cache persistence from the ordinary remote-read critical path.
Reuse remote bytes that have already been fetched but are still waiting for local persistence.
Avoid duplicate background writes for the same cache block.
Keep queued work bounded and make overload fall back to a cache miss rather than block a query.
Prevent stale tasks from recreating cache data after clear/remove operations.
Preserve the existing synchronous behavior for explicit cache-population callers.
Provide query-profile and per-cache-disk observability for the new path.

Non-goals of this phase

This is not remote-read singleflight. Two cold readers that overlap before either publishes its completed remote buffer can still issue separate remote reads.
This does not change local file-cache durability guarantees; persistence remains best effort.
This does not make warm-up or prefetch completion asynchronous.
This does not enable the feature by default.

Architecture

The query thread owns read planning and result assembly. It performs one batch inflight lookup and one read-only probe for the complete aligned request, then represents each aligned block with a small source classification. Background workers own cache-cell acquisition and persistence. The two sides exchange immutable, reference-counted buffers through an inflight index and a bounded per-disk service.

flowchart LR
    Caller[Scan / query caller]
    Reader[CachedRemoteFileReader]
    Remote[Remote FileReader]

    subgraph CacheDisk[One file-cache disk]
        Probe[BlockFileCache read-only probe]
        Index[InflightWriteBufferIndex]
        Service[AsyncCacheWriteService]
        Queue[Bounded MPMC queue]
        Workers[Resizable write workers]
        Storage[BlockFileCache and local storage]
    end

    Metrics[Runtime profile and per-disk bvars]

    Caller -->|read request| Reader
    Reader -->|lookup completed remote buffers| Index
    Reader -->|observe cache state without ownership| Probe
    Reader -->|one remaining middle-range read| Remote
    Remote -->|aligned bytes| Reader
    Reader -->|complete caller buffer| Caller
    Reader -->|publish true MISS buffers| Index
    Reader -->|non-blocking submit| Service
    Service --> Queue
    Queue --> Workers
    Workers -->|revalidate, acquire, append, finalize| Storage
    Workers -->|conditional cleanup| Index
    Storage -.->|clear/remove advances write epoch| Service
    Reader -.-> Metrics
    Index -.-> Metrics
    Service -.-> Metrics

Component responsibilities

Component	Responsibility	Important guarantee
`CachedRemoteFileReader`	Resolves synchronous versus asynchronous mode for every read, performs one inflight lookup and one whole-range cache probe, classifies aligned blocks, performs the required remote read, fills the caller buffer, and submits only true misses.	The caller never observes an incomplete buffer. Cache-write submission is not allowed to turn a successful remote read into a query failure.
`BlockFileCache::probe`	Reports downloaded, downloading, empty, deleting, and missing ranges without creating cache cells or taking downloader ownership.	Planning does not mutate cache state or transfer thread-affine `FileBlock` downloader ownership. LRU touch happens only after a block is actually consumed.
`InflightWriteBufferIndex`	Stores shard-protected references to completed remote buffers that are pending persistence, keyed by cache key and block offset and tagged with a write epoch.	Insert-if-absent selects one background-write owner. Conditional removal prevents an old completion callback from removing a newer entry.
`AsyncCacheWriteService`	Owns the per-disk bounded MPMC queue, pending count, tracked-buffer accounting, resizable workers, task-age checks, shutdown, and service-level metrics.	Submission is non-blocking. Backpressure rejects work and rolls back its inflight entry instead of blocking the query thread.
Async write worker	Rechecks task age and epoch, calls the normal cache admission/cell-creation path, skips blocks already downloaded or owned by another downloader, and appends/finalizes only eligible empty blocks.	Background work follows current cache state; it does not blindly overwrite or resurrect blocks.
Cache write epoch	Represents the generation of work accepted by a cache instance. Clear/remove paths advance it.	A task accepted under an older generation is dropped before or during persistence.
Profile and bvar integration	Records query-local submission/rejection/inflight reuse and per-disk queue, memory, latency, skip, failure, and stale-task information.	Operators can distinguish remote-read latency from asynchronous cache-write health and pressure.

Important flows

1. Ordinary read with mixed coverage

Each aligned block is classified in this order:

completed remote buffers in InflightWriteBufferIndex;
existing cache blocks through the read-only probe;
remaining blocks as misses.

The lookup and probe each cover the complete aligned request; the planner does not build or repeatedly probe individual uncovered runs. It identifies the first and last real cache miss, materializes only inflight/cache blocks outside those boundaries, and retains the existing wait behavior for a DOWNLOADING block outside the remote span. If misses remain, everything from the first through the last miss is fetched with one aligned remote read. Existing cache, inflight, or downloading blocks inside that span are intentionally covered by the same remote result instead of being filled or waited one by one. Only the blocks that were real misses are published and submitted for asynchronous persistence.

sequenceDiagram
    participant C as Caller
    participant R as CachedRemoteFileReader
    participant I as Inflight index
    participant F as BlockFileCache
    participant O as Remote storage
    participant S as Async write service
    participant W as Worker

    C->>R: read_at(offset, size)
    R->>I: lookup aligned blocks for current epoch
    I-->>R: reusable pending buffers
    R->>F: one read-only probe for the aligned request
    F-->>R: downloaded / downloading / miss coverage
    R->>R: materialize blocks outside the first-to-last miss span

    alt all requested bytes are covered
        R-->>C: return completed caller buffer
    else a middle range remains
        R->>O: one aligned middle-range read
        O-->>R: remote bytes
        R->>R: copy requested overlap into caller buffer
        loop each true MISS block in the remote span
            R->>I: insert completed block buffer if absent
            alt another owner already published it
                I-->>R: existing entry, skip duplicate write
            else this reader owns persistence
                R->>S: try_submit without waiting
                alt queue accepts task
                    S-->>R: accepted
                else queue or shutdown rejects task
                    S-->>R: rejected
                    R->>I: conditional rollback
                end
            end
        end
        R-->>C: return completed caller buffer
        S->>W: dequeue in background
        W->>F: revalidate epoch and current block state
        W->>F: append and finalize eligible empty blocks
        W->>I: conditional completion cleanup
    end

2. Later reader reuses a pending buffer

After the first remote read publishes a buffer and before its worker finishes persistence, a later reader can copy the requested bytes directly from the inflight entry. It does not wait for the local file to be finalized and does not submit another write task for that block.

This index deduplicates pending cache writes and bridges the interval between remote-read completion and local persistence. It intentionally does not claim to deduplicate remote requests that began before publication.

3. Backpressure and asynchronous failure

The per-disk pending-task limit bounds queued work. Buffer allocation, queue submission, or shutdown can reject a task. In that case the reader conditionally removes its own inflight entry and records the rejection, but still returns the already-read data to the caller.

Likewise, an append/finalize failure is logged and counted by the worker; it does not retroactively fail a query whose remote read succeeded. A future read simply observes another cache miss and may retry population.

4. Clear/remove and stale-task handling

Cache invalidation advances the service's write epoch. Both inflight lookup/replacement and worker processing compare their entry or task against the current epoch. Workers check the epoch before acquiring cache blocks and again while processing them. This closes the window where old queued work could recreate blocks after a clear/remove operation.

Completion cleanup uses pointer-conditional removal. Therefore, even if a newer generation has published an entry for the same block offset, an older task cannot erase it.

5. Mode selection and compatibility

CacheWriteMode is resolved for every read so an online configuration switch affects existing readers. Ordinary reads may select the asynchronous path when the feature is enabled. Dry-run, warm-up, peer-cache, segment-index population, downloader, and explicit prefetch paths select or override synchronous mode because their completion semantics depend on the cache being populated before returning.

The default remains synchronous because enable_async_file_cache_write defaults to false.

6. Worker lifecycle

Each BlockFileCache creates its inflight index and async-write service state during initialization. When asynchronous writeback is disabled, the service does not create persistent worker loops. Enabling the feature online explicitly starts all initialized per-disk services. Each worker keeps its own MPMC consumer cursor and performs dequeue, cache-state revalidation, downloader acquisition, append, and finalize in the same thread; there is no second persistence pool or cross-thread FileBlock ownership transfer. Worker count can be resized online. Shutdown first stops new submitters, waits for already registered submitters, drains accepted tasks, joins workers, and only then destroys state referenced by task-finalization callbacks.

Configuration

Configuration	Default	Purpose
`enable_async_file_cache_write`	`false`	Enables the ordinary asynchronous write path.
`async_file_cache_write_workers_per_disk`	`16`	Number of persistence workers for each cache disk; supports online resize. Workers are started only when asynchronous writeback is enabled.
`async_file_cache_write_max_pending_tasks_per_disk`	`256`	Bounds accepted and queued tasks per disk.
`async_file_cache_write_batch_size`	`16`	Maximum tasks a worker handles per dequeue loop.
`async_file_cache_write_watchdog_warn_secs`	`30`	Warns when a task waited too long.
`async_file_cache_write_watchdog_drop_secs`	`120`	Drops excessively old best-effort work.
`enable_inflight_write_buffer_index`	`true`	Enables pending-buffer reuse and duplicate-write suppression.
`inflight_write_buffer_index_shard_count`	`64`	Controls index lock sharding at cache initialization.

Observability

Query runtime profiles expose asynchronous submissions, rejections, buffer-allocation failures, stale-epoch drops, and inflight hits/misses. Per-cache-disk bvars additionally expose queue depth, tracked buffer bytes, write latency, task-age timeouts, append failures, state-based skips, index size/memory, conditional-removal results, and backpressure rollbacks.

Tests

Latest concurrency and lifecycle update: 12/12 targeted ASAN BE unit tests passed.
- eight write workers consume distinct MPMC tasks concurrently;
- disabled services reject submissions without retaining pending work;
- online enablement starts initialized per-disk services;
- runtime resize, shutdown, watchdog, epoch cleanup, and reader backpressure rollback remain covered.
Latest read-planning refactor: 6/6 targeted ASAN BlockFileCacheTest cases passed.
- inflight-buffer reuse and downloaded-block reads;
- DOWNLOADING wait outside the remote span;
- cached prefix/suffix assembly;
- one first-to-last-miss remote read with an existing middle block;
- true-miss-only submission and backpressure rollback;
- per-read cache-write mode resolution.
BE ASAN targeted unit tests: 26/26 passed across four suites.
- read-only probe behavior;
- inflight insert/lookup/epoch replacement and concurrent publication;
- queue limits, shutdown, worker resize, watchdog, epoch invalidation, multi-cell writes, and partial overlap;
- mixed cache coverage, one-middle-read behavior, pending-buffer reuse, backpressure rollback, and per-read mode resolution;
- runtime-profile delta aggregation.
Docker cloud regression: 1/1 suite passed.
- a cold query submits asynchronous cache writes;
- after the queue drains, a second query observes probe hits;
- switching the feature off restores synchronous behavior without asynchronous submissions.
Build and style:
- ./build.sh --be --fe --cloud -j100
- ./build.sh --be -j100
- build-support/check-format.sh
- clang-tidy was intentionally not run for this change.

Release note

Add an opt-in asynchronous file-cache write path controlled by enable_async_file_cache_write. It is disabled by default.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (Docker cloud regression and runtime configuration switch)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes. When explicitly enabled, ordinary file-cache misses persist eligible cache blocks through background workers. Default behavior and explicit cache-population completion remain synchronous.
Does this need documentation?
- No. The feature is experimental and disabled by default.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: On a file-cache miss, CachedRemoteFileReader currently reads the requested data from remote storage and then appends and finalizes the corresponding local cache blocks on the query thread. The remote read is required to satisfy the query, but the subsequent local writes are best-effort cache population. Coupling the two makes local filesystem latency and backpressure part of foreground scan latency. Moving only the append call to a background thread is not sufficient. Concurrent readers may fetch the same missing range, FileBlock downloader ownership has thread-affine cleanup semantics, and cache clear/remove can invalidate queued work. Warm-up, prefetch, dry-run, and other explicit cache-population callers also require synchronous completion semantics. This change introduces an opt-in asynchronous write path with the following architecture: 1. BlockFileCache provides a read-only probe API that reports downloaded, downloading, empty, and missing ranges without creating cache cells or taking downloader ownership. 2. Each cache instance owns an InflightWriteBufferIndex. It publishes remote-read buffers with insert-if-absent semantics so later readers can reuse bytes that have already been fetched and are awaiting persistence. 3. Each cache disk owns an AsyncCacheWriteService with a bounded MPMC queue, tracked-buffer accounting, dynamically resizable workers, task-age protection, and an explicit shutdown protocol. 4. The ordinary read path combines inflight buffers and downloaded cache blocks, reads the remaining middle range from remote storage once, copies all required bytes into the caller buffer, and submits background tasks only for true cache misses. 5. Workers revalidate the cache write epoch and current block state before writing. Conditional index removal and epoch changes prevent stale callbacks or queued tasks from deleting newer entries or recreating data after cache invalidation. Queue rejection, tracked-buffer pressure, or asynchronous persistence failure does not fail a remote read that has already produced the requested data. The inflight entry is rolled back and the operation falls back to best-effort cache behavior. Explicit cache-population paths continue to use the synchronous implementation. The feature is disabled by default and can be switched online. Worker counts, pending-task limits, batch size, and task-age thresholds are validated through configuration. New bvars and runtime-profile counters expose submissions, inflight reuse, probe results, rejections, failures, queue depth, buffer memory, and write latency. The asynchronous reader implementation is isolated in cached_remote_file_reader_async_write.cpp. The top-level read function only orchestrates planning, covered-range materialization, a single remote middle read, and task submission; the detailed steps are kept in cohesive helper functions. ### Release note Add an opt-in asynchronous file-cache write path controlled by `enable_async_file_cache_write`. It is disabled by default. ### Check List (For Author) - Test: - [x] Regression test - Docker cloud suite `test_async_file_cache_write`: 1 suite passed - [x] Unit Test - BE ASAN targeted tests: 26 cases from 4 suites passed - [x] Build - `./build.sh --be --fe --cloud -j100` - `./build.sh --be -j100` - [x] Code style - `build-support/check-format.sh` - clang-tidy was intentionally not run for this change - Behavior changed: - [x] Yes. When explicitly enabled, ordinary file-cache misses return after the caller buffer is complete and persist missing cache blocks in background workers. The default and explicit cache-population behavior are unchanged. - Does this need documentation: - [x] No. The feature is experimental and disabled by default.

hello-stephen · 2026-07-15T10:43:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The phase-one asynchronous file-cache reader built detailed coverage runs, maintained multiple cursors, and materialized individual holes inside a read even though most read_at requests span only one or two cache blocks. This made the query-side orchestration difficult to review and maintain without providing meaningful value for the common case. Replace that logic with one aligned inflight lookup, one read-only cache probe, and a simple per-block source plan. The reader still gives inflight buffers priority and still distinguishes downloaded, downloading, and missing cache blocks. Downloading blocks outside the remote span retain their wait behavior. When real misses exist, the reader takes the first through last miss as one remote range, intentionally rereads any cache or inflight blocks inside that range, and submits background writes only for the blocks that were actual misses. A cache-side race falls back to one full aligned remote read. This preserves caller-buffer completeness, inflight deduplication, existing-block reads, cache wait semantics, non-blocking write submission, and backpressure rollback while substantially reducing the amount of control flow in CachedRemoteFileReader::_read_async_write_path and its helpers. ### Release note None ### Check List (For Author) - Test: - Unit Test: six targeted BlockFileCacheTest cases passed under ASAN, covering inflight reuse, DOWNLOADING wait, cached sides, one remote middle span, real-miss-only submission, backpressure rollback, and per-read mode selection - Build: ./build.sh --be -j100 passed - Style check: build-support/check-format.sh and git diff --check passed - Behavior changed: No. This refactor preserves the phase-one asynchronous cache-write behavior while simplifying how the read range is assembled. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The asynchronous cache read planner called BlockFileCache::probe before consulting the inflight write-buffer index. BlockFileCache::probe acquires the cache mutex, so a request already covered entirely by inflight buffers still contended on BlockFileCache even though it needed no cache metadata. Build the aligned block list and perform the batch inflight lookup first. If every requested block is covered for the current write epoch, return the plan immediately and materialize the caller buffer directly from inflight memory. If any block is not covered, retain the existing mixed-source behavior by issuing one whole-range read-only cache probe and classifying only the non-inflight blocks as downloaded, downloading, or remote misses. Make the probe result optional in the read plan so ownership matches the conditional probe. Extend the inflight reuse unit test to hold the BlockFileCache mutex during the second read; the read must still complete, directly proving that the full-inflight fast path does not enter BlockFileCache::probe. ### Release note None ### Check List (For Author) - Test: - Unit Test: six targeted BlockFileCacheTest cases passed under ASAN, including full inflight coverage while the BlockFileCache mutex is held, partial cache coverage, downloading waits, middle-span reads, backpressure rollback, and per-read mode selection - Build: ./build.sh --be -j100 passed - Style check: build-support/check-format.sh and git diff --check passed - Behavior changed: Yes. Reads fully covered by current-epoch inflight buffers no longer call BlockFileCache::probe or acquire its cache mutex; partial inflight coverage still probes and combines existing cache blocks. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: AsyncCacheWriteService previously used a follow_global_config flag to switch between fixed test options and direct reads of mutable BE configuration. That made queue admission, batching, and watchdog behavior depend on global state that was not visible in the service interface. It also split online updates across two mechanisms: worker-count changes were forwarded by FileCacheFactory, while the remaining settings were read implicitly from worker and submission paths. Make configuration ownership explicit. A newly initialized BlockFileCache constructs a complete per-disk options snapshot, and FileCacheFactory registers update callbacks for all five mutable async-write settings. Each callback captures one complete configuration snapshot and forwards it through FileCacheFactory::update_async_write_options to AsyncCacheWriteService::update_options. The service validates the snapshot, applies the requested worker count, and atomically publishes immutable queue, batch, and watchdog settings. Submission and worker paths now consume service-owned snapshots and no longer include or reference common/config.h. Update unit tests to configure isolated services through the explicit interface, and add coverage proving that config::set_config propagates every mutable setting through the factory into an initialized per-disk service. ### Release note None ### Check List (For Author) - Test: - Unit Test: `./run-be-ut.sh --run --filter=AsyncCacheWriteServiceTest.*:BlockFileCacheTest.async_write_backpressure_rolls_back_inflight_entry -j 100` passed all 11 tests under ASAN - Build: `./build.sh --be -j100` passed - Style check: `build-support/check-format.sh` and `git diff --check` passed - Behavior changed: No. Online mutable settings keep their existing behavior but are propagated through explicit update interfaces. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The phase-one asynchronous file-cache write service used two persistent workers per cache disk. The synchronous path it replaces persisted cache blocks directly on scanner threads, so its effective per-disk write concurrency could scale with the external scanner concurrency, whose default per-context upper bound is 16, and could grow further across concurrent query contexts. A two-worker default therefore serialized writeback far more aggressively than the former path and could fill the bounded pending queue during ordinary scan fan-out. Increase the default to 16 workers per cache disk. Keep one MPMC queue and let each worker dequeue, revalidate, claim the FileBlock downloader, and write the block in the same thread. Splitting consumption and persistence into separate pools would add a full-task handoff without an independent processing stage, and claiming a downloader before that handoff would violate FileBlock's thread-bound ownership contract. Each worker now uses its own ConsumerToken so concurrent consumers maintain independent producer-stream cursors instead of rescanning streams for every task. Avoid creating 16 persistent per-disk worker loops while asynchronous writeback is disabled. The cache still constructs the service state and inflight index, but starts workers only when the feature is enabled. A false-to-true online configuration update explicitly starts all initialized services through the factory interface, while mutable service options continue to flow through the explicit factory/service update API. Service readiness is published only after all configured worker loops have been accepted, so query threads reject best-effort submissions instead of enqueueing work to an unready service. Add deterministic coverage for eight workers consuming distinct tasks concurrently, disabled-service rejection, online enablement, and the existing runtime resize, shutdown, watchdog, inflight cleanup, and reader backpressure rollback behavior. ### Release note Increase the default asynchronous file-cache write concurrency from 2 to 16 workers per cache disk. Worker threads are created only after asynchronous file-cache writeback is enabled. ### Check List (For Author) - Test: Unit Test - `./build.sh --be -j100` - `./run-be-ut.sh --run --filter=AsyncCacheWriteServiceTest.*:BlockFileCacheTest.async_write_backpressure_rolls_back_inflight_entry -j 100` (12 tests passed) - `build-support/check-format.sh` - `git diff --check` - Behavior changed: Yes. The default per-disk asynchronous write concurrency is 16, and disabled services no longer keep worker loops resident. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The phase-one async file-cache write implementation had strong happy-path coverage, but several correctness boundaries were not exercised through the complete component interactions. Missing coverage included cache-file disappearance and whole-key self-heal cleanup, wait timeout fallback, direct-read prefix preservation, final concurrent publication deduplication, tracked-buffer allocation failure, external-table cache reuse, worker ownership of existing or deleting cells, remove/write epoch races, runtime worker growth and shrink, and complete propagation of the new profile and synchronous cache-population semantics. Add compact scenario-oriented BE unit tests that drive the real reader, cache, inflight index, async service, worker, removal, downloader, and index-preload paths. The tests verify both returned data and persistent cache state, including metadata and physical file deletion. Narrow test-only sync points make allocation failures and race windows deterministic without changing normal runtime behavior. ### Release note None ### Check List (For Author) - Test: Unit Test - 51 related ASAN BE unit tests passed with run-be-ut.sh and -j100 - 7 focused changed-path tests passed with run-be-ut.sh and -j100 - BE build passed with build.sh --be -j100 - build-support/check-format.sh passed - Behavior changed: No; only test coverage and deterministic test injection points are added - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The async cache-write planner queried one block-aligned range, but `BlockFileCache::probe` returned a `FileBlocksHolder` of cache hits plus an independent gap list. The planner then had to scan all hits and gaps for every logical read block even though the read-plan blocks and probe slots use the same block boundaries. This obscured the alignment invariant and introduced unnecessary nested matching logic on the query read path. Change the probe contract to return one ordered nullable `FileBlock` pointer per aligned input block. A non-null slot is asserted to have the exact corresponding range, except that the final block may end at EOF, while a null slot directly represents a cache miss. The planner now preserves its inflight-first fast path and joins probe results to plan blocks by index; materialization also reads the matching slot directly. Remove `FileBlocksHolder` and the independent gaps from `FileBlocksProbeResult`. Preserve the existing deferred cleanup semantics for EMPTY and deleting cache blocks through a shared cache-user reference release helper rather than embedding a holder in the probe result. Update focused and end-to-end tests for hit/miss slots, a short final block, retained block states, self-heal cleanup, and an aligned direct-cache prefix followed by an async-written suffix. ### Release note None ### Check List (For Author) - Test: Unit Test - 51 related ASAN BE unit tests passed with `run-be-ut.sh` and `-j100` - 2 focused probe/direct-prefix ASAN BE unit tests passed with `run-be-ut.sh` and `-j100` - `build-support/check-format.sh` passed - Behavior changed: No; this simplifies an internal probe/planning contract without changing user-visible cache semantics - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: Existing async file-cache write tests covered a single pending-limit rejection and runtime worker resizing independently, but they did not exercise the complete dynamic backpressure lifecycle. Without that coverage, regressions could allow the MPMC backlog to exceed its bound, lose submissions under producer concurrency, fail to expose sustained rejection at capacity, or leave accepted work stranded after consumers are scaled up. Add one deterministic service-level BE unit test that stalls the initial worker before cache mutation and drives four producers in controlled waves. The test verifies that the actual queued backlog grows through 4, 8, 12, and 16 tasks, that pending count includes the blocked active task, and that a subsequent 48-task producer burst is rejected without changing the bounded queue or accepted-task count. After producers stop, the test increases worker concurrency from one to four and batch size from one to four, then releases the artificial write delay. It samples the MPMC backlog independently from pending count, verifies an intermediate lower watermark and an empty queue, and confirms that all accepted tasks finalize with pending count returning to zero. The synchronization point makes both phases deterministic without adding a production-only observation API. ### Release note None ### Check List (For Author) - Test: Unit Test - New dynamic MPMC backpressure ASAN BE unit test passed with run-be-ut.sh and -j100 - All 14 AsyncCacheWriteServiceTest ASAN BE unit tests passed with run-be-ut.sh and -j100 - build-support/check-format.sh passed - Behavior changed: No; this adds deterministic test coverage only - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The async file cache write settings were mixed into the broader block file cache configuration section, while the inflight write buffer index settings did not carry the feature name. This made the feature difficult to locate as one configuration group and made name-based filtering incomplete. Move all async file cache write declarations, definitions, and validators into a dedicated contiguous section. Keep the primary enable_async_file_cache_write switch unchanged, rename only the inflight index enable and shard-count settings with the async_file_cache_write prefix, and update runtime consumers plus BE and regression test configuration. Defaults, mutability, validation, and runtime behavior remain unchanged. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=AsyncCachedRemoteFileReaderTest.*:BlockFileCacheTest.*async_write*:BlockFileCacheTest.cache_write_mode_is_resolved_for_each_read_context:AsyncCacheWriteServiceTest.* -j100 (26 tests passed) - build-support/check-format.sh - Behavior changed: No - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The dynamic async-write backpressure test created four producer threads during queue growth, but each producer submitted only one task per fill wave. That exercised simultaneous entry only weakly and did not model sustained concurrent production before backpressure. Start every producer in a wave through a barrier and let each producer submit four consecutive tasks. The test now observes deterministic queue growth through 16, 32, 48, and 64 queued tasks, verifies a subsequent 128-task concurrent burst is rejected at the pending limit, then confirms the enlarged and accelerated consumer side drains the queue to zero. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=AsyncCachedRemoteFileReaderTest.*:BlockFileCacheTest.*async_write*:BlockFileCacheTest.cache_write_mode_is_resolved_for_each_read_context:AsyncCacheWriteServiceTest.* -j100 (26 tests passed) - build-support/check-format.sh - Behavior changed: No - Does this need documentation: No

bobhan1 · 2026-07-17T05:28:01Z

/review

github-actions

Request changes: the async-write design is broadly implemented and the happy-path coverage is substantial, but nine blocking issues remain.

Reader correctness: legal noncanonical/restored cache cells can abort probe(), and a live peer-cache enable transition can make the async path abort the BE with a null remote buffer.
Concurrency and lifecycle: accepted-but-unstarted persistent worker loops can deadlock shrink; invalidation can enqueue a generation-blind delete that removes a newer refill.
Configuration and errors: runtime/startup validation does not validate a coherent incoming option set; concurrent callbacks can apply stale snapshots; online enablement reports success after service-start failure.
Admission: async handoff drops both file_cache_query_limit_bytes and the lifetime needed to enforce file_cache_query_limit_percent.
Performance: the default idle workers poll every millisecond, around 16,000 timed wakeups per second per cache disk, while the current notification protocol is not safe for a naive indefinite wait.

Critical checkpoint conclusions:

Goal and test proof: ordinary aligned misses do return promptly and persistence is handed to a bounded service. Unit/regression coverage is broad, but it omits the negative and concurrency cases identified inline.
Scope, parallel paths, and compatibility: downloader, index-loader, prefetch, dry-run, warm-up, and stable peer paths retain synchronous completion. No FE/BE wire, transaction, EditLog, MoW, or visible-version contract changes were found. Existing writer/restored cache layouts are not compatible with the new exact-boundary probe.
Concurrency, lifecycle, configuration, and errors: shutdown/destruction order, thread context, tracker ownership, lock order, and pointer-conditional inflight cleanup are sound. The worker-start/shrink, generation deletion, validation, snapshot ordering, and enablement failures remain blocking.
Data and persistence: source data is immutable/remote and no transactional persistence boundary changes. The generation race can delete a newer derived cache file/meta and force self-heal, but does not corrupt the remote source.
Observability and performance: new query/service metrics are propagated through merge/diff/reporting and scanner detection. The 1 ms idle wake protocol is the remaining observability/performance concern.
Tests and validation: static review only, as the review contract forbids builds and source changes. .worktree_initialized and thirdparty/installed are also absent. Existing review threads/comments were empty, and all three round-two reviewers returned NO_NEW_VALUABLE_FINDINGS against this exact nine-comment set.
User focus: no additional focus was supplied; the whole PR was reviewed.

github-actions · 2026-07-17T06:12:53Z

+            cached_block->second.file_block->range().left <= expected_range.right) {
+            file_block = cached_block->second.file_block;
+            DORIS_CHECK(file_block->range().left == expected_range.left);
+            DORIS_CHECK(file_block->range().right == expected_range.right);


probe() cannot require existing cells to have the new planner's exact slot boundaries. get_or_set() accepts arbitrary ranges, and the S3 write-through path allocates each mutable s3_write_buffer_size part without requiring divisibility by file_cache_each_block_size (persisted cells can also retain an older block size). For example, 4 MiB cache blocks plus a 5 MiB upload leave a legal [4,5 MiB) cell; an async query probes [4,8 MiB) and aborts here. Please plan from actual intersecting ranges or fall back to the established path for noncanonical layouts, and cover a non-divisible writer/restored layout.

github-actions · 2026-07-17T06:12:53Z

+    }
+
+    const std::vector<FileBlockSPtr> no_peer_blocks;
+    RETURN_IF_ERROR(_execute_remote_read(no_peer_blocks, remote_left, remote_size, *remote_buffer,


This call re-evaluates mutable peer-read configuration after _resolve_cache_write_mode() already selected the async/S3-only path. If peer reads are enabled in that window, no_peer_blocks makes the peer fetch return OK without producing remote_buffer (and the peer race can likewise win), so the next DORIS_CHECK(*remote_buffer != nullptr) aborts the BE process. Preserve the dispatch-time peer decision or call an explicitly S3-only helper here.

github-actions · 2026-07-17T06:12:53Z

+    _cv.notify_all();
+
+    if (worker_count < old_count) {
+        std::unique_lock state_lock(_worker_state_mutex);


This wait can be permanent after a partial OS-thread creation failure. Doris ThreadPool::do_submit() enqueues a task and returns OK when create_thread() fails but another pool thread exists; _schedule_worker() then leaves that queued id marked scheduled. Here the existing thread is occupied indefinitely by worker 0, so queued higher-id loops can never start and clear their flags, while a shrink to 1 waits for exactly those flags. Please distinguish accepted from actually started loops and provide a cancellation/retirement path for queued persistent workers.

github-actions · 2026-07-17T06:12:53Z

+                 [](int32_t value) { return value > 0 && value <= 128; });
+DEFINE_Validator(async_file_cache_write_max_pending_tasks_per_disk,
+                 [](int64_t value) { return value > 0; });
+DEFINE_Validator(async_file_cache_write_batch_size, [](int32_t value) { return value > 0; });


These new validators cannot enforce the advertised runtime/startup contract through the current config framework. Runtime UPDATE_FIELD calls the zero-argument validator before assigning new_value, so e.g. batch_size=0 validates the old 16, commits 0, returns OK, and only logs when the service rejects it; a later cache construction hits its DORIS_CHECK. At startup the field map is alphabetical, so drop_secs is loaded before warn_secs: a valid final pair warn=5, drop=10 is rejected against the default warn=30. Please validate the incoming coherent option set before committing it and add invalid-runtime plus lowered-pair startup tests.

github-actions · 2026-07-17T06:12:54Z

+    if (factory == nullptr) {
+        return;
+    }
+    Status status = factory->update_async_write_options(load_async_write_options_from_config());


The factory mutex orders service application, but it does not order config mutation with this full-snapshot read. Two concurrent update requests can therefore apply stale state last: A commits batch=32 and captures pending=256, B commits pending=512 and applies {32,512}, then A acquires _mtx and reapplies {32,256}. Both requests report OK while services no longer match the final globals. Serialize snapshot/version creation with application, or reject older generations.

github-actions · 2026-07-17T06:12:54Z

+    if (factory == nullptr) {
+        return;
+    }
+    Status status = factory->start_async_write_services();


A service-start failure is swallowed after the global flag has already been committed. The HTTP update still reports OK, _resolve_cache_write_mode() selects async solely from that flag, and disks whose first worker failed to start reject every write submission; setting true again is also a callback no-op because old==new. Make enablement transactional/error-reporting, or dispatch per cache only after that service is ready and fall back to synchronous writeback on failed disks.

github-actions · 2026-07-17T06:12:54Z

+    DORIS_CHECK(remote_buffer != nullptr);
+
+    CacheContext cache_context(io_ctx);
+    CacheAdmissionContext admission_context {


This handoff loses both query-scoped admission contracts. The worker reconstructs a context without RemoteScanCacheWriteLimiter, so positive file_cache_query_limit_bytes never accumulates and never switches the query to remote-only-on-miss. Retaining only query_id also fails the older percentage limit: once query teardown removes its QueryFileCacheContext, a queued task reserves from global LRU without file_cache_query_limit_percent. Admission needs to happen while the query state is alive (or the task must retain lifetime-safe admission state); please add async tests for both limits.

github-actions · 2026-07-17T06:12:54Z

+            continue;
+        }
+        const size_t buffer_offset = block->range().left - task.file_offset;
+        TEST_SYNC_POINT_CALLBACK("AsyncCacheWriteService::_write_one:before_append", &task);


There is still an invalidation window after these last epoch/deleting checks. An E worker can pause here, be marked deleting by E+1, then append/finalize; holder cleanup queues a delayed removal containing only the cache key. If an E+1 task refills the same (hash, offset) before background GC consumes that key, GC unconditionally deletes the shared file/meta for the new block, leaving a DOWNLOADED cell whose storage is gone. Make finalization/removal generation-aware (or share a linearization boundary), and test refill-before-recycle for this existing sync point.

github-actions · 2026-07-17T06:12:54Z

+        }
+        if (processed == 0) {
+            std::unique_lock lock(_cv_mutex);
+            _cv.wait_for(lock, std::chrono::milliseconds(1), [&]() {


With the default 16 workers per disk, this 1 ms timeout causes about 16,000 idle timed wakeups per second per cache disk (and scales linearly with disks), even when no cache write exists. It also appears to be masking a wakeup-protocol gap: enqueue notifies without synchronizing the queue predicate on _cv_mutex, so merely changing this to an indefinite wait could lose a notification between the predicate check and sleep. Please use a proper semaphore/CV sequence (or synchronize predicate updates) and block normally; at minimum, make any safety poll genuinely coarse.

### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The async file cache write path exposed only an aggregate pending count and a limited set of outcome counters. When throughput degraded, operators could not distinguish producer backpressure, MPMC queue buildup, unavailable workers, inflight-index lock contention, BlockFileCache metadata contention, append/finalize latency, or watchdog and stale-epoch drops. Add exact atomically maintained queue, active-task, running-worker, configured-capacity, and active-stage gauges. Add reason-specific rejection and watchdog counters, submitted and persisted byte/block throughput, and latency recorders for submission, allocation, queue wait, worker processing, get-or-set, append, finalize, probing, read-plan construction, and write submission. Instrument inflight shard lock wait and hold time, and route probe/get-or-set locking through the existing BlockFileCache lock-wait metric. Keep queue monitoring passive and exact instead of adding a sampling thread. Remove the unused AsyncCacheWriteService::stats snapshot API and use the service state and metrics directly in tests. Do not expose an inflight index metadata memory estimate because it can be mistaken for total payload memory; async_cache_write_buffer_memory_bytes remains the payload-memory metric. Extend the BE unit tests to verify live queue and stage gauges, metric counters and latency samples, lock instrumentation, and the concurrent MPMC growth, rejection, scale-up, and drain flow. ### Release note Add monitoring metrics for async file cache write queue pressure, worker activity, stage latency, throughput, rejection causes, and lock contention. ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh -j100 --run --filter=AsyncCacheWriteServiceTest.*:InflightWriteBufferIndexTest.*:BlockFileCacheTest.Probe*:AsyncCachedRemoteFileReaderTest.* (29 tests passed under ASAN_UT) - Behavior changed: Yes. Adds observability only; async cache read and write semantics are unchanged. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The existing file-cache microbenchmark either includes object-store and network latency through CachedRemoteFileReader or stops at BlockFileCache::get_or_set. It cannot isolate phase-1 asynchronous writeback, distinguish caller return time from background drain, or expose queue saturation and inflight-index contention. Add a standalone Release microbenchmark beside the existing tool. It combines a deterministic in-memory remote reader with a real filesystem-backed BlockFileCache and covers three layers: synchronous versus asynchronous cold-miss reader latency with complete-range persistence verification; producer admission, bounded MPMC queue behavior, worker scaling, backpressure, and real get_or_set/append/finalize persistence; and sharded miss/hit versus single-hot-key InflightWriteBufferIndex contention. Each case emits machine-readable latency percentiles, throughput, accepted/rejected/persisted counts, and pending/queued/inflight high-water marks. Benchmark data defaults to output/ so it remains untracked and uses the larger workspace disk. ### Release note None ### Check List (For Author) - Test: Manual test - ./build.sh --be --file-cache-microbench -j100 - Existing file_cache_get_or_set benchmark with 1 and 32 threads - Full async benchmark in all mode with 16 producers and worker counts 1,4,16 - build-support/check-format.sh - git diff --check - Behavior changed: No (benchmark tooling only) - Does this need documentation: No (tool README updated)

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The asynchronous file-cache write microbenchmark previously emitted only one sample per case and did not establish the storage baseline of the cache filesystem. These short concurrent cases are sensitive to scheduler activity, page-cache state, filesystem metadata, and background writeback, so a single number can hide material variance and make worker-scaling conclusions unreliable. Run every selected reader, service, and inflight-index case five times by default and add the one-based repetition to each machine-readable RESULT line. Add an installed runner that automatically detects fio, measures direct 1 MiB sequential QD1 and random QD16 writes on the same filesystem, then starts the benchmark. The runner uses a unique sibling directory under the selected cache path, unlinks fio data, and keeps direct I/O out of the page cache. The benchmark now rejects non-empty cache paths instead of recursively clearing them, and suppresses INFO logging so merged stdout and stderr cannot corrupt RESULT records. Expand the tool README with the component flow, coverage and non-goals of each group, default workload, field semantics, fio controls, cache-path ownership, repetition methodology, and interpretation guidance. Median is the primary value and the observed min-to-max range is retained. Release experiment on /dev/nvme11n1 ext4, 1 MiB blocks, 64 KiB caller reads, 16 producers, 128 reader operations, 256 service attempts, and five repetitions: - fio direct baseline: sequential QD1 was 2513 MiB/s with 161 us p95 completion latency; random QD16 was 3106 MiB/s with 10.552 ms p95 completion latency. - Reader foreground throughput in ops/s, median [min, max]: sync 5645 [5124, 7649], async 6739 [4739, 8298]. Median average latency was 1459 us for sync and 912 us for async. The median throughput improved 19.4% and median average latency fell 37.5%, while the overlapping ranges show why repeated samples are required. - Verified service completion in MiB/s, median [min, max]: 1 worker 798 [730, 968], 4 workers 1562 [1260, 1751], and 16 workers 7825 [5255, 13203]. Median drain time fell from 0.300 s to 0.153 s and 0.014 s. These are buffered append/finalize completions without fsync and are not durable-media throughput. - Backpressure accepted 76 [64, 101] and rejected 180 [155, 192] tasks, with peak pending fixed at the configured limit of 64, peak queued fixed at 48, and peak inflight 65 [65, 67]. Every accepted task was verified as persisted. - Inflight lookup throughput in ops/s, median [min, max]: sharded miss 5.531M [4.028M, 6.621M], sharded hit 4.198M [3.270M, 6.036M], and hot-key hit 1.104M [1.090M, 1.268M]. - All 45 RESULT records were complete and parseable. All reader ranges and all accepted service tasks passed final BlockFileCache coverage verification. ### Release note None ### Check List (For Author) - Test: Manual test - ./build.sh --be --file-cache-microbench -j100 (Release) - ./output/be/bin/run-async-file-cache-write-microbench.sh --benchmark_mode=all --cache_path=./output/async_file_cache_write_microbench_repeat_5_clean --producer_threads=16 --reader_workers=16 --worker_counts=1,4,16 --repetitions=5 - Non-empty cache-path rejection with sentinel preservation - build-support/clang-format.sh - build-support/check-format.sh - bash -n and shellcheck for the runner - git diff --check - Behavior changed: No (benchmark tooling only) - Does this need documentation: No (tool README updated)

### What problem does this PR solve? Issue Number: None Related PR: apache#65658 Problem Summary: The asynchronous file-cache write microbenchmark previously emitted only one sample per case and did not establish the storage baseline of the cache filesystem. These short concurrent cases are sensitive to scheduler activity, page-cache state, filesystem metadata, and background writeback, so a single number can hide material variance and make worker-scaling conclusions unreliable. Run every selected reader, service, and inflight-index case five times by default and add the one-based repetition to each machine-readable RESULT line. Add an installed runner that can measure direct 1 MiB sequential QD1 and random QD16 writes on the same filesystem before starting the benchmark. The fio behavior is explicit and does not make fio a mandatory dependency: | RUN_FIO | fio available | Behavior | | --- | --- | --- | | auto (default) | Yes | Run both disk baselines, then run the cache benchmark | | auto (default) | No | Print DISK_BASELINE skipped and continue directly with the cache benchmark | | 1 | No | Fail because the caller explicitly required fio | | 0 | Any | Skip fio and run the cache benchmark | The runner uses a unique sibling directory under the selected cache path, unlinks fio data, and keeps direct I/O out of the page cache. The benchmark rejects non-empty cache paths instead of recursively clearing them, and suppresses INFO logging so merged stdout and stderr cannot corrupt RESULT records. Expand the tool README with the component flow, coverage and non-goals of each group, default workload, field semantics, fio controls, cache-path ownership, repetition methodology, and interpretation guidance. Median is the primary value and the observed minimum and maximum are retained. Release experiment configuration: /dev/nvme11n1 ext4, 1 MiB blocks, 64 KiB caller reads, 16 producers, 128 reader operations, 256 service attempts, and five repetitions. fio direct-I/O baseline: | Workload | Bandwidth | p95 completion latency | | --- | ---: | ---: | | 1 MiB sequential write, QD1 | 2513 MiB/s | 161 us | | 1 MiB random write, QD16 | 3106 MiB/s | 10.552 ms | CachedRemoteFileReader foreground results: | Write mode | Median ops/s | Minimum ops/s | Maximum ops/s | Median average latency | | --- | ---: | ---: | ---: | ---: | | Synchronous | 5645 | 5124 | 7649 | 1459 us | | Asynchronous | 6739 | 4739 | 8298 | 912 us | The asynchronous median was 19.4% higher in throughput and 37.5% lower in average latency. The overlapping ranges are retained because they show why a single run is insufficient. AsyncCacheWriteService verified completion results: | Workers | Median MiB/s | Minimum MiB/s | Maximum MiB/s | Median drain time | | ---: | ---: | ---: | ---: | ---: | | 1 | 798 | 730 | 968 | 0.300 s | | 4 | 1562 | 1260 | 1751 | 0.153 s | | 16 | 7825 | 5255 | 13203 | 0.014 s | These values measure buffered append and finalize completion without fsync. They are not durable-media throughput and are not directly comparable with the direct-I/O fio baseline. Bounded backpressure results: | Metric | Median | Minimum | Maximum | | --- | ---: | ---: | ---: | | Accepted tasks | 76 | 64 | 101 | | Rejected tasks | 180 | 155 | 192 | | Peak pending | 64 | 64 | 64 | | Peak queued | 48 | 48 | 48 | | Peak inflight | 65 | 65 | 67 | Every accepted task was verified as persisted, and peak pending stayed at the configured limit. InflightWriteBufferIndex lookup results: | Workload | Median ops/s | Minimum ops/s | Maximum ops/s | Median average latency | | --- | ---: | ---: | ---: | ---: | | Sharded miss | 5.531M | 4.028M | 6.621M | 2.688 us | | Sharded hit | 4.198M | 3.270M | 6.036M | 3.171 us | | Hot-key hit | 1.104M | 1.090M | 1.268M | 13.701 us | All 45 RESULT records were complete and parseable. All reader ranges and all accepted service tasks passed final BlockFileCache coverage verification. ### Release note None ### Check List (For Author) - Test: Manual test - ./build.sh --be --file-cache-microbench -j100 (Release) - ./output/be/bin/run-async-file-cache-write-microbench.sh --benchmark_mode=all --cache_path=./output/async_file_cache_write_microbench_repeat_5_clean --producer_threads=16 --reader_workers=16 --worker_counts=1,4,16 --repetitions=5 - Non-empty cache-path rejection with sentinel preservation - build-support/clang-format.sh - build-support/check-format.sh - bash -n and shellcheck for the runner - git diff --check - Behavior changed: No (benchmark tooling only) - Does this need documentation: No (tool README updated)

bobhan1 added 9 commits July 16, 2026 13:54

github-actions Bot requested changes Jul 17, 2026

View reviewed changes

bobhan1 added 2 commits July 17, 2026 15:44

bobhan1 force-pushed the feature/async-file-cache-write-phase1 branch from 6792d8e to 733ee93 Compare July 17, 2026 09:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[feature](be) Add asynchronous file cache writes#65658

[feature](be) Add asynchronous file cache writes#65658
bobhan1 wants to merge 13 commits into
apache:masterfrom
bobhan1:feature/async-file-cache-write-phase1

bobhan1 commented Jul 15, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

bobhan1 commented Jul 17, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

github-actions Bot Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bobhan1 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Goals

Non-goals of this phase

Architecture

Component responsibilities

Important flows

1. Ordinary read with mixed coverage

2. Later reader reuses a pending buffer

3. Backpressure and asynchronous failure

4. Clear/remove and stale-task handling

5. Mode selection and compatibility

6. Worker lifecycle

Configuration

Observability

Tests

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 15, 2026

Uh oh!

bobhan1 commented Jul 17, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bobhan1 commented Jul 15, 2026 •

edited

Loading