Skip to content

perf(history): stop storing the request payload a third time - #4

Merged
adhikjoshi merged 1 commit into
mainfrom
perf/dont-store-payload-in-task-history
Aug 18, 2026
Merged

adhikjoshi merged 1 commit into
mainfrom
perf/dont-store-payload-in-task-history

Conversation

@adhikjoshi

@adhikjoshi adhikjoshi commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

What

Enqueueing writes the request payload three times:

key read back?
ml_tasks yes — the worker pops it and needs the args
task:{id} yes — requeueStuckProcessingTasks() rebuilds a stuck job from it
task_history:{id} no

getTaskHistory() and getTaskStats() only look at status and task_name. getTaskDetails() reads task:{id} and never touches history. So the third copy is written, held for TASK_HISTORY_RETENTION, and never used.

This PR strips payload from the history write only.

Why it matters

A generation carrying an inline base64 image pins three times its own size for a full retention window.

In production one caller pushed 17 requests of 11–17 MB in 30 seconds. The database returned:

OOM command not allowed when used memory > 'maxmemory'
Redis::rPush(): Send of 16923690 bytes failed with errno=32 Broken pipe

These databases are ~450 MB and many queues share one host, so the damage was not confined to that caller. Other server types began failing to enqueue with read error on connection, fell back to the direct-server path, and surfaced errors to unrelated users.

For reference, the database I could inspect was already doing everything right — maxmemory-policy=allkeys-lru, 102,548 keys already evicted, and a TTL on 6,296 of 6,297 keys. Eviction was not the missing piece; the volume was. Its key census also shows the duplication plainly: task 2,933 vs task_history 2,856, near 1:1.

What this does not fix

Multi-megabyte payloads should not travel through Redis at all. The durable fix is to upload them and enqueue a URL. This removes a third of the cost per request — the third that is pure retention — and buys headroom; it is not the cure.

Deliberately not stripped

  • ml_tasks — the worker pops it and needs the args.
  • task:{id} — requeueStuckProcessingTasks() rebuilds from it, so stripping here would silently requeue an argument-less job.

One coupling worth knowing about

The frontend resolves a finished generation through payload.data.args.0.output on whatever getTaskDetails() returns. That is safe only because getTaskDetails() reads task:{id}, which keeps its payload.

If getTaskDetails() is ever given a task_history:{id} fallback — and ModelQTest::testTaskHistoryWithErrorDetails already expects one — a stripped history would resolve those links to an empty array and report completed work as producing no output. The docblock says so at the point of the change.

Tests

New tests/Integration/TaskHistoryPayloadTest.php, 5 cases:

  • history loses the payload (marker absent, payload key gone)
  • history keeps the fields its readers use (status, task_name, task_id)
  • queue and task:{id} still carry the payload — the control case
  • history no longer scales with payload size
  • getTaskStats() still reports the task

Targeted run: 5 passed, 12 assertions.

Mutation-tested: revert the unset() and 2 of the 5 go red; restore it and all pass.

Full suite is 142 tests, 13 failures — all pre-existing on main (ModelQTest payload-shape assertions, StreamingTest, WorkerTest). Confirmed by stashing this change and re-running: baseline is 15 failures, which is these same 13 plus the 2 new tests failing without the fix. PHPStan is 35 errors before and after, and does not flag the new method.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Enqueueing writes the payload three times: into `ml_tasks` for the worker to
pop, into `task:{id}` because requeueStuckProcessingTasks() rebuilds a stuck
job from that key, and into `task_history:{id}`. Only the first two are read
back. getTaskHistory() and getTaskStats() look at `status` and `task_name`;
getTaskDetails() reads `task:{id}` and never touches history.

The third copy is what makes a large request expensive. A generation carrying
an inline base64 image pins three times its own size for TASK_HISTORY_RETENTION.
In production one caller pushed 17 requests of 11-17MB in 30 seconds, which was
enough to exhaust the database and return "OOM command not allowed when used
memory > 'maxmemory'". Those databases are ~450MB and are shared by many queues,
so the failure was not confined to the caller: other server types on the same
host began failing to enqueue with "read error on connection", fell back to the
direct-server path, and surfaced errors to unrelated users.

Dropping the history copy removes a third of the cost per request and all of
the part that is pure retention. It does not fix the underlying problem, which
is that multi-megabyte payloads travel through Redis at all; the durable fix is
to upload them and enqueue a URL.

Deliberately not stripped:
  - `ml_tasks`  — the worker pops it and needs the args.
  - `task:{id}` — requeueStuckProcessingTasks() rebuilds from it, so stripping
                  here would silently requeue an argument-less job.

The docblock records why history must stay payload-free only while
getTaskDetails() reads `task:{id}`: the frontend resolves finished work through
`payload.data.args.0.output`, so a history fallback plus a stripped history
would report completed generations as producing no output.

Tests cover both directions — history loses the payload, queue and task key keep
it — plus the fields the history readers actually use. Verified by reverting the
strip: two of the new tests go red, and pass again once restored.
@adhikjoshi
adhikjoshi merged commit 02690d4 into main Aug 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant