Repository navigation
perf(history): stop storing the request payload a third time - #4
Merged
Merged
Conversation
Enqueueing writes the payload three times: into `ml_tasks` for the worker to
pop, into `task:{id}` because requeueStuckProcessingTasks() rebuilds a stuck
job from that key, and into `task_history:{id}`. Only the first two are read
back. getTaskHistory() and getTaskStats() look at `status` and `task_name`;
getTaskDetails() reads `task:{id}` and never touches history.
The third copy is what makes a large request expensive. A generation carrying
an inline base64 image pins three times its own size for TASK_HISTORY_RETENTION.
In production one caller pushed 17 requests of 11-17MB in 30 seconds, which was
enough to exhaust the database and return "OOM command not allowed when used
memory > 'maxmemory'". Those databases are ~450MB and are shared by many queues,
so the failure was not confined to the caller: other server types on the same
host began failing to enqueue with "read error on connection", fell back to the
direct-server path, and surfaced errors to unrelated users.
Dropping the history copy removes a third of the cost per request and all of
the part that is pure retention. It does not fix the underlying problem, which
is that multi-megabyte payloads travel through Redis at all; the durable fix is
to upload them and enqueue a URL.
Deliberately not stripped:
- `ml_tasks` — the worker pops it and needs the args.
- `task:{id}` — requeueStuckProcessingTasks() rebuilds from it, so stripping
here would silently requeue an argument-less job.
The docblock records why history must stay payload-free only while
getTaskDetails() reads `task:{id}`: the frontend resolves finished work through
`payload.data.args.0.output`, so a history fallback plus a stripped history
would report completed generations as producing no output.
Tests cover both directions — history loses the payload, queue and task key keep
it — plus the fields the history readers actually use. Verified by reverting the
strip: two of the new tests go red, and pass again once restored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Enqueueing writes the request payload three times:
ml_taskstask:{id}requeueStuckProcessingTasks()rebuilds a stuck job from ittask_history:{id}getTaskHistory()andgetTaskStats()only look atstatusandtask_name.getTaskDetails()readstask:{id}and never touches history. So the third copy is written, held forTASK_HISTORY_RETENTION, and never used.This PR strips
payloadfrom the history write only.Why it matters
A generation carrying an inline base64 image pins three times its own size for a full retention window.
In production one caller pushed 17 requests of 11–17 MB in 30 seconds. The database returned:
These databases are ~450 MB and many queues share one host, so the damage was not confined to that caller. Other server types began failing to enqueue with
read error on connection, fell back to the direct-server path, and surfaced errors to unrelated users.For reference, the database I could inspect was already doing everything right —
maxmemory-policy=allkeys-lru, 102,548 keys already evicted, and a TTL on 6,296 of 6,297 keys. Eviction was not the missing piece; the volume was. Its key census also shows the duplication plainly:task2,933 vstask_history2,856, near 1:1.What this does not fix
Multi-megabyte payloads should not travel through Redis at all. The durable fix is to upload them and enqueue a URL. This removes a third of the cost per request — the third that is pure retention — and buys headroom; it is not the cure.
Deliberately not stripped
ml_tasks— the worker pops it and needs the args.task:{id}—requeueStuckProcessingTasks()rebuilds from it, so stripping here would silently requeue an argument-less job.One coupling worth knowing about
The frontend resolves a finished generation through
payload.data.args.0.outputon whatevergetTaskDetails()returns. That is safe only becausegetTaskDetails()readstask:{id}, which keeps its payload.If
getTaskDetails()is ever given atask_history:{id}fallback — andModelQTest::testTaskHistoryWithErrorDetailsalready expects one — a stripped history would resolve those links to an empty array and report completed work as producing no output. The docblock says so at the point of the change.Tests
New
tests/Integration/TaskHistoryPayloadTest.php, 5 cases:payloadkey gone)status,task_name,task_id)task:{id}still carry the payload — the control casegetTaskStats()still reports the taskTargeted run: 5 passed, 12 assertions.
Mutation-tested: revert the
unset()and 2 of the 5 go red; restore it and all pass.Full suite is 142 tests, 13 failures — all pre-existing on
main(ModelQTestpayload-shape assertions,StreamingTest,WorkerTest). Confirmed by stashing this change and re-running: baseline is 15 failures, which is these same 13 plus the 2 new tests failing without the fix. PHPStan is 35 errors before and after, and does not flag the new method.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.