Skip to content

style(workflow): dedupe unexpected timerFired warnings (fixes upstream #980)#1

Open
Aydeing wants to merge 1 commit into
mainfrom
fix/issue-980-timer-warning-spam
Open

style(workflow): dedupe unexpected timerFired warnings (fixes upstream #980)#1
Aydeing wants to merge 1 commit into
mainfrom
fix/issue-980-timer-warning-spam

Conversation

@Aydeing

@Aydeing Aydeing commented Jun 16, 2026

Copy link
Copy Markdown
Owner

Summary

Targets upstream issue dapr/python-sdk#980.

A misbehaving runtime can replay the same unexpected timerFired event many times across activity retries, producing 1000+ identical warning lines for a single workflow request. Fix dedupes the warning per (instance_id, timer_id) pair using a bounded LRU cache on _OrchestrationExecutor.

Changes

  • dapr/ext/workflow/_durabletask/worker.py
    • Add _UNEXPECTED_TIMER_WARN_CAPACITY = 1024 module constant.
    • Add _warned_unexpected_timers: OrderedDict[tuple[str, int], None] on _OrchestrationExecutor.__init__.
    • Add _warn_unexpected_timer_once(instance_id, timer_id) helper that emits the warning at most once per pair, evicting the oldest entry when capacity is exceeded.
    • Replace inline warning in the timerFired branch of process_event with the helper.
  • tests/ext/workflow/durabletask/test_orchestration_executor.py — 4 new tests:
    • same key warns once across 50 repeats
    • distinct keys each warn once
    • LRU evicts oldest when capacity reached (verified via monkeypatch on _UNEXPECTED_TIMER_WARN_CAPACITY)
    • end-to-end via process_event with 100 repeated timerFired events emits a single warning

Validation

  • uv run ruff check — clean
  • uv run ruff format — clean
  • uv run mypySuccess: no issues found in 175 source files
  • uv run pytest -m "not e2e" ./tests/ext/workflow/391 passed, 33 deselected

Safety / scope notes

  • Helper is intentionally scoped to the timerFired warning (matches issue title style: limit durabletask warnings for timers). Identical patterns exist for taskCompleted, taskFailed, childWorkflowInstanceCompleted, and childWorkflowInstanceFailed — they can be refactored to the same helper in a follow-up.
  • Capacity 1024 caps memory at ~tens of KB per worker. Entries are not cleared on orchestration completion (intentional — runs are typically short and the LRU bound makes manual cleanup unnecessary).
  • DCO sign-off included.

Repeated timerFired events with no matching pending task could flood logs
with thousands of identical warnings per workflow request (see dapr#980),
because the same unexpected event was re-emitted across activity retries.

Track each (instance_id, timer_id) pair seen by _OrchestrationExecutor in
a bounded LRU and emit the warning only on first occurrence. Capacity is
capped at 1024 entries so memory cannot grow without bound.

Fixes dapr#980

Signed-off-by: Aydeing <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant