Skip to content

bench-300: adding retries, respawn and timeouts - #12

Merged
jamesrut1996 merged 5 commits into
mainfrom
bench-300-retries
Nov 17, 2026
Merged

jamesrut1996 merged 5 commits into
mainfrom
bench-300-retries

Conversation

@jamesrut1996

@jamesrut1996 jamesrut1996 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Add retry, respawn and timeout logic. Also added testing for the status function
This PR has oom and cgroup changes that have already been approved

  • editted .z.pc to try respawn a secondary when it goes down using .primary.respawnsecondary
  • a secondary will only respwn if it has not been respawned more than RESPAWN_MAX
  • a failed task will be retried if its failure reason is in RETRY_REASONS (we wouldnt want to retry something that caused oom)
  • a task will fail if it takes longer than TASK_TIMEOUT_MS

examples

// kill secondary while a task is running
.primary.newtask ([tasktype:`sleep; args:10])
2
\kill -9 1029249
q)2026.11.16D11:22:42.939085109 info PID[1029231] HOST[homer.aquaq.co.uk] secondary 1 has closed its connection to the primary
2026.11.16D11:22:42.939224785 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 2 from running to failed
2026.11.16D11:22:42.939353873 info PID[1029231] HOST[homer.aquaq.co.uk] retrying task 2, retry 1 of 3
2026.11.16D11:22:42.939452112 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 2 from failed to queued
2026.11.16D11:22:42.939654633 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 2 from queued to dispatched
2026.11.16D11:22:42.939746697 info PID[1029231] HOST[homer.aquaq.co.uk] sent task 2 (sleep) to secondary 3
2026.11.16D11:22:42.939815387 warn PID[1029231] HOST[homer.aquaq.co.uk] task 2 was on secondary 1 marking it as failed due to lost
2026.11.16D11:22:42.939889540 info PID[1029231] HOST[homer.aquaq.co.uk] restarting secondary 1, restart 1 of 3
2026.11.16D11:22:42.945670530 info PID[1029231] HOST[homer.aquaq.co.uk] running system command: /opt/kdbx/5.0/2026.07.23/bin/q src/secondary.q -id 1 -s 1 < /dev/null >> logs/sec1.2026.11.16D11.22.42.log 2>&1 & echo $!
2026.11.16D11:22:42.945802737 info PID[1029231] HOST[homer.aquaq.co.uk] spawned secondary 1 , pid 1033376
2026.11.16D11:22:42.946142168 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 2 from dispatched to running
2026.11.16D11:22:42.972899384 info PID[1029231] HOST[homer.aquaq.co.uk] secondary 1 registered at: 2026.11.16D11:22:42.972815863

q)2026.11.16D11:22:52.966813474 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 2 from running to done
select taskid, state, reason, retrycount, secondaryid from tasks
taskid state reason retrycount secondaryid
------------------------------------------
1      done         0          1          
2      done         1          3          
q).primary.showstatus[]
== primary status 2026.11.16D11:23:00.416933209  uptime 0D00:00:51.300100383 ==
memory: 17MB in use, 4096MB cap, 4079MB headroom
dispatch: running, paused 0 times for 0D00:00:00.000000000
secondaries: 8 free, 0 busy, 0 stale, 0 closed
id pid     status spawned task memmb restarts lastseen    
----------------------------------------------------------
1  1033376 free   1            1.9   1        00:00:00.540
2  1029256 free   1            1.9   0        00:00:00.353
3  1029263 free   1            2.1   0        00:00:01.048
4  1029271 free   1            1.9   0        00:00:00.038
5  1029275 free   1            1.9   0        00:00:00.336
6  1029281 free   1            1.9   0        00:00:01.031
7  1029286 free   1            1.9   0        00:00:00.925
8  1029290 free   1            1.9   0        00:00:00.419
tasks: 0 queued, 0 dispatched, 0 running, 2 done, 0 failed

// task fails with a timeout
q).primary.newtask ([tasktype:`sleep; args:10; timeout:00:00:02.000])
3
q)2026.11.16D11:23:12.311433792 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 3 from queued to dispatched
2026.11.16D11:23:12.311543430 info PID[1029231] HOST[homer.aquaq.co.uk] sent task 3 (sleep) to secondary 1
2026.11.16D11:23:12.311994078 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 3 from dispatched to running
2026.11.16D11:23:14.626300708 warn PID[1029231] HOST[homer.aquaq.co.uk] task 3 on secondary 1 has run past its timeout of 00:00:02.000, killing pid 1033376
2026.11.16D11:23:14.626529158 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 3 from running to failed
2026.11.16D11:23:14.629397966 info PID[1029231] HOST[homer.aquaq.co.uk] secondary 1 has closed its connection to the primary
2026.11.16D11:23:14.629515029 info PID[1029231] HOST[homer.aquaq.co.uk] restarting secondary 1, restart 2 of 3
2026.11.16D11:23:14.634473344 info PID[1029231] HOST[homer.aquaq.co.uk] running system command: /opt/kdbx/5.0/2026.07.23/bin/q src/secondary.q -id 1 -s 1 < /dev/null >> logs/sec1.2026.11.16D11.23.14.log 2>&1 & echo $!
2026.11.16D11:23:14.634559287 info PID[1029231] HOST[homer.aquaq.co.uk] spawned secondary 1 , pid 1038111
2026.11.16D11:23:14.665035995 info PID[1029231] HOST[homer.aquaq.co.uk] secondary 1 registered at: 2026.11.16D11:23:14.664943071
select taskid, state, reason, retrycount from tasks where taskid=3
taskid state  reason  retrycount
--------------------------------
3      failed timeout 0  

// a failed task doesnt get retried
q).primary.newtask ([tasktype:`fail; args:1])
4
q)2026.11.16D11:23:31.311720253 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 4 from queued to dispatched
2026.11.16D11:23:31.311835110 info PID[1029231] HOST[homer.aquaq.co.uk] sent task 4 (fail) to secondary 1
2026.11.16D11:23:31.312208405 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 4 from dispatched to running
2026.11.16D11:23:31.336982830 info PID[1029231] HOST[homer.aquaq.co.uk] updated task id 4 from running to failed
q)select taskid, state, reason, retrycount from tasks where taskid=4
taskid state  reason retrycount
-------------------------------
4      failed error  0         

// kill a secondary enough times and it wont come back up
q)exec id, pid from secondaries
id | 1       3       2       4       5       6       7       8      
pid| 1038111 1029263 1029256 1029271 1029275 1029281 1029286 1029290
q)\kill -9 1038111
q)2026.11.16D11:24:31.071061589 info PID[1029231] HOST[homer.aquaq.co.uk] secondary 1 has closed its connection to the primary
2026.11.16D11:24:31.071185935 info PID[1029231] HOST[homer.aquaq.co.uk] restarting secondary 1, restart 3 of 3
2026.11.16D11:24:31.076935019 info PID[1029231] HOST[homer.aquaq.co.uk] running system command: /opt/kdbx/5.0/2026.07.23/bin/q src/secondary.q -id 1 -s 1 < /dev/null >> logs/sec1.2026.11.16D11.24.31.log 2>&1 & echo $!
2026.11.16D11:24:31.077064461 info PID[1029231] HOST[homer.aquaq.co.uk] spawned secondary 1 , pid 1048058
2026.11.16D11:24:31.103931975 info PID[1029231] HOST[homer.aquaq.co.uk] secondary 1 registered at: 2026.11.16D11:24:31.103844034
exec id, pid from secondaries
id | 1       3       2       4       5       6       7       8      
pid| 1048058 1029263 1029256 1029271 1029275 1029281 1029286 1029290
q)kill -9 1048058
'kill
  [0]  kill -9 1048058
       ^
q)\kill -9 1048058
q)2026.11.16D11:24:43.168550666 info PID[1029231] HOST[homer.aquaq.co.uk] secondary 1 has closed its connection to the primary
2026.11.16D11:24:43.168623661 warn PID[1029231] HOST[homer.aquaq.co.uk] secondary 1 has been restarted 3 times already, max secondary restarts is 3

q)secondaries
id| handle pid     host              status lastheartbeat                 consecutivefails healthy spawned starttime                     registeredtime                tasktypes currenttask rssanon vmrss   memtime                      
--| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 |        1048058 homer.aquaq.co.uk closed 2026.11.16D11:24:42.707644492 0                1       1       2026.11.16D11:24:31.103817144 2026.11.16D11:24:31.103816991                       1978368 7294976 2026.11.16D11:24:42.707645565
3 | 12     1029263 homer.aquaq.co.uk free   2026.11.16D11:24:47.268464186 0                1       1       2026.11.16D11:22:09.166136426 2026.11.16D11:22:09.166136256                       2224128 7528448 2026.11.16D11:24:47.268465307
2 | 11     1029256 homer.aquaq.co.uk free   2026.11.16D11:24:46.362566604 0                1       1       2026.11.16D11:22:09.166267666 2026.11.16D11:22:09.166267528                       1978368 7245824 2026.11.16D11:24:46.823485696
4 | 13     1029271 homer.aquaq.co.uk free   2026.11.16D11:24:46.978468556 0                1       1       2026.11.16D11:22:09.172988417 2026.11.16D11:22:09.172988229                       1982464 7241728 2026.11.16D11:24:46.978469576
5 | 14     1029275 homer.aquaq.co.uk free   2026.11.16D11:24:46.780283614 0                1       1       2026.11.16D11:22:09.175332503 2026.11.16D11:22:09.175332357                       1978368 7245824 2026.11.16D11:24:46.823485696
6 | 15     1029281 homer.aquaq.co.uk free   2026.11.16D11:24:46.485934733 0                1       1       2026.11.16D11:22:09.180551062 2026.11.16D11:22:09.180550919                       1978368 7303168 2026.11.16D11:24:46.823485696
7 | 16     1029286 homer.aquaq.co.uk free   2026.11.16D11:24:46.791908824 0                1       1       2026.11.16D11:22:09.187022517 2026.11.16D11:22:09.187022308                       1978368 7245824 2026.11.16D11:24:46.823485696
8 | 17     1029290 homer.aquaq.co.uk free   2026.11.16D11:24:46.397486581 0                1       1       2026.11.16D11:22:09.192664609 2026.11.16D11:22:09.192664442                       1978368 7131136 2026.11.16D11:24:46.823485696
q)\\

@jamesrut1996
jamesrut1996 merged commit 90db366 into main Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants