Skip to content

bench-300: oom scoring and starting with cgroups - #13

Merged
jamesrut1996 merged 1 commit into
bench-300-retriesfrom
bench-300-oom
Nov 17, 2026
Merged

jamesrut1996 merged 1 commit into
bench-300-retriesfrom
bench-300-oom

Conversation

@jamesrut1996

Copy link
Copy Markdown
Contributor

Adding oom scoring and starting the primary/secondary with cgroups in the bash scripts. Also added a stop all script

  • oom score for secondaries set to 1000 so they are killed first in a oom error we use .secondary.oomscore for this
  • for a task that causes an oom it is not retried

Manual testing below

# start the primary
$ bin/startprimary.sh    
startprimary: port 45000, cgroup cap 5368709120 bytes, framework cap 4294967296, config config/default.env, pid 2684293, log logs/primary.2026.11.17D15.41.46.log

# connect to the primary and run a task
$ x
KDB-X 5.0 2026.07.23 Copyright (C) 1993-2026 Kx Systems
l64/ 64()core 385394MB jrutledge homer.aquaq.co.uk 127.0.1.1 EXPIRE 2027.03.26 dataintellect.com KXMS #104376

q)h:hopen 45000
q)h".primary.status[]`seccounts" 
free  | 8
busy  | 0
stale | 0
closed| 0
q)h(`.primary.newtask; ([tasktype:`sumtil; args:1000]))
1
q)h".primary.status[]`taskcounts" 
queued    | 0
dispatched| 0
running   | 0
done      | 1
failed    | 0

# start a secondary manually
$ bin/startsecondary.sh 9
startsecondary: id 9, threads 1, primary port 45000, config config/default.env, pid 2716045, log logs/sec9.2026.11.17D15.45.54.log

# can now see 9 secondaries
$ x
KDB-X 5.0 2026.07.23 Copyright (C) 1993-2026 Kx Systems
l64/ 64()core 385394MB jrutledge homer.aquaq.co.uk 127.0.1.1 EXPIRE 2027.03.26 dataintellect.com KXMS #104376

q)h:hopen 45000
q)h".primary.status[]`seccounts" 
free  | 9
busy  | 0
stale | 0
closed| 0

# kill all processes with the stop all script
$ bin/stopall.sh  
stopall: running for PRIMARY_PORT 45000: pids 2684293 2684349 2684355 2684360 2684366 2684372 2684377 2684383 2684392 2716045
stopall: asking the primary to exit
stopall: all stopped


# cause an oom failure, set cap at 1GB
$ set -a; source config/default.env; set +a
$ CGROUP_MEM_LIMIT=1073741824 GLOBAL_MEM_LIMIT= systemd-run --user --scope -p MemoryMax=1073741824 -p MemorySwapMax=0 $KDBX_HOME/bin/q src/primary.q -p $PRIMARY_PORT
...

q).primary.newtask ([tasktype:`hold; args:([mb:2000; secs:5])])
1
q)2026.11.17D15:47:12.304315639 info PID[2724047] HOST[homer.aquaq.co.uk] updated task id 1 from queued to dispatched
2026.11.17D15:47:12.304432425 info PID[2724047] HOST[homer.aquaq.co.uk] sent task 1 (hold) to secondary 1
2026.11.17D15:47:12.304866396 info PID[2724047] HOST[homer.aquaq.co.uk] updated task id 1 from dispatched to running
2026.11.17D15:47:13.136519795 info PID[2724047] HOST[homer.aquaq.co.uk] secondary 1 has closed its connection to the primary: killed by the kernel for using too much memory
2026.11.17D15:47:13.136675139 info PID[2724047] HOST[homer.aquaq.co.uk] updated task id 1 from running to failed
2026.11.17D15:47:13.136871492 warn PID[2724047] HOST[homer.aquaq.co.uk] task 1 was on secondary 1 marked it as failed due to oom
2026.11.17D15:47:13.136963076 info PID[2724047] HOST[homer.aquaq.co.uk] restarting secondary 1, restart 1 of 3
2026.11.17D15:47:13.143284941 info PID[2724047] HOST[homer.aquaq.co.uk] running system command: /opt/kdbx/5.0/2026.07.23/bin/q src/secondary.q -id 1 -s 1 < /dev/null >> logs/sec1.2026.11.17D15.47.13.log 2>&1 & echo $!
2026.11.17D15:47:13.143385905 info PID[2724047] HOST[homer.aquaq.co.uk] spawned secondary 1 , pid 2725459
2026.11.17D15:47:13.176066176 info PID[2724047] HOST[homer.aquaq.co.uk] secondary 1 registered at: 2026.11.17D15:47:13.175984832

q)select taskid, state, reason, retrycount, memafter from tasks 
taskid state  reason retrycount memafter  
------------------------------------------
1      failed oom    0          1073741824
q).primary.showstatus[]   
== primary status 2026.11.17D15:47:21.736034719  uptime 0D00:00:16.926231259 ==
memory: 17MB in use, no cap
dispatch: running, paused 0 times for 0D00:00:00.000000000
secondaries: 8 free, 0 busy, 0 stale, 0 closed
id pid     status spawned task memmb restarts lastseen    
----------------------------------------------------------
1  2725459 free   1            1.9   1        00:00:00.160
2  2724065 free   1            1.9   0        00:00:00.971
3  2724069 free   1            1.9   0        00:00:00.873
4  2724073 free   1            1.9   0        00:00:00.958
5  2724077 free   1            1.9   0        00:00:00.952
6  2724084 free   1            1.9   0        00:00:00.648
7  2724096 free   1            1.9   0        00:00:00.641
8  2724105 free   1            1.9   0        00:00:00.636
tasks: 0 queued, 0 dispatched, 0 running, 0 done, 1 failed
q).primary.newtask ([tasktype:`sumtil; args:1000]) 
2
q)2026.11.17D15:47:27.204074366 info PID[2724047] HOST[homer.aquaq.co.uk] updated task id 2 from queued to dispatched
2026.11.17D15:47:27.204185100 info PID[2724047] HOST[homer.aquaq.co.uk] sent task 2 (sumtil) to secondary 1
2026.11.17D15:47:27.204584111 info PID[2724047] HOST[homer.aquaq.co.uk] updated task id 2 from dispatched to running
2026.11.17D15:47:27.222640934 info PID[2724047] HOST[homer.aquaq.co.uk] updated task id 2 from running to done

q).primary.showstatus[]   
== primary status 2026.11.17D15:47:29.943793583  uptime 0D00:00:25.133990116 ==
memory: 17MB in use, no cap
dispatch: running, paused 0 times for 0D00:00:00.000000000
secondaries: 8 free, 0 busy, 0 stale, 0 closed
id pid     status spawned task memmb restarts lastseen    
----------------------------------------------------------
1  2725459 free   1            2.1   1        00:00:00.969
2  2724065 free   1            1.9   0        00:00:00.479
3  2724069 free   1            1.9   0        00:00:00.579
4  2724073 free   1            1.9   0        00:00:00.665
5  2724077 free   1            1.9   0        00:00:00.759
6  2724084 free   1            1.9   0        00:00:00.155
7  2724096 free   1            1.9   0        00:00:00.248
8  2724105 free   1            1.9   0        00:00:00.243
tasks: 0 queued, 0 dispatched, 0 running, 1 done, 1 failed
q)\\
2026.11.17D15:47:32.535604978 info PID[2724047] HOST[homer.aquaq.co.uk] stopping secondaries
2026.11.17D15:47:37.572190052 info PID[2724047] HOST[homer.aquaq.co.uk] stopped

@mkeenan-di mkeenan-di left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good to me

@jamesrut1996
jamesrut1996 merged commit 3a0e0b8 into bench-300-retries Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants