Skip to content

bench-300: adding jobs and task estimation - #14

Open
jamesrut1996 wants to merge 1 commit into
mainfrom
bench-300-jobs
Open

jamesrut1996 wants to merge 1 commit into
mainfrom
bench-300-jobs

Conversation

@jamesrut1996

@jamesrut1996 jamesrut1996 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Adding jobs which split up an approved task into individual tasks which run. These tasks will be saved down to disk under the jobId that ran them. The task estimation also uses past tasks to estimate if a new job/task will go over the approved limit turning it down. If part of a job fails the rest of that jobs tasks will be cancelled.

  • we run jobs with .primary.newjob
  • we can get their result paths with .primary.jobpaths which takes the job id number. We can then raze this if its a table to combine results. Results are saved somewhere like path/to/project/results/primary start time/job id/ task id
  • a task ran outside a job is saved somewhere like path/to/project/results/primary start time/nojob/ task id
  • .primary.estimatemem is used for estimating how much memory is required for a task based off previous runs. It finds the max memory used for that task type.
$ x src/primary.q -p $PRIMARY_PORT

// a job end to end
q)j:.primary.newjob[`sumtil; 1000 2000 3000; ()!()]
2026.11.18D11:35:40.362778939 info PID[1037028] HOST[homer.aquaq.co.uk] new job 1 for jrutledge: 3 sumtil tasks (1..3), priority 0
2026.11.18D11:35:40.384142623 info PID[1037028] HOST[homer.aquaq.co.uk] job 1 is done: 3 tasks, 0MB

q) jobs j
tasktype     | `sumtil
user         | `jrutledge
ntasks       | 3i
priority     | 0i
state        | `done
submittedtime| 2026.11.18D11:35:40.362697977
finishedtime | 2026.11.18D11:35:40.384052551
resultbytes  | 33
err          | ""
q) .primary.jobpaths j
`:/home/jrutledge/internalWork/Parallelisation-Project/results/2026.11.18D11.35.31.313968744/1/1`:/home/jrutledge/internalWork/Parallelisation-Project/results/2026.11.18D11.35.31.313968744/1/2`:/home/jrutledge/internalWork/Parallelisation-Project/results/202..
q) raze get each .primary.jobpaths j
499500 1999000 4498500


// a task outside a job
q)t:.primary.newtask ([tasktype:`sumtil; args:10])
q)tasks[t]`resultpath
`:/home/jrutledge/internalWork/Parallelisation-Project/results/2026.11.18D11.35.31.313968744/nojob/4


// failing job cancels any queued tasks
q)jf:.primary.newjob[`fail; til 20; ()!()]
2026.11.18D11:38:54.714392822 info PID[1037028] HOST[homer.aquaq.co.uk] new job 3 for jrutledge: 20 fail tasks (8..27), priority 0
2026.11.18D11:38:54.714806498 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 8 from queued to dispatched
2026.11.18D11:38:54.714906162 info PID[1037028] HOST[homer.aquaq.co.uk] sent task 8 (fail) to secondary 2
2026.11.18D11:38:54.715032592 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 9 from queued to dispatched
2026.11.18D11:38:54.715113422 info PID[1037028] HOST[homer.aquaq.co.uk] sent task 9 (fail) to secondary 1
2026.11.18D11:38:54.715227362 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 10 from queued to dispatched
...
2026.11.18D11:38:54.716720262 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 9 from dispatched to running
2026.11.18D11:38:54.716824809 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 12 from dispatched to running
2026.11.18D11:38:54.716919542 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 13 from dispatched to running
...
2026.11.18D11:38:54.741544802 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 8 from running to failed
2026.11.18D11:38:54.741805369 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 16 from queued to failed
2026.11.18D11:38:54.742074020 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 17 from queued to failed
...
2026.11.18D11:38:54.744030837 warn PID[1037028] HOST[homer.aquaq.co.uk] job 3: cancelled 12 queued tasks
2026.11.18D11:38:54.744098601 warn PID[1037028] HOST[homer.aquaq.co.uk] job 3 failed: task 8 failed (error): fail
2026.11.18D11:38:54.744274025 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 10 from running to failed
2026.11.18D11:38:54.744501290 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 9 from running to failed
2026.11.18D11:38:54.744706032 info PID[1037028] HOST[homer.aquaq.co.uk] updated task id 14 from running to failed
...
q) jobs jf
tasktype     | `fail
user         | `jrutledge
ntasks       | 20i
priority     | 0i
state        | `failed
submittedtime| 2026.11.18D11:38:54.714244960
finishedtime | 2026.11.18D11:38:54.741705268
resultbytes  | 0N
err          | "task 8 failed (error): fail"
q)select count i by reason from tasks where jobid=jf
reason   | x 
---------| --
cancelled| 12
error    | 8 


// the memory estimate
q).primary.newjob[`hold; enlist ([mb:300; secs:1]); ()!()]
q).primary.newjob[`hold; enlist ([mb:100; secs:1]); ()!()]
q) select taskid, memused:memafter-membefore, memestimate from tasks where tasktype=`hold
taskid memused   memestimate
----------------------------
28     319500288            
29     109719552 319500288  
q).primary.memcap:200*1048576
q).primary.newjob[`hold; enlist ([mb:10; secs:1]); ()!()]
2026.11.18D11:39:42.178620397 warn PID[1037028] HOST[homer.aquaq.co.uk] job refused for jrutledge: hold tasks need about 305MB, at or over the cap of 200MB
'job refused: hold tasks need about 305MB, at or over the cap of 200MB
  [0]  .primary.newjob[`hold; enlist ([mb:10; secs:1]); ()!()]
       ^

// start with a cgroup mem limit
$ CGROUP_MEM_LIMIT=1073741824 GLOBAL_MEM_LIMIT= systemd-run --user --scope -p MemoryMax=1073741824 -p MemorySwapMax=0 $KDBX_HOME/bin/q src/primary.q -p $PRIMARY_PORT

q) j:.primary.newjob[`hold; enlist ([mb:2000; secs:5]); ()!()]
2026.11.18D11:39:58.660984699 info PID[1067763] HOST[homer.aquaq.co.uk] new job 1 for jrutledge: 1 hold tasks (1..1), priority 0
2026.11.18D11:39:58.661266997 info PID[1067763] HOST[homer.aquaq.co.uk] updated task id 1 from queued to dispatched
2026.11.18D11:39:58.661365985 info PID[1067763] HOST[homer.aquaq.co.uk] sent task 1 (hold) to secondary 1
q)2026.11.18D11:39:58.661709540 info PID[1067763] HOST[homer.aquaq.co.uk] updated task id 1 from dispatched to running
2026.11.18D11:39:59.370530179 info PID[1067763] HOST[homer.aquaq.co.uk] secondary 1 has closed its connection to the primary: killed by the kernel for using too much memory
2026.11.18D11:39:59.370696972 info PID[1067763] HOST[homer.aquaq.co.uk] updated task id 1 from running to failed
2026.11.18D11:39:59.370895224 warn PID[1067763] HOST[homer.aquaq.co.uk] job 1 failed: task 1 failed (oom): secondary killed by the kernel for using too much memory
2026.11.18D11:39:59.371065760 warn PID[1067763] HOST[homer.aquaq.co.uk] task 1 was on secondary 1 marked it as failed due to oom
2026.11.18D11:39:59.371162785 info PID[1067763] HOST[homer.aquaq.co.uk] restarting secondary 1, restart 1 of 3
2026.11.18D11:39:59.375187325 info PID[1067763] HOST[homer.aquaq.co.uk] running system command: /opt/kdbx/5.0/2026.07.23/bin/q src/secondary.q -id 1 -s 1 < /dev/null >> logs/sec1.2026.11.18D11.39.59.log 2>&1 & echo $!
2026.11.18D11:39:59.375287738 info PID[1067763] HOST[homer.aquaq.co.uk] spawned secondary 1 , pid 1068517
2026.11.18D11:39:59.409569112 info PID[1067763] HOST[homer.aquaq.co.uk] secondary 1 registered at: 2026.11.18D11:39:59.409486746

q )jobs j
tasktype     | `hold
user         | `jrutledge
ntasks       | 1i
priority     | 0i
state        | `failed
submittedtime| 2026.11.18D11:39:58.660900903
finishedtime | 2026.11.18D11:39:59.370816761
resultbytes  | 0N
err          | "task 1 failed (oom): secondary killed by the kernel for using too much memory"
q) .primary.newjob[`hold; enlist ([mb:2000; secs:5]); ()!()]
2026.11.18D11:40:06.946206997 warn PID[1067763] HOST[homer.aquaq.co.uk] job refused for jrutledge: 1 of this job's hold args were killed for memory before, under a 1024MB cap (e.g. `mb`secs!2000 5); the kernel's cap is 1024MB
'job refused: 1 of this job's hold args were killed for memory before, under a 1024MB cap (e.g. `mb`secs!2000 5); the kernel's cap is 1024MB
  [0]  .primary.newjob[`hold; enlist ([mb:2000; secs:5]); ()!()]
       ^
q) .primary.cgroupcap:4294967296
q).primary.newjob[`hold; enlist ([mb:2000; secs:5]); ()!()]
2026.11.18D11:40:13.514056868 info PID[1067763] HOST[homer.aquaq.co.uk] new job 2 for jrutledge: 1 hold tasks (2..2), priority 0
2026.11.18D11:40:13.514313790 info PID[1067763] HOST[homer.aquaq.co.uk] updated task id 2 from queued to dispatched
2026.11.18D11:40:13.514420494 info PID[1067763] HOST[homer.aquaq.co.uk] sent task 2 (hold) to secondary 1
2
q)2026.11.18D11:40:13.514896887 info PID[1067763] HOST[homer.aquaq.co.uk] updated task id 2 from dispatched to running
2026.11.18D11:40:14.210289586 info PID[1067763] HOST[homer.aquaq.co.uk] secondary 1 has closed its connection to the primary: killed by the kernel for using too much memory
2026.11.18D11:40:14.210445286 info PID[1067763] HOST[homer.aquaq.co.uk] updated task id 2 from running to failed
2026.11.18D11:40:14.210620535 warn PID[1067763] HOST[homer.aquaq.co.uk] job 2 failed: task 2 failed (oom): secondary killed by the kernel for using too much memory
2026.11.18D11:40:14.210770326 warn PID[1067763] HOST[homer.aquaq.co.uk] task 2 was on secondary 1 marked it as failed due to oom
2026.11.18D11:40:14.210863284 info PID[1067763] HOST[homer.aquaq.co.uk] restarting secondary 1, restart 2 of 3
2026.11.18D11:40:14.214037336 info PID[1067763] HOST[homer.aquaq.co.uk] running system command: /opt/kdbx/5.0/2026.07.23/bin/q src/secondary.q -id 1 -s 1 < /dev/null >> logs/sec1.2026.11.18D11.40.14.log 2>&1 & echo $!
2026.11.18D11:40:14.214121861 info PID[1067763] HOST[homer.aquaq.co.uk] spawned secondary 1 , pid 1071127
2026.11.18D11:40:14.247419897 info PID[1067763] HOST[homer.aquaq.co.uk] secondary 1 registered at: 2026.11.18D11:40:14.247335264

q)select taskid, state, reason, memestimate from tasks where tasktype=`hold
taskid state  reason memestimate
--------------------------------
1      failed oom               
2      failed oom    1073741824 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants