Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,34 @@ A job is refused before anything runs if:
2. Add its name to `TASK_TYPES`.
3. Restart.

## Status

`.primary.status[]` returns a snapshot of the primary as a dict, for code to read:

| Key | What it holds |
|---|---|
| `time`, `uptime` | when the snapshot was taken (UTC) and how long the primary has been running |
| `memory` | `total` in use (the primary plus every live secondary, RssAnon, bytes), `cap` (`GLOBAL_MEM_LIMIT`), `headroom` (cap minus total), `primary` (the primary's own) |
| `dispatch` | `paused`, `pausecount`, `pausedtotal` (includes the current pause), `pausedsince` |
| `secondaries` | one row per secondary: id, pid, status, spawned, current task, rssanon, restarts, time since its last heartbeat (`lastseen`) |
| `seccounts` | how many secondaries are free, busy, stale and closed |
| `taskcounts` | how many tasks are queued, dispatched, running, done and failed |

`.primary.showstatus[]` prints the same snapshot for a person at the console:

== primary status 2026.11.13D13:51:09.408361778 uptime 0D00:01:29.987809969 ==
memory: 17MB in use, 4096MB cap, 4079MB headroom
dispatch: running, paused 0 times for 0D00:00:00.000000000
secondaries: 8 free, 0 busy, 0 stale, 0 closed
id pid status spawned task memmb restarts lastseen
----------------------------------------------------------
1 3775257 free 1 2.1 0 00:00:00.641
...
tasks: 0 queued, 0 dispatched, 0 running, 1 done, 1 failed

`restarts` stays 0 until restarting lost secondaries is added. A closed secondary keeps its
last memory reading in the table but is not counted in the total.

## Memory polling

All memory figures come from the kernel, in `/proc/<pid>/status`, in bytes. The framework doesn't use `.Q.w[]`: the kernel's figures are what it actually enforces, and they can be read from outside a process, which matters because a secondary running a task can't answer.
Expand Down
33 changes: 33 additions & 0 deletions src/primary.q
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ system"l src/common/cgroup.q";
.primary.pausedsince:0Np; // when the current pause started
.primary.pausedtotal:0D; // total time spent paused
.primary.pausecount:0; // how many times dispatch has paused
.primary.secondarystates:`free`busy`stale`closed;
.primary.taskstates:`queued`dispatched`running`done`failed;


// dict of secondaries ids to their pids that the primary started
Expand All @@ -50,6 +52,9 @@ system"l src/common/cgroup.q";
// the time column that gets updated at each task status
.primary.statustime:([dispatched: `dispatchedtime; running: `startedtime; done: `finishedtime; failed: `finishedtime]);

// restarts so far, by secondary id
.primary.respawns:(`long$())!`long$();

// when a secondary starts add it to the secondaries table
// example run:
// .primary.register ([id:5; pid:4000i; host:`localhost])
Expand Down Expand Up @@ -78,6 +83,34 @@ system"l src/common/cgroup.q";
.log.warn "no heartbeat from secondary ",(", " sv string stale)];
};

// how many of each state appear in the status col, done like this so can have 0 for failed say if no failed in the status col
.primary.countby:{[states;statuscol] states!sum each statuscol=/:states};

// a snapshot of the primarys status
// memory is in bytes headroom how far under the cap we are
.primary.status:{
secs:select id, pid, status, spawned, currenttask, rssanon, restarts:0^.primary.respawns id, lastseen:`time$.z.p-lastheartbeat from 0!secondaries;
tot:.primary.totalmem[];
mem:([total:tot; cap:.primary.memcap; headroom:.primary.memcap-tot; primary:.primary.selfmem]);
pt:.primary.pausedtotal+$[.primary.paused; .z.p-.primary.pausedsince; 0D];
disp:([paused:.primary.paused; pausecount:.primary.pausecount; pausedtotal:pt; pausedsince:.primary.pausedsince]);
([time: .z.p; uptime: .z.p - .primary.starttime; memory: mem; dispatch: disp; secondaries: secs; seccounts: .primary.countby[.primary.secondarystates;exec status from secondaries]; taskcounts: .primary.countby[.primary.taskstates;exec state from tasks]])
};

// the same snapshot, printed for a person at the console
.primary.showstatus:{
pristatus:.primary.status[];
fmt:{[d] ", " sv {[d;k] string[d k]," ",string k}[d] each key d};
-1 "== primary status ",string[pristatus`time]," uptime ",string[pristatus`uptime]," ==";
m:pristatus`memory;
-1 "memory: ",.primary.mb[m`total]," in use, ",$[null m`cap; "no cap"; .primary.mb[m`cap]," cap, ",.primary.mb[m`headroom]," headroom"];
d:pristatus`dispatch;
-1 "dispatch: ",$[d`paused; "PAUSED (memory over the cap) since ",string d`pausedsince; "running"],", paused ",string[d`pausecount]," times for ",string d`pausedtotal;
-1 "secondaries: ",fmt pristatus`seccounts;
show `id xasc select id, pid, status, spawned, task:currenttask, memmb:0.1*floor 0.5+10*rssanon%1048576, restarts, lastseen from pristatus`secondaries;
-1 "tasks: ",fmt pristatus`taskcounts;
};

// used to start a secondary process
.primary.spawncmd:{
" " sv (.primary.qbin; "src/secondary.q"; "-id"; string x; "-s"; string .primary.secondarythreads)
Expand Down