Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 21 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ One **primary** process and N **secondary** processes run on one machine.

**Pre-requisites**
- KDB-X, with `KDBX_HOME` set. The framework uses its `kx.log`, `di.timer` and `di.os` modules.
- Linux. Memory figures are read from `/proc`, and the kernel's memory cap uses cgroups v2.
- For the kernel's memory cap, a systemd user session (`systemd-run --user`). This part is optional: without it, start with `USE_CGROUP=0` and only the framework's cap applies.

**Set up:**
```bash
Expand All @@ -27,7 +29,7 @@ Then edit `config/default.env`. These are the settings to look at first:
| `PRIMARY_PORT` | the port the primary listens on |
| `NUM_SECONDARYS` | how many secondaries the primary starts |
| `GLOBAL_MEM_LIMIT` | the frameworks memory cap in bytes, for the primary and every secondary together; empty means no cap |
| `CGROUP_MEM_LIMIT` | the kernals memory cap in bytes, used when starting under a cgroup: above it, the kernal kills a secondary. Must be above `GLOBAL_MEM_LIMIT`, roughly x1.25 |
| `CGROUP_MEM_LIMIT` | the kernel's memory cap in bytes, used when starting under a cgroup: above it, the kernel kills a secondary. Must be above `GLOBAL_MEM_LIMIT`, roughly x1.25 |
| `RESULTS_DIR` | where task results are saved |
| `TASK_TYPES` | the task types users may run |
| `HDB_DIR` | an HDB for the secondaries to load, if your tasks need one |
Expand All @@ -48,7 +50,19 @@ x src/primary.q -p $PRIMARY_PORT
systemd-run --user --scope -p MemoryMax=$CGROUP_MEM_LIMIT -p MemorySwapMax=0 $KDBX_HOME/bin/q src/primary.q -p $PRIMARY_PORT
```

There are two caps. `GLOBAL_MEM_LIMIT` is the framework's: above it, dispatch pauses. `CGROUP_MEM_LIMIT` is the kernel's: above it, the kernel kills a process in the cgroup (a secondary, never the primary). The kernel's cap must be higher. With swap off, the kernel acts the moment a process tries to go past its cap, so if the two were equal, the framework would never get to pause. The gap gives tasks already running room to keep growing after dispatch has paused.
There are two caps. `GLOBAL_MEM_LIMIT` is the framework's: above it, dispatch pauses. `CGROUP_MEM_LIMIT` is the kernel's: above it, the kernel kills a process in the cgroup (a secondary, never the primary). The kernel's cap must be higher. With swap off, the kernel acts the moment a process tries to go past its cap, so if the two were equal, the framework would never get to pause. The gap gives tasks already running room to keep growing after dispatch has paused. Every secondary sets its `oom_score_adj` to 1000 when it starts, and the primary keeps 0, so when the cgroup runs out of memory the kernel picks a secondary.

Or use the scripts, which load the config themselves and run the primary in the background:
```bash
bin/startprimary.sh # under a cgroup capped at CGROUP_MEM_LIMIT
USE_CGROUP=0 bin/startprimary.sh # without the kernel's cap
bin/startsecondary.sh 9 # start one more secondary by hand
bin/stopall.sh # stop the primary and every secondary for this config
```
Each script takes the config file as its last argument (default `config/default.env`). Logs go to `logs/`, and the primary's pid to `logs/primary.pid`. To check the kernel's cap on a running primary:
```bash
cat /sys/fs/cgroup$(cut -d: -f3 /proc/$(cat logs/primary.pid)/cgroup)/memory.max
```


After starting the primary it will spin up secondaries itself, current default amount is 8.
Expand All @@ -59,7 +73,7 @@ To check that it's up, in the primary:
```

### Stopping primary and secondaries
To stop everything, quit the primary (`\\`): it asks its secondaries to exit, and kills any still running after `STOP_GRACE_PERIOD_SECS`. To start one more secondary from the primary, use `.primary.spawn <next available id>`.
To stop everything, quit the primary (`\\`): it asks its secondaries to exit, and kills any still running after `STOP_GRACE_PERIOD_SECS`. To start one more secondary from the primary, use `.primary.spawn <next available id>`. From a shell, `bin/stopall.sh` stops everything the same way, and kills anything still running after the grace period.

## Running a task

Expand Down Expand Up @@ -144,13 +158,16 @@ last memory reading in the table but is not counted in the total.
secondary. The secondary is restarted, up to `RESPAWN_MAX` times.
- **A task that runs too long:** it fails as `timeout` and isn't retried. Its secondary is
killed, since a busy q process can't be interrupted, and then restarted.
- **A secondary the kernel kills for memory** (over `CGROUP_MEM_LIMIT`): its task fails as `oom`
and isn't retried, since the same args would need the same memory again. Its `memafter` records
the kernel's cap. The secondary is restarted, up to `RESPAWN_MAX` times.

Per task, `timeout` and `retriesmax` can be set when it's submitted; otherwise
`TASK_TIMEOUT_MS` and `TASK_RETRIES_MAX` apply:
```q
.primary.newtask ([tasktype:`sleep; args:10; timeout:00:00:02.000; retriesmax:1i])
```
More detail, including the order things happen in, is in [How it works](docs/howitworks.md#failures).
More detail, including the order things happen in, is in [How it works](docs/howitworks.md#failures), and how a memory kill is spotted is in [Memory kills](docs/howitworks.md#memory-kills).

## Memory polling

Expand Down
27 changes: 21 additions & 6 deletions bin/startprimary.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,32 @@ set -a
source "$CONFIG"
set +a

: "${PRIMARY_PORT:?startPRIMARY: PRIMARY_PORT is not set in $CONFIG}"
: "${PRIMARY_PORT:?startprimary: PRIMARY_PORT is not set in $CONFIG}"
export QHOME="$KDBX_HOME"

# used for creating logs
# todo: have it create the absolute path
mkdir -p logs
echo "start primary: port $PRIMARY_PORT, config $CONFIG"
LOG="logs/primary.$(date -u +%Y.%m.%dD%H.%M.%S).log"

# nohup to run in backgorund if want to run interactively just use x src/primary.q
nohup "$KDBX_HOME/bin/q" src/primary.q -p "$PRIMARY_PORT" < /dev/null >> "$LOG" 2>&1 &
# Claude
# the command, wrapped in systemd-run when starting under a cgroup
USE_CGROUP="${USE_CGROUP:-1}"
CMD=("$KDBX_HOME/bin/q" src/primary.q -p "$PRIMARY_PORT")
if [[ "$USE_CGROUP" == 1 && -n "${CGROUP_MEM_LIMIT:-}" ]]; then
if ! systemctl --user show-environment > /dev/null 2>&1; then # checks if there is a systemd user manager so we can create a cgroup
echo "startprimary: USE_CGROUP=1 but there's no systemd user session to make a cgroup in; set USE_CGROUP=0 to start without the memory cap" >&2
exit 1
fi
# --scope runs the command itself in a new cgroup; MemorySwapMax=0 means kill rather than swap; --quiet keeps systemd-run's own message out of the log
# the kernel's cap is CGROUP_MEM_LIMIT, above GLOBAL_MEM_LIMIT, so the framework pauses dispatch before the kernel kills
CMD=(systemd-run --user --scope --quiet -p MemoryMax="$CGROUP_MEM_LIMIT" -p MemorySwapMax=0 "${CMD[@]}")
CAP="cgroup cap $CGROUP_MEM_LIMIT bytes, framework cap ${GLOBAL_MEM_LIMIT:-none}"
else
CAP="no cgroup cap"
fi

# nohup to run in background if want to run interactively just use x src/primary.q
nohup "${CMD[@]}" < /dev/null >> "$LOG" 2>&1 &

echo $! > logs/primary.pid
echo "startprimary: port $PRIMARY_PORT, config $CONFIG, pid $!, log $LOG"
echo "startprimary: port $PRIMARY_PORT, $CAP, config $CONFIG, pid $!, log $LOG"
21 changes: 8 additions & 13 deletions bin/startsecondary.sh
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,18 +1,16 @@
#!/usr/bin/env bash
# start one secondary process
# bin/startsecondary.sh 1 config/default.env
# id is 1, 2, 3 ... and decides the port: SECONDARY_BASE_PORT + id - 1
# start one secondary process by hand (the primary normally spawns them)
# usage: bin/startsecondary.sh <id> [config]
# secondaries don't listen on a port: each one connects to the primary, which sends it work down that connection
set -euo pipefail


# the id is required and must be a whole number, 1 or more
# ${1:-} gives "" when no argument was passed, so set -u doesn't abort before the message
ID="${1:-}"
CONFIG="${2:-config/default.env}"

# if no id provided exit
if [[ ! "$ID" =~ ^[1-9][0-9]*$ ]]; then
echo "no id provided usage example: bash bin/startsecondary.sh <id>" >&2
echo "startsecondary: usage: bin/startsecondary.sh <id> [config], id is 1 or more" >&2
exit 1
fi

Expand All @@ -21,20 +19,17 @@ set -a
source "$CONFIG"
set +a


: "${PRIMARY_PORT:?startsecondary: PRIMARY_PORT is not set in $CONFIG}"
export QHOME="$KDBX_HOME"
THREADS="${THREADS:-1}"

# used for creating logs
# todo: have it create the absolute path
mkdir -p logs
LOG="logs/sec$ID.$(date -u +%Y.%m.%dD%H.%M.%S).log"

# if THREADS not set it defaults to 1
THREADS=${THREADS:-1}

# nohup to run in backgorund if want to run interactively just use x src/secondary.q -id 1
# nohup to run in the background; to run interactively just use x src/secondary.q -id <id>
nohup "$KDBX_HOME/bin/q" src/secondary.q -id "$ID" -s "$THREADS" < /dev/null >> "$LOG" 2>&1 &


echo $! > "logs/sec$ID.pid"
echo "start secondary: id $ID, config $CONFIG, pid $!, log $LOG"
echo "startsecondary: id $ID, threads $THREADS, primary port $PRIMARY_PORT, config $CONFIG, pid $!, log $LOG"
60 changes: 60 additions & 0 deletions bin/stopall.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env bash

# Claude
# stop the primary and every secondary started with one config

# forces script to exit as soon a the command fails, or an unset variable is used, or a pipe fails
set -euo pipefail

CONFIG="${1:-config/default.env}"

# set -a exports every variable the file sets, so q can read them with getenv
set -a
source "$CONFIG"
set +a

# this user's primary and secondary processes for this config: PRIMARY_PORT in their environment,
# or, for a primary started without the config loaded, -p PRIMARY_PORT on its command line
ourpids() {
for pid in $(pgrep -u "$USER" -f 'src/(primary|secondary)\.q' || true); do
if tr '\0' '\n' < "/proc/$pid/environ" 2>/dev/null | grep -qx "PRIMARY_PORT=$PRIMARY_PORT" \
|| tr '\0' ' ' < "/proc/$pid/cmdline" 2>/dev/null | grep -q -- "-p $PRIMARY_PORT "; then
echo "$pid"
fi
done
}

before=$(ourpids)
if [[ -z "$before" ]]; then
echo "stopall: nothing running for PRIMARY_PORT $PRIMARY_PORT"
exit 0
fi
echo "stopall: running for PRIMARY_PORT $PRIMARY_PORT: pids $(echo $before)"

# 1. ask the primary to exit (2s to connect); if it isn't there or doesn't answer, go straight to the wait and kill
"$KDBX_HOME/bin/q" -q <<'EOF' || true
h:@[hopen; (`$"::",getenv`PRIMARY_PORT; 2000); 0Ni];
if[null h; -1 "stopall: could not connect to the primary, will kill instead"; exit 1];
-1 "stopall: asking the primary to exit";
neg[h]"exit 0"; neg[h][];
exit 0
EOF

# 2. wait for everything to exit: the primary's own grace period, plus 5s
wait=$(( ${STOP_GRACE_PERIOD_SECS:-5} + 5 ))
for ((i=0; i<wait; i++)); do
[[ -z "$(ourpids)" ]] && { echo "stopall: all stopped"; exit 0; }
sleep 1
done

# 3. anything still running: kill, then kill -9
left=$(ourpids)
echo "stopall: still running after ${wait}s, killing: pids $(echo $left)"
kill $left 2>/dev/null || true
sleep 2
left=$(ourpids)
if [[ -n "$left" ]]; then
echo "stopall: killing with -9: pids $(echo $left)"
kill -9 $left 2>/dev/null || true
fi
echo "stopall: all stopped"
23 changes: 20 additions & 3 deletions src/primary.q
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ system"l src/common/cgroup.q";
.primary.taskstates:`queued`dispatched`running`done`failed;
.primary.retryreasons:`$"," vs getenv`RETRY_REASONS;
.primary.respawnmax:"J"$getenv`RESPAWN_MAX; // restarts allowed per spawned secondary; 0 or unset = never
.primary.cgroupcap:"J"$getenv`CGROUP_MEM_LIMIT; // the kernel's cap (MemoryMax), recorded as an oom-killed task's memafter
.primary.oomkills:0N; // the cgroup's oom_kill count at the last check

// dict of secondaries ids to their pids that the primary started
.primary.spawned:(`long$())!`int$();
Expand Down Expand Up @@ -149,7 +151,7 @@ system"l src/common/cgroup.q";
.primary.stop:{
.log.info "stopping secondaries";
handles:exec handle from secondaries where not null handle, not status=`closed;
{[h] @[neg[h];(`.secondary.stop;::);{[e] ::}]} each handles;
{[h] @[{[h] neg[h](`.secondary.stop;::); neg[h][]}; h; {[e] ::}]} each handles;
.os.sleep .primary.stopgraceperiod;
left:p where .os.exists each "/proc/",/:string p:value .primary.spawned;
if[count left;
Expand Down Expand Up @@ -289,6 +291,15 @@ system"l src/common/cgroup.q";
.primary.spawn sid;
};

// checks if the kernal has killed a process in our cgroup since the last check
.primary.oomkilled:{
n:.cgroup.oomkills .primary.cgroupdir;
if[null n; :0b];
if[n<=.primary.oomkills; :0b];
.primary.oomkills+:1;
1b
};

// Claude with minor edits from me
// read our own memory and every live secondary's from /proc in one call, store it, and add a history row
// works for busy secondaries too: nothing is asked of them. A pid that has gone gets nulls until .z.pc closes it
Expand Down Expand Up @@ -330,14 +341,18 @@ system"l src/common/cgroup.q";


// if a secondary loses connection mark it as closed, fail its task and try to respawn it if under RESPAWN_MAX
// a task that fails due to oom is not retried
.z.pc:{[h]
secid: first exec id from secondaries where handle=h;
if[null secid; :()];
tid: first exec currenttask from `secondaries where id=secid;
update status:`closed, handle:0Ni, currenttask: 0N from `secondaries where id=secid;
.log.info "secondary ",string[secid]," has closed its connection to the primary";
oom:.primary.oomkilled[];
.log.info "secondary ",string[secid]," has closed its connection to the primary",$[oom; ": killed by the kernel for using too much memory"; ""];
if[not null tid;
r:([err:"secondary lost"; reason:`lost]);
r:$[oom;
([err:"secondary killed by the kernel for using too much memory"; reason:`oom; memafter:.primary.memcap^.primary.cgroupcap]);
([err:"secondary lost"; reason:`lost])];
.primary.taskupdate[tid; `failed; r];
.log.warn"task ",string[tid]," was on secondary ",string[secid]," marked it as failed due to ",string r`reason;
];
Expand All @@ -356,6 +371,8 @@ system"l src/common/cgroup.q";
.os.mkdir .primary.resultsdir;
.log.info "task results will be saved in ",.primary.resultsdir;
.primary.pollmem[];
.primary.cgroupdir:: .cgroup.dir[];
.primary.oomkills:: .cgroup.oomkills .primary.cgroupdir;
if[.primary.spawnsecondaries; .primary.spawn each 1+til .primary.numsecondaries];
.timer.addjob.custom[`stalecheck; {.primary.check[]}; (); 1; 2; ()!()];
.timer.addjob.custom[`dispatch; {.primary.dispatch[]}; (); 1; 2; ()!()];
Expand Down
9 changes: 9 additions & 0 deletions src/secondary.q
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,14 @@ if[count .secondary.hdb:getenv `HDB_DIR;
@[{[p;r] p set r; (1b; p; hcount p)}[path]; res; {[e] (0b; e)}]
};

// we will set any secondarys oom_score to 1000 so it will be shut down first if the OS is out of memory
.secondary.oomscore:{
oomscorepath:"/proc/",string[.z.i],"/oom_score_adj";
system "sh -c 'echo 1000 > ",oomscorepath,"'";
score:first system "cat ",oomscorepath;
$[score~"1000"; .log.info "oom_score_adj set to 1000"; .log.warn "could not set oom_score_adj, it is ",score];
};


// if the primary's connection closes kill the secondary
.z.pc:{[h]
Expand All @@ -95,6 +103,7 @@ if[count .secondary.hdb:getenv `HDB_DIR;

// start sequence
.secondary.start:{
.secondary.oomscore[];
.secondary.handle:: .secondary.connect[];
.secondary.resultsdir:: @[.secondary.handle; (`.primary.register; .secondary.info); {[error] .log.error "registering with the primary failed with error: ",error,". Will exit the process now"; exit 1}];
.log.info "registered with the primary, saving results in ",.secondary.resultsdir;
Expand Down
10 changes: 10 additions & 0 deletions tests/integration/oomscore_spec.q
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
// check the oom score of the secondaries
system "l tests/helpers/it.q";

.tst.desc["check the score of all secondaries oom score"]{
should["return 1000 for all secondaries"]{
pids:exec pid from .it.sec[];
paths:{"/proc/",string[x],"/oom_score_adj"} each pids;
1b musteq all raze 1000="J"$ {system x}each "cat ",/:paths
}
};
Loading