Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ One **primary** process and N **secondary** processes run on one machine.

**Pre-requisites**
- KDB-X, with `KDBX_HOME` set. The framework uses its `kx.log`, `di.timer` and `di.os` modules.
- Linux. Memory figures are read from `/proc`, and the kernel's memory cap uses cgroups v2.
- For the kernel's memory cap, a systemd user session (`systemd-run --user`). This part is optional: without it, start with `USE_CGROUP=0` and only the framework's cap applies.

**Set up:**
```bash
Expand All @@ -27,7 +29,7 @@ Then edit `config/default.env`. These are the settings to look at first:
| `PRIMARY_PORT` | the port the primary listens on |
| `NUM_SECONDARYS` | how many secondaries the primary starts |
| `GLOBAL_MEM_LIMIT` | the frameworks memory cap in bytes, for the primary and every secondary together; empty means no cap |
| `CGROUP_MEM_LIMIT` | the kernals memory cap in bytes, used when starting under a cgroup: above it, the kernal kills a secondary. Must be above `GLOBAL_MEM_LIMIT`, roughly x1.25 |
| `CGROUP_MEM_LIMIT` | the kernel's memory cap in bytes, used when starting under a cgroup: above it, the kernel kills a secondary. Must be above `GLOBAL_MEM_LIMIT`, roughly x1.25 |
| `RESULTS_DIR` | where task results are saved |
| `TASK_TYPES` | the task types users may run |
| `HDB_DIR` | an HDB for the secondaries to load, if your tasks need one |
Expand All @@ -48,7 +50,19 @@ x src/primary.q -p $PRIMARY_PORT
systemd-run --user --scope -p MemoryMax=$CGROUP_MEM_LIMIT -p MemorySwapMax=0 $KDBX_HOME/bin/q src/primary.q -p $PRIMARY_PORT
```

There are two caps. `GLOBAL_MEM_LIMIT` is the framework's: above it, dispatch pauses. `CGROUP_MEM_LIMIT` is the kernel's: above it, the kernel kills a process in the cgroup (a secondary, never the primary). The kernel's cap must be higher. With swap off, the kernel acts the moment a process tries to go past its cap, so if the two were equal, the framework would never get to pause. The gap gives tasks already running room to keep growing after dispatch has paused.
There are two caps. `GLOBAL_MEM_LIMIT` is the framework's: above it, dispatch pauses. `CGROUP_MEM_LIMIT` is the kernel's: above it, the kernel kills a process in the cgroup (a secondary, never the primary). The kernel's cap must be higher. With swap off, the kernel acts the moment a process tries to go past its cap, so if the two were equal, the framework would never get to pause. The gap gives tasks already running room to keep growing after dispatch has paused. Every secondary sets its `oom_score_adj` to 1000 when it starts, and the primary keeps 0, so when the cgroup runs out of memory the kernel picks a secondary.

Or use the scripts, which load the config themselves and run the primary in the background:
```bash
bin/startprimary.sh # under a cgroup capped at CGROUP_MEM_LIMIT
USE_CGROUP=0 bin/startprimary.sh # without the kernel's cap
bin/startsecondary.sh 9 # start one more secondary by hand
bin/stopall.sh # stop the primary and every secondary for this config
```
Each script takes the config file as its last argument (default `config/default.env`). Logs go to `logs/`, and the primary's pid to `logs/primary.pid`. To check the kernel's cap on a running primary:
```bash
cat /sys/fs/cgroup$(cut -d: -f3 /proc/$(cat logs/primary.pid)/cgroup)/memory.max
```


After starting the primary it will spin up secondaries itself, current default amount is 8.
Expand All @@ -59,7 +73,7 @@ To check that it's up, in the primary:
```

### Stopping primary and secondaries
To stop everything, quit the primary (`\\`): it asks its secondaries to exit, and kills any still running after `STOP_GRACE_PERIOD_SECS`. To start one more secondary from the primary, use `.primary.spawn <next available id>`.
To stop everything, quit the primary (`\\`): it asks its secondaries to exit, and kills any still running after `STOP_GRACE_PERIOD_SECS`. To start one more secondary from the primary, use `.primary.spawn <next available id>`. From a shell, `bin/stopall.sh` stops everything the same way, and kills anything still running after the grace period.

## Running a task

Expand Down Expand Up @@ -137,6 +151,24 @@ A job is refused before anything runs if:
`restarts` stays 0 until restarting lost secondaries is added. A closed secondary keeps its
last memory reading in the table but is not counted in the total.

## Failures

- **Errors** in a task's own code are reported (`failed`, reason `error`) and not retried.
- **A secondary that dies mid-task:** its task fails as `lost` and is retried on another
secondary. The secondary is restarted, up to `RESPAWN_MAX` times.
- **A task that runs too long:** it fails as `timeout` and isn't retried. Its secondary is
killed, since a busy q process can't be interrupted, and then restarted.
- **A secondary the kernel kills for memory** (over `CGROUP_MEM_LIMIT`): its task fails as `oom`
and isn't retried, since the same args would need the same memory again. Its `memafter` records
the kernel's cap. The secondary is restarted, up to `RESPAWN_MAX` times.

Per task, `timeout` and `retriesmax` can be set when it's submitted; otherwise
`TASK_TIMEOUT_MS` and `TASK_RETRIES_MAX` apply:
```q
.primary.newtask ([tasktype:`sleep; args:10; timeout:00:00:02.000; retriesmax:1i])
```
More detail, including the order things happen in, is in [How it works](docs/howitworks.md#failures), and how a memory kill is spotted is in [Memory kills](docs/howitworks.md#memory-kills).

## Memory polling

All memory figures come from the kernel, in `/proc/<pid>/status`, in bytes. The framework doesn't use `.Q.w[]`: the kernel's figures are what it actually enforces, and they can be read from outside a process, which matters because a secondary running a task can't answer.
Expand Down Expand Up @@ -166,6 +198,7 @@ select id, status, rssanon, memtime from secondaries

**A known overstatement.** A task's `memafter − membefore` includes file pages it read: the peak counts them, but "before" doesn't. For a task that reads an HDB, that's the heap it used **plus** the pages it touched: on a 31-day query, about 145MB more. For tasks that only compute, it's accurate. The kernel keeps no peak for the heap alone, and q's `.Q.w[]`peak` can't be reset for each task, so VmHWM is the only per-task peak available. The error is on the safe side: the cgroup counts file pages too, and an estimate thats too high holds tasks back rather than letting too many run.


## Tests

```bash
Expand Down
27 changes: 21 additions & 6 deletions bin/startprimary.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,32 @@ set -a
source "$CONFIG"
set +a

: "${PRIMARY_PORT:?startPRIMARY: PRIMARY_PORT is not set in $CONFIG}"
: "${PRIMARY_PORT:?startprimary: PRIMARY_PORT is not set in $CONFIG}"
export QHOME="$KDBX_HOME"

# used for creating logs
# todo: have it create the absolute path
mkdir -p logs
echo "start primary: port $PRIMARY_PORT, config $CONFIG"
LOG="logs/primary.$(date -u +%Y.%m.%dD%H.%M.%S).log"

# nohup to run in backgorund if want to run interactively just use x src/primary.q
nohup "$KDBX_HOME/bin/q" src/primary.q -p "$PRIMARY_PORT" < /dev/null >> "$LOG" 2>&1 &
# Claude
# the command, wrapped in systemd-run when starting under a cgroup
USE_CGROUP="${USE_CGROUP:-1}"
CMD=("$KDBX_HOME/bin/q" src/primary.q -p "$PRIMARY_PORT")
if [[ "$USE_CGROUP" == 1 && -n "${CGROUP_MEM_LIMIT:-}" ]]; then
if ! systemctl --user show-environment > /dev/null 2>&1; then # checks if there is a systemd user manager so we can create a cgroup
echo "startprimary: USE_CGROUP=1 but there's no systemd user session to make a cgroup in; set USE_CGROUP=0 to start without the memory cap" >&2
exit 1
fi
# --scope runs the command itself in a new cgroup; MemorySwapMax=0 means kill rather than swap; --quiet keeps systemd-run's own message out of the log
# the kernel's cap is CGROUP_MEM_LIMIT, above GLOBAL_MEM_LIMIT, so the framework pauses dispatch before the kernel kills
CMD=(systemd-run --user --scope --quiet -p MemoryMax="$CGROUP_MEM_LIMIT" -p MemorySwapMax=0 "${CMD[@]}")
CAP="cgroup cap $CGROUP_MEM_LIMIT bytes, framework cap ${GLOBAL_MEM_LIMIT:-none}"
else
CAP="no cgroup cap"
fi

# nohup to run in background if want to run interactively just use x src/primary.q
nohup "${CMD[@]}" < /dev/null >> "$LOG" 2>&1 &

echo $! > logs/primary.pid
echo "startprimary: port $PRIMARY_PORT, config $CONFIG, pid $!, log $LOG"
echo "startprimary: port $PRIMARY_PORT, $CAP, config $CONFIG, pid $!, log $LOG"
21 changes: 8 additions & 13 deletions bin/startsecondary.sh
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,18 +1,16 @@
#!/usr/bin/env bash
# start one secondary process
# bin/startsecondary.sh 1 config/default.env
# id is 1, 2, 3 ... and decides the port: SECONDARY_BASE_PORT + id - 1
# start one secondary process by hand (the primary normally spawns them)
# usage: bin/startsecondary.sh <id> [config]
# secondaries don't listen on a port: each one connects to the primary, which sends it work down that connection
set -euo pipefail


# the id is required and must be a whole number, 1 or more
# ${1:-} gives "" when no argument was passed, so set -u doesn't abort before the message
ID="${1:-}"
CONFIG="${2:-config/default.env}"

# if no id provided exit
if [[ ! "$ID" =~ ^[1-9][0-9]*$ ]]; then
echo "no id provided usage example: bash bin/startsecondary.sh <id>" >&2
echo "startsecondary: usage: bin/startsecondary.sh <id> [config], id is 1 or more" >&2
exit 1
fi

Expand All @@ -21,20 +19,17 @@ set -a
source "$CONFIG"
set +a


: "${PRIMARY_PORT:?startsecondary: PRIMARY_PORT is not set in $CONFIG}"
export QHOME="$KDBX_HOME"
THREADS="${THREADS:-1}"

# used for creating logs
# todo: have it create the absolute path
mkdir -p logs
LOG="logs/sec$ID.$(date -u +%Y.%m.%dD%H.%M.%S).log"

# if THREADS not set it defaults to 1
THREADS=${THREADS:-1}

# nohup to run in backgorund if want to run interactively just use x src/secondary.q -id 1
# nohup to run in the background; to run interactively just use x src/secondary.q -id <id>
nohup "$KDBX_HOME/bin/q" src/secondary.q -id "$ID" -s "$THREADS" < /dev/null >> "$LOG" 2>&1 &


echo $! > "logs/sec$ID.pid"
echo "start secondary: id $ID, config $CONFIG, pid $!, log $LOG"
echo "startsecondary: id $ID, threads $THREADS, primary port $PRIMARY_PORT, config $CONFIG, pid $!, log $LOG"
60 changes: 60 additions & 0 deletions bin/stopall.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#!/usr/bin/env bash

# Claude
# stop the primary and every secondary started with one config

# forces script to exit as soon a the command fails, or an unset variable is used, or a pipe fails
set -euo pipefail

CONFIG="${1:-config/default.env}"

# set -a exports every variable the file sets, so q can read them with getenv
set -a
source "$CONFIG"
set +a

# this user's primary and secondary processes for this config: PRIMARY_PORT in their environment,
# or, for a primary started without the config loaded, -p PRIMARY_PORT on its command line
ourpids() {
for pid in $(pgrep -u "$USER" -f 'src/(primary|secondary)\.q' || true); do
if tr '\0' '\n' < "/proc/$pid/environ" 2>/dev/null | grep -qx "PRIMARY_PORT=$PRIMARY_PORT" \
|| tr '\0' ' ' < "/proc/$pid/cmdline" 2>/dev/null | grep -q -- "-p $PRIMARY_PORT "; then
echo "$pid"
fi
done
}

before=$(ourpids)
if [[ -z "$before" ]]; then
echo "stopall: nothing running for PRIMARY_PORT $PRIMARY_PORT"
exit 0
fi
echo "stopall: running for PRIMARY_PORT $PRIMARY_PORT: pids $(echo $before)"

# 1. ask the primary to exit (2s to connect); if it isn't there or doesn't answer, go straight to the wait and kill
"$KDBX_HOME/bin/q" -q <<'EOF' || true
h:@[hopen; (`$"::",getenv`PRIMARY_PORT; 2000); 0Ni];
if[null h; -1 "stopall: could not connect to the primary, will kill instead"; exit 1];
-1 "stopall: asking the primary to exit";
neg[h]"exit 0"; neg[h][];
exit 0
EOF

# 2. wait for everything to exit: the primary's own grace period, plus 5s
wait=$(( ${STOP_GRACE_PERIOD_SECS:-5} + 5 ))
for ((i=0; i<wait; i++)); do
[[ -z "$(ourpids)" ]] && { echo "stopall: all stopped"; exit 0; }
sleep 1
done

# 3. anything still running: kill, then kill -9
left=$(ourpids)
echo "stopall: still running after ${wait}s, killing: pids $(echo $left)"
kill $left 2>/dev/null || true
sleep 2
left=$(ourpids)
if [[ -n "$left" ]]; then
echo "stopall: killing with -9: pids $(echo $left)"
kill -9 $left 2>/dev/null || true
fi
echo "stopall: all stopped"
Loading