Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 33 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ One **primary** process and N **secondary** processes run on one machine.

- **The primary** owns the task queue. It decides which task runs where and watches memory against one cap for the primary and every secondary together (`GLOBAL_MEM_LIMIT`). It holds back new work while memory is over that cap.
- **The secondaries** run tasks, one at a time. They report back to the primary: status, memory, results. After each task they run `.Q.gc[]`, so its memory is returned before the next one.
- **Jobs:** work is submitted as a job, one task type and a list of args. The primary makes one task per arg and spreads them across the secondaries. Each secondary saves its result to disk and returns only the path, so results never pass through the primary.
- **Global memory cap:** the primary reads its own and every secondary's memory every second, busy secondaries included. While the total is over `GLOBAL_MEM_LIMIT`, no new task is sent; dispatch resumes once memory falls back under. Tasks already running carry on.

## Getting started
Expand Down Expand Up @@ -75,9 +76,10 @@ To check that it's up, in the primary:
### Stopping primary and secondaries
To stop everything, quit the primary (`\\`): it asks its secondaries to exit, and kills any still running after `STOP_GRACE_PERIOD_SECS`. To start one more secondary from the primary, use `.primary.spawn <next available id>`. From a shell, `bin/stopall.sh` stops everything the same way, and kills anything still running after the grace period.

## Running a task
## Running a job

Tasks are submitted as **jobs**, from any q session.
Work is submitted as **jobs**, from any q session. A job is one task type and a list of args: the primary
makes one task per arg and runs them across the secondaries.
```q
h:hopen 45000

Expand All @@ -91,15 +93,16 @@ h({jobs x}; j)
paths:h(`.primary.jobpaths; j)
raze get each paths
```
For several inputs per task, pass a table: each row becomes one task's args, as a dict.

**Params,** for every task of the job:
**Params,** for every task of the job (`()!()` for none):

| Param | Default | Meaning |
|---|---|---|
| `priority` | 0 | higher goes first; ties go oldest first |
| `timeout` | `TASK_TIMEOUT_MS` | a task running longer than this has its secondary killed |
| `retriesmax` | `TASK_RETRIES_MAX` | how many retries, for the reasons in `RETRY_REASONS` |
| `memestimate` | learnt | the memory each task is expected to use, in bytes |
| Param | Type | Default | Meaning |
|---|---|---|---|
| `priority` | int | 0 | higher goes first; ties go oldest first |
| `timeout` | time | `TASK_TIMEOUT_MS` | a task running longer than this has its secondary killed |
| `retriesmax` | int | `TASK_RETRIES_MAX` | how many retries, for the reasons in `RETRY_REASONS` |
| `memestimate` | long | learnt | the memory each task is expected to use, in bytes; learnt from earlier tasks of the type if not given |

For example:
```q
Expand All @@ -108,15 +111,28 @@ h(`.primary.newjob; `vwap; 2014.04.21 2014.04.22; ([priority:10; timeout:00:00:3

**Other calls:**
```q
h(`.primary.canceljob; j) / cancel the job's queued tasks
h({select taskid, state, reason, err from tasks where jobid=x}; j)
h".primary.status[]" / the status as a dict
h({select taskid, state, reason, err from tasks where jobid=x}; j) / the job's tasks
h(`.primary.canceljob; j) / cancel its queued tasks
h".primary.status[]" / the status as a dict
```

A job is refused before anything runs if:
- its task type isn't approved; or
- earlier tasks of the type needed more memory than the cap; or
- any of its args were killed for memory before, under the same cap or a lower one. Raising the cap lets them run again.
**A job is refused before anything runs** (the call errors and no task is made) if:
- its task type isn't approved, or it has no args;
- a param is unknown, or of the wrong type (`priority` and `retriesmax` accept any whole number);
- earlier tasks of its type needed at least `GLOBAL_MEM_LIMIT`, so it could never fit;
- any of its args were killed for memory before, under a kernel cap (`CGROUP_MEM_LIMIT`) at least as
high as now. Raising that cap lets them run again.

**A job fails** as soon as one of its tasks fails for good, that is, after any retries. Its queued tasks
are then cancelled and never sent; tasks already running finish. `err` on the job says which task failed
and why.

**Results** are saved under `RESULTS_DIR/<primary start time>/<jobid>/<taskid>`. A task submitted on its
own, outside a job, goes under `RESULTS_DIR/<primary start time>/nojob/<taskid>`.

Each job and its tasks record the user who submitted it (`.z.u`: the name the client gives, not checked).

More detail, including the memory estimate and every error message, is in [Jobs](docs/jobs.md).

**Adding a task type:**
1. Add the function to `src/approvedtasks.q`. It takes exactly one argument; for several inputs, use a dict.
Expand Down Expand Up @@ -148,8 +164,7 @@ A job is refused before anything runs if:
...
tasks: 0 queued, 0 dispatched, 0 running, 1 done, 1 failed

`restarts` stays 0 until restarting lost secondaries is added. A closed secondary keeps its
last memory reading in the table but is not counted in the total.
A closed secondary keeps its last memory reading in the table but is not counted in the total.

## Failures

Expand Down
149 changes: 149 additions & 0 deletions docs/Approved Tasks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# Approved tasks

A task type is a q function the secondaries are allowed to run. Only approved task types can run, and
they're checked twice:
- **The primary** refuses any task or job whose type isn't in `TASK_TYPES`.
- **Each secondary** refuses any task type that isn't in its own `.secondary.approvedtasks` dict, even if
the primary sent it. The task fails with `unknown tasktype: <name>`.

So nothing outside this list can be run on a secondary, whatever a client sends.

## Adding a task type

1. **Write the function in `src/approvedtasks.q`** as `.task.<name>`, then add it to the approved dict:
```q
// what it does, and an example call
// .task.myavg ([tbl:`trades; d:2014.04.21])
.task.myavg:{[a] select avg price by sym from a[`tbl] where date=a`d};
.secondary.approvedtasks[`myavg]:.task.myavg;
```
2. **Add its name to `TASK_TYPES`** in `config/default.env` (and in `config/integration.env` if the tests
use it):
```bash
TASK_TYPES=sumtil,sleep,fail,hold,vwap,myavg
```
3. **Restart the primary.** It reads `TASK_TYPES` at start, and the secondaries it spawns load
`src/approvedtasks.q` when they start.
4. **Try it:**
```q
.primary.newtask ([tasktype:`myavg; args:([tbl:`trades; d:2014.04.21])])
.primary.newjob[`myavg; ([] tbl:2#`trades; d:2014.04.21 2014.04.22); ()!()]
```

### Rules for a task function

- **It takes exactly one argument.** For several inputs, take a dict: `{[a] ... a`mb ... a`secs}`. A job
given a table passes each row as a dict, so `([] mb:100 200; secs:1 2)` makes two tasks.
- **Its return value is the result.** The secondary saves it to disk with `set` and only the path goes back
to the primary. So it must be something `set` can write: atoms, lists, dicts and tables are all fine.
Return `::` for no result: no file is saved.
- **An error fails the task.** Anything it signals (`'oops`) is caught: the task fails with reason `error`,
and the message goes in `err`. Errors are not retried.
- **It mustn't keep memory in globals.** After each task the secondary runs `.Q.gc[]` to give the memory
back. That only works for memory nothing still refers to: a task that sets a global keeps that memory
for the life of the secondary, and every later task's memory figures include it.
- **Long tasks need a long enough `timeout`.** A task running past `TASK_TIMEOUT_MS` (or its own `timeout`)
fails, and its secondary is killed. Give a slow task type its own `timeout` when you submit it.
- **It can use secondary threads.** Each secondary starts with `-s THREADS` (default 1). With more, a task
can use `peach` inside itself.
- **An HDB is loaded for it if `HDB_DIR` is set.** Each secondary loads that HDB at start, so a task can
query its tables directly (see `vwap`).

### Naming

- The function is `.task.<name>`, and the approved key is the same name: `` `myavg ``.
- Names are lowercase, with no underscores.

### Testing a task without the framework

Load the file in a plain q session and call the function:
```bash
x src/approvedtasks.q
```
```q
.task.sumtil 1000 / 499500
.secondary.approvedtasks[`hold] ([mb:100; secs:1])
\l /path/to/hdb / for a task that reads an HDB
.task.vwap 2014.04.21
```

## The approved tasks

| Task type | Argument | Returns | Used for |
|---|---|---|---|
| `sumtil` | a number `n` | `sum til n` | quick CPU work; tests, demos |
| `sleep` | a number of seconds | an empty list | a task that takes a known time; timeouts, priority, busy secondaries |
| `fail` | anything (ignored) | never returns: always fails | the failure path: `failed`, reason `error`, cancelling a job |
| `hold` | a dict `` `mb`secs `` | the number of longs it held | a known amount of memory for a known time; memory gating, gc, OOM |
| `vwap` | a date | a table: `vwap` by `date`, `sym` | a real HDB query, one task per date |

### `sumtil`

```q
.task.sumtil:{sum til x};
```
Adds up `0` to `n-1`. Bigger `n` takes longer and uses more memory: `til n` is a list of `n` longs (8
bytes each), so `n` of 50,000,000 uses about 400MB at its peak.
```q
.primary.newtask ([tasktype:`sumtil; args:1000]) / result 499500
.primary.newjob[`sumtil; 1000 2000 3000; ()!()] / 499500 1999000 4498500
```

### `sleep`

```q
.task.sleep:{system"sleep ",string x};
```
Waits `x` seconds, using no memory or CPU. Useful for keeping secondaries busy, and for testing timeouts
and priority.
```q
.primary.newtask ([tasktype:`sleep; args:10; timeout:00:00:02.000]) / fails as timeout after ~2s
.primary.newjob[`sleep; 8#5; ()!()] / keeps 8 secondaries busy for 5s
```

### `fail`

```q
.task.fail:{'`fail};
```
Always signals `'fail`. The task fails with reason `error` and `err` "fail", and it isn't retried. In a
job, the first one to fail makes the job fail and cancels its queued tasks.
```q
.primary.newtask ([tasktype:`fail; args:1])
.primary.newjob[`fail; til 20; ()!()]
```

### `hold`

```q
.task.hold:{[a] v:(a[`mb]*131072)#0; system "sleep ",string a`secs; count v};
```
Takes `mb` MB of memory and keeps it for `secs` seconds:
- 131,072 longs is 1MB, and filling them with 0 writes every page, so the memory really is used and the
kernel counts it (RssAnon).
- It returns the number of longs, not the list, so the saved result is tiny.

It's how the memory features are tested:
- **The global cap:** several 300MB tasks pause dispatch under a low cap.
- **gc:** memory before, at the peak and after `.Q.gc[]`.
- **OOM:** a 2000MB task under a 1GB kernel cap is killed and labelled `oom`.
- **Memory estimates:** the next `hold` job is estimated at the biggest seen.
```q
.primary.newtask ([tasktype:`hold; args:([mb:300; secs:2])])
.primary.newjob[`hold; ([] mb:300 300 300; secs:2 3 4); ()!()]
```

### `vwap`

```q
.task.vwap:{[d]
select vwap: size wavg price by date,sym from trades where date = d
};
```
The volume-weighted average price for every symbol on one date, from the HDB's `trades` table. It needs
`HDB_DIR` set to a date-partitioned HDB with a `trades` table (`date`, `sym`, `price`, `size`). One task per
date, then join the results:
```q
j:.primary.newjob[`vwap; 2014.04.21 2014.04.22 2014.04.23; ()!()]
raze get each .primary.jobpaths j / one table: vwap by date, sym
```
Loading