Skip to content

plugin-dev.md

Latest commit

 

History

History
117 lines (93 loc) · 3.76 KB

plugin-dev.md

File metadata and controls

117 lines (93 loc) · 3.76 KB
Raw

Custom filter rules

tok is a Go library (github.com/GrayCodeAI/tok). It has no standalone CLI and no plugin/agent install system — the "plugins" that earlier drafts described (per-command output rewriting keyed on match_command, shell hooks, and a tok doctor validator) belonged to a CLI that no longer exists.

What is real, and the subject of this page, is the custom filter DSL: a TOML file of regex find/replace rules you load with tok.LoadFilterRules and apply through tok.WithCustomFilters. This runs as a stage of the normal compression pipeline.

go get github.com/GrayCodeAI/tok
import "github.com/GrayCodeAI/tok"

File format

Each rule is a [[rule]] array-of-tables entry:

# ~/.tok/filters.toml
[[rule]]
name        = "collapse-uuids"
pattern     = "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"
replacement = "<uuid>"
priority    = 10
enabled     = true
applies_to  = "*.log"   # optional content-type or glob hint

[[rule]]
name        = "strip-timestamps"
pattern     = "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}Z\\s*"
replacement = ""
priority    = 20

Schema

Field Type Meaning
name string Human-readable identifier, used in logged warnings.
pattern regex A Go (RE2) regular expression to match. Required.
replacement string Substituted for each match. May reference capture groups via $1 / ${name}. Empty deletes the match.
priority int Application order: lower runs first. Defaults to 0.
enabled bool Gates the rule. Omitting the field defaults to enabled (opt-out).
applies_to string Optional content-type label or filename glob (e.g. "*.log", "json"). Empty applies the rule to all input.

Rules are applied in ascending priority order. Invalid regex patterns are skipped with a logged warning rather than crashing the load — a single bad rule never disables the rest. Malformed TOML returns a non-nil error.

Loading and applying rules

home, _ := os.UserHomeDir()
rules, err := tok.LoadFilterRules(filepath.Join(home, ".tok", "filters.toml"))
if err != nil {
    log.Fatal(err)
}

out, stats := tok.Compress(text, tok.WithCustomFilters(rules))
_ = stats
fmt.Println(out)

tok.WithCustomFilters composes with the other options, so you can combine custom rules with the built-in modes:

out, _ := tok.Compress(text,
    tok.WithCustomFilters(rules),
    tok.Aggressive,
)

Testing your rules

Because rules are plain Go values, test them with the standard library — load the file and assert on tok.Compress output:

func TestUUIDCollapse(t *testing.T) {
    rules, err := tok.LoadFilterRules("testdata/filters.toml")
    if err != nil {
        t.Fatal(err)
    }
    in := "request 550e8400-e29b-41d4-a716-446655440000 ok"
    out, _ := tok.Compress(in, tok.WithCustomFilters(rules))
    if !strings.Contains(out, "<uuid>") {
        t.Fatalf("uuid not collapsed: %q", out)
    }
}

Run with go test ./....

Profiles

A related loader, tok.LoadProfile(path), reads a saved set of compression settings (a "profile") from disk so you can keep tuned configurations in version control and apply them consistently. tok.LoadProfiles(dir) loads every profile in a directory.

Command-output filtering

If what you actually want is to compress the output of shell commands (git log, terraform plan, kubectl, …) using per-command rules, that lives in Hawk (github.com/GrayCodeAI/hawk), which embeds this library and ships the per-command builtin filters. See hawk tok compress and the Hawk docs. tok itself remains a library: it gives you tok.Compress, tok.PromptCompress, the estimators (tok.EstimateTokensForModel, tok.EstimateCost), and the custom-filter DSL described above.