Skip to content

examples

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
filters.toml
filters.toml
 
 

README.md

Tok Examples

Tok is a Go library (github.com/GrayCodeAI/tok) that compresses prompts and filters text to reduce LLM token costs by 60-90%. There is no standalone tok binary — you call it from Go code, or use the CLI verbs exposed through Hawk, which embeds this library.

go get github.com/GrayCodeAI/tok
import "github.com/GrayCodeAI/tok"

Basic Usage

Compress a prompt

in := "Hey, could you please help me figure out why this React component keeps " +
    "re-rendering every time the props change?"

out, stats := tok.Compress(in, tok.Aggressive)
// out: "React component re-renders on prop change. Why?"
// stats: compression ratio, token savings, etc.

PromptCompress lets you pick an intensity level directly:

out, stats := tok.PromptCompress(in, tok.IntensityFull)
// intensities: tok.IntensityLite, tok.IntensityFull, tok.IntensityUltra

Estimate tokens and cost

cost := tok.EstimateCost(out, "claude-3-5-sonnet")

Advanced Examples

Custom compression rules

Filter rules are TOML files loaded with tok.LoadFilterRules and applied via tok.WithCustomFilters:

rules, err := tok.LoadFilterRules("rules.toml")
if err != nil {
    log.Fatal(err)
}

out, _ := tok.Compress(in, tok.WithCustomFilters(rules))

Code-aware and perplexity-guided compression

// Preserve code structure for a given language.
out, _ := tok.Compress(src, tok.WithCodeAware("go"))

// Drop the lowest-information tokens using your own scorer.
out, _ = tok.Compress(in, tok.WithPerplexityGuided(scorer, 0.5))

Profiles

profile, err := tok.LoadProfile("profile.toml")
out, _ := tok.Compress(in, profile.Options()...)

Secret detection and sensitive files

det := tok.NewSecretDetector()
findings := det.Scan(text)

if tok.IsSensitiveFilename(".env") {
    // skip or redact
}

Truncation and JSON extraction

short := tok.SmartTruncate(longText, 500)
payload, err := tok.ExtractJSON(mixedOutput)

Usage tracking

tracker := tok.NewTracker(ctx)
// record and report token/cost usage over a run

Using the CLI verbs (via Hawk)

The tok ... subcommands you may remember (compress, estimate, etc.) are provided by Hawk, which embeds this library:

hawk tok compress < prompt.txt
hawk tok estimate < prompt.txt

See Hawk for the full command surface.

Benchmarks

./benchmarks/run.sh
./evals/pipeline-bench.sh

Both wrap go test -bench.

See the main README for full documentation.