perf: CLI shell completion generation is slow

Describe the bug

I am sensitive to my shell's startup time. I want to enable gh completions for my shell, but doing so adds 50ms to my startup time which is uncomfortable to me.

gh version 2.57.0 (2024-09-16)
https://github.com/cli/cli/releases/tag/v2.57.0

Steps to reproduce the behavior

$ hyperfine -w2 "gh completion -s bash" "bash -c 'source <(gh completion -s bash)'" "gh completion -s zsh" "zsh -c 'source <(gh completion -s zsh)'" "fzf --zsh" "zsh -c 'source <(fzf --zsh)'" "jump shell" "zsh -c 'source <(jump shell)'"
Benchmark 1: gh completion -s bash
  Time (mean ± σ):      43.6 ms ±   1.0 ms    [User: 42.2 ms, System: 6.6 ms]
  Range (min … max):    42.4 ms …  49.1 ms    62 runs

Benchmark 2: bash -c 'source <(gh completion -s bash)'
  Time (mean ± σ):      48.9 ms ±   1.0 ms    [User: 44.5 ms, System: 8.9 ms]
  Range (min … max):    47.2 ms …  52.1 ms    55 runs

Benchmark 3: gh completion -s zsh
  Time (mean ± σ):      44.0 ms ±   3.1 ms    [User: 42.6 ms, System: 6.6 ms]
  Range (min … max):    41.7 ms …  66.2 ms    60 runs

Benchmark 4: zsh -c 'source <(gh completion -s zsh)'
  Time (mean ± σ):      49.9 ms ±   1.2 ms    [User: 43.2 ms, System: 10.4 ms]
  Range (min … max):    47.6 ms …  52.9 ms    56 runs

Benchmark 5: fzf --zsh
  Time (mean ± σ):       2.8 ms ±   0.6 ms    [User: 1.1 ms, System: 1.1 ms]
  Range (min … max):     1.5 ms …   5.8 ms    414 runs

  Warning: Command took less than 5 ms to complete. 

Benchmark 6: zsh -c 'source <(fzf --zsh)'
  Time (mean ± σ):       8.9 ms ±   2.0 ms    [User: 3.1 ms, System: 3.5 ms]
  Range (min … max):     6.5 ms …  33.8 ms    235 runs


Benchmark 7: jump shell
  Time (mean ± σ):       2.2 ms ±   1.2 ms    [User: 0.8 ms, System: 0.9 ms]
  Range (min … max):     1.4 ms …  24.5 ms    518 runs

  Warning: Command took less than 5 ms to complete. 

Benchmark 8: zsh -c 'source <(jump shell)'
  Time (mean ± σ):       9.7 ms ±   1.2 ms    [User: 4.1 ms, System: 3.2 ms]
  Range (min … max):     7.7 ms …  14.3 ms    206 runs

Summary
  jump shell ran
    1.24 ± 0.75 times faster than fzf --zsh
    3.97 ± 2.37 times faster than zsh -c 'source <(fzf --zsh)'
    4.34 ± 2.47 times faster than zsh -c 'source <(jump shell)'
   19.45 ± 10.80 times faster than gh completion -s bash
   19.66 ± 10.99 times faster than gh completion -s zsh
   21.82 ± 12.12 times faster than bash -c 'source <(gh completion -s bash)'
   22.29 ± 12.38 times faster than zsh -c 'source <(gh completion -s zsh)'

Expected vs actual behavior

Tools like fzf and jump incur a much lower overhead to my startup time. I expected gh to be similar.

@vegerot

@vegerot thanks for opening this. What would be the maximum amount of time you'd find acceptable? There's probably some low hanging fruit we could address here but we'll probably hit diminishing returns pretty quickly since we're at the mercy of our dependencies. Finger in the air I don't think we'd get much below 15-20ms (computer dependent of course).

As a very quick win you can set TCELL_MINIMIZE=1 which is by far the biggest time sink:

➜ hyperfine -w2 "gh completion -s bash" "TCELL_MINIMIZE=1 gh completion -s bash"
Benchmark 1: gh completion -s bash
  Time (mean ± σ):      42.4 ms ±   1.1 ms    [User: 42.0 ms, System: 6.3 ms]
  Range (min … max):    41.1 ms …  47.4 ms    63 runs

Benchmark 2: TCELL_MINIMIZE=1 gh completion -s bash
  Time (mean ± σ):      21.9 ms ±   1.3 ms    [User: 23.5 ms, System: 6.6 ms]
  Range (min … max):    19.8 ms …  26.9 ms    119 runs

Summary
  TCELL_MINIMIZE=1 gh completion -s bash ran
    1.94 ± 0.13 times faster than gh completion -s bash

We had done some work previously to make this permanent but looking at it I think it's flaky. We can definitely revisit that.

If this is still too much for you, I think the only real solution is to produce the completions once each time we release a new version and put that into your shell startup scripts.

If this is still too much for you, I think the only real solution is to produce the completions once each time we release a new version and put that into your shell startup scripts.

To add to this, cobra generally recommends executing once anyway. Though I definitely recognize that it could be less convenient than executing each time.

@williammartin

@williammartin thank you! This made a big improvement.

$ hyperfine -w2 "gh completion -s bash" "gh completion -s zsh" "TCELL_MINIMIZE=1 gh completion -s zsh" "fzf --zsh" "jump shell"
Benchmark 1: gh completion -s bash
  Time (mean ± σ):      45.0 ms ±   1.4 ms    [User: 43.3 ms, System: 7.0 ms]
  Range (min … max):    43.1 ms …  52.0 ms    57 runs

Benchmark 2: gh completion -s zsh
  Time (mean ± σ):      45.0 ms ±   1.0 ms    [User: 42.6 ms, System: 6.9 ms]
  Range (min … max):    43.6 ms …  49.0 ms    58 runs

Benchmark 3: TCELL_MINIMIZE=1 gh completion -s zsh
  Time (mean ± σ):      23.9 ms ±   1.4 ms    [User: 23.6 ms, System: 7.2 ms]
  Range (min … max):    21.7 ms …  28.8 ms    102 runs

Benchmark 4: fzf --zsh
  Time (mean ± σ):       3.3 ms ±   0.9 ms    [User: 1.2 ms, System: 1.4 ms]
  Range (min … max):     1.0 ms …   6.8 ms    351 runs

  Warning: Command took less than 5 ms to complete. 

Benchmark 5: jump shell
  Time (mean ± σ):       3.0 ms ±   0.7 ms    [User: 1.0 ms, System: 1.4 ms]
  Range (min … max):     1.5 ms …   8.9 ms    357 runs

  Warning: Command took less than 5 ms to complete. 
  Warning: Statistical outliers were detected.

Summary
  jump shell ran
    1.11 ± 0.40 times faster than fzf --zsh
    7.95 ± 2.02 times faster than TCELL_MINIMIZE=1 gh completion -s zsh
   15.00 ± 3.72 times faster than gh completion -s zsh
   15.00 ± 3.74 times faster than gh completion -s bash

I think the only real solution is to produce the completions once each time we release a new version and put that into your shell startup scripts.

Why? fzf and jump can quickly generate the completions

Why? fzf and jump can quickly generate the completions

The completion generation isn't the slow part, it only takes a few ms. I think the rest of the time is spent by the runtime initialising packages. There might be a few more quick wins but as you can see it diminishes pretty quickly (and I think these are parallelised where possible, though I haven't confirmed that).

> GODEBUG=inittrace=1 ./bin/gh 2>&1 | awk -F'[, ]+' '{print $2, $5 " ms"}' | sort -k2 -n
...
github.com/gorilla/css/scanner 0.26 ms
github.com/microcosm-cc/bluemonday/css 0.41 ms
github.com/yuin/goldmark/util 0.46 ms
github.com/microcosm-cc/bluemonday 0.60 ms
github.com/alecthomas/chroma/v2/lexers 3.0 ms
github.com/alecthomas/chroma/v2/styles 4.0 ms
github.com/gdamore/tcell/v2 30 ms

fzf and jump have significantly fewer dependencies.

Still, I'll think on it!

feat(zsh): add gh completion

That makes sense. Thanks! It ended up adding around 25ms to my start time.

You can see vegerot/dotfiles@87b5ba9 for more info

@vegerot

👋 @vegerot I spoke to @williammartin more on this and we opened #9724 to investigate gh start-up time more broadly in the future.

With that, I will close this in favour of #9724

Navigation Menu

Repository navigation

perf: CLI shell completion generation is slow #9686

Describe the bug

Steps to reproduce the behavior

Expected vs actual behavior

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

perf: CLI shell completion generation is slow #9686

Description

Describe the bug

Steps to reproduce the behavior

Expected vs actual behavior

Activity

williammartin commented on Oct 3, 2024

BagToad commented on Oct 3, 2024

vegerot commented on Oct 3, 2024

williammartin commented on Oct 3, 2024

vegerot commented on Oct 4, 2024

BagToad commented on Oct 7, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions