Allow gh api --paginate --slurp to be used with --jq

@heaths

Describe the feature or problem you’d like to solve

Currently, when trying to use gh api with --paginate --slurp alongside --jq I get the following error:

the --slurp option is not supported with --jq or --template

I would have expected to be able to use these together, most likely getting a top-level array of all of the responses, that I could then process however I like with --jq syntax.

For example, using only --paginate but not --slurp:

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=1 -F per_page=50 --paginate --jq 'length'

50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
50
14

Using --paginate --slurp with --jq:

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=1 -F per_page=50 --paginate --slurp --jq 'length' | subl

the `--slurp` option is not supported with `--jq` or `--template`

Using --paginate --slurp and then needing to call out to external jq for further processing:

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=1 -F per_page=50 --paginate --slurp | jq 'length'
24

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=1 -F per_page=50 --paginate --slurp | jq 'map(length)'
[
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  50,
  14
]

For a more real-world example of what I was trying to achieve with --paginate and --jq; that I was then expecting --slurp would help me with (note that I am starting from page 20 here to reduce the length of the output for demonstration purposes):

⇒ gh api notifications --method GET -F participating=false -F all=true -F page=20 -F per_page=50 --paginate --jq '
  # Filter for notifications whose subject type is 'Release',
  # group them by repository, and output repo name and count sorted descending.
  [ .[] | select(.subject.type == "Release") ]
  | group_by(.repository.full_name)
  | map({ repo: .[0].repository.full_name, count: length })
  | sort_by(.count) | reverse
  #| map("\(.repo)  \(.count)")
  # | .[]
'

[{"count":3,"repo":"invoke-ai/InvokeAI"},{"count":2,"repo":"refined-github/refined-github"},{"count":2,"repo":"reduxjs/redux-toolkit"},{"count":2,"repo":"ollama/ollama"},{"count":2,"repo":"aws/aws-cdk"},{"count":1,"repo":"supabase/supabase"},{"count":1,"repo":"remix-run/react-router"},{"count":1,"repo":"obsidianmd/obsidian-releases"},{"count":1,"repo":"langflow-ai/langflow"},{"count":1,"repo":"facebook/react"},{"count":1,"repo":"OpenInterpreter/open-interpreter"}]
[{"count":6,"repo":"invoke-ai/InvokeAI"},{"count":3,"repo":"holochain/holochain"},{"count":2,"repo":"stripe/stripe-js"},{"count":2,"repo":"aws/aws-cdk"},{"count":1,"repo":"reduxjs/redux-toolkit"},{"count":1,"repo":"jehna/humanify"},{"count":1,"repo":"firebase/firebase-js-sdk"},{"count":1,"repo":"actions/checkout"},{"count":1,"repo":"Aider-AI/aider"}]
[{"count":4,"repo":"invoke-ai/InvokeAI"},{"count":3,"repo":"holochain/holochain"},{"count":1,"repo":"stripe/stripe-js"},{"count":1,"repo":"refined-github/refined-github"},{"count":1,"repo":"microsoft/autogen"},{"count":1,"repo":"firebase/firebase-js-sdk"},{"count":1,"repo":"aws/aws-cdk"},{"count":1,"repo":"actions/checkout"},{"count":1,"repo":"Readarr/Readarr"},{"count":1,"repo":"ProxymanApp/Proxyman"},{"count":1,"repo":"Aider-AI/aider"}]
[{"count":3,"repo":"invoke-ai/InvokeAI"},{"count":2,"repo":"ollama/ollama"},{"count":2,"repo":"aws/aws-cdk"},{"count":1,"repo":"refined-github/refined-github"},{"count":1,"repo":"holochain/holochain"},{"count":1,"repo":"firebase/firebase-js-sdk"},{"count":1,"repo":"Whisky-App/Whisky"},{"count":1,"repo":"JedWatson/react-select"},{"count":1,"repo":"Heroic-Games-Launcher/HeroicGamesLauncher"},{"count":1,"repo":"Aider-AI/aider"}]
[{"count":1,"repo":"supabase/supabase"},{"count":1,"repo":"stripe/stripe-js"},{"count":1,"repo":"holochain/holochain"}]

Essentially, the --paginate data seems to be processed individually by --jq; so the output is sort of reminiscent of JSON Lines:

https://jsonlines.org/

Because each page of the --paginate'd data is treated separately by --jq, when I try and aggregate/count, it doesn't really work, as in this case I get ~24 pages of counts by repo, rather than 1 full count by repo.

By my reading of the vast history of related issues/PR's (linked below), I believe this was kept as there was a desire to not introduce potentially breaking changes before v3 (even if the likelihood of this having any real world user impact was probably low, but somewhat unquantifiable). I believe this was the reason that --slurp was introduced, as a potentially temporary addon so that --paginate's old functionality remained, but we could opt in to a single JSON return value by using --slurp.

Proposed solution

When I was first reading through #8620 (after eventually getting to it through all the past related context linked below), I was expecting that the whole point of that PR was so I could use --paginate --slurp with --jq to avoid needing to rely on an external program (eg. jq); as otherwise, I believe we could have just always relied on an external jq --slurp with the old --paginate implementation.

I would really love to be able to use --paginate, and then process that with --jq; and have it all treated as a 'single pass', rather than the JSON Lines style processing we currently get. My understanding of --slurp is that that would be ideal for this; but we can't use that with --jq currently.

Additional context

I spent a bunch of time reading through the long history of related issues.. (with a LOT of effort put in by @heaths, along with a bunch of others along the way) and while I'm not necessarily deeply wiser about the full spectrum of implementation choices explored and iterated through; here is my attempted shortlist of related issues/PR's in hopefully roughly relevant historical order:

Output a single JSON document from api --paginate #1268
- Closed as completed in Merge JSON responses from gh api #8620
Merge JSON responses from gh api #5652
- Closing in lieu of Merge JSON responses from gh api #8620
api: output a single JSON array in REST pagination mode #7190
Support merging JSON arrays, objects go-gh#148
- Closing per discussion above and in Merge JSON responses from gh api #8620
Merge JSON responses from gh api #8620
- Merge JSON responses from gh api #8620 (comment)
  - Like before, it will not be used for --jq or --template as this is implemented as an io.Writer.
    
    I suppose another way would be to implement this as a reader similar to the existing REST merge, but then you'd have more problems sending incomplete JSON documents to jsoncolor. The only advantage is --template and --jq would get the same JSON array container representation if --slurp were provided. Without a complete overhaul of how JSON is effectively merged, there are pros and cons either way. This way, at least, seems fairly straight forward.
Allow RESTClient and GQLClient to support automatic pagination go-gh#23

Tangentially related, but probably not relevant (including because it recently touched a similar area, so not sure if there was a regression, but based on above context, I think probably isn't relevant):

Incorrect error shown (the --slurp option is not supported with --jq or --template) when passing --jq + --template without --slurp #10260
- [gh api] Fix mutual exclusion messages of --slurp flag #10332

Similarly, I'm not sure if this is only tangentially related or not.. but it feels in a similar space of things, so linking in case the additional context is helpful:

Support getting both JSON and template output #8415

Tangentially Related

There have been a number of other json/jq Quality of Life issues I've opened recently; and some awesome PRs associated with some of them, made by @heaths . I'm not sure of the deeper specifics as to whether any of them would crossover with this at an implementation level; but from a user perspective, it's kind of weird/confusing feeling when different CLI commands seem to handle json/jq differently.

Allow --jq and --template to be used together in gh api #10261

Support getting both JSON and template output #8415

Support filtering before applying template go-gh#177

Process --jq before --template #10273

Add custom formatting functions to --jq similar to those added to --template #10262

Add useful template functions to jq filters go-gh#178

Support useful template functions and modules in jq filters #10275

Allow passing additional arguments through to --jq script, just like we can with jq --arg #10263

Originally posted by @0xdevalias in #10385

Reasons for this are well documented in the bug related to my PR. The timing doesn't make jq templates make sense because of the decision of how pages are combined. I think at some point you just have to pipe to jq instead of making gh the same as jq. Being able to combine commands makes all those commands more flexible. Do-it-all commands don't do anyone particular task very well.

@heaths

Reasons for this are well documented in the bug related to my PR.

@heaths As there are many issues and PR's related to this feature, are you able to be more specific/link to which 'bug related to PR' you're referring to specifically?

The timing doesn't make jq templates make sense because of the decision of how pages are combined.

I'm not deeply familiar with all of this, beyond the issues I read through today, but if I remember correctly, I think one of the reasons for timing was wanting to keep the ability to stream the output or similar? Which definitely seems to be occurring with --paginate on it's own; but from what I saw in my testing this afternoon, if I remember correctly, seemingly not with --paginate --slurp.

Which aspect of 'timing' and 'decision of how pages are combined' makes this a blocker?

Being able to combine commands makes all those commands more flexible. Do-it-all commands don't do anyone particular task very well.

Sure.. but the same argument could be made about any feature/enhancement/issue that's been raised related to any of these features over the years. I don't personally feel like that's a particularly strong reason to not have this one.

[bin] gh-notifications: add --mode release-count + refactor to DRY AP…

Hey folks, perusing the monstrous discussion on #8620, I left the following comment:

So the current paginatedArrayReader works by dropping the first [ in the first paginated response, and for every subsequent response it drops the initial [. This allows for responses to be streamed as they come in.

However, in the case of --slurp I think it would be quite reasonable to say "no streaming, only the final slurped object gets printed". Do you agree, and if so do you think it might be simpler in the case of --slurp to have a reader that collects all the paginated responses and then sends it to jsoncolor in one go? This might also open the door for --jq and --template in the future. This might be something you considered when you said "complete overhaul" but it doesn't seem like a wildly big change at first glance.

I'm content to say ^ is non-blocking (if someone out there starts using this and json parses the output in a streaming manner then I'll handle that). However, since you've been in the code maybe you have some opinions.

@heaths responded:

The thought had crossed my mind, but paged responses can be pretty massive in some cases. Then again, it's still written into memory as cells in the terminal if written to a TTY - we'd just be duplicating. Not the end of the world. I can look deeper, but off hand, that might be more refactoring because a lot in the modified code path already assumes things are written out as they come. It's not that I'm averse to refactoring - look at how much I did in various iterations of my previous PR - but it does increase the likelihood of regressions. I've been adding test cases where I've accidentally made regressions, but it means there were - and likely still are - some scenarios that aren't well-tested e.g., TTY output. I should probably add some, but I admit with as many VT sequences as there are it's been daunting.

And I responded:

Let's punt on doing a larger refactoring. I want to get this in and not stall it any further.

So I think the core reason we chose not to support --jq and --template is that getting --slurp shipped was the highest priority. It seems like both heaths and I thought it would be possible to support this if we gave up streaming but existing code was heavily dependent on streaming writers, and the refactoring was high risk and lower value with no one having asked for it yet. It may also be a breaking change but I expressed the first time round that I would consider breaking in this case.

Very, very fuzzy memory on this one though, and as your in-depth investigation shows, this discussion was all over the place.

I'm open to revisiting this but I have to admit, it's pretty daunting. I think I would want to see the api command significantly reworked first unless someone has a very clear path forward to achieve this.

The original PR your predecessors rejected did merge all pages together. It was ready, then someone complained it didn't merge nested arrays, to which I pointed out there's no reasonable way to know unequivocally whether we can. We can only know for sure for top-level arrays. So I was made to change it to be "safe".

The whole point of slurp was to emit valid JSON. If you don't, you can't use PowerShell's ConvertFrom-JSON, which we use a ton in our pipelines. Before my change, each page was a top-level object our array and only jq --slurp would handle that.

But if you run jq expressions on each page, you can't properly aggregate objects and aggregation functions produce multiple restore instead of a single result. I believe proof of that was all on my comments prior. That's not what pale would expect and produces incorrect results.

I still think of you want full jq, use jq.

But if you run jq expressions on each page, you can't properly aggregate objects and aggregation functions produce multiple restore instead of a single result. I believe proof of that was all on my comments prior. That's not what pale would expect and produces incorrect results.

I think what you're describing is that given a single GQL page you end up with a structure like:

[
 { "data": ... }
]

And with --slurp you end up with a structure like:

[
 { "data": ... },
 { "data": ... }
]

So that you cannot write the same --jq query and use it with and without --slurp, which might be surprising.

In the case of REST it looks like:

[
 { },
 { },
]

And with --slurp you end up with a structure like:

[
  [
    { },
    { },
  ],
  [
    { },
    { },
  ]
]

With --slurp defined as:

--slurp Use with "--paginate" to return an array of all pages of either JSON arrays or objects

Users may reasonably infer that the data is being concatenated in a way that doesn't preserve the original shape.

I probably need to spend a good bit more time getting all the context into my brain again. I think both of us gratefully put this out of our mind after that PR was merged 🙈

Without slurp, for GraphQL you ended up with:

{
}
{
}

After a couple years of deliberations we agreed to nest them into an array. But then that breaks aggregating functions. Someone said you'd consider my original PR that merged top-level objects and arrays so avoid possibly (though unlikely) breakage.

My old PR handled both REST and GraphQL merging but Nate ended up just shaving the end of the array off for REST responses, leaving GraphQL responses as invalid JSON.

I remember this all too well.

You're right, no [] surrounding {} when paginating GQL.

Though actually I was intending to compare no flags with --paginate --slurp, but I also got that wrong 🤦

@williammartin

It was ready, then someone complained it didn't merge nested arrays, to which I pointed out there's no reasonable way to know unequivocally whether we can. We can only know for sure for top-level arrays. So I was made to change it to be "safe".

I remember reading some of this stuff, and I also remember some later discussions that seemed to go back and forth on deeper merging, vs just collecting everything in a top level array and leaving processing that up to the user; and some thoughts about how the latter would break GraphQL conventions/expectations/etc.

My personal /2c on all of that is that while some kind of 'magically perfectly merged' might be nice if it's possible, at the end of the day, the main thing I need is the ability to apply the tools I have to the output. So for me, even if the entire GraphQL response was collected as is, and naively wrapped into an array, that would be fine; so long as --jq was able to process that array as a single document such that I could process/manipulate the entire response how I like.

It wouldn't matter to me that there would be a slight difference in format between processing a single response vs a --paginate --slurp response; as realistically that's a decision I will be making at time of writing my script, and am unlikely to be swapping back and forth between paginated or not very much; and even if I was, it's fairly minimal logic to add to the jq script to be able to handle both.

The current way that --jq is applied 'per page' in a streaming fashion feels to me (naively) more like a legacy aspect more than an explicit design choice. Sort of feels like the CLI was probably written with a single request in mind, and then at some point --paginate was added and streamed the responses because that was closest to how the code was written before, and then maybe --jq was tacked on at some point and just made to work with the code as it was because changing it was too hard or similar, and then it hasn't come up much because for a lot of use cases you can kind of get away with the streaming --jq hack because if you're just outputting/processing single lines/etc it doesn't make a huge difference.

Basically, this:

But if you run jq expressions on each page, you can't properly aggregate objects and aggregation functions produce multiple restore instead of a single result. I believe proof of that was all on my comments prior. That's not what people would expect and produces incorrect results.

So that you cannot write the same --jq query and use it with and without --slurp, which might be surprising.

I'm not deeply versed in GraphQL conventions and semantics, so my opinion here may be based on that naivety; but as mentioned above, that would barely even register outside my expectations; particularly since --slurp mentions something about collecting the pages in an array in it's help text.

I would almost go as far as to see it would feel like a logical conclusion to me; with --paginate getting multiple pages, the first way I would think for that to work would be to have them in an array; and I definitely wouldn't be expecting any sort of 'magical merging' there (again, maybe that's a naive view if GraphQL has some default/standard conventions for where the 'data part' of paged responses ends up)

With --slurp defined as:

--slurp Use with "--paginate" to return an array of all pages of either JSON arrays or objects

Users may reasonably infer that the data is being concatenated in a way that doesn't preserve the original shape.

@williammartin Agreed. That's how I read/understand it.

I remember this all too well.

@heaths That doesn't surprise me in the least, even just from reading through the issues yesterday, it's kind of amazing how much you stuck with it through all the painful back and forth over the years. And I totally understand why you may be hesitant to have someone open the discussion again.

This is probably off topic/may not add anything new/relevant here; but a while back (based on enjoying how gh api's --paginate worked, at least in basic cases) I implemented a curl wrapper for adding --paginate; and made it so I could at least somewhat customize which keys it was looking at for paging/the 'main data chunk'/etc. That may not be as relevant here since for most cases the shape of responses may already be known since it all hits the GitHub API; but figured I would link it just in case it sparks any ideas/etc:

https://github.com/0xdevalias/dotfiles/blob/4233c263e0b6a0a3c0c3cad3682b49ae4a00a189/bin/paginate-fetch#L62-L68

It looks like with --slurp I was just buffering the pages of records as it ran, and otherwise went with a very basic/naive merge using jq -s 'add':

https://github.com/0xdevalias/dotfiles/blob/4233c263e0b6a0a3c0c3cad3682b49ae4a00a189/bin/paginate-fetch#L175-L180

# Merge records if not slurping
if [[ "$SLURP" == true ]]; then
  response_combined+=("$records")
else
  merged_output=$(echo "$merged_output $records" | jq -s 'add')
fi

And then when outputting with --slurp, I just naively wrapped it with [ ]:

https://github.com/0xdevalias/dotfiles/blob/4233c263e0b6a0a3c0c3cad3682b49ae4a00a189/bin/paginate-fetch#L195-L200

# Output results
if [[ "$SLURP" == true ]]; then
  echo "["$(IFS=,; echo "${response_combined[*]}")"]" | jq
else
  echo "$merged_output" | jq
fi

And I totally understand why you may be hesitant to have someone open the discussion again.

That's not it at all. As I've stated, if you want use jq, use jq. Making this a monolith when you can combine tools - command line APIs, if you will - just creates bloat that gets more difficult to maintain. Of course it's not my tool, but after almost 40 years of writing software I've learned a few lessons. The power of a she'll - bash, powershell, etc. - is making chaining commands easy. Sed, awk, grep, jq, etc. PowerShell takes this a step further with piping objects, and is why I fixed the output with --slurp so that it was valid to read as JSON.

I've submitted a few PRs to improve existing functionality. I'm happy to make any changes for feedback but I'm not interested in complicating one tool when piping content to the appropriate tool is already possible. I just don't have the time.

@heaths

That's not it at all. As I've stated, if you want use jq, use jq. Making this a monolith when you can combine tools - command line APIs, if you will - just creates bloat that gets more difficult to maintain.

I guess my /2c on this, as a random user (so take it with a grain of salt), is that if you're going to include a feature like --jq, and base that on a library designed to largely mirror jq, then it's a reasonable expectation that it would work like that tool; or at the very least would document how it differs from it.

In that regard, I feel like a user expecting that the input the CLI passes through to --jq would be a single document, rather than a stream of documents, isn't unreasonable.

If these comments were being made on #10263 for example, I would probably agree with you, as that feels far more 'extra and has a decent enough workaround' than what I am raising here, which feels more 'core/standard expectation' (and if the outcome to that ends up being "no", then fair enough).

I've submitted a few PRs to improve existing functionality. I'm happy to make any changes for feedback but I'm not interested in complicating one tool when piping content to the appropriate tool is already possible. I just don't have the time.

@heaths Which is totally fair. I didn't open this with any expectation of you doing so; and only mentioned you on it originally to acknowledge the effort you put in over the years to get to where things are now.

[BEHAVIORAL] Fix PR #237 review feedback: --slurp, stdin JSON bodies,…

ci(fuzz): R2 — revert jq slice (paginate runs jq per-page), use tail

ci(fuzz): prune older fuzz-corpus cache entries to bound namespace (#252

ci(setup): pipe dismissal review query through jq

ci(setup): merge-gate follow-ups — codex visibility, dispatch dedupe,…

Address review: slurp all pages with jq -s in the verdict-query one-l…

fix: CLAUDE.md prescribed a verdict query that returns a stale review (…

Navigation Menu

Repository navigation

Allow `gh api` `--paginate --slurp` to be used with `--jq` #10459

Describe the feature or problem you’d like to solve

Proposed solution

Additional context

Tangentially Related

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow gh api --paginate --slurp to be used with --jq #10459

Description

Describe the feature or problem you’d like to solve

Proposed solution

Additional context

Tangentially Related

Activity

heaths commented on Feb 17, 2025

0xdevalias commented on Feb 17, 2025

williammartin commented on Feb 17, 2025

heaths commented on Feb 17, 2025

williammartin commented on Feb 17, 2025

heaths commented on Feb 17, 2025

williammartin commented on Feb 17, 2025

0xdevalias commented on Feb 18, 2025

heaths commented on Feb 18, 2025

0xdevalias commented on Feb 18, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Allow `gh api` `--paginate --slurp` to be used with `--jq` #10459