Repository navigation
--paginate doesn't work well with /enterprises/{enterprise}/consumed-licenses and ?page= #8419
Description
Describe the bug
Version:
gh --version
gh version 2.39.2 (2023-11-27)
https://github.com/cli/cli/releases/tag/v2.39.2
When running gh api "https://api.github.com/enterprises/{enterprise}/consumed-licenses" --paginate, it returns a broken json object:
{
... # page 1
}
{
... # page 2
}
...
So the user must loop through the pages themselves and concatenate the content like so:
$page = 1
do {
$result = (& gh api "https://api.github.com/enterprises/{enterprise}/consumed-licenses?per_page=100&page=$page" | ConvertFrom-Json)
if ($result.users)
{
$knownEnterpriseUsers += $result.users
$page = $page + 1
}
} while ($result.users)Steps to reproduce the behavior
- Type this
gh api "https://api.github.com/enterprises/{enterprise}/consumed-licenses" --paginate - Output:
(& gh api "https://api.github.com/enterprises/{enterprise}/consumed-licenses" --paginate | ConvertFrom-Json)
ConvertFrom-Json: Conversion from JSON failed with error: Additional text encountered after finished reading JSON content: {. Path '', line 1, position 72640.
Expected vs actual behavior
The content is correctly concatenated and a single json object is returned
Related: #6044
Activity
jessehouwing commentedon Dec 7, 2023
Funny, if you add
--jq '.users[]'it actually stiches the results together and gives me proper JSON.andyfeller commentedon Dec 7, 2023
@jessehouwing : I think you're experiencing a common behavior of
gh api --paginatewhich is that we stream the output rather than aggregating all of it into a single response back. #1268 is an issue we're tracking about how we can improve that experience.In the mean time, you would need to leverage
jq -sto slurp up all of the fragments depending on your use case:That said, passing page and per_page to the
gh apicall works as expected:GH_DEBUG=api gh api /enterprises/{enterprise}/consumed-licenses --paginate -F page=2 -X GET -F per_page=10jessehouwing commentedon Dec 8, 2023
That's quite unexpected. But reading through it, I understand the complexity. Relying on the presence of jq is indeed not obvious, since it's not a common utility on windows. The fact that there is a
--jqargument would immediately lead me to think I should be able to solve it there...andyfeller commentedon Dec 8, 2023
That is a fair assessment.
Is there anything you would suggest we do outside of the larger work in #1268 for the short term to avoid the confusion?
For example, what might you expect to see in the
gh apilong description about the nature of the command and this workaround to make it more apparent?jessehouwing commentedon Dec 8, 2023
If it could add a separator somehow it would become much easier to split the resulting text into parts so that other scripting languages can more easily work with the result sets.
But for now I'd expect this behavior to be documented here:
And it mentions nothing of the sort. Many parts of
ghreturn json objects, so the fact that some commands return malformed json by design is an interesting choice. to me I started with a small dataset, so the issue only started to appear after the dataset grew. Quite frustrating ;).Ideally
ghwould register a powershell alias that would auto split the pages into pipeline objects so that it's transparent to PowerShell that this behavior exists.jessehouwing commentedon Dec 8, 2023
Copilot suggests doing something akin to:
which could probably be
andyfeller commentedon Dec 8, 2023
I think the way to look at it is that
gh api --paginateis returning a stream of valid JSON as it is processing bucket of data rather than a single aggregate response. You say malformed which isn't technically correct.jessehouwing commentedon Dec 8, 2023
True. Agreed. Yet there are few tools that will be able to parse the output by default. To them it's not something they understand.
samcoe commentedon Dec 11, 2023
Having read through this I am fairly confident that this issue is the same as #1268 which is potentially being addressed in #5652. I am going to close this out as a duplicate, but please let me know if you think that is in error.