Support getting both JSON and template output

Describe the feature or problem you’d like to solve

I'd like commands such as gh pr list to be able to give me both JSON output and to be able to apply a template.

The comment at #3012 (comment) notes as an aside:

Since filter and template are mutually exclusive ...

I'm using gh pr list in a (Bash) script that both generates pretty output for the user, and also does some other processing of the PRs.

I'd like to avoid having to run gh pr list twice.

What doesn't really work for me: add {{ . }} in the template. This gives me output in Go syntax, which I can't otherwise use.

Proposed solution

I have no particular preference for a solution. Obvious thought: add an option to output the JSON to a file.

Or, a completely different approach: add a new command that takes JSON input (from stdin and/or a file) and applies a template.

@rwalkerands

@rwalkerands : thank you for opening up this issue! 🤗

How would you expect to separate out multiple output streams if --template and --json both go to standard output?

I don't really see a clean way to achieve what you're hoping for unfortunately. That said, I can talk about how I typically work with JSON for scripting without duplicate gh calls.

Like mona-actions/gh-repo-stats extension, you can dump the JSON to a file on disk and work with it as base-64 strings allowing you to loop over entries with normal shell scripting:

  REPOS=$(echo "${PARSE_DATA}" | jq -r '.data.organization.repositories.nodes')
  for REPO_DATA in $(echo -n "${REPOS}" | jq -r '.[] | @base64'); do
    _jq() {
    echo -n "${REPO_DATA}" | base64 --decode | jq -r "${1}"
    }

    OWNER=$(_jq '.owner.login' | tr '[:upper:]' '[:lower:]')
    REPO_NAME=$(_jq '.name' | tr '[:upper:]' '[:lower:]')

This way, you can pull and even paginate data while limiting the number of calls you're trying to make.

I followed this issue because I had the same feature request, and am surprised this is considered too difficult to implement.

The template feature is just wrapping templating functionality in Go to my understanding, and the feature request is that the input to this template can be filtered using jq.

At least, in my case that was all I wanted, and was surprised to find it incompatible (so I had to pipe to jq manually, despite this being bundled in gh, making my script less portable).

I don't need to handle "multiple output streams" and I don't want to rely on writing to disk, this seems pretty simple: just pipe the output of the jq filter into the template format.

How would you expect to separate out multiple output streams if --template and --json both go to standard output?

I wouldn't, which is why I offered two proposed solutions which are different from that.

To repeat them:

Add an option to output the JSON to a file (i.e., with the filename specified as the parameter of the option).
Or, a completely different approach: add a new command that takes JSON input (from stdin and/or a file) and applies a template.

To expand on (2): add a new top-level command, say gh template --template <template>, that takes JSON input either from standard input or from a file specified as another command-line option (or, I guess the JSON could also be provided as a command-line argument), and applies the template provided as the command-line argument. (By "JSON input" I mean, of course, the output of a previous gh ... -jq ... command.)

I happen to like the implementation of templates provided by gh. Here's part of my script that produces a report on open PRs for a repo:

fields=assignees,author,baseRefName,body,createdAt,headRefName,headRepositoryOwner,id,labels,number,reviewDecision,reviews,statusCheckRollup,title,updatedAt,url
#PR=$(gh pr list -R "${REPO}" --json "${fields}"  --jq '.[]')

gh pr list -R "${REPO}" --json "${fields}" \
--template \
'====================================================================================================
{{range .}}
{{printf "#%v" .number | color "green"}} {{.title}}
{{.url}}
Author: {{.author.name}} (@{{.author.login}} https://github.com/{{.author.login}})
Created at: {{timeago .createdAt}}
Updated at: {{timeago .updatedAt}}
{{if .reviewDecision}}Review decision: {{.reviewDecision}}{{end -}}

This is a request to merge: {{.headRepositoryOwner.login}}/{{.headRefName}} -> {{.baseRefName}}

Description:

{{.body}}

{{if .labels}}{{tablerow "LABEL" "DESCRIPTION"}}{{range .labels}}{{tablerow .name .description}}{{end}}{{tablerender}}
{{end -}}
{{if .assignees}}{{tablerow "ASSIGNEE" "NAME"}}{{range .assignees}}{{tablerow .login .name}}{{end}}{{tablerender}}
{{end -}}
{{if .reviews}}{{tablerow "REVIEWER" "STATE" "COMMENT"}}{{range .reviews}}{{tablerow .author.login .state .body}}{{end}}{{tablerender}}
{{end -}}
{{if .statusCheckRollup}}{{tablerow "CHECK" "START_TIME" "STATE" "URL"}}{{range .statusCheckRollup}}{{tablerow .context .startedAt .state .targetUrl}}{{end}}{{tablerender}}
{{end -}}
====================================================================================================
{{end}}

'

Yes, I probably could implement this template using Bash/jq. But I'd rather not have to.

@rwalkerands

@rwalkerands : Thanks for clarifying your thoughts further as initially it seemed you might be asking for something that was impossible: having JSON and template output both streaming through standard out. Seeing that you and @lmmx were considering multi-phase processing via JSON + JQ then template processing, I believe this issue has come up previously as I recall asking a similar question.

At the time, the feedback I received was:

--jq and --template were both designed to convert JSON into something more convenient for terminals. I guess you could in theory use --jq to massage data passed to --template (if we allowed the combination of flags) but I think allowing that would be fiddly and not lead to great user experiences. it would be better to add helpers to --template to make massaging data there more easier

Personally, I'm open to the idea still though the --jq support isn't 1:1 to jq; see https://github.com/itchyny/gojq#difference-to-jq for more information. Understanding how this will play out with @heaths changes in #1268 for being able to provide a single merged JSON response from the GitHub API will be interesting.

Or, a completely different approach: add a new command that takes JSON input (from stdin and/or a file) and applies a template.

This is certainly outside of the scope of what the GitHub CLI team and maintainers can support. It is something I myself have looked into for similar reasons however I didn't truly find anything that allowed users to arbitrary define Go templates and pass in JSON via file or standard input.

Add an option to output the JSON to a file (i.e., with the filename specified as the parameter of the option).

Currently, this wouldn't work or work well given commands that require a --paginate flag or have to iterate on multiple pages of data from the API.

Why not just have gh emit JSON, tee that, and send stdout through jq or some Go templating app. Off hand I'm not aware of one, but you could use a similar design to my extension https://github.com/heaths/gh-merge-json (gh ext install heaths/gh-merge-json) that is an alternative to #1268 until I was allowed to complete that work (soon).

E.g.,

gh api ... | tee output.json | jq ...

Having a CLI do "everything" seems not only a maintenance issue, but also limiting: it's not always going to work the way people want, and as a consequence devs keep having to add knobs which increases maintenance costs for complexity. Keeping it flexible might be better.

If you want something easy to use in scripts, consider at least keeping it in the ecosystem as an extension. You could easily have an extension both tee and format JSON using Go templates and install that if needed with the --force switch I added back in #7173 for that very purpose that I use it for in many of our scripts:

#!/usr/bin/env bash

# this will install or uprade the extension, or no-op if already installed
gh ext install --force heaths/gh-merge-json

# using `gh ext exec` is good for scripts and helps avoid future possible ambiguity, which happened to me a while back
gh api ... | gh ext exec merge-json

Just my $0.02.

I added support to merge-json in heaths/gh-merge-json#6 and pushed a new release. Seems appropriate, given that the merge-json extension requires JSON input anyway...for which I also fixed an input validation bug prior to this PR.

Example usage:

gh ext install --force heaths/gh-merge-json > /dev/null # or just $null in pwsh
gh api graphql --paginate -f query='
    query($endCursor: String) {
      viewer {
        repositories(first: 100, after: $endCursor) {
          nodes { nameWithOwner }
          pageInfo {
            hasNextPage
            endCursor
          }
        }
      }
    }
  ' | gh merge-json --template '{{range .data.viewer.repositories.nodes}}{{tablerow .nameWithOwner}}{{end}}' --tee repos.json

Why not just have gh emit JSON, tee that, and send stdout through jq ...

Because using jq to implement the sort of template(s) I want to have (see #8415 (comment)) appears to be quite cumbersome ... if it is even possible. (E.g., how to do {{timeago .createdAt}}? An inability to do that wouldn't be a showstopper, but it's nice to be able to mimic what's shown in the GitHub web interface.)

or some Go templating app. Off hand I'm not aware of one,

You answered your own question. I already looked, but didn't find. Very happy to be pointed at a small standalone-app that does JSON through a Go-like template to text.

Indeed, thank you for the pointer to https://github.com/heaths/gh-merge-json ; I have now tried it and confirmed that it works for me.

@heaths

Thanks for sharing your insights, @heaths ❤️ There is definitely a fine line of how GitHub CLI aligns with other tools in the Unix design principle we're trying to exercise being an extremely small team.

All of that to say, @rwalkerands , I hope you have some options for short/medium term workarounds as discussed above while I bring this up for discussion with the other maintainers on ramifications of refactoring the code involved:

cli/pkg/cmdutil/json_flags.go

Lines 185 to 220 in 57f6787

    
           // Write serializes data into JSON output written to w. If the object passed as data implements exportable, 
        
           // or if data is a map or slice of exportable object, ExportData() will be called on each object to obtain 
        
           // raw data for serialization. 
        
           func (e *jsonExporter) Write(ios *iostreams.IOStreams, data interface{}) error { 
        
           	buf := bytes.Buffer{} 
        
           	encoder := json.NewEncoder(&buf) 
        
           	encoder.SetEscapeHTML(false) 
        
           	if err := encoder.Encode(e.exportData(reflect.ValueOf(data))); err != nil { 
        
           		return err 
        
           	} 
        
           	w := ios.Out 
        
           	if e.filter != "" { 
        
           		indent := "" 
        
           		if ios.IsStdoutTTY() { 
        
           			indent = "  " 
        
           		} 
        
           		if err := jq.EvaluateFormatted(&buf, w, e.filter, indent, ios.ColorEnabled()); err != nil { 
        
           			return err 
        
           		} 
        
           	} else if e.template != "" { 
        
           		t := template.New(w, ios.TerminalWidth(), ios.ColorEnabled()) 
        
           		if err := t.Parse(e.template); err != nil { 
        
           			return err 
        
           		} 
        
           		if err := t.Execute(&buf); err != nil { 
        
           			return err 
        
           		} 
        
           		return t.Flush() 
        
           	} else if ios.ColorEnabled() { 
        
           		return jsoncolor.Write(w, &buf, "  ") 
        
           	} 
        
           	_, err := io.Copy(w, &buf) 
        
           	return err 
        
           }

@rwalkerands

All of that to say, @rwalkerands, I hope you have some options for short/medium term workarounds as discussed above while I bring this up for discussion with the other maintainers ...

Indeed, I am now using gh merge-json --template and it does what I need.

@rwalkerands

[shameless plug] @rwalkerands if you like it, consider starring heaths/gh-merge-json to to help discovery on https://github.com/topics/gh-extension 😉 If not, that's cool too. Won't change support or feature ideas unlike some repos in the news recently. Just helps others find it easier in case it might help them.

On that note: @andyfeller it'd be cool if GitHub defaulted a search from a topic to include that topic when you hit / or click in the search box e.g. topic:gh-extension merge json. I use search a lot and didn't even realize topic: was a valid prefix. Just guessed it might work and tried it with pleasant surprise!

@andyfeller

On that note: @andyfeller it'd be cool if GitHub defaulted a search from a topic to include that topic when you hit / or click in the search box e.g. topic:gh-extension merge json. I use search a lot and didn't even realize topic: was a valid prefix. Just guessed it might work and tried it with pleasant surprise!

Kind of like jumping to search within a repo will add repo:..., owner:..., or repo:... path:....

@rwalkerands

At the time, the feedback I received was:

--jq and --template were both designed to convert JSON into something more convenient for terminals. I guess you could in theory use --jq to massage data passed to --template (if we allowed the combination of flags) but I think allowing that would be fiddly and not lead to great user experiences. it would be better to add helpers to --template to make massaging data there more easier

I recently raised this, which was closed as a dupe of this issue:

Allow --jq and --template to be used together in gh api #10261

I'd like to add my support for having whatever data manipulations I provide in --jq passed through to --template as a 'pipeline'.

It might be fiddly, but that's something I as a user should be allowed to choose to opt into IMO. There shouldn't be any backwards compatibility issues as --jq + --template is currently disallowed.

Personally I find the fact that I can't use both together a worse user experience.

I have no strong opinions on whether --template could also be improved to make massaging data there easier.. but IMO that should be a separate enhancement; that would then give users more choice as to whether they want to use the fiddly --jq + --template, or if they want to use whatever new features may eventually end up in --template.

tl;dr: Supporting --jq + --template feels like a quick/easy win, whereas changes to --template feels like a larger scope with more potential for delays/scope creep/etc.

Because using jq to implement the sort of template(s) I want to have (see #8415 (comment)) appears to be quite cumbersome ... if it is even possible. (E.g., how to do {{timeago .createdAt}}?

@rwalkerands Since I hacked something together for this recently, this was how I solved it:

https://github.com/0xdevalias/dotfiles/blob/843359eb91e2530b27d27baccb5a08f707bd1a63/bin/gh-notifications#L75-L87

def timeago:
  . | fromdateiso8601 | now - . | . / 60 | floor |
  if . < 60 then
    (. | tostring + \" minutes ago\")
  elif . < 1440 then
    (. / 60 | floor | tostring + \" hours ago\")
  elif . < 43200 then
    (. / 1440 | floor | tostring + \" days ago\")
  elif . < 525600 then
    (. / 43200 | floor | tostring + \" months ago\")
  else
    (. / 525600 | floor | tostring + \" years ago\")
  end;

I also raised this, which might be helpful if possible/implemented

Add custom formatting functions to --jq similar to those added to --template #10262

Most commands use github.com/cli/cli/cmdutil.AddJSONFlags or .../cmdutil.AddFormatFlags, which do not enforce --jq and --template to be mutually exclusive:

gh issue list --json number,title --jq 'sort_by(.title)' --template '{{range .}}{{.title}}{{end}}

However, in this example since --jq is processed first, it writes directly to stdout. Given that, supporting --jq processing before --template could be considered a breaking change; but, I still think it's a good idea if you want to do additional processing.

gh api does treat --jq and --template as mutually exclusive, though. With a little refactoring it could be possible to serialize processing --jq then --template.

Process `--jq` before `--template`

Support filtering before applying template

@andyfeller

@andyfeller after an initial attempt to do this entirely in cli/cli, while it was pretty straight forward - though it felt a little tacky - in the cmdutil package, the api command would've felt even more hacky given where the template was initialized and how even more disjoint the input and output buffers were across functions.

With that in mind, and because I think jq filters offer a lot more flexibility for extension developers than Go's simplistic (in mostly good ways) template engine, I created cli/go-gh#177 with a very simple change to use the existing jq package and is much less code churn than in cli/cli (see the first commit for this PR, which I left to give you some idea; imagine worse for the api command).

@heaths

@heaths @0xdevalias we see the value of this, and it doesn't seem particularly tricky to implement based on the proposed PRs. That said, we are a bit swamped at the moment with other work so it may take a little time to ensure we are confident in the UX and documentation that needs to go with it.

@heaths do you see value for extension authors putting this in go-gh, or was it mainly a matter of ease/simplicity in the solution?

Both. It definitely keeps the code in the CLI cleaner, but also allows extension authors to easily filter JSON output before applying templates (baked in or otherwise).

E.g., say you wanted to output the top 10 issues by vote in a nice, neat table. If your GraphQL statement had something like:

reactions(content: THUMBS_UP) {
  totalCount
}

You can easily morph and filter:

t := template.New(ios.Out, ios.Width(), ios.ColorEnabled());
t.Parse("{{range .}}{{tablerow .number .title .votes}}{{end}}")
template.ExecuteFiltered(r,
    "map({number, title, votes: .reactions.totalCount}) | sort_by(.votes) | reverse | .[:2]")

Much easier than all the Go code to do the same.

@williammartin

@williammartin I wanted to circle back and see if there's been any further discussion internally on whether to accept this and related PRs.

There hasn't been any further discussion around this issue specifically. We have discussed that we are uncomfortable accepting changes to gh api until we try to disentangle some of it. Even though the changes in #10273 are relatively small, every time we accept the addition of another conditional check, we make it harder for ourselves to address in the future.

I'm borderline on this one, (comparatively, I would be a no on #10459 until we fix it). I'm going to use the discuss label on this so we get back to you with a clear answer.

Not trying to revive a dead post (6 months looool), but just wanted to chime in saying that this would be a very very nice feature for my own workflow and it's a shame it's stalled out so long.

While it'd obviously be nice were this resolved soon, given how slowly this has seemed to progress, I made my own gh extension as a workaround/fix in the meantime¹ (which effectively just passes the jq output directly to go-gh/pkg/template).

I had already started work on it out of sheer annoyance before stumbling across this issue lol ↩

Navigation Menu

Repository navigation

Support getting both JSON and template output #8415

Describe the feature or problem you’d like to solve

Proposed solution

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// Write serializes data into JSON output written to w. If the object passed as data implements exportable,
	// or if data is a map or slice of exportable object, ExportData() will be called on each object to obtain
	// raw data for serialization.
	func (e jsonExporter) Write(ios iostreams.IOStreams, data interface{}) error {
	buf := bytes.Buffer{}
	encoder := json.NewEncoder(&buf)
	encoder.SetEscapeHTML(false)
	if err := encoder.Encode(e.exportData(reflect.ValueOf(data))); err != nil {
	return err
	}

	w := ios.Out
	if e.filter != "" {
	indent := ""
	if ios.IsStdoutTTY() {
	indent = " "
	}
	if err := jq.EvaluateFormatted(&buf, w, e.filter, indent, ios.ColorEnabled()); err != nil {
	return err
	}
	} else if e.template != "" {
	t := template.New(w, ios.TerminalWidth(), ios.ColorEnabled())
	if err := t.Parse(e.template); err != nil {
	return err
	}
	if err := t.Execute(&buf); err != nil {
	return err
	}
	return t.Flush()
	} else if ios.ColorEnabled() {
	return jsoncolor.Write(w, &buf, " ")
	}

	_, err := io.Copy(w, &buf)
	return err
	}

Support getting both JSON and template output #8415

Description

Describe the feature or problem you’d like to solve

Proposed solution

Activity

andyfeller commented on Jan 2, 2024

lmmx commented on Jan 2, 2024

rwalkerands commented on Jan 8, 2024

andyfeller commented on Jan 9, 2024

heaths commented on Jan 9, 2024

heaths commented on Jan 9, 2024

rwalkerands commented on Jan 10, 2024

andyfeller commented on Jan 16, 2024

rwalkerands commented on Jan 16, 2024

heaths commented on Jan 16, 2024

1 remaining item

andyfeller commented on Feb 12, 2024

0xdevalias commented on Jan 18, 2025

heaths commented on Jan 19, 2025

heaths commented on Jan 19, 2025

williammartin commented on Jan 24, 2025

heaths commented on Jan 24, 2025

heaths commented on Mar 13, 2025

williammartin commented on Mar 19, 2025

Bertie690 commented on Oct 1, 2025

Footnotes

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions