unexpected end of JSON input when sending a HEAD request with gh api #9927

Activity

1. Arthri

   added

   bugSomething isn't workingSomething isn't working

   on Nov 16, 2024
2. cliAutomation

   added

   needs-triageneeds to be reviewedneeds to be reviewed

   on Nov 16, 2024
3. andyfeller

   added

   gh-apirelating to the gh api commandrelating to the gh api command

   on Nov 18, 2024
4. andyfeller commented on Nov 18, 2024

   

   andyfeller

   on Nov 18, 2024

   Contributor

   More actions

   @Arthri : Could you go into greater detail why this shouldn't be an error?

   The GitHub API documentation doesn't officially state how HEAD method works with various endpoints such as Get a tag endpoint, however it is returning 404 response code because this tag does not exist.

   As you can see from pkg/api/rest_client.go in cli/go-gh, all 4XX and 5XX responses from the API are treated as client or server side errors respectively:

   // RequestWithContext issues a request with type specified by method to the
   // specified path with the specified body.
   // The response is returned rather than being populated
   // into a response argument.
   func (c *RESTClient) RequestWithContext(ctx context.Context, method string, path string, body io.Reader) (*http.Response, error) {
   	url := restURL(c.host, path)
   	req, err := http.NewRequestWithContext(ctx, method, url, body)
   	if err != nil {
   		return nil, err
   	}
   
   	resp, err := c.client.Do(req)
   	if err != nil {
   		return nil, err
   	}
   
   	success := resp.StatusCode >= 200 && resp.StatusCode < 300
   	if !success {
   		defer resp.Body.Close()
   		return nil, HandleHTTPError(resp)
   	}
   
   	return resp, err
   }

   gh can't determine from context whether a given endpoint and user's circumstances are expected or not. In most cases, working with a non-existing GitHub primitive is unintentional and thus an error; situations where people are attempting to discover information typically needing to implement special logic to avoid these response codes or exit codes to continue on.

   Anything you can do to elaborate on why HEAD requests for non-existent primitives shouldn't be treated as an error including excerpts from other tools and what use cases you're attempt to solve would be helpful!
5. andyfeller

   added

   more-info-neededMore info needed from user/contributorMore info needed from user/contributor

   on Nov 18, 2024
6. Arthri commented on Nov 19, 2024

   

   Arthri

   on Nov 19, 2024

   Author

   More actions

   Could you go into greater detail why this shouldn't be an error?

   This should be an error.

   I apologize for the misunderstanding. In retrospect, both commands I've noted are working as expected. I understand now that using gh api to read non-existing resources will cause the command to exit with a status code of 1 regardless of whether -X HEAD is specified, which is the normal behavior

   Anything you can do to elaborate on why HEAD requests for non-existent primitives shouldn't be treated as an error including excerpts from other tools and what use cases you're attempt to solve would be helpful!

   I use the API request to determine whether a tag exists without pulling all tags from the remote repository. In which case, I want it to fail specifically with a status code of 404. And, I realize now the paradoxical expectation I had in the issue

   ---

   I suppose that I was confused by the message unexpected end of JSON input, for two reasons

   1. I falsely believed that the message directly led to an exit code of 1, when it was instead the 404 status code that led to such
   2. The second command and API call (which had a status code of 200), didn't have a similar message, indicating that the response handler supported HEAD requests to an extent and terminated early instead of attempting to read an empty body

   Thank you for your response and clarification. Should I close the issue? I believe that my issue is resolved, however, I wouldn't mind if you'd like to keep the issue open to track the last point I mentioned

   ❤️React with ❤️1Reacted by Andy Feller
7. andyfeller commented on Nov 19, 2024

   

   andyfeller

   on Nov 19, 2024

   Last edited by andyfeller

   Contributor

   More actions

   I suppose that I was confused by the message unexpected end of JSON input

   Thank you for your response and clarification. Should I close the issue? I believe that my issue is resolved, however, I wouldn't mind if you'd like to keep the issue open to track the last point I mentioned

   Yeah, that isn't a great experience 🤔 unsure what can be done to improve handling the error raised in this scenario. Pivoting to curl to see what difference there is behind the responses.

   curl --verbose "https://api.github.com/repos/cli/cli/releases/tags/v2.62.0" -X HEAD

   Warning: Setting custom HTTP method to HEAD with -X/--request may not work the 
   Warning: way you want. Consider using -I/--head instead.
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host api.github.com:443 was resolved.
   * IPv6: (none)
   * IPv4: 140.82.114.6
   *   Trying 140.82.114.6:443...
   * Connected to api.github.com (140.82.114.6) port 443
   * ALPN: curl offers h2,http/1.1
   * (304) (OUT), TLS handshake, Client hello (1):
   } [319 bytes data]
   *  CAfile: /etc/ssl/cert.pem
   *  CApath: none
   * (304) (IN), TLS handshake, Server hello (2):
   { [122 bytes data]
   * (304) (IN), TLS handshake, Unknown (8):
   { [19 bytes data]
   * (304) (IN), TLS handshake, Certificate (11):
   { [3136 bytes data]
   * (304) (IN), TLS handshake, CERT verify (15):
   { [78 bytes data]
   * (304) (IN), TLS handshake, Finished (20):
   { [36 bytes data]
   * (304) (OUT), TLS handshake, Finished (20):
   } [36 bytes data]
   * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
   * ALPN: server accepted h2
   * Server certificate:
   *  subject: CN=*.github.com
   *  start date: Mar  7 00:00:00 2024 GMT
   *  expire date: Mar  7 23:59:59 2025 GMT
   *  subjectAltName: host "api.github.com" matched cert's "*.github.com"
   *  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Domain Validation Secure Server CA
   *  SSL certificate verify ok.
   * using HTTP/2
   * [HTTP/2] [1] OPENED stream for https://api.github.com/repos/cli/cli/releases/tags/v2.62.0
   * [HTTP/2] [1] [:method: HEAD]
   * [HTTP/2] [1] [:scheme: https]
   * [HTTP/2] [1] [:authority: api.github.com]
   * [HTTP/2] [1] [:path: /repos/cli/cli/releases/tags/v2.62.0]
   * [HTTP/2] [1] [user-agent: curl/8.7.1]
   * [HTTP/2] [1] [accept: */*]
   > HEAD /repos/cli/cli/releases/tags/v2.62.0 HTTP/2
   > Host: api.github.com
   > User-Agent: curl/8.7.1
   > Accept: */*
   > 
   * Request completely sent off
   < HTTP/2 200 
   < date: Tue, 19 Nov 2024 20:40:49 GMT
   < content-type: application/json; charset=utf-8
   < cache-control: public, max-age=60, s-maxage=60
   < vary: Accept,Accept-Encoding, Accept, X-Requested-With
   < etag: W/"7141dcef2b4dcfa7c00ebdefd3a2d4056b52d22e2938364fc40ef11dac82ff4f"
   < last-modified: Thu, 14 Nov 2024 16:12:31 GMT
   < x-github-media-type: github.v3; format=json
   < x-github-api-version-selected: 2022-11-28
   < access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
   < access-control-allow-origin: *
   < strict-transport-security: max-age=31536000; includeSubdomains; preload
   < x-frame-options: deny
   < x-content-type-options: nosniff
   < x-xss-protection: 0
   < referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
   < content-security-policy: default-src 'none'
   < server: github.com
   < x-ratelimit-limit: 60
   < x-ratelimit-remaining: 50
   < x-ratelimit-reset: 1732051961
   < x-ratelimit-resource: core
   < x-ratelimit-used: 10
   < accept-ranges: bytes
   < content-length: 44168
   < x-github-request-id: C8DD:34DB9A:E64EAB:1C1B7B9:673CF7D7
   < 
   { [0 bytes data]
   * transfer closed with 44168 bytes remaining to read
   
     0 44168    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
   * Connection #0 to host api.github.com left intact
   curl: (18) transfer closed with 44168 bytes remaining to read

   curl --verbose "https://api.github.com/repos/cli/cli/releases/tags/nonexistent" -X HEAD

   Warning: Setting custom HTTP method to HEAD with -X/--request may not work the 
   Warning: way you want. Consider using -I/--head instead.
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host api.github.com:443 was resolved.
   * IPv6: (none)
   * IPv4: 140.82.114.6
   *   Trying 140.82.114.6:443...
   * Connected to api.github.com (140.82.114.6) port 443
   * ALPN: curl offers h2,http/1.1
   * (304) (OUT), TLS handshake, Client hello (1):
   } [319 bytes data]
   *  CAfile: /etc/ssl/cert.pem
   *  CApath: none
   * (304) (IN), TLS handshake, Server hello (2):
   { [122 bytes data]
   * (304) (IN), TLS handshake, Unknown (8):
   { [19 bytes data]
   * (304) (IN), TLS handshake, Certificate (11):
   { [3136 bytes data]
   * (304) (IN), TLS handshake, CERT verify (15):
   { [79 bytes data]
   * (304) (IN), TLS handshake, Finished (20):
   { [36 bytes data]
   * (304) (OUT), TLS handshake, Finished (20):
   } [36 bytes data]
   * SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
   * ALPN: server accepted h2
   * Server certificate:
   *  subject: CN=*.github.com
   *  start date: Mar  7 00:00:00 2024 GMT
   *  expire date: Mar  7 23:59:59 2025 GMT
   *  subjectAltName: host "api.github.com" matched cert's "*.github.com"
   *  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Domain Validation Secure Server CA
   *  SSL certificate verify ok.
   * using HTTP/2
   * [HTTP/2] [1] OPENED stream for https://api.github.com/repos/cli/cli/releases/tags/nonexistent
   * [HTTP/2] [1] [:method: HEAD]
   * [HTTP/2] [1] [:scheme: https]
   * [HTTP/2] [1] [:authority: api.github.com]
   * [HTTP/2] [1] [:path: /repos/cli/cli/releases/tags/nonexistent]
   * [HTTP/2] [1] [user-agent: curl/8.7.1]
   * [HTTP/2] [1] [accept: */*]
   > HEAD /repos/cli/cli/releases/tags/nonexistent HTTP/2
   > Host: api.github.com
   > User-Agent: curl/8.7.1
   > Accept: */*
   > 
   * Request completely sent off
   < HTTP/2 404 
   < date: Tue, 19 Nov 2024 20:40:49 GMT
   < content-type: application/json; charset=utf-8
   < x-github-media-type: github.v3; format=json
   < x-github-api-version-selected: 2022-11-28
   < access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
   < access-control-allow-origin: *
   < strict-transport-security: max-age=31536000; includeSubdomains; preload
   < x-frame-options: deny
   < x-content-type-options: nosniff
   < x-xss-protection: 0
   < referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
   < content-security-policy: default-src 'none'
   < vary: Accept-Encoding, Accept, X-Requested-With
   < server: github.com
   < x-ratelimit-limit: 60
   < x-ratelimit-remaining: 49
   < x-ratelimit-reset: 1732051961
   < x-ratelimit-resource: core
   < x-ratelimit-used: 11
   < content-length: 147
   < x-github-request-id: C8DE:22DBE9:D600CE:1A28693:673CF7D7
   < 
   { [0 bytes data]
   * transfer closed with 147 bytes remaining to read
   
     0   147    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
   * Connection #0 to host api.github.com left intact
   curl: (18) transfer closed with 147 bytes remaining to read

   Let me raise the question to other maintainers as I'm unsure what can be done here.
8. andyfeller commented on Nov 19, 2024

   

   andyfeller

   on Nov 19, 2024

   Last edited by andyfeller

   Contributor

   More actions

   Stepping through the debugger, gh api commands are relying upon different logic than I previously assumed for processing REST and GraphQL calls. While it does treat any 4XX or 5XX responses as errors, I think it assumes the JSON response contains content when it doesn't:

   cli/pkg/cmd/api/api.go

   Lines 447 to 468 in 9177b22

   	func processResponse(resp *http.Response, opts *ApiOptions, bodyWriter, headersWriter io.Writer, template *template.Template, isFirstPage, isLastPage bool) (endCursor string, err error) {
   	if opts.ShowResponseHeaders {
   	fmt.Fprintln(headersWriter, resp.Proto, resp.Status)
   	printHeaders(headersWriter, resp.Header, opts.IO.ColorEnabled())
   	fmt.Fprint(headersWriter, "\r\n")
   	}
   	if resp.StatusCode == 204 {
   	return
   	}
   	var responseBody io.Reader = resp.Body
   	defer resp.Body.Close()
   	isJSON, _ := regexp.MatchString(`[/+]json(;|$)`, resp.Header.Get("Content-Type"))
   	var serverError string
   	if isJSON && (opts.RequestPath == "graphql" || resp.StatusCode >= 400) {
   	responseBody, serverError, err = parseErrorResponse(responseBody, resp.StatusCode)
   	if err != nil {
   	return
   	}
   	}

   See More

   cli/pkg/cmd/api/api.go

   Lines 610 to 624 in 9177b22

   	func parseErrorResponse(r io.Reader, statusCode int) (io.Reader, string, error) {
   	bodyCopy := &bytes.Buffer{}
   	b, err := io.ReadAll(io.TeeReader(r, bodyCopy))
   	if err != nil {
   	return r, "", err
   	}
   	var parsedBody struct {
   	Message string
   	Errors json.RawMessage
   	}
   	err = json.Unmarshal(b, &parsedBody)
   	if err != nil {
   	return bodyCopy, "", err
   	}

   In the following screenshot from debug session, you can see the b is empty when reading the response from GitHub API:

   
9. andyfeller

   removed

   more-info-neededMore info needed from user/contributorMore info needed from user/contributor

   on Nov 19, 2024
10. jsoref commented on Feb 7, 2025

    

    jsoref

    on Feb 7, 2025

    Contributor

    More actions

    I tripped on this with:

    gh api -X HEAD https://api.github.com/repos/cli/cli/commits/b77778291f650de438fc2072ff0c57340b40f00f
    unexpected end of JSON input

    There's no valid excuse to try to parse the body of a response to a HEAD request. The whole point of issuing a HEAD request is to tell the client and server not to provide a body.
11. jtmcg commented on Feb 7, 2025

    

    jtmcg

    on Feb 7, 2025

    Contributor

    More actions

    I tripped on this with:

    gh api -X HEAD https://api.github.com/repos/cli/cli/commits/b77778291f650de438fc2072ff0c57340b40f00f
    unexpected end of JSON input
    There's no valid excuse to try to parse the body of a response to a HEAD request. The whole point of issuing a HEAD request is to tell the client and server not to provide a body.

    Nice catch, @jsoref, and an elegant fix.

    That said, I can think of a valid excuse that we're trying to parse the body, here, and that's simply that this got overlooked before among the multitudes of priorities we juggle implementing and maintaining this project. You yourself as an open-source maintainer can surely empathize.

    That's why we appreciate folks like you and @Arthri bringing things like this to our attention and finding the quirks like this that have missed our attention. I'd request that you are a bit more gracious when highlighting these types of discoveries in the future.
12. jtmcg commented on Feb 7, 2025

    

    jtmcg

    on Feb 7, 2025

    Contributor

    More actions

    Acceptance Criteria

    Given I make an api call with the -X HEAD flag
    when the response status is >= 400
    then the response body is not returned
13. jtmcg

    added

    priority-3Affects a small number of users or is largely cosmeticAffects a small number of users or is largely cosmetic

    and removed

    needs-triageneeds to be reviewedneeds to be reviewed

    on Feb 7, 2025
14. jtmcg

    closed this as completedin #10388on Feb 7, 2025