Skip to content

fix: retry once on stale keep-alive sockets [CHA-4943] - #80

Closed
itsmeadi wants to merge 2 commits into
masterfrom
cha-stale-keepalive-retry
Closed

itsmeadi wants to merge 2 commits into
masterfrom
cha-stale-keepalive-retry

Conversation

@itsmeadi

Copy link
Copy Markdown
Collaborator

Ticket

Summary

Follow-up to #78 (idle_timeout 55→25). PrizePicks still hit GetStreamRuby::TransportError on reused keep-alive sockets (SSL_read EOF, Connection reset by peer, Net::ReadTimeout on a closed TCPSocket). Faraday/net_http_persistent does not retry those; Go net/http does for unused idle connections.

Retry the request once, with no backoff, when the failure looks like a dead pooled socket. Applies to POST as well as GET. DNS failures and real read timeouts are not retried. The opt-in retry_config: policy is unchanged.

Previous PR

Checklist

  • The changed code has been covered with unit tests
  • API endpoints are covered with client tests
  • The internal documentation (./docs) has been updated

Notes for review

  • Multipart uploads are not retried (they skip Client#request before the retry loop).
  • PrizePicks is still on 8.0.2; this needs a gem release after merge.

Made with Cursor

…4943]

Faraday/net_http_persistent reuses dead sockets and surfaces SSL EOF, RST,
or ReadTimeout-on-closed-socket as TransportError. Match Go net/http by
retrying unused idle connections once with no backoff, including POST.
DNS failures and real read timeouts are not retried.

Co-authored-by: Cursor <[email protected]>
SSLError and ConnectionFailed also cover cert failures and connection
refused. Retry once only when the message says the pooled socket is
already dead.

Co-authored-by: Cursor <[email protected]>
@itsmeadi itsmeadi changed the title [CHA-4943] Retry once on stale keep-alive sockets fix: retry once on stale keep-alive sockets [CHA-4943] Aug 31, 2026

@mogita mogita left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment, needs discussion. 3 Must Fix, 2 Should Fix. Focused and full unit tests pass. The new targeted cases are clear and stable.

handle_response(response)
rescue Faraday::Error => e
error = TransportError.new("Request failed: #{e.message}", error_type: ErrorMapping.classify_faraday_error(e))
if stale_retries.zero? && ErrorMapping.stale_keep_alive?(e)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This retries every HTTP method after errors that can arrive while reading the response. The server may already have applied a POST, PATCH, or DELETE, so the retry can repeat a write. I reproduced a POST whose complete body reached the server twice. Restrict automatic retries to safe methods, or require a server enforced idempotency key for writes.

STALE_KEEP_ALIVE_PATTERN = /
connection\ reset\ by\ peer
|unexpected\ eof
|tcpsocket:\(closed\)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TCPSocket:(closed) is also the normal message after a real Net::ReadTimeout because Net::HTTP closes the socket before Faraday formats the exception. A local real timeout sent two requests through this branch. Remove this match or detect a stale pooled socket without using the closed socket suffix.

error = TransportError.new("Request failed: #{e.message}", error_type: ErrorMapping.classify_faraday_error(e))
if stale_retries.zero? && ErrorMapping.stale_keep_alive?(e)
log_retry_attempt(method, path, error, stale_retries, started)
stale_retries += 1

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stale_retries does not increment attempt, so this retry sits outside retry_config.max_attempts. With max_attempts: 3, I reproduced four requests. Count this retry in the same total attempt budget.

Comment thread CHANGELOG.md
Detected from the error message, not Faraday class (so cert failures and
connection refused are not retried). Applies to POST as well as GET; DNS
failures and real read timeouts are not retried. No backoff; the opt-in
`retry_config:` policy is unchanged.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README.md still says retries are opt in, disabled clients make exactly one attempt, and writes are never retried. This change makes one retry automatic and retries writes. Update the public retry section to describe the new default.

|broken\ pipe
|connection\ is\ closed
|end\ of\ file\ reached
|tls_retry_write_records

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Four accepted messages have no tests: broken pipe, connection is closed, end of file reached, and tls_retry_write_records. The wrapped exception message branch is also only tested for DNS. Add table cases for every accepted message, including one stale message from wrapped_exception.

@itsmeadi

itsmeadi commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

@itsmeadi itsmeadi closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants