Skip to content

WebSocket upgrades fail intermittently under pm2: ERR_HTTP_SOCKET_ASSIGNED (@pm2/io httpMetrics vs @fastify/websocket) #545

Description

@melvincarvalho

Symptom

On a production JSS (0.0.204, node v22.21.1, Ubuntu) running under pm2, WebSocket upgrades to /.webrtc fail in intermittent bursts. Clients see either a failed connect or — worse — a half-open socket (TLS front returns 101, but the inner upgrade has already died, so the client gets silence: no messages, no error). Fresh-socket clients (curl, Node WebSocket) almost always succeed; browsers holding keep-alive connections to the same host intermittently hit it.

Logs

{"level":40,"err":{"type":"Error","message":"ServerResponse has an already assigned socket",
 "stack":"Error [ERR_HTTP_SOCKET_ASSIGNED]: ServerResponse has an already assigned socket
    at ServerResponse.assignSocket (node:_http_server:291:11)
    at Server.onUpgrade (.../node_modules/@fastify/websocket/index.js:56:19)
    at Server.emit (node:events:531:35)
    at Server.<anonymous> (.../lib/node_modules/pm2/node_modules/@pm2/io/build/main/metrics/httpMetrics.js:153:37)
    at onParserExecuteCommon (node:_http_server:977:14)",
 "code":"ERR_HTTP_SOCKET_ASSIGNED"},"msg":"websocket upgrade failed"}

Bursts of these correlate exactly with client-observed dead signaling connections.

Analysis

pm2's @pm2/io httpMetrics instrumentation wraps the HTTP server's events and sits in the upgrade path ahead of @fastify/websocket's onUpgrade. "Already assigned socket" points at upgrades arriving on reused keep-alive sockets (a connection that previously served a normal HTTP response), where the wrapper's ServerResponse handling collides with the upgrade's assignSocket. That matches the observed pattern: fresh sockets fine, browser keep-alive connections intermittent.

Possibly aggravated in this deployment by a pm2 daemon/CLI version mismatch (in-memory 6.0.14 vs local 4.5.6), so the injected @pm2/io came from a different pm2 than the one managing the process.

Suggested directions

  • register the upgrade handler so it runs ahead of (or is resilient to) external 'request'/'upgrade' instrumentation — e.g. guard onUpgrade against an already-assigned socket and destroy/retry instead of throwing through
  • document pm2 deployment guidance for websocket-bearing instances (metrics-off configuration, or systemd as the recommended supervisor when --webrtc/--tunnel/notifications are enabled)
  • a health probe: the /.webrtc route could expose a trivial HTTP GET response (instead of 404) so operators can distinguish "flag off" from "upgrade path broken"

Environment

  • jss 0.0.204 under pm2 (fork mode), node v22.21.1
  • mitigation being applied operationally: pm2 update + restart; will report recurrence here

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions