Symptom
On a production JSS (0.0.204, node v22.21.1, Ubuntu) running under pm2, WebSocket upgrades to /.webrtc fail in intermittent bursts. Clients see either a failed connect or — worse — a half-open socket (TLS front returns 101, but the inner upgrade has already died, so the client gets silence: no messages, no error). Fresh-socket clients (curl, Node WebSocket) almost always succeed; browsers holding keep-alive connections to the same host intermittently hit it.
Logs
{"level":40,"err":{"type":"Error","message":"ServerResponse has an already assigned socket",
"stack":"Error [ERR_HTTP_SOCKET_ASSIGNED]: ServerResponse has an already assigned socket
at ServerResponse.assignSocket (node:_http_server:291:11)
at Server.onUpgrade (.../node_modules/@fastify/websocket/index.js:56:19)
at Server.emit (node:events:531:35)
at Server.<anonymous> (.../lib/node_modules/pm2/node_modules/@pm2/io/build/main/metrics/httpMetrics.js:153:37)
at onParserExecuteCommon (node:_http_server:977:14)",
"code":"ERR_HTTP_SOCKET_ASSIGNED"},"msg":"websocket upgrade failed"}
Bursts of these correlate exactly with client-observed dead signaling connections.
Analysis
pm2's @pm2/io httpMetrics instrumentation wraps the HTTP server's events and sits in the upgrade path ahead of @fastify/websocket's onUpgrade. "Already assigned socket" points at upgrades arriving on reused keep-alive sockets (a connection that previously served a normal HTTP response), where the wrapper's ServerResponse handling collides with the upgrade's assignSocket. That matches the observed pattern: fresh sockets fine, browser keep-alive connections intermittent.
Possibly aggravated in this deployment by a pm2 daemon/CLI version mismatch (in-memory 6.0.14 vs local 4.5.6), so the injected @pm2/io came from a different pm2 than the one managing the process.
Suggested directions
- register the upgrade handler so it runs ahead of (or is resilient to) external 'request'/'upgrade' instrumentation — e.g. guard
onUpgrade against an already-assigned socket and destroy/retry instead of throwing through
- document pm2 deployment guidance for websocket-bearing instances (metrics-off configuration, or systemd as the recommended supervisor when
--webrtc/--tunnel/notifications are enabled)
- a health probe: the
/.webrtc route could expose a trivial HTTP GET response (instead of 404) so operators can distinguish "flag off" from "upgrade path broken"
Environment
- jss 0.0.204 under pm2 (fork mode), node v22.21.1
- mitigation being applied operationally:
pm2 update + restart; will report recurrence here
Symptom
On a production JSS (0.0.204, node v22.21.1, Ubuntu) running under pm2, WebSocket upgrades to
/.webrtcfail in intermittent bursts. Clients see either a failed connect or — worse — a half-open socket (TLS front returns 101, but the inner upgrade has already died, so the client gets silence: no messages, no error). Fresh-socket clients (curl, NodeWebSocket) almost always succeed; browsers holding keep-alive connections to the same host intermittently hit it.Logs
Bursts of these correlate exactly with client-observed dead signaling connections.
Analysis
pm2's
@pm2/iohttpMetrics instrumentation wraps the HTTP server's events and sits in the upgrade path ahead of@fastify/websocket'sonUpgrade. "Already assigned socket" points at upgrades arriving on reused keep-alive sockets (a connection that previously served a normal HTTP response), where the wrapper's ServerResponse handling collides with the upgrade'sassignSocket. That matches the observed pattern: fresh sockets fine, browser keep-alive connections intermittent.Possibly aggravated in this deployment by a pm2 daemon/CLI version mismatch (in-memory 6.0.14 vs local 4.5.6), so the injected
@pm2/iocame from a different pm2 than the one managing the process.Suggested directions
onUpgradeagainst an already-assigned socket and destroy/retry instead of throwing through--webrtc/--tunnel/notifications are enabled)/.webrtcroute could expose a trivial HTTP GET response (instead of 404) so operators can distinguish "flag off" from "upgrade path broken"Environment
pm2 update+ restart; will report recurrence here