fix(conneg): HEAD mirrors GET's negotiated content type for files — closes #552 by melvincarvalho · Pull Request #553 · JavaScriptSolidServer/JavaScriptSolidServer

melvincarvalho · 2026-06-10T19:40:46Z

Closes #552.

Bug

GET negotiates file content types (q-aware since #325), but HEAD's file branch reported the stored type regardless of Accept — an RFC 9110 §9.3.2 violation. Clients probing with HEAD saw a content type the subsequent GET never returned:

# GET negotiates:
curl -s /public/test.jsonld -H 'Accept: text/turtle, application/ld+json' …  # → text/turtle
# HEAD did not:
curl -sI /public/test.jsonld -H 'Accept: text/turtle, application/ld+json' … # → application/ld+json

The directory branch of handleHead already negotiated (#325 covered it) — files were the gap.

Fix

New negotiateHeadFileContentType helper mirrors handleGet's file-branch decision tree exactly: negotiated Turtle/JSON-LD for RDF-stored files (gated on the same JSON-parse-success check GET applies), HTML-data-island → Turtle (island presence + island parseability checked, like GET), and the extensionless octet-stream HTML sniff GET applies even without conneg.

Design points (each documented in code)

Concern	Resolution
HEAD reading files	Bounded: reads only when the stored type makes content relevant AND size ≤ `HEAD_SNIFF_MAX_BYTES` (1 MiB). GET reads everything anyway (it sends the body); HEAD on a multi-GB media file must not slurp it to compute a header. Above the cap, stored type.
304 ordering	GET 304s files before any read; HEAD now defers negotiation until after its `If-None-Match` check, so a 304 pays no negotiation I/O.
Content-Length honesty	When GET would convert the body (Turtle conversion / JSON-LD re-serialization), its length ≠ on-disk size — HEAD now omits Content-Length in exactly those cases instead of claiming `stats.size` for a body GET never sends. The extensionless HTML sniff merely relabels as-is bytes, so it keeps Content-Length. (This was a pre-existing inaccuracy that correct content-types would have made more visible.)
Residual divergence	If GET's conversion fails after a successful JSON parse (`fromJsonLd` error), GET falls back to raw bytes while HEAD has committed to the negotiated type. Requires a parseable-but-unconvertible document; a full conversion dry-run per HEAD isn't worth it. Documented in the helper docstring.

Tests

test/head-conneg.test.js (5 cases):

Parity loop — HEAD media type byte-equals GET's across four Accept variants (ld+json-first, turtle-first, q-weighted, no header). Charset parameter excluded with a documented reason: Fastify appends ; charset=utf-8 only when serializing a string body, which HEAD lacks — it's a serialization artifact, not a negotiation outcome.
The conneg: HEAD ignores the Accept header — content-type diverges from GET on the same resource #552 turtle-preference repro directly.
Content-Length omitted on converted responses.
Content-Length kept on as-is responses (extensionless HTML sniff).
If-None-Match revalidation still 304s.

Full suite: 942/942 passing.

Refs

Authenticated GET on a container returns Turtle body with application/ld+json content-type, ignoring Accept #325 — the GET-side q-weight fix this brings HEAD into line with
Conneg ignores Accept header priority #71 — closed earlier today; its verification surfaced this gap
Content type is coupled to document extension #460 — extensionless content-type coupling (adjacent, separately tracked; the sniff-parity here matches current GET behaviour, whatever Content type is coupled to document extension #460 decides later applies to both paths through the shared decision logic)

GET negotiates file content types (q-aware since #325), but HEAD's file branch reported the STORED type regardless of Accept — an RFC 9110 §9.3.2 violation (HEAD should send the same header fields as GET). Clients probing with HEAD saw a content type the subsequent GET never returned. The directory branch already negotiated (#325); files were the gap. Fix: new negotiateHeadFileContentType helper mirrors handleGet's file-branch decision tree — negotiated Turtle/JSON-LD for RDF-stored files (gated on the same JSON-parse-success check GET applies), HTML-data-island → Turtle (island presence + parseability checked, like GET), and the extensionless octet-stream HTML sniff that GET applies even without conneg. Design points: - Bounded read: HEAD only reads the file when the stored type makes content relevant AND size <= HEAD_SNIFF_MAX_BYTES (1 MiB). GET reads everything anyway (it sends the body); HEAD on a multi-GB media file must not slurp it to compute a header. Above the cap HEAD reports the stored type. - 304 ordering preserved: GET 304s files BEFORE any read, so HEAD defers negotiation until after its If-None-Match check — a 304 pays no negotiation I/O. - Content-Length honesty: when GET would CONVERT the body (Turtle conversion or JSON-LD re-serialization via fromJsonLd), its body length is not the on-disk size — HEAD now omits Content-Length in exactly those cases instead of claiming stats.size for a body GET never sends. The extensionless HTML sniff only relabels as-is bytes, so it keeps Content-Length. - Known residual divergence (documented in the helper docstring): if GET's conversion fails AFTER a successful JSON parse, GET falls back to raw bytes while HEAD has committed to the negotiated type. Requires a parseable-but-unconvertible document; not worth a full conversion dry run per HEAD. Tests (test/head-conneg.test.js, 5 cases): HEAD/GET media-type parity loop across four Accept variants (charset parameter excluded — Fastify appends it only when serializing a string body, which HEAD lacks), the #552 turtle-preference repro, Content-Length omitted on converted / kept on as-is responses, and If-None-Match revalidation still 304s. Full suite: 942/942 passing. Closes #552.

Copilot

Pull request overview

This PR fixes HTTP content negotiation parity between HEAD and GET for file resources so that HEAD reports the same negotiated Content-Type as a corresponding GET (addressing #552 / RFC 9110 §9.3.2).

Changes:

Add negotiateHeadFileContentType to mirror handleGet’s file-branch content-type decision logic for HEAD.
Defer file negotiation until after If-None-Match handling to avoid unnecessary I/O on 304 responses.
Update HEAD Content-Length behavior to omit it when GET would convert/re-serialize the body; add tests covering parity and Content-Length rules.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`src/handlers/resource.js`	Adds file negotiation helper for `HEAD`, adjusts ordering around 304s, and omits `Content-Length` when the body would be converted.
`test/head-conneg.test.js`	New tests asserting `HEAD`/`GET` media-type parity for files plus `Content-Length` and 304 revalidation behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

melvincarvalho · 2026-06-10T19:58:13Z

+  const contentRelevant =
+    (connegEnabled && (isRdfContentType(storedContentType) || storedContentType === 'text/html')) ||
+    storedContentType === 'application/octet-stream';
+  if (!contentRelevant || stats.size > HEAD_SNIFF_MAX_BYTES) {
+    return { contentType: storedContentType, converted: false };
+  }


Legitimate catch — fixed in 1f95667 with a per-case degradation instead of the flat cap:

RDF-stored above the cap (your exact example): returns the negotiated type WITHOUT the parse gate — optimistic. The gate only mirrors GET's corrupt-file fallback, and a corrupt >1 MiB RDF document is far rarer than a valid one, so parity holds for the common case and the divergence shrinks to that corner. New test: large valid JSON-LD + Accept: text/turtle → text/turtle on HEAD, Content-Length omitted.

Extensionless HTML sniff: only needs the first bytes — now runs at ANY size via a bounded 1 KiB ranged read (storage.createReadStream({start, end}), the Range handler's API). New test: >1 MiB extensionless HTML sniffs to text/html with Content-Length kept.

HTML + data island above the cap: stays conservative (text/html) — the island can sit anywhere in the file, so presence can't be verified without the full read this cap exists to avoid. Documented as a residual divergence in the helper docstring alongside the other two corners.

Constant renamed HEAD_SNIFF_MAX_BYTES → HEAD_FULL_READ_MAX_BYTES since it now caps full reads only. 944/944 passing.

…otiation Copilot's finding was legitimate: the flat HEAD_SNIFF_MAX_BYTES early return regressed parity for >1 MiB resources — a large VALID JSON-LD file with Accept: text/turtle fell back to the stored type on HEAD, and the extensionless HTML sniff was disabled entirely above the cap. Upgrade: - RDF-stored above the cap: return the negotiated type WITHOUT the parse gate (optimistic). The gate only mirrors GET's corrupt-file fallback; a corrupt >1 MiB RDF document is far rarer than a valid one, so optimism keeps parity for the common case and confines the divergence to that corner. (This is exactly Copilot's example, fixed.) - Extensionless HTML sniff: only the first bytes matter — now runs at ANY size via a bounded 1 KiB ranged read (readFirstBytes via storage.createReadStream, same API the Range handler uses). - HTML + data island above the cap: stays conservative (text/html) — the island can sit anywhere in the file, so presence can't be checked without a full read. Documented residual. - Also restored from the first draft: the island check keys off CONTENT, not the stored type, so an extensionless HTML file with an island negotiates to Turtle like GET does (the restructure had briefly gated it on text/html-stored). Constant renamed HEAD_SNIFF_MAX_BYTES → HEAD_FULL_READ_MAX_BYTES: it now caps full-content reads only; O(1) chunk sniffs are always allowed. Helper docstring rewritten to enumerate the per-case large-file behaviour and the three residual divergences. Two new tests: large valid RDF negotiates on HEAD (Copilot's case, with Content-Length omitted), large extensionless HTML sniffs via the ranged read (with Content-Length kept). 944/944 passing.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

melvincarvalho · 2026-06-10T20:10:39Z

+ * GET's decision depends on file CONTENT in three places — the
+ * HTML-data-island sniff, the JSON-parse-success gate before conneg
+ * conversion, and the extensionless-file HTML sniff — so this may read
+ * the file, but only when the stored type makes content relevant and
+ * the file is within HEAD_SNIFF_MAX_BYTES.
+ *


Fixed in 605d4bd — the docstring now references HEAD_FULL_READ_MAX_BYTES. Fair catch; the constant was renamed in pass 1 and this reference was missed.

melvincarvalho · 2026-06-10T20:10:41Z

@@ -712,8 +870,10 @@
  });

  // Content-Length: only set when the file size matches the response body.
-  // Mashlib HTML and containers are dynamically generated.
-  if (!stats.isDirectory && !isMashlibResponse) {
+  // Mashlib HTML and containers are dynamically generated, and a
+  // conneg-converted body (Turtle / re-serialized JSON-LD, #552) has a
+  // different length than the on-disk file — omit rather than lie.
+  if (!stats.isDirectory && !isMashlibResponse && !negotiationConverted) {


Done in 605d4bd. HEAD now mirrors GET's uniform rule: isRdfContentType(contentType) → RDF_CACHE_CONTROL. One scope note: your comment mentioned files, but GET also sets it on container listings (resource.js:243/257/364/383), so the mirror covers containers too — same parity contract. The test parity loop now compares Cache-Control across all four Accept variants, plus a new container-listing parity case. 945/945 passing.

Two Copilot findings, both legitimate: 1. The helper docstring still referenced HEAD_SNIFF_MAX_BYTES after the constant was renamed to HEAD_FULL_READ_MAX_BYTES in pass 1. Fixed the reference. 2. GET applies RDF_CACHE_CONTROL ('private, no-cache, must-revalidate') wherever the response content type is RDF — container listings (Turtle/JSON-LD) AND files (converted or as-is) — but HEAD set it nowhere, leaving headers divergent for negotiated RDF responses. HEAD now mirrors with the same uniform rule: isRdfContentType(contentType) → RDF_CACHE_CONTROL. (Copilot's comment scoped this to files; GET also sets it on container listings at resource.js:243/257/364/383, so the mirror covers containers too — the parity contract is the same.) Tests: the parity loop now also compares Cache-Control across all four Accept variants, plus a new container-listing parity case (media type + Cache-Control). 945/945 passing.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

melvincarvalho · 2026-06-10T20:21:16Z

+    const islandCandidate =
+      storedContentType === 'text/html' || storedContentType === 'application/octet-stream';
+    if (islandCandidate && wantsTurtle && fitsFullRead) {


Legitimate again — fixed in 422f8aa. The stored-type gate is gone: HTML-ness is now decided by a 1 KiB ranged sniff on ANY stored type (O(1) regardless of file size), and only HTML-looking content pays the full read for the island check — same content-only gating as GET, without unconditionally reading every file. My earlier 'keys off CONTENT' comment claimed this while the code only half-delivered; the code now matches the comment. New regression test: .xhtml (stored application/xhtml+xml) with a parseable island + Turtle-preferring Accept → HEAD media type equals GET's (text/turtle). 946/946 passing.

…ET's Copilot pass-3 finding, legitimate: GET's data-island branch is gated on CONTENT only — any file whose body starts with <!DOCTYPE/<html gets the island→Turtle conversion regardless of stored type — but the helper's islandCandidate gate (text/html || octet-stream) missed HTML-ish stored types like .xhtml (application/xhtml+xml). My own 'keys off CONTENT' comment claimed what the code didn't fully do. Fix: drop the stored-type gate. A 1 KiB ranged sniff decides HTML-ness for O(1) cost on any file ≤ the full-read cap; only HTML-looking content pays the full read for the island check. The extensionless relabel sniff still applies on fall-through. Docstring updated: the conservative large-file case is now phrased content-based ('HTML-looking content, any stored type'). New test: .xhtml file with a parseable island + Turtle-preferring Accept — HEAD media type must equal GET's (text/turtle). 946/946.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

melvincarvalho · 2026-06-10T20:31:31Z

+    if (wantsTurtle && fitsFullRead) {
+      const head = await readFirstBytes(storagePath, HEAD_SNIFF_CHUNK_BYTES);
+      const headTrimmed = head === null ? '' : head.trimStart();
+      if (headTrimmed.startsWith('<!DOCTYPE') || headTrimmed.startsWith('<html')) {
+        // GET converts an HTML data island to Turtle only when the
+        // island exists AND its JSON parses; otherwise it serves the
+        // document as-is.
+        const content = await storage.read(storagePath);
+        if (content !== null) {
+          const jsonLdMatch = content.toString().match(/<script\s+type=["']application\/ld\+json["']\s*>([\s\S]*?)<\/script>/i);
+          if (jsonLdMatch) {
+            try {
+              JSON.parse(jsonLdMatch[1]);
+              return { contentType: 'text/turtle', converted: true };
+            } catch { /* unparseable island → GET serves as-is; fall through */ }
+          }
+        }
+      }
+      // No island conversion → fall through to the as-is path below
+      // (HTML-looking extensionless files still get the relabel sniff).
+    }


Fixed in 277c2b3. When the 1 KiB chunk trims to empty AND the file continues past the chunk, HEAD escalates to the full read (the file already fits HEAD_FULL_READ_MAX_BYTES — that budget exists precisely so this stays bounded) and runs GET's exact trim + marker check. Restructured to avoid a double read when the escalation already loaded the content. The ranged-read-failure case resolves to the stored type, unchanged — if the ranged read fails, the full read GET performs would likely fail too. New regression test: 2 KiB-whitespace-padded .xhtml island document with a Turtle-preferring Accept → HEAD media type equals GET's. 947/947 passing.

…l read Copilot pass-4: a file with >1 KiB of leading whitespace before <!DOCTYPE/<html — GET trims the FULL body and detects it; HEAD's 1 KiB chunk trimmed to empty and missed the marker. Pathological input, but the fix is cheap and bounded: when the chunk is entirely whitespace AND the file continues past it, escalate to the full read (the file already fits HEAD_FULL_READ_MAX_BYTES — that budget exists precisely so reads like this stay bounded) and decide exactly like GET. Restructured to avoid a double read when the escalation path already loaded the content. New test: 2 KiB-whitespace-padded .xhtml island doc with a Turtle-preferring Accept — HEAD media type equals GET's. 947/947.

Eight PRs merged since 0.0.206 — IdP hardening, protocol conformance, and the de-Googled-phone (#46) arc: IdP / auth - #558 passkey login degrades cleanly on stale WebViews / insecure contexts instead of crashing — drops a redundant browser crypto.randomUUID and guards both ceremonies on secure-context + WebAuthn (closes #556) - #554 IdP login-form error now survives the POST→redirect→GET cycle (rode in a non-persisted field that Interaction.save() dropped); failed sign-ins show the error instead of a silent re-render (closes #514) - #551 RFC 9207 'iss' authorization-response param normalized to match the discovery issuer, so strict OIDC clients (solid-oidc) complete sign-in (closes #524) Tunnel - #555 opt-in, per-tunnel credential passthrough (Cookie / Authorization / Set-Cookie) so authenticated access works through a tunnel; the relay's own IdP session cookies are isolated from the tunnel client (closes #530) Content negotiation / git - #553 HEAD now mirrors GET's negotiated Content-Type / Content-Length / Cache-Control for files (RFC 9110 §9.3.2 parity) (closes #552) - #550 git WAC preHandler 401/402/403 responses carry the git CORS headers, so browser git clients see the status, not a CORS error (closes #548) - #549 first HTTP-contract coverage for the git handler + fixes a DATA_ROOT test-pollution bug (closes #375) Docs / metadata - #547 README tagline + npm keywords surface the agentic positioning (closes #406)

melvincarvalho requested a review from Copilot June 10, 2026 19:43

Copilot started reviewing on behalf of melvincarvalho June 10, 2026 19:43 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

melvincarvalho requested a review from Copilot June 10, 2026 20:00

Copilot started reviewing on behalf of melvincarvalho June 10, 2026 20:00 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

melvincarvalho requested a review from Copilot June 10, 2026 20:11

Copilot started reviewing on behalf of melvincarvalho June 10, 2026 20:11 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

melvincarvalho requested a review from Copilot June 10, 2026 20:22

Copilot started reviewing on behalf of melvincarvalho June 10, 2026 20:22 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

melvincarvalho merged commit f40978b into gh-pages Jun 10, 2026
1 check passed

melvincarvalho deleted the issue-552-head-conneg-parity branch June 10, 2026 20:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(conneg): HEAD mirrors GET's negotiated content type for files — closes #552#553

fix(conneg): HEAD mirrors GET's negotiated content type for files — closes #552#553
melvincarvalho merged 5 commits into
gh-pagesfrom
issue-552-head-conneg-parity

melvincarvalho commented Jun 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

melvincarvalho Jun 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

melvincarvalho Jun 10, 2026

Uh oh!

melvincarvalho Jun 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

melvincarvalho Jun 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

melvincarvalho Jun 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

melvincarvalho commented Jun 10, 2026

Bug

Fix

Design points (each documented in code)

Tests

Refs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

melvincarvalho Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

melvincarvalho Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

melvincarvalho Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

melvincarvalho Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

melvincarvalho Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants