Index subdomain-mode pods (#411)#413
Merged
Merged
Conversation
The well-known did:nostr indexer was silently dropping every subdomain-mode pod. The original `profilePathFromWebId` derived the on-disk path from `webIdUrl.pathname` only, which works for path-mode named pods (`example.com/alice/...` → `<dataRoot>/alice/...`) and root pods (`example.com/profile/...` → `<dataRoot>/profile/...`) but loses the subdomain in the third case: WebID: https://alice.example.com/profile/card.jsonld#me pathname: /profile/card.jsonld derived: <dataRoot>/profile/card.jsonld actual: <dataRoot>/alice/profile/card.jsonld ← profile lives here On solid.social (subdomain-mode), every subdomain user fell through to the typed-username fallback because the local index never found their profile. "Sign in with Schnorr" with no typing just didn't work. Fix: - New `profilePathCandidates(dataRoot, webId)` returns the ORDERED list of candidate filesystem paths to probe: 1. `<dataRoot>/<pathname>` (path mode + root pod) 2. `<dataRoot>/<host-first-label>/<pathname>` (subdomain mode) Subdomain candidate only emitted for hosts with ≥2 DNS labels. Containment-checked: every candidate stays under dataRootAbs. - rebuildPubkeyIndex now PROBES candidates and accepts the first one whose `@id` matches `account.webId`. Multiple candidates can exist on disk simultaneously (root pod + named pods coexisting; subdomain pod with a coincidentally- named path-mode dir), so committing to the first candidate-that-stats was a bug — could pick someone else's profile, fail the @id check, log "no match," skip the actual matching candidate. Now: stat → size cap → parse → @id check, fall through to next candidate on any failure. - Removed the unused `findProfilePathOnDisk` helper. Tests: - Six new unit tests on `profilePathCandidates` covering path mode, root pod, subdomain mode, single-label hosts, unparseable input, and containment across all of them. - One new integration test that synthesizes a subdomain-mode pod (profile at `<TEST_DATA_DIR>/sub/profile/card.jsonld`, webId at `http://sub.example.test/profile/...`) and asserts the well-known endpoint resolves it. The test exercises the "first candidate exists but belongs to a different account" case because the prior test in the same suite leaves a root-pod profile at `<TEST_DATA_DIR>/profile/card.jsonld` — the subdomain account's path-mode candidate would hit it, fail the @id check, and the loop must fall through to the subdomain candidate. (This caught the original findProfilePathOnDisk bug during development.) Test count: 752 → 767 in full suite (+ 7 unit + 1 integration + 7 from the same test file's organic growth across the run).
There was a problem hiding this comment.
Pull request overview
Fixes local did:nostr indexing for subdomain-mode deployments by probing multiple on-disk profile path candidates derived from a WebID, ensuring subdomain pods (stored under <DATA_ROOT>/<podname>/...) are discovered and indexed correctly.
Changes:
- Add
profilePathCandidates()to generate ordered on-disk profile-path candidates (path/root-pod first, then subdomain-derived). - Update
rebuildPubkeyIndex()to iterate candidates and accept the first profile whose@idmatchesaccount.webId. - Extend tests with unit coverage for candidate generation and an integration test simulating a subdomain-mode on-disk layout.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/idp/well-known-did-nostr.js |
Switch index rebuild logic to probe multiple candidate profile paths and introduce profilePathCandidates() for subdomain-mode support. |
test/well-known-did-nostr.test.js |
Add unit + integration tests covering subdomain-mode profile path derivation and candidate probing behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+329
to
+333
| // directory under dataRoot. Only meaningful for hosts with at | ||
| // least two labels (`alice.example.com`); a bare host like | ||
| // `example.com` falls back to candidate #1 above (root/path mode). | ||
| const hostLabels = url.hostname.split('.'); | ||
| if (hostLabels.length >= 2 && hostLabels[0]) { |
Real concern with my pass-0 implementation: `profilePathCandidates` emitted the "host first label as pod dir" candidate for ANY host with ≥2 DNS labels. For a root-pod WebID (`https://example.com/profile/card.jsonld`) on a normal domain, that produced `<dataRoot>/example/profile/card.jsonld` as a secondary candidate — which could be a totally unrelated account's pod dir. The downstream `@id` check rejected mis-indexing, so it was never exploitable, but: - extra wasted disk probes per non-subdomain account - confusing log entries blaming the wrong path on rebuild miss - defense rested entirely on @id check, no precision in candidate generation itself Fix: add an optional `podName` parameter. The subdomain candidate is only emitted when `<podName>.` is the WebID host's actual first label (case-insensitive — DNS is). `rebuildPubkeyIndex` passes `account.podName` since it has the account record in hand. For root pods (podName='me' from src/server.js's single-user seed, host is the bare baseDomain), the gate fails and only the path-mode/root candidate is emitted. For a true subdomain pod (podName='alice', host='alice.example.com'), the gate matches and we emit `<dataRoot>/alice/profile/card.jsonld` as the secondary candidate. Path mode (no subdomain in host) still gets just the path-mode candidate. Tests: - "subdomain-mode pod: emits ... when host first label matches podName" (renamed; existing behavior, now explicit about the matching condition) - NEW "does NOT emit a subdomain candidate when podName is omitted" — backward-safe behavior for any other caller - NEW "does NOT emit a subdomain candidate when podName does not match the host first label" — the case Copilot flagged: root-pod on example.com with podName='me' must NOT produce `<dataRoot>/example/profile/...` - "single-label host" test now passes podName explicitly to confirm gating is consistent - Containment loop now includes podName per case Test count: 767 → 769 in full suite.
Comment on lines
+190
to
+197
| if (!profile) { | ||
| // Nothing on disk matched. Surface the issue for operators — | ||
| // either the profile genuinely doesn't exist (ENOENT on every | ||
| // candidate) or each candidate's `@id` mismatched. Either way | ||
| // worth logging once per hour per account. | ||
| const tried = candidates.length ? candidates.join(' | ') : '(no candidates)'; | ||
| const err = firstError || { code: 'ENOENT', message: 'no candidate matched account.webId' }; | ||
| logProfileFailure(accountId, tried, err); |
Comment on lines
+327
to
+329
| const add = (...parts) => { | ||
| const r = path.resolve(dataRootAbs, ...parts); | ||
| if (insideRoot(r) && !out.includes(r)) out.push(r); |
Two findings, both about losing operator-visible information:
1. The fallback `err = { code: 'ENOENT', message: 'no candidate
matched account.webId' }` was misleading. The actual reason
could be:
- profile genuinely missing (ENOENT)
- oversized file
- JSON parse failure
- `@id` mismatch (most common when subdomain config is wrong)
- candidate rejected by containment
Hard-coding ENOENT erased that signal.
Fix: rebuild loop now accumulates per-candidate failure
reasons in a list, and the fallback log emits ALL of them
joined with ` | `, prefixed with the actual code:
- `<path>: ENOENT` / stat-error / not-a-regular-file
- `<path>: oversized (NN > MAX)`
- `<path>: parse-error (msg)`
- `<path>: @id-mismatch (declared=...)`
- `<path>: outside-dataRoot`
Top-level err code changed to `NO_CANDIDATE_MATCHED` so it's
distinguishable from a single ENOENT.
2. `profilePathCandidates` was silently dropping out-of-dataRoot
candidates. Pre-#411, `profilePathFromWebId` logged "resolves
outside dataRoot" loudly. Operators lost that signal —
traversal/misconfig and "profile not on disk" looked
identical in the failure log.
Fix: function now returns `{ paths, skipped }`. `paths` is
the containment-passed candidates (caller probes these);
`skipped` is `[{ path, reason }]` for any rejected candidate
(today only "outside-dataRoot"). The rebuild loop folds
`skipped` into its diagnostics list so containment
rejections show up in the per-account failure log alongside
ENOENT/parse/mismatch reasons.
Test changes:
- All `profilePathCandidates` callers updated for the
`{ paths, skipped }` shape (destructuring `paths`).
- The "returns []" test became "returns empty paths" with the
new shape.
- New "returns the shape so the caller can surface
diagnostics" test confirms the API contract: paths and
skipped are both arrays, normal URL input produces empty
skipped.
Test count: 769 → 770 in full suite (+1 shape test, -0 — the
shape rewrite consolidated the `[].length === 0` assertion
into the new shape test).
Comment on lines
170
to
176
| if (stat.size > MAX_PROFILE_BYTES) { | ||
| reasons.push(`${candidate}: oversized (${stat.size} > ${MAX_PROFILE_BYTES})`); | ||
| console.error( | ||
| `well-known-did-nostr: skipping account ${accountId} ` + | ||
| `— profile size ${stat.size} > ${MAX_PROFILE_BYTES} bytes`, | ||
| `well-known-did-nostr: skipping candidate ${candidate} for ` + | ||
| `account ${accountId} — size ${stat.size} > ${MAX_PROFILE_BYTES} bytes`, | ||
| ); | ||
| continue; |
Comment on lines
+202
to
+204
| logProfileFailure(accountId, summary, { | ||
| code: 'NO_CANDIDATE_MATCHED', | ||
| message: `no candidate profile matched account.webId for ${account.webId}`, |
Comment on lines
+329
to
+331
| const subWebId = 'http://sub.example.test/profile/card.jsonld#me'; | ||
| const subProfilePath = path.join(TEST_DATA_DIR, 'sub', 'profile', 'card.jsonld'); | ||
| const VM_ID = 'http://sub.example.test/profile/card.jsonld#k'; |
Comment on lines
+707
to
+738
| const { paths } = profilePathCandidates(DATA_ROOT, 'https://example.com/alice/profile/card.jsonld#me'); | ||
| assert.ok(paths.includes('/srv/jss/data/alice/profile/card.jsonld'), | ||
| `expected path-mode candidate; got ${paths.join(', ')}`); | ||
| }); | ||
|
|
||
| it('root pod: <dataRoot>/profile/card.jsonld', () => { | ||
| const { paths } = profilePathCandidates(DATA_ROOT, 'https://example.com/profile/card.jsonld#me'); | ||
| assert.ok(paths.includes('/srv/jss/data/profile/card.jsonld'), | ||
| `expected root-pod candidate; got ${paths.join(', ')}`); | ||
| }); | ||
|
|
||
| it('subdomain-mode pod: emits <dataRoot>/<podName>/profile/... when host first label matches podName', () => { | ||
| const { paths } = profilePathCandidates(DATA_ROOT, 'https://test.solid.social/profile/card.jsonld#me', 'test'); | ||
| assert.ok(paths.includes('/srv/jss/data/profile/card.jsonld'), | ||
| `expected path-mode candidate; got ${paths.join(', ')}`); | ||
| assert.ok(paths.includes('/srv/jss/data/test/profile/card.jsonld'), | ||
| `expected subdomain candidate; got ${paths.join(', ')}`); | ||
| }); | ||
|
|
||
| it('does NOT emit a subdomain candidate when podName is omitted', () => { | ||
| const { paths } = profilePathCandidates(DATA_ROOT, 'https://test.solid.social/profile/card.jsonld#me'); | ||
| assert.deepStrictEqual(paths, ['/srv/jss/data/profile/card.jsonld']); | ||
| }); | ||
|
|
||
| it('does NOT emit a subdomain candidate when podName does not match the host first label', () => { | ||
| const { paths } = profilePathCandidates(DATA_ROOT, 'https://example.com/profile/card.jsonld#me', 'me'); | ||
| assert.deepStrictEqual(paths, ['/srv/jss/data/profile/card.jsonld']); | ||
| }); | ||
|
|
||
| it('does NOT emit a subdomain candidate for a single-label host', () => { | ||
| const { paths } = profilePathCandidates(DATA_ROOT, 'http://localhost/profile/card.jsonld#me', 'localhost'); | ||
| assert.deepStrictEqual(paths, ['/srv/jss/data/profile/card.jsonld']); |
Four findings, all real.
1. Oversize-profile branch was emitting a direct `console.error`,
bypassing the per-account rate limit in `logProfileFailure`.
On every TTL rebuild, an oversized profile would re-log; if
the same account had a later candidate that matched, the
error would also be misleading. Now folded into the same
`reasons` accumulator used for ENOENT/parse/mismatch — the
rate-limited `logProfileFailure` only fires when NO candidate
matches, and the oversize hit is part of the diagnostics in
`err.message`.
2. `logProfileFailure(accountId, profilePath, err)` formats logs
as `(profile=${profilePath})`. We were passing a multi-
candidate summary string (`<path>: ENOENT | <path>: ...`) as
the `profilePath` argument, which made the log field
misleading and unsearchable. Now passes the placeholder
`(multiple candidates)` for the path and puts the full
summary in `err.message`. Operators searching for
`profile=/some/path` get back the actual paths; the summary
travels with the err.
3. The subdomain-mode integration test had an implicit
dependency on test ordering. The "first candidate exists but
belongs to a different account" path was only exercised
because a prior test left a root-pod profile at
`<TEST_DATA_DIR>/profile/card.jsonld`. Reordering tests would
have silently lost that coverage. Now self-contained: the
test creates its OWN decoy profile (with an unrelated key
and webId) at the path-mode candidate location, runs the
subdomain assertion, then cleans up both fixtures.
4. The new `profilePathCandidates` unit tests hard-coded POSIX
paths like `/srv/jss/data/alice/...`, which would fail on
Windows (different separator + drive root). DATA_ROOT is now
`path.resolve('/srv/jss/data')` and every expected value is
built via `path.join(DATA_ROOT, ...)`. Portable.
Test count: same 770; the subdomain test became self-contained
and added two fs writes + cleanup but no new test cases.
Comment on lines
+379
to
+383
| // Cleanup: leave neither the decoy nor the index entry behind | ||
| // so the test isolates correctly even when re-run / reordered. | ||
| await fs.remove(decoyProfilePath); | ||
| delete idx[subWebId]; | ||
| await fs.writeJson(indexPath, idx, { spaces: 2 }); |
One finding, real test-isolation gap.
The pass-3 subdomain-mode test created a decoy profile at
`<TEST_DATA_DIR>/profile/card.jsonld` and then unconditionally
deleted it during cleanup. That same path is also used by the
earlier "indexes root-level pods" test, which leaves a real
root-pod profile + an account entry in `_webid_index.json`.
Running the subdomain test would silently wipe the root-pod
fixture, and any later TTL rebuild would log "no candidate
matched" for the orphaned root-pod account — order-dependent,
brittle, and noisy.
Fix:
- Snapshot the decoy file's content (if any) BEFORE writing
over it. ENOENT on snapshot means "didn't exist; remove on
cleanup"; any other error rethrows.
- On cleanup, restore the snapshotted content (if there was
one) or delete the file (if it was created by this test
only).
- Also remove `subProfilePath` (the subdomain candidate file
this test wrote) and prune the now-empty `sub/profile/`
+ `sub/` dirs (best-effort — failures swallowed).
Test count: same 770; the test stays correct + now isolates
its filesystem side effects regardless of execution order.
Comment on lines
+332
to
+410
| const decoyPk = getPublicKey(generateSecretKey()); // unrelated key | ||
| const decoyProfilePath = path.join(TEST_DATA_DIR, 'profile', 'card.jsonld'); | ||
| const decoyWebId = `${baseUrl}/profile/card.jsonld#decoy`; | ||
| let decoySnapshot = null; | ||
| try { | ||
| decoySnapshot = await fs.readFile(decoyProfilePath, 'utf8'); | ||
| } catch (e) { | ||
| if (e.code !== 'ENOENT') throw e; | ||
| // didn't exist — nothing to restore | ||
| } | ||
| await fs.ensureDir(path.dirname(decoyProfilePath)); | ||
| await fs.writeJson(decoyProfilePath, { | ||
| '@context': 'https://www.w3.org/ns/solid/v1', | ||
| '@id': decoyWebId, | ||
| verificationMethod: [{ | ||
| id: `${decoyWebId.replace('#decoy', '')}#k`, | ||
| type: 'Multikey', | ||
| controller: decoyWebId, | ||
| publicKeyMultibase: fformMultikey(decoyPk), | ||
| }], | ||
| authentication: [`${decoyWebId.replace('#decoy', '')}#k`], | ||
| }, { spaces: 2 }); | ||
|
|
||
| const sk = generateSecretKey(); | ||
| const subPk = getPublicKey(sk); | ||
| const subWebId = 'http://sub.example.test/profile/card.jsonld#me'; | ||
| const subProfilePath = path.join(TEST_DATA_DIR, 'sub', 'profile', 'card.jsonld'); | ||
| const VM_ID = 'http://sub.example.test/profile/card.jsonld#k'; | ||
| await fs.ensureDir(path.dirname(subProfilePath)); | ||
| await fs.writeJson(subProfilePath, { | ||
| '@context': 'https://www.w3.org/ns/solid/v1', | ||
| '@id': subWebId, | ||
| verificationMethod: [{ | ||
| id: VM_ID, | ||
| type: 'Multikey', | ||
| controller: subWebId, | ||
| publicKeyMultibase: fformMultikey(subPk), | ||
| }], | ||
| authentication: [VM_ID], | ||
| }, { spaces: 2 }); | ||
|
|
||
| const accountsDir = path.join(TEST_DATA_DIR, '.idp', 'accounts'); | ||
| const indexPath = path.join(accountsDir, '_webid_index.json'); | ||
| const idx = await fs.readJson(indexPath); | ||
| const accountId = 'subdomain-pod-test-account'; | ||
| idx[subWebId] = accountId; | ||
| await fs.writeJson(indexPath, idx, { spaces: 2 }); | ||
| await fs.writeJson(path.join(accountsDir, `${accountId}.json`), { | ||
| id: accountId, | ||
| podName: 'sub', | ||
| webId: subWebId, | ||
| email: '[email protected]', | ||
| }, { spaces: 2 }); | ||
|
|
||
| const r = await fetch(`${baseUrl}/.well-known/did/nostr/${subPk}.json`); | ||
| assert.strictEqual(r.status, 200, 'subdomain-mode pod must be findable'); | ||
| const doc = await r.json(); | ||
| assert.strictEqual(doc.id, `did:nostr:${subPk}`); | ||
| assert.strictEqual(doc.alsoKnownAs[0], subWebId); | ||
|
|
||
| // Cleanup: leave neither the decoy nor the subdomain | ||
| // fixture behind. Restore (or delete) the decoy file so | ||
| // any prior fixture state at that path is preserved. | ||
| if (decoySnapshot !== null) { | ||
| await fs.writeFile(decoyProfilePath, decoySnapshot, 'utf8'); | ||
| } else { | ||
| await fs.remove(decoyProfilePath); | ||
| } | ||
| await fs.remove(subProfilePath); | ||
| // Also remove the parent `<TEST_DATA_DIR>/sub` dir if empty | ||
| // (won't be if other tests added siblings — that's fine). | ||
| try { | ||
| await fs.rmdir(path.dirname(subProfilePath)); | ||
| await fs.rmdir(path.dirname(path.dirname(subProfilePath))); | ||
| } catch { /* not empty / already gone — fine */ } | ||
| delete idx[subWebId]; | ||
| await fs.writeJson(indexPath, idx, { spaces: 2 }); | ||
| await fs.remove(path.join(accountsDir, `${accountId}.json`)); | ||
| }); |
One finding, real test-isolation gap. The pass-4 subdomain-mode test snapshotted/restored the decoy profile but the cleanup ran in the happy path only. A failing fetch or assertion would skip the cleanup block and leave: - the decoy profile overwritten on disk - subProfilePath created - `_webid_index.json` carrying a synthetic account entry - `subdomain-pod-test-account.json` left in accountsDir turning the whole suite into an order-dependent mess on the next run. Fix: wrap the fixture mutations + assertions in `try`, with ALL restoration in `finally`. The snapshot read happens BEFORE the try block (it's a read; if it throws non-ENOENT we want to fail before mutating anything). Each cleanup step is itself wrapped in a swallow-errors try so one failed restoration doesn't prevent the others (best effort). The decoy snapshot/restore semantics are preserved: - file existed before → restore the original content - file didn't exist → delete Test count: same 770; the test now isolates correctly even when an assertion throws mid-run.
Comment on lines
+155
to
157
| let profile = null; | ||
| let profilePath = null; | ||
| let mtimeMs = 0; |
Comment on lines
+408
to
+411
| // Subdomain fixture file + empty parent dirs. | ||
| try { await fs.remove(subProfilePath); } catch { /* best effort */ } | ||
| try { await fs.rmdir(path.dirname(subProfilePath)); } catch { /* not empty / gone */ } | ||
| try { await fs.rmdir(path.dirname(path.dirname(subProfilePath))); } catch { /* not empty / gone */ } |
Two findings, both small.
1. `profilePath` in rebuildPubkeyIndex was assigned when a
candidate matched but never read afterward — the only
downstream consumers care about `profile` (the parsed JSON)
and `mtimeMs` (used in the index entry). Pre-restructure
it was the path used for the next steps; after pass-2's
accept-or-fall-through loop the variable became orphaned.
Removed the declaration and the assignment.
2. `fs.rmdir` (node:fs/promises) emits a deprecation warning
in modern Node — it's slated for removal. Switched to
`fs-extra`'s `fs.remove` which:
- is already in use elsewhere in this test
- handles both files and (recursively) directories
- is a no-op on missing paths (matches our best-effort
cleanup semantics)
Replaces `await fs.rmdir(<dir>)` with `await fs.remove(<dir>)`
for the two parent-dir cleanup steps.
Test count: same 770.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #411.
The well-known did:nostr indexer was silently dropping every subdomain-mode pod. Surfaced during live testing of #408 on solid.social: account
b0b1707f-...with WebIDhttps://test.solid.social/profile/card.jsonld#melives on disk at<dataRoot>/test/profile/card.jsonld, but the indexer was looking at<dataRoot>/profile/card.jsonldand ENOENT-ing.Root cause
profilePathFromWebIdderived the on-disk path fromwebIdUrl.pathnamealone, which works for path-mode named pods and root pods but loses the subdomain when host =<podname>.<basedomain>:So every subdomain user on solid.social fell through to the typed-username fallback (#403/#405), defeating the zero-typing UX promise of #408.
Fix
profilePathCandidates(dataRoot, webId)returns an ordered list of candidates: path-mode/root-pod first, then<dataRoot>/<host-first-label>/<pathname>if the host has ≥2 DNS labels. Containment-checked.rebuildPubkeyIndexnow probes candidates and accepts the first whose@idmatchesaccount.webId. Multiple candidates can exist simultaneously (root pod + named pods coexisting), so committing to the first stat-able candidate was buggy — would pick someone else's profile, fail the @id check, skip without trying the next candidate.Test plan
profilePathCandidatescovering all three deployment shapes (path, root, subdomain) + edge cases (single-label host, unparseable webId, containment invariant)findProfilePathOnDiskgot wrong/.well-known/did/nostr/<pubkey>.json; "Sign in with Schnorr" works without typing a username