Skip to content

Add SwiftMail provider qualification harness - #308

Merged
rajzik merged 7 commits into
mainfrom
agent/swiftmail-provider-qualification
Aug 10, 2026
Merged

rajzik merged 7 commits into
mainfrom
agent/swiftmail-provider-qualification

Conversation

@gipity-bot

@gipity-bot gipity-bot Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • add a standalone Swift package pinned to SwiftMail 1.10.0 at commit c907f871bb23812895274f4c7ae17bf343171c1e
  • exercise the iCloud Mail and Fastmail provider contract, including role discovery, missing-role creation, metadata synchronization, body search, mutations, IDLE, SMTP, verified Sent append, ADR 0027 budgets, and targeted cleanup
  • add a manual-only protected-environment workflow that keeps provider credentials out of pull-request jobs and uploads redacted JSON evidence
  • document environment setup, secret names, invocation, evidence requirements, and the manual soak checklist

Why

Issue #280 requires secrets-backed vertical qualification against disposable iCloud Mail and Fastmail accounts before SwiftMail can be admitted as a production dependency. The harness is intentionally isolated from the app dependency graph so this pull request adds no production SwiftMail linkage.

Impact

Maintainers can prepare the 10,000-message qualification datasets and run both providers through an opt-in GitHub Actions workflow. The issue remains open until both live reports pass and the manual network-loss soak evidence is attached.

Validation

  • swift test --package-path tools/swiftmail-provider-qualification
  • production Swift package build
  • strict swift-format and SwiftLint for the qualification sources and tests
  • mise exec -- pnpm lint
  • mise exec -- pnpm format
  • mise exec -- pnpm turbo run check-types
  • mise exec -- pnpm test
  • mise exec -- pnpm fallow
  • mise exec -- zsh scripts/check-apple-lint.zsh
  • workflow YAML parse and git diff --check

No changeset is included because this is internal qualification tooling and documentation rather than production runtime behavior or an exported package API.

Refs #280


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@coderabbitai

coderabbitai Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 42 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: c0868633-57ea-47f9-89d7-d529d5d67759

📥 Commits

Reviewing files that changed from the base of the PR and between 3317e38 and 00fe654.

📒 Files selected for processing (8)
  • .github/workflows/swiftmail-provider-qualification.yml
  • README.md
  • docs/qualification/swiftmail-provider.md
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/ProviderQualificationRunner.swift
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationMessages.swift
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationMetrics.swift
  • tools/swiftmail-provider-qualification/Sources/swiftmail-provider-qualification/SwiftMailProviderQualificationCommand.swift
  • tools/swiftmail-provider-qualification/Tests/SwiftMailProviderQualificationTests/QualificationTests.swift
📝 Walkthrough

Summary

  • Added an isolated macOS Swift package for SwiftMail 1.10.0 provider qualification.
  • Added iCloud Mail and Fastmail checks for mailbox roles, synchronization, search, mutations, IDLE, SMTP, Sent append, cleanup, and ADR 0027 budgets.
  • Added manual-only protected-environment GitHub Actions execution with redacted JSON evidence uploads.
  • Added setup, secret handling, invocation, evidence, and manual soak documentation.
  • Kept SwiftMail outside the production dependency graph and ordinary CI.
  • Added environment-based provider configuration, including credentials, mailbox mappings, and dataset options. Credentials are excluded from reports and evidence.
  • Apple area: new Swift Package Manager qualification tool under tools/swiftmail-provider-qualification.
  • TypeScript area: no TypeScript files or production TypeScript behavior changed.
  • Validation covered Swift formatting, linting, type checks, tests, workflow parsing, dependency pin verification, and related repository checks.

Walkthrough

Adds an isolated SwiftMail 1.10.0 qualification package for iCloud and Fastmail. The package runs IMAP, SMTP, mailbox, synchronization, performance, cleanup, and evidence checks through a manual macOS workflow.

Changes

SwiftMail qualification

Layer / File(s) Summary
Qualification contracts and configuration
tools/swiftmail-provider-qualification/Package.swift, tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/*, tools/swiftmail-provider-qualification/Tests/*
Adds provider configuration, mailbox-role resolution, message generation, metrics, reports, budgets, package targets, and unit tests.
Provider qualification execution
tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/ProviderQualificationRunner.swift
Adds end-to-end IMAP and SMTP qualification, synchronization checks, IDLE monitoring, performance validation, cleanup, failure handling, and JSON evidence construction.
CLI and workflow integration
tools/swiftmail-provider-qualification/Sources/swiftmail-provider-qualification/main.swift, .github/workflows/swiftmail-provider-qualification.yml, docs/qualification/swiftmail-provider.md, README.md, .gitignore
Adds command-line execution, manual iCloud/Fastmail workflow jobs, dependency verification, artifact upload, aggregate gating, runbook documentation, and generated-artifact ignore rules.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Workflow
  participant CLI as SwiftMailProviderQualificationCommand
  participant Runner as ProviderQualificationRunner
  participant MailProvider
  participant Report as QualificationReport
  Workflow->>CLI: invoke provider qualification
  CLI->>Runner: execute()
  Runner->>MailProvider: authenticate and run IMAP/SMTP checks
  MailProvider-->>Runner: return qualification results
  Runner->>Report: build evidence
  Report-->>Workflow: write JSON report
Loading

Possibly related issues

Possibly related PRs

Suggested labels: codex, codex-automation

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The description clearly explains the isolated SwiftMail qualification harness, workflow, documentation, validation, and production dependency scope.
Title check ✅ Passed The title clearly and concisely identifies the main change: adding a SwiftMail provider qualification harness.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch agent/swiftmail-provider-qualification

Comment @coderabbitai help to get the list of available commands.

@gipity-bot

gipity-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@gipity-bot

gipity-bot Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@gipity-bot

gipity-bot Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

@rajzik
rajzik marked this pull request as ready for review August 9, 2026 21:40
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor
⚠️ Action not completed

Already reviewed.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 13

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/swiftmail-provider-qualification.yml:
- Around line 116-122: Update the custom shell configuration at
.github/workflows/swiftmail-provider-qualification.yml:36 to use errexit and
pipefail (for example, zsh -e -o pipefail {0}), ensuring assertion failures
propagate in both the provider qualification script at lines 116-122 and the
version/revision assertions at lines 49-53; no direct changes are required at
either script site.
- Around line 49-53: Update the workflow’s custom zsh shell configuration at the
`defaults.run.shell` setting to enable `errexit` and `pipefail`, using `zsh -e
-o pipefail {0}` or equivalent script flags. Preserve both `resolved_version`
and `resolved_revision` assertions so either mismatch fails the qualification
step.
- Around line 38-41: Pin both third-party actions in
.github/workflows/swiftmail-provider-qualification.yml to immutable full commit
SHAs: update actions/checkout at lines 38-41 and actions/upload-artifact at
lines 101-103 to the intended release SHAs, preserving trailing version
comments.

In `@docs/qualification/swiftmail-provider.md`:
- Around line 1-3: Update the terminology in the SwiftMail qualification
documentation: replace “provider qualification” with “Provider Compatibility
Run,” refer to dedicated accounts as “Provider Test Mailboxes/Tenants,” and call
generated redacted JSON “Mail Test Evidence.” Apply these terms consistently
throughout the document, including the additionally referenced section.

In `@README.md`:
- Line 171: Update the README provider-run section to use the canonical terms
“Provider Compatibility Run,” “Provider Test Mailboxes/Tenants,” and “Mail Test
Evidence,” or explicitly define “provider qualification” as their alias in
CONTEXT.md. Also state that this boundary is owned by Apple/Swift and remains
separate from the TypeScript validation commands, keeping terminology aligned
with the Swift package, workflow, and runbook.

In
`@tools/swiftmail-provider-qualification/Sources/swiftmail-provider-qualification/main.swift`:
- Around line 5-7: Resolve the executable entry-point conflict by renaming
main.swift to SwiftMailProviderQualificationCommand.swift while retaining the
`@main` annotation on SwiftMailProviderQualificationCommand.

In
`@tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/ProviderQualificationRunner.swift`:
- Around line 55-59: Make the final imap.disconnect() in
ProviderQualificationRunner’s qualification flow best-effort: catch and suppress
disconnect errors after cleanup completes, then set passed = true regardless of
disconnect success. Keep qualification failures and cleanup error handling
unchanged.
- Around line 218-226: Update the complete-history backfill handling around
validateBackfill and the "complete-history backfill" QualificationCheck to
explicitly record its decoded-bytes exemption and bound. Preserve the direct
metrics assignment only if it clearly documents that decodedBytes is
intentionally unevaluated and capped by the stated 5 MiB maximum; otherwise use
the existing recordBudgetedMetrics mechanism with the appropriate explicit
exemption, and update the check detail/report to state the bound.
- Around line 918-920: Update isSeen(_:) to check message.flags for direct
Flag.seen membership using contains(.seen), replacing the description-string
comparison while preserving the existing Boolean result.

In
`@tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationMessages.swift`:
- Around line 39-56: Update paddedMessage to include a valid RFC 5322 Date
header alongside the existing From header, using the project’s established
date-formatting mechanism if available. Extend the message tests to verify the
Date header is present and correctly formatted without changing the existing
padding or byte-count behavior.

In
`@tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationMetrics.swift`:
- Around line 60-64: Replace the lifetime high-water-mark implementation in
peakResidentMemoryBytes with a current resident-memory query using task_info,
updating both measure call sites to use that current-size helper and adding the
MachO import if required for the Mach symbols. Preserve the existing Int64
return type and zero fallback when the query fails so sequential measurements
independently report memory increases.
- Around line 21-28: Update the heartbeat and qualification measurement flow so
mainThreadStallMilliseconds captures main-actor execution caused by the measured
operation: run the relevant work on MainActor, including
ProviderQualificationRunner or IMAPServer operations as needed, before recording
stalls. Do not leave the metric measuring unrelated scheduling delay;
alternatively, explicitly document it as harness-only evidence and avoid
applying QualificationBudget.adr0027 to it.

In
`@tools/swiftmail-provider-qualification/Tests/SwiftMailProviderQualificationTests/QualificationTests.swift`:
- Around line 7-22: The test
configurationNamesMissingProtectedValuesWithoutEchoingSecrets currently discards
the error and does not exercise a secret-bearing environment. Update its iCloud
case to provide a known password while omitting the email variable, capture the
thrown error’s localizedDescription, and assert it names the missing email
variable without containing the supplied password.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a054df06-a7f9-4459-a366-7bbd1f40f635

📥 Commits

Reviewing files that changed from the base of the PR and between bf42eaa and 3317e38.

⛔ Files ignored due to path filters (1)
  • tools/swiftmail-provider-qualification/Package.resolved is excluded by !**/Package.resolved
📒 Files selected for processing (13)
  • .github/workflows/swiftmail-provider-qualification.yml
  • .gitignore
  • README.md
  • docs/qualification/swiftmail-provider.md
  • tools/swiftmail-provider-qualification/Package.swift
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/MailboxRoleResolver.swift
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/ProviderQualificationRunner.swift
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationConfiguration.swift
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationMessages.swift
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationMetrics.swift
  • tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationModels.swift
  • tools/swiftmail-provider-qualification/Sources/swiftmail-provider-qualification/main.swift
  • tools/swiftmail-provider-qualification/Tests/SwiftMailProviderQualificationTests/QualificationTests.swift
📜 Review details
🧰 Additional context used
📓 Path-based instructions (5)
README.md

📄 CodeRabbit inference engine (AGENTS.md)

Update README.md when developer setup, common commands, or high-level project navigation changes.

Files:

  • README.md
**/*.md

⚙️ CodeRabbit configuration file

**/*.md: Treat docs as product truth, not marketing. Prefer precise domain language
from CONTEXT.md; flag undefined terms, synonym drift, and conflicts with
existing ADRs under docs/adr/. For ADR changes, require a clear decision,
consequences, and privacy or security boundary impact; flag silent
reversals of prior ADRs without an explicit supersession note. For
docs/agents/ and bootstrap docs, verify agent workflow, validation, and
triage guidance stay aligned with AGENTS.md. Flag broken cross-links,
speculative future claims presented as shipped behavior, and docs that
omit the Apple vs TypeScript ownership of a boundary they describe.

Files:

  • README.md
  • docs/qualification/swiftmail-provider.md
**/*Tests/**/*.swift

⚙️ CodeRabbit configuration file

**/*Tests/**/*.swift: Verify externally observable behavior, privacy invariants, error paths,
cancellation or restart behavior where relevant, and deterministic
isolation. Flag fragile sleeps, ordering dependencies, and tests that
duplicate the implementation instead of exercising it.

Files:

  • tools/swiftmail-provider-qualification/Tests/SwiftMailProviderQualificationTests/QualificationTests.swift
.github/workflows/*.{yml,yaml}

📄 CodeRabbit inference engine (AGENTS.md)

.github/workflows/*.{yml,yaml}: CI must run, for non-draft pull requests and the default branch, pnpm install --frozen-lockfile, lint, format, type checks, tests, Fallow, Swift Format, SwiftLint, and the documented Xcode tests when the project exists.
Keep TypeScript, Fallow, and Apple validation in separate CI jobs; use the Fallow fallow-rs/fallow@v3 action in audit mode, failing only findings introduced since the relevant base commit.
Apple CI must use an affected-project matrix with matching path filters, run Debug tests separately from the Release-only local-mail performance fixture, use configuration-specific DerivedData caches, and keep the compatibility gate dependent on both jobs.
The hosted Apple commands must preserve the documented exclusions, serial testing, simulator destination, derived-data paths, compilation conditions, and separate mixed-connection and Release performance test invocations.

Files:

  • .github/workflows/swiftmail-provider-qualification.yml
.github/workflows/**/*.{yml,yaml}

⚙️ CodeRabbit configuration file

.github/workflows/**/*.{yml,yaml}: Check least-privilege permissions, immutable third-party action pins,
cache safety, secret handling, path filters, and parity with the local
validation commands documented in AGENTS.md.

Files:

  • .github/workflows/swiftmail-provider-qualification.yml
🪛 zizmor (1.29.0)
.github/workflows/swiftmail-provider-qualification.yml

[warning] 75-75: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)


[warning] 96-96: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)


[error] 39-39: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)


[error] 103-103: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🔇 Additional comments (28)
tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationConfiguration.swift (1)

4-89: LGTM!

tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/MailboxRoleResolver.swift (1)

3-57: LGTM!

tools/swiftmail-provider-qualification/Package.swift (1)

15-18: 🔒 Security & Privacy

Establish SwiftPM lockfile behavior before changing the dependency requirement.

Package.resolved records the documented revision, and the workflow checks that revision before tests and execution. The remaining question is whether swift package resolve can replace a locked revision when the 1.10.0 tag moves.

tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationModels.swift (5)

3-28: LGTM!


30-71: LGTM!


73-114: LGTM!


116-129: LGTM!


131-149: LGTM!

tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/QualificationMetrics.swift (2)

30-51: LGTM!


66-86: LGTM!

tools/swiftmail-provider-qualification/Sources/SwiftMailProviderQualification/ProviderQualificationRunner.swift (13)

7-21: LGTM!


95-114: LGTM!


116-135: LGTM!


137-175: LGTM!


230-287: LGTM!


289-350: LGTM!


420-481: LGTM!


483-551: LGTM!


553-577: LGTM!


579-657: LGTM!


659-758: LGTM!


364-399: 🩺 Stability & Availability

Keep the APPEND calls on the same IMAPServer.

SwiftMail 1.10.0 uses primaryConnection for non-IDLE commands and separate idleConnections for IDLE sessions. No second IMAPServer is required.

			> Likely an incorrect or invalid review comment.

792-799: 🗄️ Data Integrity & Integration

Retain UID-scoped cleanup

SwiftMail 1.10.0 maps expunge(messages:) to UID EXPUNGE and throws IMAPError.commandNotSupported when UIDPLUS is absent. It does not fall back to plain EXPUNGE.

			> Likely an incorrect or invalid review comment.
tools/swiftmail-provider-qualification/Sources/swiftmail-provider-qualification/main.swift (2)

17-32: LGTM!


48-87: LGTM!

.github/workflows/swiftmail-provider-qualification.yml (2)

3-26: LGTM!


55-99: 📐 Maintainability & Code Quality

No documentation issue found.

The runbook documents the qualification workflow invocation and all qualification environment variables. AGENTS.md contains no qualification-specific local command set to match.

.gitignore (1)

156-159: LGTM!

Comment thread .github/workflows/swiftmail-provider-qualification.yml
Comment thread .github/workflows/swiftmail-provider-qualification.yml
Comment thread .github/workflows/swiftmail-provider-qualification.yml
Comment thread docs/qualification/swiftmail-provider.md Outdated
Comment thread README.md Outdated
@rajzik
rajzik merged commit 3cd5f7c into main Aug 10, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant