Skip to content

feat: system attestation types#225

Closed
mintlify[bot] wants to merge 13 commits into
mainfrom
mintlify/system-attestation-types-preview
Closed

feat: system attestation types#225
mintlify[bot] wants to merge 13 commits into
mainfrom
mintlify/system-attestation-types-preview

Conversation

@mintlify
Copy link
Copy Markdown
Contributor

@mintlify mintlify Bot commented May 13, 2026

DO NOT MERGE YET OR APPROVE PLEASE - DOCS DRIVEN DEVELOPMENT

Mirror of feat/system-attestation-types opened through Mintlify so the preview build runs.

No content changes — this branch contains the same diff as the original feat/system-attestation-types branch, just opened via the Mintlify-authored flow so the Mintlify preview check is triggered.

Original PR: #224

@mintlify mintlify Bot requested a review from a team as a code owner May 13, 2026 15:01
@mintlify
Copy link
Copy Markdown
Contributor Author

mintlify Bot commented May 13, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
kosli 🟢 Ready View Preview May 13, 2026, 3:03 PM

The page was moved from administration/managing_custom_attestation_types/
overview.md to attestation_types/custom/managing.mdx as part of consolidating
the custom-type docs. The redirect is already in place in config/redirects.json
and the navigation entry was removed; the old file was reintroduced by the
rebase and is now redundant. Delete it so visitors only resolve the new path.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
AlexKantor87 and others added 2 commits May 13, 2026 16:29
Replace the single GitHub example with an AccordionGroup covering all four
supported providers (GitHub / GitHub Enterprise, GitLab / self-hosted GitLab,
Bitbucket Cloud, Azure DevOps). Each accordion has a copy-pasteable example
and a flag table noting which flags are required, which auto-default in CI,
and which only apply to self-hosted / enterprise variants.

Also add a "What gets recorded" subsection clarifying that auth tokens are
never stored in the attestation — the flags are CLI inputs, not attestation
content. The pull_requests array shape is identical across providers.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
System-type compliance is now described in customer-observable terms, not
implementation. Each per-type page states what the compliance check actually
is for that type:

- system:sarif — schema-driven (records facts, no built-in opinion)
- system:junit — every suite has zero failures and zero errors
- system:jira — at least one referenced issue, all exist
- system:sonar — quality-gate status is OK
- system:pull_request — at least one PR for the commit
- system:generic — caller-supplied is_compliant flag

The legacy-type compliance rules preserve the behaviour of the dedicated
commands they replace (kosli attest junit etc.), so existing customers can
migrate to kosli attest system <type> without changes. For new system types
(starting with system:sarif) compliance is schema-only; opinions live in
kosli evaluate + rego on a separate attestation.

Removes the jq mentions and "structural only" framing from customer-facing
copy. jq remains as an optional internal implementation detail for legacy
types — never surfaced to customers.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Switch the customer-facing model to schema-only compliance for every system
type. The earlier "legacy types carry their existing compliance behaviour
forward" framing is gone — system types record facts; opinions live in
kosli evaluate + rego on a separate attestation.

Migration from a legacy kosli attest <type> command does NOT preserve the
built-in verdict (e.g. kosli attest junit was non-compliant on any failed
test; system:junit is compliant once results are recorded). This is loud
on every per-type page and in a new dedicated migration guide that ships a
copy-paste rego policy for each legacy command to reproduce its verdict
via kosli evaluate.

Changes:

- system/overview.mdx: compliance section is now "schema-validated only".
  Adds a Warning callout pointing at the new migration page.
- system/{junit,jira,sonar,pull_request}.mdx: compliance sections rewritten
  to schema-only with a pointer to the migration page anchor.
- system/sarif.mdx and system/generic.mdx tightened for consistency.
- attestation_types/overview.mdx and custom/overview.mdx comparison
  tables: System row simplified to "schema-validated only; use kosli
  evaluate for opinions".
- system/catalogue.mdx: legacy types are now described as deprecated,
  not the recommended path, with a link to the migration guide.
- system/legacy_migration.mdx (NEW): per-type migration guide. For each
  legacy command, diff-style attest call migration plus a copy-paste rego
  policy reproducing the legacy compliance rule, ready to run via
  kosli evaluate trail. Covers snyk, junit, sonar, jira, pullrequest_*,
  generic.
- config/navigation.json: new page registered under System attestation types,
  between overview and the available-types subgroup.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
The legacy-command migration content lived under attestation_types/system/.
Move it to tutorials/ in the "Attesting" group where it belongs — it's a
workflow, not a type reference.

While moving, restructure for the actual customer use case:

- Reframe as a four-step workflow: replace the attest call, write the rego,
  run kosli evaluate trail with --output json, and record the verdict as a
  system:generic attestation with --compliant/--attachments/--user-data.
  Step 4 is what the old page was missing — without it the rego verdict
  doesn't end up on the trail at all.

- Show the kosli evaluate output (RESULT/VIOLATIONS) so readers know what
  they should see in CI.

- Per-type recipes go in collapsible accordions: snyk, junit, sonar, jira,
  pullrequest_*, generic. Each accordion holds just the diff and the
  type-specific rego; the workflow shape is shared at the top so we don't
  repeat steps 3 and 4 six times.

Also:
- Add a redirect from the old /attestation_types/system/legacy_migration
  path to the new /tutorials/migrating_to_system_types so any external link
  still resolves.
- Update every per-type page's cross-reference (jira, junit, sonar, sarif,
  pull_request, generic), plus catalogue and the overview Warning.
- Rewire navigation.json: drop the old page from the system types group;
  add the new tutorial between attest_sarif and attest_snyk under the
  Attesting group.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Jon's review feedback: every rego recipe hardcoded the attestation name
(e.g. "security-scan", "unit-tests") inside the policy, but the name
varies per customer. Same pattern as the OPA tutorial's
data.params.max_high / max_medium thresholds.

Read attestation_name := data.params.attestation_name at the top of each
policy and reference [attestation_name] in the lookups. One rego file
works regardless of what --name the customer set on the system attestation.
If the param is missing the lookup is undefined, the compliance condition
fails to fire, and allow stays false — correct fail-safe behaviour.

Step 3 of the general workflow now shows --params '{"attestation_name":
"security-scan"}' on the kosli evaluate trail call, and every accordion's
"run steps 3 and 4" line tells the reader which value to pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Jon's review: --artifact-type docker my_company/backend:latest is valid but
hard to scan because the positional sits in the middle of the flag values.
Convention elsewhere on the page (the kosli attest artifact example just
above this one) puts the positional right after the command. Match it.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…ypes

Jon's review: the system-types overview said system types "know the shape
of the input, parse it, and render it in the trail" — true on the server
side but missing the bigger client-side win. The Kosli CLI takes the raw
output from your tool (JUnit XML directory, SARIF file, Sonar project key,
Jira API, git provider API), parses it, posts the structured attestation,
AND uploads the source file to the Evidence Vault as an attachment. One
command, no glue scripting. Custom types require you to do that work
yourself before attesting.

Updates four pages so the contrast is visible everywhere a customer might
land:

- attestation_types/system/overview.mdx — adds a "First-class CLI support
  for the source format" bullet and a contrast paragraph about custom
  types parsing being the caller's problem.

- attestation_types/overview.mdx — System Card description rewritten;
  comparison table grows a "Data preparation" column.

- attestation_types/custom/overview.mdx — comparison table mirrored to
  carry the same "Data preparation" row.

- getting_started/attestations.md — System types accordion leads with the
  CLI-parsing-and-vault story; Custom types accordion adds an explicit
  contrast paragraph noting the caller produces the JSON. Also picks up
  the "structural" → "schema-validated" wording fix while in the same
  block, and adds a link to the per-type migration tutorial we just shipped.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Already implied by the rest of the page — and the customer experience is
better served by letting the table speak for itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Surface the default value on the per-type page so callers don't get caught
out — kosli attest system generic with no --compliant flag produces a
compliant attestation, matching the legacy kosli attest generic default.
Also note the --compliant=false vs --compliant false gotcha that bites with
Cobra boolean flags (same warning already exists in the migration tutorial
and the OPA tutorial).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Jon's review: kosli evaluate trail has a --no-assert flag that makes the
command exit 0 even when the policy denies, so the surrounding "|| true"
and stderr redirect are unnecessary. The new shape is cleaner and surfaces
real stderr if something goes wrong (vs. silently swallowing it).

Also add a one-sentence explanation of why --no-assert is needed for the
two-step "evaluate then attest" flow — the deny is captured in the JSON
report instead of as an exit code, which step 4 reads from.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@AlexKantor87
Copy link
Copy Markdown
Contributor

Closing as not planned. We're parking the system-types initiative for now — no immediate customer or business driver makes it the right thing to ship over other work.

Nothing here is wasted:

  • The docs branch (PR feat: system attestation types #225) captured the full customer experience we'd ship: schema-only compliance for system types, per-type pages, @type-version/type_version: notation, per-provider accordions for system:pull_request, the legacy-command migration tutorial with copy-paste rego policies, and the CLI-side parsing + Evidence Vault story. Anyone resuming this work has the customer-facing spec already drafted.
  • Mike Long's server PR https://github.com/kosli-dev/server/pull/4871 stays open as the foundation if/when we come back to it.
  • The decisions made along the way (schema-only over jq for system types, --type-version over @vN, splitting CLI parsing as the user-visible win, layered kosli evaluate for compliance opinions) are preserved in the ticket bodies and the docs branch.

If a customer driver emerges that justifies prioritising this, reopen the master ticket #5647 and the sub-issues; the docs branch is mergeable as a starting point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant