-
Notifications
You must be signed in to change notification settings - Fork 1.5k
feat: add generated experiments reference (DOCS-139) #29311
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| --- | ||
| # Code generated by make gen. DO NOT EDIT. | ||
| title: Experiments | ||
| description: "Reference for the experimental features a Coder deployment can enable, with the key each one uses." | ||
| icon_path: "./images/icons/puzzle.svg" | ||
| --- | ||
|
|
||
| <!-- DO NOT EDIT | GENERATED CONTENT --> | ||
|
|
||
| An experiment is a feature that is not ready for production. | ||
| Experiments are disabled by default, are not guaranteed to be backward compatible, and can be renamed or removed at any time. | ||
|
|
||
| Enable one by passing its key to `coder server`: | ||
|
|
||
| ```shell | ||
| coder server --experiments=<experiment-key> | ||
| ``` | ||
|
|
||
| The same keys work through the `CODER_EXPERIMENTS` environment variable. | ||
| For how experiments relate to beta and generally available features, refer to [Feature stages](../install/releases/feature-stages.md). | ||
|
|
||
| ## The wildcard value | ||
|
|
||
| `--experiments=*` enables only the experiments Coder marks as safe for general opt-in, not every experiment on this page. | ||
|
|
||
| No experiment currently carries that mark, so the wildcard enables nothing. | ||
| Enable an experiment by naming its key. | ||
|
|
||
| ## Available experiments | ||
|
|
||
| | Experiment | Key | Description | | ||
| |---------------------------------|-------------------------------|------------------------------------------------------------------------------------------| | ||
| | Agent Lifecycle Hooks | `agent-lifecycle-hooks` | Enables chat lifecycle hook webhooks for agent chats. | | ||
| | AI Gateway Seat Exclusion | `ai-gateway-seat-exclusion` | Excludes AI Gateway (AI Bridge) usage from AI Governance seat consumption. | | ||
| | Auto-fill Template Parameters | `auto-fill-parameters` | This should not be taken out of experiments until we have redesigned the feature. | | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3 [CRF-3] The generator publishes raw constant comments as the Description column, so an internal maintainer note ships as user documentation. (Ryosuke P3, Leorio P3, Chopper P3, Pariston P3, Mafuuu P3, Mafu-san P3, Netero P3, Knov P3) The page's stated job is to tell an operator what each experiment does. For The PR body acknowledges this and says a follow-up was "filed," but no ticket is linked in the diff. Under the no-follow-up assumption the misleading rows ship as the official public description. This needs a human decision: link the tracking ticket, rewrite the two constant comments in
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The missing-ticket premise is false: DOCS-905 exists, is linked to DOCS-139, and specifically tracks these two comments. I kept the wording follow-up there because behavior verification needs product ownership. 553b35a makes missing descriptions fail closed.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Accepted as a ticketed deferral (DOCS-905). The two internal-note descriptions (auto-fill-parameters, example) still ship on the rendered page; tracking the wording change under product ownership is a reasonable disposition.
|
||
| | Chat Advisor | `chat-advisor` | Enables the advisor tool for root agent chats. | | ||
| | Chat Virtual Desktop | `chat-virtual-desktop` | Enables virtual desktop and computer use provider for agents. | | ||
| | Example Experiment | `example` | This isn't used for anything. | | ||
| | MCP HTTP Server Functionality | `mcp-server-http` | Enables the MCP HTTP server functionality. | | ||
| | MCP Tool Search | `mcp-tool-search` | Defers MCP tool schemas behind a searchable catalog in agent chats. | | ||
| | NATS Pubsub | `nats_pubsub` | Enables embedded NATS pubsub. | | ||
| | SMTP and Webhook Notifications | `notifications` | Sends notifications via SMTP and webhooks following certain events. | | ||
| | Workspace Build Updates Channel | `workspace-build-updates` | Enables publishing workspace build updates to the all builds pubsub channel. | | ||
| | Workspace-Capable Licensing | `workspace-capable-licensing` | Counts only users holding the workspace-create permission toward the license seat limit. | | ||
| | Workspace Usage Tracking | `workspace-usage` | Enables the new workspace usage tracking. | | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,249 @@ | ||
| // Command experimentsdocgen generates the experiments reference at | ||
| // docs/reference/experiments.md from codersdk. | ||
| // | ||
| // Two sources are combined. The set of experiments, their display names, and | ||
| // whether each one is safe to enable through the wildcard come from the | ||
| // codersdk package at run time. The per-experiment descriptions live as | ||
| // trailing comments on the Experiment constants, so they are read from the | ||
| // syntax tree of the file that declares them. | ||
| package main | ||
|
|
||
| import ( | ||
| "flag" | ||
| "fmt" | ||
| "go/ast" | ||
| "go/parser" | ||
| "go/token" | ||
| "slices" | ||
| "strconv" | ||
| "strings" | ||
|
|
||
| "golang.org/x/xerrors" | ||
|
|
||
| utilstrings "github.com/coder/coder/v2/coderd/util/strings" | ||
| "github.com/coder/coder/v2/codersdk" | ||
| "github.com/coder/coder/v2/scripts/atomicwrite" | ||
| "github.com/coder/coder/v2/scripts/docgenenv" | ||
| "github.com/coder/flog" | ||
| ) | ||
|
|
||
| // routeTitles is the manifest breadcrumb to this page's route. The page | ||
| // mirrors that route's metadata into its front matter, so the manifest stays | ||
| // the single source of the title and description. | ||
| var routeTitles = []string{"Reference", "Experiments"} | ||
|
|
||
| const intro = `An experiment is a feature that is not ready for production. | ||
| Experiments are disabled by default, are not guaranteed to be backward compatible, and can be renamed or removed at any time. | ||
|
|
||
| Enable one by passing its key to ` + "`coder server`" + `: | ||
|
|
||
| ` + "```shell" + ` | ||
| coder server --experiments=<experiment-key> | ||
| ` + "```" + ` | ||
|
|
||
| The same keys work through the ` + "`CODER_EXPERIMENTS`" + ` environment variable. | ||
| For how experiments relate to beta and generally available features, refer to [Feature stages](../install/releases/feature-stages.md). | ||
|
|
||
| ` | ||
|
|
||
| const wildcardSection = `## The wildcard value | ||
|
|
||
| ` + "`--experiments=*`" + ` enables only the experiments Coder marks as safe for general opt-in, not every experiment on this page. | ||
|
|
||
| ` | ||
|
|
||
| const tableSection = `## Available experiments | ||
|
|
||
| ` | ||
|
|
||
| func main() { | ||
| manifestPath := flag.String("manifest", "docs/manifest.json", "path to the docs manifest that supplies the page metadata") | ||
| source := flag.String("source", "codersdk/deployment.go", "path to the Go file declaring the Experiment constants") | ||
| out := flag.String("out", "docs/reference/experiments.md", "path to write the generated reference page") | ||
| flag.Parse() | ||
|
|
||
| manifest, err := docgenenv.LoadManifest(*manifestPath) | ||
| if err != nil { | ||
| flog.Fatalf("%v", err) | ||
| } | ||
| route := manifest.FindRoute(routeTitles...) | ||
| if route == nil { | ||
| flog.Fatalf("manifest %q has no route %q", *manifestPath, strings.Join(routeTitles, " > ")) | ||
| } | ||
|
|
||
| descriptions, err := readDescriptions(*source) | ||
| if err != nil { | ||
| flog.Fatalf("%v", err) | ||
| } | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note [CRF-8] Structural alternative: a Reusing the trailing comment is what forces the maintainer-note problem in CRF-3 and the doc-comment leak in CRF-7.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Considered. A Description() API would remove AST coupling, but changes codersdk product metadata and needs owner agreement. Filed DOCS-907 to evaluate it; current hardening remains AST-scoped.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Accepted as a ticketed deferral (DOCS-907). Agreed the Description() direction changes codersdk product metadata and needs owner agreement; keeping the current hardening AST-scoped is reasonable.
|
||
| displayNames, err := readDisplayNames(*source) | ||
| if err != nil { | ||
| flog.Fatalf("%v", err) | ||
| } | ||
|
|
||
| content, err := render(*route, codersdk.ExperimentsKnown, codersdk.ExperimentsSafe, descriptions, displayNames) | ||
| if err != nil { | ||
| flog.Fatalf("render experiments reference: %v", err) | ||
| } | ||
| if err := atomicwrite.File(*out, []byte(content)); err != nil { | ||
| flog.Fatalf("write %s: %v", *out, err) | ||
| } | ||
| flog.Successf("wrote %s", *out) | ||
| } | ||
|
|
||
| // readDescriptions returns the trailing comment on each Experiment constant, | ||
| // keyed by the experiment's string value. | ||
| func readDescriptions(path string) (map[string]string, error) { | ||
| fset := token.NewFileSet() | ||
| file, err := parser.ParseFile(fset, path, nil, parser.ParseComments) | ||
| if err != nil { | ||
| return nil, xerrors.Errorf("parse %q: %w", path, err) | ||
| } | ||
|
|
||
| descriptions := map[string]string{} | ||
| for node := range ast.Preorder(file) { | ||
| spec, ok := node.(*ast.ValueSpec) | ||
| if !ok { | ||
| continue | ||
| } | ||
| ident, ok := spec.Type.(*ast.Ident) | ||
| if !ok || ident.Name != "Experiment" || len(spec.Values) != 1 { | ||
| continue | ||
| } | ||
| lit, ok := spec.Values[0].(*ast.BasicLit) | ||
| if !ok || lit.Kind != token.STRING { | ||
| continue | ||
| } | ||
| value, err := strconv.Unquote(lit.Value) | ||
| if err != nil { | ||
| continue | ||
| } | ||
| descriptions[value] = commentSentence(spec) | ||
| } | ||
|
|
||
| if len(descriptions) == 0 { | ||
| return nil, xerrors.Errorf("no Experiment constants found in %q", path) | ||
| } | ||
| return descriptions, nil | ||
| } | ||
|
|
||
| func readDisplayNames(path string) (map[string]bool, error) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit [CRF-21] The function returns
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note [CRF-16] Both functions call
|
||
| fset := token.NewFileSet() | ||
| file, err := parser.ParseFile(fset, path, nil, 0) | ||
| if err != nil { | ||
| return nil, xerrors.Errorf("parse %q: %w", path, err) | ||
| } | ||
|
|
||
| constants := map[string]string{} | ||
| for node := range ast.Preorder(file) { | ||
| spec, ok := node.(*ast.ValueSpec) | ||
| if !ok || len(spec.Names) != 1 || len(spec.Values) != 1 { | ||
| continue | ||
| } | ||
| ident, ok := spec.Type.(*ast.Ident) | ||
| if !ok || ident.Name != "Experiment" { | ||
| continue | ||
| } | ||
| lit, ok := spec.Values[0].(*ast.BasicLit) | ||
| if !ok || lit.Kind != token.STRING { | ||
| continue | ||
| } | ||
| value, err := strconv.Unquote(lit.Value) | ||
| if err != nil { | ||
| continue | ||
| } | ||
| constants[spec.Names[0].Name] = value | ||
| } | ||
|
|
||
| displayNames := map[string]bool{} | ||
| for node := range ast.Preorder(file) { | ||
| decl, ok := node.(*ast.FuncDecl) | ||
| if !ok || decl.Name.Name != "DisplayName" || decl.Recv == nil || len(decl.Recv.List) != 1 { | ||
| continue | ||
| } | ||
| receiver, ok := decl.Recv.List[0].Type.(*ast.Ident) | ||
| if !ok || receiver.Name != "Experiment" { | ||
| continue | ||
| } | ||
| for node := range ast.Preorder(decl.Body) { | ||
| clause, ok := node.(*ast.CaseClause) | ||
| if !ok { | ||
| continue | ||
| } | ||
| for _, expr := range clause.List { | ||
| ident, ok := expr.(*ast.Ident) | ||
| if !ok { | ||
| continue | ||
| } | ||
| if value, ok := constants[ident.Name]; ok { | ||
| displayNames[value] = true | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| return displayNames, nil | ||
| } | ||
|
|
||
| func commentSentence(spec *ast.ValueSpec) string { | ||
| if spec.Comment == nil { | ||
| return "" | ||
| } | ||
| return sentence(spec.Comment.Text()) | ||
| } | ||
|
|
||
| func render(route docgenenv.Route, known, safe codersdk.Experiments, descriptions map[string]string, displayNames map[string]bool) (string, error) { | ||
| var b strings.Builder | ||
| // The front matter title renders as the page heading, so the body starts | ||
| // at the intro and its sections begin at level two. | ||
| _, _ = b.WriteString(docgenenv.GeneratedHeader(route)) | ||
| _, _ = b.WriteString(intro) | ||
| _, _ = b.WriteString(wildcardSection) | ||
|
|
||
| if len(safe) > 0 { | ||
| _, _ = b.WriteString("These experiments are safe to enable with the wildcard:\n\n") | ||
| for _, exp := range safe { | ||
| _, _ = fmt.Fprintf(&b, "- `%s`\n", exp) | ||
| } | ||
| _, _ = b.WriteString("\n") | ||
| } else { | ||
| _, _ = b.WriteString("No experiment currently carries that mark, so the wildcard enables nothing.\nEnable an experiment by naming its key.\n\n") | ||
| } | ||
|
|
||
| _, _ = b.WriteString(tableSection) | ||
| _, _ = b.WriteString("| Experiment | Key | Description |\n|------------|-----|-------------|\n") | ||
|
|
||
| known = slices.Clone(known) | ||
| slices.Sort(known) | ||
| for _, exp := range known { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3 [CRF-20] The page's completeness rests on the hand-maintained The page promises every experiment a deployment can enable, and render derives that set from
|
||
| key := string(exp) | ||
| desc, ok := descriptions[key] | ||
| if !ok || desc == "" { | ||
| return "", xerrors.Errorf("experiment %q has no description comment on its constant", key) | ||
| } | ||
| displayName := exp.DisplayName() | ||
| if !displayNames[key] { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P3 [CRF-19] The display-name gate checks that an explicit case exists, not that it returns an authored non-empty name, so it is narrower than the description gate it mirrors. (Knov P3, Hisoka Note) Two residual gaps on the CRF-1 fix:
|
||
| return "", xerrors.Errorf("experiment %q has no explicit display name", key) | ||
| } | ||
| _, _ = fmt.Fprintf(&b, "| %s | `%s` | %s |\n", markdownCell(displayName), key, markdownCell(desc)) | ||
| } | ||
|
|
||
| return strings.TrimRight(b.String(), "\n") + "\n", nil | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Note [CRF-15] The table is sorted by experiment key, so the leading "Experiment" (display name) column is not alphabetical. (Chopper) Rows are ordered by
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Declined. Key order is the stable operator-facing identifier order and matches the page's command input; at 13 rows, a separate display-name sort does not improve lookup enough to outweigh that consistency.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Defense accepted; closing. The panel judged 10/10 that sorting by key keeps the Key column (what an operator types into --experiments) alphabetical, and the unsorted display-name column is bounded friction at 13 rows. No re-raise.
|
||
| } | ||
|
|
||
| func markdownCell(value string) string { | ||
| return strings.ReplaceAll(value, "|", `\|`) | ||
| } | ||
|
|
||
| // sentence collapses a comment to one line, capitalizes it, and gives it | ||
| // terminal punctuation so it reads as a sentence in a table cell. | ||
| func sentence(s string) string { | ||
| s = strings.Join(strings.Fields(s), " ") | ||
| if s == "" { | ||
| return s | ||
| } | ||
| s = utilstrings.Capitalize(s) | ||
| if !strings.HasSuffix(s, ".") { | ||
| s += "." | ||
| } | ||
| return s | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
P3 [CRF-4] The hand-written wildcard note restates a live fact the generated page derives, so it drifts the moment
ExperimentsSafeis populated. (Hisoka P3, Ryosuke P3, Knov P3)The generated
experiments.mdcomputes the wildcard's effect fromcodersdk.ExperimentsSafeat run time: non-empty prints the safe list, empty prints "the wildcard enables nothing." This page hardcodes the empty case in prose: "no experiment currently carries that mark, so name the experiments you want instead." Nothing regenerates this file. The whole PR exists because hand-maintained experiment docs drift from source; this change fixes the drift in the reference page and reintroduces it here, on a security-relevant opt-in, with no mechanical guard. The day someone marks an experiment safe,make genupdates the reference and this guide asserts the opposite.Keep the durable clause (the wildcard enables only the experiments Coder marks safe, not all) and drop the time-bound one, pointing at the generated reference, which already sits two lines down.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Accepted. Removed the time-bound empty-safe-set claim; the generated reference remains the source for the current wildcard set. Resolved in 553b35a.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Verified fixed in 553b35a. feature-stages.md dropped the time-bound empty-safe-set clause and links to the generated reference, which owns the dynamic wildcard fact.