-
Notifications
You must be signed in to change notification settings - Fork 1.5k
feat: add Helm chart for standalone AI Gateway #27256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
pawbana
merged 10 commits into
main
from
pawel/aigov-318-create-helm-chart-for-standalone-ai-gateway
Jul 22, 2026
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
a48dc1e
feat: add Helm chart for standalone AI Gateway
pawbana 3accb82
agentic review 1
pawbana e167ed3
agentic review 2
pawbana ac76410
agentic review 2.1
pawbana 1baf638
agentic review 3
pawbana bc0e490
min version in README
pawbana abd179c
test cleanup
pawbana 3a42089
review 1: gen-golden fail loud, fixture rename, some tests merged
pawbana 12ca765
review 1.1: remove env vars check, remove coder URL Heml option, upda…
pawbana 9170c6c
small README improvement
pawbana File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| tests |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| dependencies: | ||
| - name: libcoder | ||
| repository: file://../libcoder | ||
| version: 0.1.0 | ||
| digest: sha256:5c9a99109258073b590a9f98268490ef387fde24c0c7c7ade9c1a8c7ef5e6e10 | ||
| generated: "2026-07-14T17:15:50.670279008Z" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| apiVersion: v2 | ||
| name: coder-ai-gateway | ||
| description: Standalone AI Gateway for Coder | ||
| home: https://github.com/coder/coder | ||
| # These versions are replaced when Coder releases the chart. | ||
| type: application | ||
| version: "0.1.0" | ||
| appVersion: "0.1.0" | ||
| kubeVersion: ">= 1.23.0-0" | ||
| keywords: | ||
| - coder | ||
| - ai | ||
| sources: | ||
| - https://github.com/coder/coder/tree/main/helm/ai-gateway | ||
| icon: https://helm.coder.com/coder_logo_black.png | ||
| maintainers: | ||
| - name: Coder Technologies, Inc. | ||
| email: [email protected] | ||
| url: https://coder.com/contact | ||
|
|
||
| dependencies: | ||
| - name: libcoder | ||
| version: 0.1.0 | ||
| repository: file://../libcoder |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,189 @@ | ||
| # Coder AI Gateway Helm chart | ||
|
|
||
| This chart deploys the Coder AI Gateway as a standalone Kubernetes Deployment. | ||
| The Gateway connects to Coder using `CODER_URL` and an AI Gateway key. To forward | ||
| proxied AI traffic to the standalone Gateway, configure the Coder AI Gateway | ||
| Proxy (`aibridgeproxyd`) after installing the chart. | ||
|
|
||
| The chart does not create credentials or TLS Secrets. | ||
|
|
||
| ## Install | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| - An AI Gateway key created in Coder. | ||
| - A Coder image that includes the `coder ai-gateway start` command. The official | ||
| Coder v2.36.0 image is the first version to include this command. | ||
|
|
||
| ### Configure the chart | ||
|
|
||
| Create a `values.yaml` file with the Coder URL and an AI Gateway key source. The | ||
| following example uses a Kubernetes Secret in the Helm release namespace: | ||
|
|
||
| ```console | ||
| kubectl create secret generic coder-ai-gateway-key \ | ||
| --namespace <release-namespace> \ | ||
| --from-literal=key='<AI gateway key>' | ||
| ``` | ||
|
|
||
| ```yaml | ||
| coder: | ||
| image: | ||
| # Required when installing the chart directly from Git. | ||
| tag: "<coder version>" | ||
| env: | ||
| - name: CODER_URL | ||
| value: https://coder.example.com | ||
|
|
||
| aigateway: | ||
| keySecret: | ||
| name: coder-ai-gateway-key | ||
| ``` | ||
|
|
||
| The Gateway can also connect to Coder through an in-cluster Service, for | ||
| example: | ||
|
|
||
| ```yaml | ||
| coder: | ||
| env: | ||
| - name: CODER_URL | ||
| value: http://coder.coder.svc.cluster.local:80 | ||
| ``` | ||
|
|
||
| For HTTPS, the Coder certificate must cover the internal Service hostname and | ||
| the Gateway must trust its issuing CA. | ||
|
|
||
| Instead of `aigateway.keySecret`, set `CODER_AI_GATEWAY_KEY` or | ||
| `CODER_AI_GATEWAY_KEY_FILE` through `coder.env`. Environment variables can also | ||
| be supplied through `coder.envFrom`. The chart does not check for variable | ||
| conflicts, regardless of whether values come from Helm options, `coder.env`, or | ||
| `coder.envFrom`. | ||
|
|
||
| When installing a released chart package, the chart automatically uses the | ||
| matching Coder image version. Set `coder.image.tag` only when installing | ||
| directly from Git or overriding the image version. Custom images must provide | ||
| the `coder ai-gateway start` command. | ||
|
|
||
| ### Install the chart | ||
|
|
||
| ```console | ||
| helm install ai-gateway ./helm/ai-gateway \ | ||
| --namespace <release-namespace> \ | ||
| --values values.yaml | ||
| ``` | ||
|
|
||
| ## Connect Coder to the standalone Gateway | ||
|
|
||
| To route proxied AI requests through the standalone Gateway, configure the Coder | ||
| AI Gateway Proxy with a target URL. When `service.enable` is true, the chart | ||
| notes show the direct in-cluster Service URL, including the scheme selected by | ||
| `aigateway.listenerTLS`. Retrieve it with: | ||
|
|
||
| ```console | ||
| helm get notes ai-gateway --namespace <release-namespace> | ||
| ``` | ||
|
|
||
| The chart notes do not show an Ingress or `HTTPRoute` URL. To route through one | ||
| of these entry points, set `CODER_AI_GATEWAY_PROXY_TARGET` to its URL instead. | ||
| When `service.enable` is false, set the target to the URL of your user-managed | ||
| route to the Deployment. | ||
|
|
||
| When listener TLS uses a private CA, the AI Gateway Proxy must trust that CA to | ||
| connect directly to the Service over HTTPS. | ||
|
|
||
| ## TLS | ||
|
|
||
| For Gateway-to-Coder HTTPS with a private CA, set | ||
| `aigateway.coderTLS.caSecret`. If Coder requires client mTLS, also set | ||
| `aigateway.coderTLS.clientSecret`. | ||
|
|
||
| Prefer terminating client-facing TLS at a Kubernetes Ingress or a `Gateway` | ||
| resource from the Kubernetes Gateway API. To terminate TLS in the AI Gateway | ||
| process, set `aigateway.listenerTLS.name` to an existing TLS Secret. | ||
|
|
||
| Client-facing TLS and backend TLS are independent. The `ingress.tls` settings | ||
| configure TLS between clients and the Ingress. For `HTTPRoute`, the Gateway | ||
| listener that accepts client connections is configured outside this chart. | ||
| These settings do not configure whether the Ingress or Gateway connects to the | ||
| AI Gateway Service using HTTP or HTTPS. | ||
|
|
||
| When `aigateway.listenerTLS` is enabled behind an Ingress or `HTTPRoute`, | ||
| configure the entry point to connect to the Service using HTTPS and trust the AI | ||
| Gateway certificate. Ingress backend TLS is controller-specific and can usually | ||
| be configured with `ingress.annotations`. Gateway API backend TLS uses a | ||
| separate `BackendTLSPolicy`, which can be managed outside this chart or rendered | ||
| with `extraTemplates`. The chart does not infer or validate this | ||
| controller-specific configuration. Without backend TLS, the entry point sends | ||
| plaintext HTTP to the HTTPS listener, which typically results in a TLS handshake | ||
| error reported as HTTP 502. | ||
|
|
||
| All referenced TLS Secrets must exist in the Helm release namespace. | ||
|
|
||
| ## Networking | ||
|
|
||
| The data-plane Service, which carries LLM traffic, is a `ClusterIP` by default. | ||
| `NodePort` and `LoadBalancer` are explicit alternatives. Ingress and `HTTPRoute` | ||
| are optional and both route to the data-plane Service. If you enable Ingress or | ||
| `HTTPRoute`, use a `ClusterIP` Service unless you intentionally need a second | ||
| external entry point through a `LoadBalancer` Service. | ||
|
|
||
| ## Scaling and resources | ||
|
|
||
| Set `coder.replicaCount` to run multiple AI Gateway replicas. The default | ||
| resource requests are 1 CPU and 1 GiB of memory per replica. These requests are | ||
| a starting point, not a capacity guarantee. CPU and memory usage depend heavily | ||
| on concurrent requests and payload size. | ||
|
|
||
| Adjust `coder.resources` after observing production traffic. Consider setting | ||
| `CODER_AI_GATEWAY_MAX_CONCURRENCY` through `coder.env` to bound concurrent | ||
| requests per replica. The application default is unlimited. The chart does not | ||
| set resource limits by default, which avoids CPU throttling and fixed memory | ||
| limits for bursty workloads. Manage resources such as a Horizontal Pod | ||
| Autoscaler or PodDisruptionBudget through your platform configuration or | ||
| `extraTemplates`. | ||
|
|
||
| ## Metrics | ||
|
|
||
| Every pod runs an unauthenticated metrics listener on the named `metrics` port, | ||
| which maps to port `2112`. The chart does not create monitoring discovery | ||
| resources. Configure pod-based discovery with `coder.podAnnotations`, for | ||
| example: | ||
|
|
||
| ```yaml | ||
| coder: | ||
| podAnnotations: | ||
| prometheus.io/scrape: "true" | ||
| prometheus.io/port: "2112" | ||
| ``` | ||
|
|
||
| Alternatively, create discovery resources such as a `ServiceMonitor` through | ||
| your monitoring stack or `extraTemplates`. | ||
|
|
||
| ## Key rotation | ||
|
|
||
| 1. Create a new AI Gateway key in Coder. | ||
| 2. Update the configured key source: | ||
| - For `aigateway.keySecret`, update the referenced Secret or set `name` to a | ||
| new Secret. | ||
| - For a key supplied through `coder.env`, update the environment variable or | ||
| the file it references. | ||
| 3. If the update did not trigger a rollout, restart the Deployment, for example: | ||
|
|
||
| ```console | ||
| kubectl rollout restart deployment/coder-ai-gateway \ | ||
| --namespace <release-namespace> | ||
| ``` | ||
|
|
||
| 4. Verify every replica is ready and serving with the new key. | ||
| 5. Revoke the old key. | ||
|
|
||
| Secret updates do not change the Deployment pod template automatically. A | ||
| reloader controller can be configured through `coder.annotations` or | ||
| `coder.podAnnotations`. | ||
|
|
||
| ## Extra manifests | ||
|
|
||
| `extraTemplates` renders additional Kubernetes manifests as part of the Helm | ||
| release. Entries can be YAML strings or Kubernetes objects, and can use Helm | ||
| release values and chart helpers. Use them for small companion resources, such | ||
| as a `NetworkPolicy`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| Coder AI Gateway is installed as {{ include "coder.name" . }}. | ||
|
|
||
| {{- if .Values.service.enable }} | ||
| Configure the Coder deployment with: | ||
|
|
||
| CODER_AI_GATEWAY_PROXY_TARGET={{ ternary "https" "http" (not (empty .Values.aigateway.listenerTLS.name)) }}://{{ include "coder.name" . }}.{{ .Release.Namespace }}.svc.cluster.local:{{ .Values.service.port }} | ||
| {{- else }} | ||
| The chart-managed Service is disabled. Configure CODER_AI_GATEWAY_PROXY_TARGET | ||
| with the URL of the user-managed route to this deployment. | ||
| {{- end }} | ||
|
|
||
| The metrics listener is active on port 2112. Configure discovery with | ||
| coder.podAnnotations or resources managed by your monitoring stack. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.