Skip to content

Add Deploy-to-Azure Bicep template - #1477

Open
Josh Duffney (duffney) wants to merge 25 commits into
mainfrom
duffney-deploy-azure-button-infra
Open

Josh Duffney (duffney) wants to merge 25 commits into
mainfrom
duffney-deploy-azure-button-infra

Conversation

@duffney

@duffney Josh Duffney (duffney) commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds a "Deploy to Azure" portal button that provisions the Azure infrastructure needed to run Scope: AKS, networking, Key Vault, Cosmos DB, Redis, Storage, and a container registry. This gives anyone a one-click way to stand up a ready-to-use environment, as a sanitized, minimal, resource-group-scoped Bicep template under a new deploy/azure/ folder.

The existing infra/ folder (azd dev-CosmosDB setup) and root azure.yaml are untouched; this is a separate, independent deployment path intended for new environments, not local/dev workflows.

Approach

deploy/azure/main.bicep orchestrates 7 modules under deploy/azure/modules/ (networking, aks, keyvault, cosmosdb, redis, storage, acr), built against a shared parameter/output contract and using Azure Verified Modules (AVM) where available:

  • AKS: one System + one User node pool (Linux only), Cilium network policy, OIDC issuer + workload identity enabled, native Azure Key Vault Secrets Provider add-on for pulling secrets into the cluster.
  • Networking: VNet + subnets, with private endpoints + private DNS zones for Key Vault, Cosmos DB, Redis, and Storage.
  • Key Vault: RBAC authorization mode, workload identity granted read access via role assignment, no hardcoded secret names.
  • Cosmos DB: Mongo API only, serverless by default.
  • Redis: Azure Cache for Redis with Entra ID auth preferred over access keys where supported.
  • Storage: blob + queue only, shared-key access disabled.
  • ACR: admin user disabled, AcrPull granted to the AKS kubelet identity (used to host the worker image).
  • No observability/monitoring resources in this v1 (no Log Analytics, no Grafana, no alerting) — kept deliberately minimal for a first pass.

main.bicep is compiled to deploy/azure/azuredeploy.json (the file the Deploy-to-Azure button URL points at), alongside main.parameters.json, createUiDefinition.json (Basics + Advanced portal UI flow), and metadata.json for the template gallery. A new CI workflow (.github/workflows/deploy-azure-drift.yml) rebuilds main.bicep and fails if azuredeploy.json has drifted, plus lints all module files. deploy/azure/README.md documents prerequisites, the CLI deploy path, and explicitly what this does not automate: the Helm-based app install itself, and BYO secrets the user must populate in Key Vault before/after deploying. The root README.md links to it with the Deploy-to-Azure button.

Notable assumptions (documented in deploy/azure/README.md)

  • AKS API server stays publicly reachable (not a private cluster) and local accounts remain enabled, so az aks get-credentials + kubectl/helm work without a jumpbox; only data-plane services (Key Vault, Cosmos, Redis, Storage) are private.
  • Cosmos DB uses serverless throughput by default to minimize idle cost for a quickstart template.
  • Classic Azure Cache for Redis is used rather than Azure Managed Redis (Enterprise) for v1.
  • Redis per-principal access-policy assignment isn't exposed by the AVM module version used, so it's a documented manual post-deployment step; the primary key is retained as a fallback.

Testing

  • az bicep build on main.bicep and each of the 7 modules individually: all compile with zero errors and zero warnings.
  • Verified azuredeploy.json matches a fresh rebuild of main.bicep exactly (no drift), matching the new CI check's logic.
  • Validated createUiDefinition.json and main.parameters.json as syntactically valid JSON.
  • Not run: live az deployment group validate/what-if against a real Azure subscription (no Azure credentials available in this environment).

Documentation and compatibility

  • New deploy/azure/README.md added; root README.md updated with a "Deploy to Azure" section and button.
  • No breaking changes; this is a new, additive deployment path alongside the existing infra//azd setup.

Checklist

  • If Portal features changed, keep CLI capabilities in sync. (N/A - infra-only change)
  • If Portal components changed, update their Storybook stories. (N/A)
  • If database changes require a migration, include up() / down() and keep it CosmosDB-compatible. (N/A)
  • If dependencies changed, update the lockfile and regenerate NOTICE / NOTICE-REVIEW.txt with pnpm notice as needed. (N/A - no dependency changes)
  • Video showing the behavior before the suggested change (N/A - infra/deployment template, not a UI change)
  • Video showing the behavior after the suggested change (N/A - infra/deployment template, not a UI change)

Provisions AKS (OIDC issuer, workload identity, native Key Vault Secrets
Provider add-on, Cilium network policy), a VNet with private endpoints for
Key Vault/Cosmos DB/Redis/Storage, Cosmos DB (MongoDB API only), Azure Cache
for Redis (Entra ID auth preferred), a blob+queue storage account, and an ACR
with AcrPull granted to the AKS kubelet identity.

Explicitly excludes azd, FluxCD/GitOps, ASO, ESO, KEDA-via-Flux, CorpNet IP
allowlisting, GitHub App/Entra bootstrap scripts, and Windows node pools.
Does not touch the existing infra/ (azd dev CosmosDB) or root azure.yaml.

Includes:
- main.bicep orchestrator + 7 modules (networking, aks, keyvault, cosmosdb,
  redis, storage, acr), using Azure Verified Modules where available
- main.parameters.json with sensible defaults
- Compiled azuredeploy.json for the Deploy-to-Azure button, plus a CI
  workflow (.github/workflows/deploy-azure-drift.yml) that fails if it
  drifts from main.bicep
- createUiDefinition.json with a Basics/Advanced portal UI flow
- metadata.json for the template gallery
- deploy/azure/README.md documenting prerequisites, the CLI deploy path,
  and what this does NOT automate (the Helm app install, BYO secrets)
- Deploy-to-Azure button + link added to the root README

All Bicep files compile cleanly via 'az bicep build' with zero errors and
zero warnings.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
@github-actions github-actions Bot added type: documentation Documentation additions, corrections, and improvements. area: cicd Build, test, release, and deployment pipelines. labels Oct 8, 2026
@duffney Josh Duffney (duffney) changed the title Add sanitized Deploy-to-Azure Bicep template under deploy/azure/ Add Deploy-to-Azure Bicep template Oct 8, 2026
Josh Duffney (duffney) and others added 2 commits October 8, 2026 12:30
Upgrade avm/res/container-service/managed-cluster from 0.9.0 to 0.14.0.
The pinned 0.9.0/0.13.0 module versions generated a managedClusters
resource at API version 2024-09-02-preview, which Azure has since
retired (not present in the supported api-versions list), causing
deployments to fail with:

  NoRegisteredProviderFound: No registered resource provider found for
  location 'westus3' and API version '2024-09-02-preview' for type
  'managedClusters'.

0.14.0 generates the resource at 2025-10-01 (current, non-preview).
Updated param usage to match this version's schema: enablePrivateCluster
moved under apiServerAccessProfile, and enableWorkloadIdentity moved
under securityProfile.workloadIdentity.enabled. All module outputs
(resourceId, name, oidcIssuerUrl, kubeletIdentityObjectId,
systemAssignedMIPrincipalId) are unchanged between versions.

Verified: deploy/azure/modules/aks.bicep and deploy/azure/main.bicep
both compile with zero errors/warnings via 'az bicep build'; confirmed
2024-09-02-preview no longer appears anywhere in the compiled
azuredeploy.json.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
…t kind

The AVM database-account module (br/public:avm/res/document-db/database-account:0.10.0)
derives the account's 'kind' property from whether mongodbDatabases is
non-empty, independent of capabilitiesToAdd. Passing an empty
mongodbDatabases array left the account at kind=GlobalDocumentDB (SQL
API) even with EnableMongo set, which then rejected the MongoDB-groupId
private endpoint with:

  BadRequest: Call to Microsoft.DocumentDB/databaseAccounts failed.
  Error message: GroupId MongoDB is not supported

Fixed by provisioning a default database (name configurable via the new
'databaseName' param, default 'scope') so the module correctly resolves
kind=MongoDB. The child mongodb-database module already skips setting
throughput when EnableServerless is present, so this has no cost impact
under the default serverless configuration.

Verified: deploy/azure/modules/cosmosdb.bicep and the full
deploy/azure/main.bicep both compile with zero errors/warnings via
'az bicep build'; confirmed the compiled azuredeploy.json's kind
expression now resolves to 'MongoDB' when mongodbDatabases is populated.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, Josh! The module separation, private-endpoint wiring, and secure connection-string outputs are a good foundation. I've left three initial comments on resource naming, the portal wizard, and the post-deployment instructions.

Comment thread deploy/azure/main.bicep

var vnetName = 'vnet-${environmentName}-${resourceToken}'
var aksName = 'aks-${environmentName}-${resourceToken}'
var keyVaultName = take('kv-${environmentName}-${resourceToken}', 24)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve a valid name and the uniqueness suffix when truncating.

A 20-character environmentName is accepted by the template, but this expression produces a Key Vault name ending in a hyphen. For example, abcdefghijklmnopqrst becomes kv-abcdefghijklmnopqrst-, which Azure rejects. With a 19-character name, only one character of the uniqueness suffix survives.

Could we truncate the environment portion before appending the suffix, rather than truncating the completed resource name? The Storage account naming expression has the same suffix-truncation problem and should preserve its uniqueness suffix too.

}
]
},
"outputs": {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Put the output mappings inside parameters and connect this wizard to the buttons.

outputs is currently at the document root, but the CreateUiDefinition schema requires parameters.outputs to bind the controls to the ARM template parameters.

Also, both Deploy to Azure buttons reference only azuredeploy.json; neither references this UI definition, so users won't see the custom wizard or its validation. Could we move these mappings into parameters and wire the UI definition into both deployment links?

Comment thread deploy/azure/README.md

- **Installing Scope itself.** This template provisions infrastructure only. The
Scope application (API, workers, Judge, Portal, Token Manager) is installed
separately via Helm charts against the AKS cluster this template creates. See the

@manekinekko Wassim Chegham (manekinekko) Oct 9, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Point to an available application-install procedure.

This directs users to install Scope via Helm charts and sends them to the documentation site for instructions, but I couldn't find a Helm chart or Helm installation instructions. Is this part of a separate PR?

Josh Duffney (duffney) and others added 6 commits October 9, 2026 10:03
…rite to race cache creation

The primaryKey output called listKeys() against a resourceId() built purely
from the redisName parameter, with no symbolic reference to the redis AVM
module. Bicep only infers an implicit dependsOn from symbolic-name
references, so this output had no dependency edge on the cache's actual
provisioning - ARM could evaluate listKeys() before the long-running Redis
create finished, which explains reports of the redis-primary-key secret
being missing from Key Vault after otherwise-successful deployments (the
cache itself provisions fine; only the secret write silently failed).

Fixed by declaring an existing resource reference for the cache (a
compile-time-known ID, valid as a listKeys() target) with an explicit
dependsOn: [redis] to force correct ordering.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
…e connectivity

The private DNS zone for the Cosmos DB MongoDB API private endpoint was
named 'privatelink.mongo.cosmos.azure.net' - wrong TLD, likely a copy/paste
mix-up with the Key Vault zone (privatelink.vaultcore.azure.net). The
canonical zone name, per the public DNS CNAME chain
(*.mongo.cosmos.azure.com -> *.privatelink.mongo.cosmos.azure.com), is
'privatelink.mongo.cosmos.azure.com' (.com).

With the wrong zone name, pods resolved the Cosmos Mongo hostname to its
public IP (no matching records in the misnamed zone), and Cosmos's
firewall rejected the connection since publicNetworkAccess is Disabled -
MongoServerSelectionError connecting to the public IP on port 10255.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
…ad identity

The shared workload identity only had Key Vault Secrets User (read-only),
which is correct for the Helm chart's Secrets Store CSI mounts but blocks
the Token Manager service (apps/token-manager), which writes BYO
credentials (GitHub/Anthropic API keys, etc.) into the vault at runtime via
SecretClient.setSecret() using this same shared identity. Read-only access
403s that write path, leaving the portal's built-in credential
registration UI (POST/DELETE /api/v1/keys) non-functional out of the box
on a freshly provisioned environment.

Key Vault Secrets Officer is a superset of Secrets User (adds set/delete
alongside get/list on secrets, still scoped to secrets only - no key/cert
or permission-management access), so this replaces rather than stacks
alongside the previous role assignment.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
… Data Owner

Storage Blob Data Contributor's data actions do not include
Microsoft.Storage/storageAccounts/blobServices/containers/blobs/tags/write,
so the app's practice of setting Blob Index Tags (requestId/runId/iteration)
via the x-ms-tags header on per-iteration workspace snapshot uploads
403s with AuthorizationPermissionMismatch under Contributor alone.
100% reproducible, confirmed via raw PUT with/without x-ms-tags using a
storage-scoped AAD token exchanged from the workload identity's federated
token.

Storage Blob Data Owner includes the tags/write data action alongside
everything Contributor already grants, so this keeps tagged snapshot
uploads working without the app needing to drop tags.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
…ed Redis

Classic Azure Cache for Redis (Microsoft.Cache/redis) is being retired for
new deployments, causing DeploymentFailed errors on the redis sub-deployment
("Azure Cache for Redis is retiring, create Azure Managed Redis instance
instead").

Rewrote redis.bicep to use the AVM module avm/res/cache/redis-enterprise:0.5.0
(Microsoft.Cache/redisEnterprise), targeting the "Balanced" Azure Managed
Redis SKU tier:

- Replaced the Basic/Standard/Premium skuName + C/P skuFamily + integer
  skuCapacity parameter trio with a single skuName enum of AMR SKU strings
  (Balanced_B0/B1/B3/B5/B10; capacity is now encoded in the SKU name itself).
  Default is Balanced_B1.
- Private endpoint service/groupId changed from redisCache to redisEnterprise;
  private DNS zone changed from privatelink.redis.cache.windows.net to
  privatelink.redisenterprise.cache.azure.net.
- Entra ID (Azure AD) data-plane auth is now granted via an access policy
  assignment on the database submodule (accessPolicyAssignments), replacing
  the classic module's redisConfiguration 'aad-enabled' flag. The shared
  workload identity's principal ID is threaded through from main.bicep (same
  pattern already used for storage.bicep).
- Access-key auth (accessKeysAuthentication: 'Enabled') is kept as a
  compatibility fallback; the primary access key is still surfaced as a
  @secure() output and written to the redis-primary-key Key Vault secret, now
  sourced directly from the AVM module's own output instead of a manual
  listKeys() call — the AMR database submodule's output is correctly ordered,
  so the existing-resource + explicit dependsOn workaround from the prior
  dependsOn fix is no longer needed.
- Updated main.bicep (params, redisName max length 63->60 per
  Microsoft.Cache/redisEnterprise naming rules, module wiring, outputs),
  main.parameters.json, createUiDefinition.json (single SKU dropdown, capacity
  slider removed), metadata.json, and README.md to match.

Verified with `az bicep build` on the module and full orchestrator (clean,
zero errors/warnings), `az bicep lint`, and a rebuild-and-diff drift check
against the committed azuredeploy.json (no drift).

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Live deployment hit:

  Unable to process template language expressions for resource
  '.../redis-apa-0'... 'The language expression property name doesnt
  exist, available properties are userObjectId.' (Code: InvalidTemplate)

Root cause: the AVM redis-enterprise module's nested access-policy-assignment
deployment reads parameters('accessPolicyAssignments')[i].name directly
(not via tryGet), even though name is documented as optional/nullable on
accessPolicyAssignmentType. ARM strips omitted optional properties from the
JSON object entirely (there is no null placeholder), so direct property
access on a missing key throws InvalidTemplate. Our accessPolicyAssignments
array item only set userObjectId, omitting name as the (incorrect) docs
imply is fine.

Fix: explicitly set name: 'workload-identity-access' on the access policy
assignment entry in redis.bicep's database block, working around the
module's property-access bug.

Verified with az bicep build (module + full orchestrator, clean), az bicep
lint (clean), and a rebuild-and-diff drift check against the committed
azuredeploy.json (no drift) - confirmed the compiled ARM template now
includes the explicit name property on the access policy assignment.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Live Helm deploy testing (#1481) found that every app service
(api, report-generator, both ACP workers) failed to connect to Redis with
connect ETIMEDOUT, even though the cache resource itself was healthy.

Root cause: redis.bicep linked the private endpoint's DNS zone group to
privatelink.redisenterprise.cache.azure.net, which has zero matching record
sets for this resource. Azure Managed Redis (the Balanced_* SKU tier used
here) and the older "Redis Enterprise" product branding share the same
underlying resource type (Microsoft.Cache/RedisEnterprise) and private
endpoint subresource (redisEnterprise), but publish their hostnames under
different private DNS zones:

  - Redis Enterprise (Enterprise/EnterpriseFlash SKUs):
    privatelink.redisenterprise.cache.azure.net
  - Azure Managed Redis (Balanced/ComputeOptimized/FlashOptimized/
    MemoryOptimized SKUs, i.e. what this template deploys):
    privatelink.redis.azure.net

With the wrong zone attached, in-cluster DNS fell through to the public
record, which resolves to a public IP - unreachable since
publicNetworkAccess is Disabled on the cache.

Confirmed authoritative zone name against the Microsoft Learn private
endpoint DNS reference table
(https://learn.microsoft.com/azure/private-link/private-endpoint-dns), which
lists both product brandings separately. Also validated live on a test
environment per the linked issue - after relinking the private endpoint's DNS
zone group to privatelink.redis.azure.net, Redis connectivity came up
cleanly with no other changes.

Fix: change redis.bicep's privateDnsZoneName var from
privatelink.redisenterprise.cache.azure.net to privatelink.redis.azure.net.
The private endpoint's service/groupId (redisEnterprise) is unchanged - only
the DNS zone name was wrong.

Verified with az bicep build (module + full orchestrator, clean), az bicep
lint (clean), and a rebuild-and-diff drift check against the committed
azuredeploy.json (no drift) - confirmed the compiled ARM template now
references privatelink.redis.azure.net.

Fixes #1483.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
…t isolation

ACP worker pods execute benchmark-controlled/untrusted agent code in the same
container as the workload identity used for Azure resource access. The
previously single shared identity had Key Vault Secrets Officer (write access),
meaning a compromised worker process could read or tamper with Key Vault
secrets it has no legitimate need to touch.

Fix: split into two user-assigned managed identities, each with its own
federated credential and Kubernetes service account subject:

- Backend identity (scope-workload-identity, unchanged name/behavior): Key
  Vault Secrets Officer, Redis Entra ID access, Storage Blob/Queue data-plane
  access. Used by trusted components (api, Token Manager, scheduler, etc.).
- Worker identity (scope-worker-workload-identity, new): Storage Blob/Queue
  data-plane access ONLY - no Key Vault role, never wired into keyvault.bicep.

storage.bicep now accepts a second optional principal ID
(workerWorkloadIdentityPrincipalId) and grants the same Storage Blob Data
Owner + Storage Queue Data Contributor roles to both identities that are
non-empty.

New outputs: workerWorkloadIdentityClientId,
workerWorkloadIdentityServiceAccountNamespace,
workerWorkloadIdentityServiceAccountName - for the Helm chart to annotate the
worker pods Kubernetes service account.

New params (with defaults, exposed in createUiDefinition.json and
main.parameters.json): workerWorkloadIdentityServiceAccountNamespace,
workerWorkloadIdentityServiceAccountName.

Verified: az bicep build (clean), az bicep lint (clean), rebuild-and-diff
against committed azuredeploy.json (no drift).

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Josh Duffney (duffney) added a commit that referenced this pull request Oct 12, 2026
Both ACP workers run benchmark-controlled agent code and should not share
the backend workload identity (Key Vault Secrets Officer) or its mounted
secrets file tree with untrusted execution.

- Drop the azure.workload.identity/use label and kv-secrets CSI
  volume/mount from both worker Deployments, so no federated Azure AD
  token or Key Vault file tree reaches the untrusted agent subprocess.
- Keep the common-secret envFrom (Mongo/Redis creds), since the worker
  still needs those to claim/lock Storage Queue work — the
  queue-processor and agent executor are one process today, so a chart
  change alone can't separate them further.
- Document the residual limitation and a worker-scoped managed identity
  (Storage Blob/Queue only, planned on the Bicep side) in WORKERS.md.

Addresses part of the review finding in
#1481 (comment). Full
trusted/untrusted separation needs worker code changes and a dedicated
worker identity from deploy/azure/, which is still being worked out on
the Bicep PR (#1477) after an earlier attempt there was reverted.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff
…app breakage)

The previous split (a28ed98, reverted in 4568107) gave the worker identity
NO Key Vault role at all. That broke the app: worker pods rely on the shared
Secrets Store CSI SecretProviderClass to mount their own BYO credentials
(GitHub/Anthropic API keys) from Key Vault, which requires at least read
access - removing Key Vault access entirely from the worker identity broke
that CSI mount for every worker pod.

Corrected approach: keyvault.bicep now grants two tiers of access:

- Backend identity (unchanged): Key Vault Secrets Officer (read/write/delete),
  for Token Manager's runtime credential-registration writes.
- Worker identity (new param `readOnlyPrincipalIds`): Key Vault Secrets User
  (read-only) - enough to mount its own CSI-synced secrets, but it can never
  write, delete, or tamper with vault contents. This still satisfies the
  original security review's intent (workers must not have write access to
  Key Vault) while keeping the CSI secret mount functional.

Everything else from the original split is unchanged: a separate
workerWorkloadIdentity module/federated credential, worker Storage Blob/Queue
access via storage.bicep's generalized role-assignment logic, and the same
output names (workerWorkloadIdentityClientId,
workerWorkloadIdentityServiceAccountNamespace,
workerWorkloadIdentityServiceAccountName).

Verified: az bicep build (clean), az bicep lint (clean), rebuild-and-diff
against committed azuredeploy.json (no drift).

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Josh Duffney (duffney) added a commit that referenced this pull request Oct 12, 2026
Bicep PR #1477 landed a worker-specific managed identity (Storage
Blob Data Owner + Queue Data Contributor only — no Key Vault access),
exposed as workerWorkloadIdentityClientId/
workerWorkloadIdentityServiceAccountName.

- values.yaml: add azure.workerWorkloadIdentityClientId and
  azure.workerServiceAccountName (default scope-worker-workload-identity).
- templates/serviceaccount.yaml: add a second ServiceAccount for the
  worker identity, rendered only when a worker is enabled.
- _helpers.tpl: add scope.workerServiceAccountName helper.
- Both ACP worker Deployments now use the worker ServiceAccount (with
  azure.workload.identity/use re-added, needed for their direct
  Storage Blob/Queue SDK access via DefaultAzureCredential) instead of
  the shared backend ServiceAccount. The kv-secrets CSI mount stays
  removed from their own pods — the worker's GITHUB_TOKEN/
  ANTHROPIC_API_KEY fallback values already reach them via
  envFrom.secretRef on the common K8s Secret, which the api pod's own
  CSI mount keeps synced; the workers don't need a mount of their own
  for that.
- scripts/generate-values-from-bicep.sh: read the two new outputs,
  tolerating deployments that don't have them (older Bicep deployments,
  or ones that never provisioned the worker identity).
- WORKERS.md: document the real worker-identity wiring and keep the
  known-limitation note (full trusted/untrusted process separation
  needs worker code changes).

Continues the fix for
#1481 (comment).

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 094bb93a-ec65-4f80-8896-0dd041aedeff
Per the Helm chart (PR #1481, commit 6b4c847), ACP worker pods no longer
mount the Secrets Store CSI SecretProviderClass directly - they get
GITHUB_TOKEN/ANTHROPIC_API_KEY via envFrom/secretRef against a common
Kubernetes Secret that the api pod's CSI mount populates. Worker pods never
call Key Vault, so the read-only `Key Vault Secrets User` grant added in
b897d9a is unnecessary.

Reverts keyvault.bicep to a single role assignment (Key Vault Secrets Officer
for the backend identity only) and removes the worker identity's
`readOnlyPrincipalIds` wiring in main.bicep. The worker identity keeps its
Storage Blob Data Owner + Storage Queue Data Contributor roles and output
names (workerWorkloadIdentityClientId, etc.) are unchanged.

Verified: az bicep build (clean), az bicep lint (clean), rebuild-and-diff
against committed azuredeploy.json (no drift).

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Several coworkers hit AKS vCPU-quota errors and Azure Managed Redis
InsufficientCapacity failures deploying this template on new/free
subscriptions. Rework the defaults to prioritize "it deploys" over
performance/HA for this quickstart template, while keeping every value
overridable via parameters:

- AKS: default VM size for both node pools changed from Standard_D4s_v5 to
  Standard_B2ms (2 vCPU/8GB burstable), and default node count from 3+2 to
  1+1. Cuts the default footprint from 20 vCPUs (often exceeding a
  subscription's 10-vCPU total regional quota, or its Dsv5-family quota
  specifically) down to 4 vCPUs on a generally less-contended VM family.
- Redis: default SKU changed from Balanced_B1 to Balanced_B0 (smallest
  available), and added a new redisHighAvailability param defaulting to
  Disabled - this roughly halves the underlying node allocation Azure
  Managed Redis needs, reducing the odds of region-level
  InsufficientCapacity errors (a known constraint on this SKU family,
  independent of subscription quota).
- ACR: default SKU changed from Standard to Basic - sufficient for a single
  worker image and avoids unnecessary cost (does not affect quota).

Updated main.parameters.json (used by the Deploy-to-Azure button) and
createUiDefinition.json (portal UI defaults/tooltips/new HA dropdown) to
match. Added a new "Sizing and quota" README section explaining the
rationale and how to size up for production.

Verified: az bicep build (clean), az bicep lint (clean), rebuild-and-diff
against azuredeploy.json (no drift), cross-checked main.parameters.json
keys against the compiled template parameters (exact match).

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Avoids naming collisions when multiple people deploy the template into
the same tenant/subscription concurrently, similar to how the Azure
portal suggests a randomized default resource group name. Uses
createUiDefinition's client-side guid()/substring() functions; actual
resource names still get their own uniqueString()-based suffix in
main.bicep regardless of what's typed here.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
The button link only included the template uri (azuredeploy.json),
so the portal's Custom Deployment screen used the raw ARM template
parameter form instead of the curated createUiDefinition.json UI —
explaining why the prepopulated Environment name field (and the
Basics/Advanced grouping) wasn't showing up when testing the link.

Adds the required /createUIDefinitionUri/<encoded-uri> segment
pointing at createUiDefinition.json alongside the existing
/uri/<encoded-uri> template segment, in both README.md and
deploy/azure/README.md.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
outputs was a top-level sibling of parameters instead of nested inside
it (parameters.outputs), which is invalid per the classic
CreateUiDefinition (Microsoft.Azure.CreateUIDef / 0.1.2-preview)
schema. The Azure portal's Custom deployment page failed to parse it
as the classic schema, silently fell back to trying the newer Form
view (uiFormDefinition) schema instead, and surfaced that fallback
parser's error: 'The provided uiFormDefinition does not have a
parameters property under $.view.outputs.'

No content changes — basics, steps, and all 19 output mappings are
unchanged, just correctly nested. Re-validated as valid JSON and
cross-checked output keys against main.bicep's parameter names.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Reverts the behavior from d6a9dc7 (defaultValue back to the static
'scope', tooltip back to the original wording) while keeping the
parameters.outputs nesting fix from fe87207, which is unrelated and
still needed. This is a manual inverse edit rather than 'git revert'
because fe87207 reformatted the whole file afterward and would
conflict.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Restores README.md, deploy/azure/README.md, and
deploy/azure/createUiDefinition.json to their exact content as of
1cd20c1, undoing d6a9dc7 (env-name random suffix),
e8d216b (wiring createUIDefinitionUri into the button link),
fe87207 (outputs-nesting fix), and f846714 (partial manual revert).

The createUIDefinitionUri change kept surfacing portal errors
('uiFormDefinition does not have a parameters property under
$.view.outputs') that didn't clear even after the underlying
createUiDefinition.json structure was fixed, likely compounded by
raw.githubusercontent.com CDN caching. Reverting to the plain
Microsoft.Template/uri/<azuredeploy.json> button link (no custom UI)
restores the last confirmed-working deploy experience. The portal
falls back to its default ARM-parameter form, which is less polished
but functional; revisiting the custom createUiDefinition.json UI is
left for a future iteration.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
…lity

The AVM managed-cluster module defaults every agent pool's
availabilityZones to [1,2,3] when not explicitly set. Not all regions
support AKS availability zones (e.g. centralus), so deployments there
failed preflight validation with:

  AvailabilityZoneNotSupported: The zone(s) '1' for resource 'system'
  is not supported. The supported zones for location 'centralus' are ''.

Both the system and user node pools now explicitly request an empty
availabilityZones array, trading zone-redundant HA (already not
guaranteed with the 1-node-per-pool quota-friendly defaults) for
working deployments in any region. Set availabilityZones explicitly
per node pool for a production environment in a region that supports
them.

Verified: az bicep build clean, az bicep lint clean, rebuild-and-diff
shows no drift in azuredeploy.json, and confirmed both agent pool
profiles compile to "availabilityZones": [] in the ARM template.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Standard_B2ms (classic B-series) is not an allowed VM size in every
subscription - a live deployment failed preflight validation with:

  The VM size of Standard_B2ms is not allowed in your subscription in
  location 'centralus'.

Standard_D2s_v5 (general-purpose, non-burstable) has the same 2 vCPU /
8 GiB memory profile as the previous default, is on this subscription's
allowed-SKU list, and is a far more broadly available family across
subscriptions/regions than burstable B-series sizes (some subscriptions
only allow the newer "_v2" B-series, not classic B2ms, so guessing
another B-series variant risks the same failure mode elsewhere).

Updated: aks.bicep/main.bicep param defaults + comments,
main.parameters.json, createUiDefinition.json tooltips/recommendedSizes,
and deploy/azure/README.md sizing guidance.

Verified: az bicep build + lint clean, zero drift in azuredeploy.json,
all JSON files parse, no remaining Standard_B2ms default references.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
Live-tested by the Helm-deploy session: with 1 System + 1 User node (both
Standard_D2s_v5, 2 vCPU), the core app alone (api/portal/judge/
token-manager/scheduler/report-generator) already consumes ~85% of the
User node's allocatable CPU. An optional Phase 3 ACP worker pod
(coder-acp-copilot: 500m main + 250m MCP-gateway sidecar = 750m) then
fails to schedule with 'Insufficient cpu' on both nodes.

Bumping aksUserNodeCount default to 2 (same Standard_D2s_v5 size, no
pool recreation needed) resolves it - confirmed live by scaling the
user pool to 2 nodes, after which the worker pod scheduled and came up
2/2 Running. Total default footprint is now 3 nodes / 6 vCPUs, still
comfortably inside the common 10-vCPU regional quota.

Users running only the core app (no ACP workers) can still set
aksUserNodeCount=1.

Updated: aks.bicep/main.bicep param defaults + comments,
main.parameters.json, createUiDefinition.json slider default/tooltip,
and deploy/azure/README.md sizing guidance.

Verified: az bicep build + lint clean, zero drift in azuredeploy.json
(only the intended defaultValue/description diffs + template hashes),
all JSON files parse.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
A coworker's live deployment failed with:

  ServiceUnavailable: Sorry, we are currently experiencing high demand
  in Central US region for the zonal redundant (Availability Zones)
  accounts, and cannot fulfill your request at this time.

Root cause: cosmosdb.bicep didn't pass a 'locations' param to the AVM
document-db/database-account module, so its internal fallback
(defaultFailoverLocation) applied - which hardcodes isZoneRedundant:
true. This silently requested a zone-redundant Cosmos DB account,
which some regions (e.g. centralus) intermittently lack capacity for,
independent of subscription quota.

Fix: explicitly pass a single-region 'locations' entry with
isZoneRedundant: false, opting out of zone redundancy by default -
same rationale/pattern as the earlier AKS availabilityZones: [] fix.

Also documented in README.md that the companion Redis
'ComputeProvisioningFailed' error reported alongside this is an
unrelated, separate transient Azure capacity issue: confirmed via the
compiled AVM template that 'zones' only ever applies to
Enterprise/EnterpriseFlash Redis SKUs, not the Balanced/AMR tier this
template uses, so it is not caused by anything in this template.

Verified: az bicep build + lint clean, zero drift in azuredeploy.json
beyond the intended locations/isZoneRedundant diff and template
hashes, confirmed isZoneRedundant: false compiles into the output.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72
…eardown

Add a "Good to know before you deploy" section right after the Deploy to
Azure button that:
- Frames occasional AKS zone / Cosmos DB zonal / Azure Managed Redis
  capacity errors as expected, transient, and easily resolved by retrying
  or switching regions (links to the existing detailed Sizing and quota
  section for specifics).
- Encourages deleting the resource group once done exploring, framed as a
  lean-by-design template that's easy to try and easy to tidy up, to avoid
  surprise cost from a left-running dev environment.

Co-authored-by: Copilot App <[email protected]>
Copilot-Session: 2250d38f-729e-4dd7-b6e1-41b525e44a72

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cicd Build, test, release, and deployment pipelines. type: documentation Documentation additions, corrections, and improvements.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants