Skip to content

Per-bucket object storage credentials with key rotation (Ceph RGW accounts) - #14170

Open
MitchDrage wants to merge 2 commits into
apache:mainfrom
MitchDrage:per-bucket-credentials
Open

Per-bucket object storage credentials with key rotation (Ceph RGW accounts)#14170
MitchDrage wants to merge 2 commits into
apache:mainfrom
MitchDrage:per-bucket-credentials

Conversation

@MitchDrage

@MitchDrage MitchDrage commented Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Description

Object storage credentials are provisioned per CloudStack account today: every bucket an account owns shares one access/secret key pair, and there is no way to rotate it.
This PR introduces per-bucket keys, a rotation mechanism, and the migration from per-account to per-bucket keys.
Fixes #14167

Discussed on dev@: "[DISCUSS] Per-bucket credentials and key rotation for object storage".

This PR's changes:

  • Each bucket gets its own credential on the backend.
  • Each credential holds two independent key slots, so a key can be rotated while consumers still use the other one, and revoked when they have moved.
  • Existing buckets keep working exactly as they do now. Nothing is migrated automatically.
  • An administrator moves an account onto per-bucket credentials explicitly, per object store, then moves each bucket, then rotates the account key to finish.
  • On Ceph this is built on RGW accounts, so it needs Ceph Squid or later. Older gateways, and providers that do not offer this, keep today's behaviour and say so rather than half-working.
  • New APIs - see below.
  • New UI: a Keys tab per bucket, a Credential Scope column and filter, a migrate action on buckets, an Object Storage tab on accounts, and a readiness line on the object store itself.
    • Details of storage backend's are only shared with the root administrator, e.g. backend type, versions, overall-readiness for migration. General users are only told that the backend doesn't support per-bucket credentials.
  • MinIO, Cloudian and the Simulator are untouched and report that they do not offer per-bucket credentials.

How it works

  • Two new tables, bucket_credential (one dedicated backend identity per bucket) and bucket_credential_key (one row per key slot, secret encrypted with @Encrypt). A bucket without a bucket_credential row behaves exactly as today (account-scoped key) - the schema change is additive and no data is migrated.
  • The existing bucket.access_key/bucket.secret_key columns become a mirror of the bucket's newest active key, so every existing consumer of them (BucketTO, the policy/versioning driver calls, the UI object browser) keeps working unchanged.
  • New provider-agnostic driver operations on ObjectStoreDriver (createBucketCredential, createBucketCredentialKey, removeBucketCredentialKey, deleteBucketCredential, plus the account-level supportsBucketCredentials / accountSupportsBucketCredentials / migrateAccountForBucketCredentials). BaseObjectStoreDriverImpl provides "unsupported" defaults, so MinIO and Cloudian are untouched; the Simulator gets a stateless fake so the service layer is exercisable in CI.
  • New APIs: rotateBucketKey, revokeBucketKey, migrateBucketCredential (user-level, bucket-scoped) and, for root/domain admins, migrateObjectStoreAccount and rotateObjectStoreAccountKey. The latter completes a migration: once no bucket of the account on a store still uses the account key, it issues CloudStack a fresh account key and revokes the old one, so a shared key copied before the migration no longer opens every bucket. It is refused while legacy buckets remain. listObjectStoragePools accepts accountid and then reports, per store, whether the store supports per-bucket credentials at all, whether the account is migrated and how many buckets still use the account key. listBuckets responses gain credentialscope (bucket/account) and a keys list, and accept credentialscope as a filter so the buckets still using the account credential can be listed directly; secret keys are marked sensitive for API-log redaction.
  • Global setting object.storage.per.bucket.credentials (default true) applies only when an account is first given an identity on an object store. It never changes an account that already has one:
The account on that store true (default) false
has no identity yet is created so that each of its buckets gets its own credential is created sharing one credential across its buckets, as in earlier releases
shares one credential unchanged - moving to per-bucket credentials is always an explicit admin action unchanged
has been migrated each new bucket gets its own credential the same: buckets keep getting their own credentials. Sharing one here would mean writing the account's root key onto a bucket, where every user of the account can read it, which undoes the migration
  • CloudStack requires a credential of its own for each account on each store: the RGW account id and the account's root key. Account details are the natural place to keep it, but they are handed back to API callers as accountdetails in listAccounts responses, and this credential is CloudStack's rather than the account's.

So these rows are named with a reserved prefix, objectstore-, held as a constant (ObjectStore.ACCOUNT_DETAIL_PREFIX) so that the driver writing them and the API layer agree on it, and ApiDBUtils.getAccountDetails drops/hides rows carrying that prefix before the response is built. The secret is additionally encrypted with DBEncryptionUtil. The namespace is the only thing removed; every other account detail, including those existing providers already store, is returned exactly as before.

  • UI: a Keys tab per bucket, a Credential Scope column and filter, a migrate action on buckets, an Object Storage tab on accounts, and a readiness line on the object store itself. Each screen is listed under "What changes in the UI" below.

Points for reviewers

These are the design decisions I've made along the way:

  • I've made all references to versions (API's, etc) to the 24.0.0 version based on the vote to drop the '4'. Please also let me know if this need to go in a version other than 24.
  • Ceph Squid or later only. Per-bucket identities on plain RGW users fragment object ownership, which breaks public bucket policies and leaves the old key with access to old objects. RGW accounts fix the cause but arrived in Squid. Older gateways keep today's behaviour. A lot of the decisions and the direction of the code is based on this point. It seemed unavoidable to move to use Ceph's Account system - but at least it aligns with what the Cloudian integration has done.
  • Account migration cannot be undone. Adopting a user into an RGW account is permanent at the backend, so the action is explicit, admin-only and warned about, and CloudStack never does it implicitly.
  • The global setting applies only at first use. object.storage.per.bucket.credentials decides how an account
    is set up the first time it uses an object store. A migrated account always gets per-bucket credentials for new
    buckets, whatever the setting says. We can't revert an Account back to a shared credential as it would write that account's root key onto its bucket rows, where every user of the account can read it.
  • Shared UI code is touched. List sections gain an optionalColumns option, and a details list can now vary per record. Both are additive and inert elsewhere, but they are shared files.
  • Account details are filtered in a shared response path. ApiDBUtils.getAccountDetails now strips the reserved namespace this feature writes. Nothing else is removed, but every account response goes through that method.
  • Secrets are still returned by the API. A bucket's keys appear in listBuckets for callers who pass its access check, as the account-scoped key does today. Marking them sensitive keeps them out of logs; hiding them from the response would break the object browser and existing automation.
  • An interrupted rotation can leave an untracked key, and rotation reports it rather than deleting it, because CloudStack did not create it and it may be an operator's own.
  • listBuckets gained a per-bucket gateway call for buckets still on the account credential of a migrated account, to decide whether the migrate action applies. Accounts with no migration record answer from the database alone.
  • The driver contract gained an optional explanation (bucketCredentialsUnsupportedReason). It is a default method, so no provider has to implement it, but adding a method to ObjectStoreEntity means an incremental build has to rebuild the implementing module.
UI changes people will see

Buckets list

  • A Credential Scope column, reading "Per-Bucket" or "Account". It stays hidden until the viewer can see at least one bucket with its own credential, then appears on its own; after that the column picker decides.
  • A Credential Scope filter in the search panel, with the same two values.

A bucket's Details tab

  • Credential Scope is shown.
  • Once a bucket has its own credential, the access key and secret are no longer listed here. They move to the Keys tab, so there is one place to find them. A bucket still on the account credential is unchanged.

A bucket's Keys tab (new, only on buckets with their own credential; viewable/usable by the bucket owner, admins and domain admins)

  • Two rows, one per key slot, each showing its state, its access key with a copy button, its secret masked behind a reveal, and when it was created.
  • Per row: rotate an active slot, create a key in an empty slot, or revoke. Revoke is disabled on the last active key, with a tooltip saying why.
  • Each action confirms first, in the standard dialog.

A bucket's actions

  • Migrate to Per-Bucket Credential (new, viewable/usable by the bucket owner, admins and domain admins), on buckets still using the account credential. It only appears once the owning account has been set up on that store, so it cannot be started in a state that would be refused.

An account's Object Storage tab (new, viewable/usable by admins and domain admins)

  • One card per object store, supported stores first.
  • While a store has work outstanding: a four-stage progress display (account key, migrate buckets, rotate account key, complete), the count of buckets still on the account credential, a Migrate Account to Per-Bucket Credentials button, a Show Buckets link filtered to that store's remaining buckets, and, once every bucket has been moved, a warning that the account key still opens everything with a Rotate Account Key button.
  • When a store is finished: a single line saying per-bucket credentials are in use, with no stages and no buttons.
  • When a store cannot support the feature: a plain notice saying so and to contact the platform administrator. No backend or version or other information is named in order to hide platform internals from users.
  • The explanation of the migration only appears while some store still has a stage left. After is has been migrated, the steps to migrate are r

Infrastructure, Object Storage, a store's Details tab (root admins only)

  • Per-Bucket Credentials: Ready or Not ready.
  • When not ready, To Resolve Before Migration naming the cause: a missing info or accounts capability on the store's admin credential, a gateway that predates RGW accounts, a provider that does not offer the feature, or an unreachable admin API.

Events

  • Key rotation, key revocation, bucket credential migration, account migration and account key rotation each raise their own event, visible on the bucket's and account's Events tabs.
Changes to shared UI code, and why they are needed

Four files outside this feature are touched. Each is additive and gated so they don't apply to any other section, but they are shared so they are listed here for completeness.

File Change Effect elsewhere
views/AutogenView.vue a section may declare optionalColumns: columns that stay unticked until a condition first holds, then are ticked once and left to the viewer thereafter none unless a section declares it; the method returns on its first line when the route has no such declaration, and only the buckets section has one
config/router.js carries that declaration into the route metadata, beside the existing columns none; the property is only copied when present
components/view/DetailsTab.vue a details() function is now called with the record: details(this.resource) none; twelve of the thirteen existing definitions take no arguments and ignore it
components/view/ListView.vue renders a credentialscope cell through the translation table, so it reads "Per-Bucket" rather than bucket none; the branch is keyed on a column name no other section uses
components/view/SearchView.vue credentialscope added to the list of filters drawn as a dropdown, with its two options none; both additions are keyed on that filter name
Ceph RGW implementation - requires Ceph Squid (v19) or later

On plain RGW users (the existing CloudStack approach to RGW credentals), giving an existing bucket its own identity cannot move the ownership of the objects already in it. The admin REST API can relink a bucket to another owner, but only the radosgw-admin bucket chown command rewrites the objects inside. CloudStack accesses the gateway over REST and S3 with no shell access to it, so it has no way to migrate ownership. That leaves migrated buckets with split ownership: public bucket policies would 403 on the old objects and the old key would retain access to them. RGW accounts remove the cause: the account owns every bucket and object regardless of which identity wrote it. So the Ceph driver maps:

  • one RGW account per CloudStack account per object store (an account can be migrated on one store and still be legacy on another; the state is kept per store in account_details), whose root user is the existing account-UUID RGW user;
  • one IAM user per bucket, with an inline policy scoped to that bucket and its access keys mapping onto the two key slots;
  • a bucket-policy statement naming the IAM user, so it can also reach objects the account root wrote before the account migration (identity policies alone cannot; verified on 20.2.4). CloudStack's public/private bucket policy is regenerated together with that statement so the two features compose.

An account counts as unmigrated only when the gateway answers and says so. An unreachable admin API, a 403 while an admin credential's capabilities are being changed, or a probe that flaps during a rolling gateway upgrade all leave the migration record standing, since migration is permanent at the backend. Treating those as unmigrated would drop the account back to its shared credential and write the account root key onto its next bucket. Where a per-bucket credential genuinely cannot be issued, bucket creation fails with the gateway's error.

Verified against three RGW releases (single-container clusters, same script):

Reef 18.2.7 Squid 19.2.3 Tentacle 20.2.4
/admin/info 200 200 200
accounts capability cannot be granted granted granted
POST /admin/account 405 works; duplicate → 409 AccountAlreadyExists same
GET /admin/account 405 403 whatever capabilities are granted, as does DELETE 404/200 as expected
adoption, IAM users/keys/policies, bucket-policy grant n/a identical to Tentacle verified

Reef cannot support this at all: its account endpoint answers 405, and radosgw-admin caps add refuses the accounts capability, so no credential on it could hold what the endpoint requires. Squid is the floor.

The plugin registers its own signer (RgwIamSigner) for IAM calls. The AWS Java SDK signs the request before the HTTP layer attaches the form Content-Type, so that header never reaches the signature, and Tentacle answers 403 to a request whose content type was unsigned.

This code works out whether a store supports this by asking the gateway: /admin/info must answer, the account endpoint must not return 405, and the store's admin credential must hold both info and accounts (with accounts alone the gateway answers 403 on /admin/info). The three causes are indistinguishable from outside, so a root admin refused a migration is told which one applies, for example "the object store's admin credential is missing the 'info' capability". The probe re-checks automatically, so upgrading a gateway is picked up with no CloudStack configuration change.

Incidental fix

listBuckets has accepted an objectstorageid parameter since 4.19.0 that seems to have never worked. It declared entityType = StoragePoolResponse.class (primary storage), so the UUID could never resolve and the call failed with "entity does not exist"; and searchForBucketsInternal never read the parameter at all, so once the UUID did resolve every bucket was still returned. Both are fixed here: the entity type is corrected to ObjectStoreResponse.class, and the search now applies the filter. The account's Object Storage tab relies on it for its "Show Buckets" link (root admin only, matching the parameter's existing authorization, which is unchanged).
I haven't found an issue raised against this, but I needed this fixed for my tests to work so I have fixed it here.

Documentation

Operator documentation is being raised separately against the documentation repository, and will be linked here once it is up. It covers the capabilities the store's admin credential needs (info and accounts), what account migration does and that it cannot be undone, and the stages an administrator works through.

Open question: how long should the account's Object Storage tab existin the Account page?

The tab, and the migration it drives, exist only because deployments have accounts that predate per-bucket credentials, and because some object storage cannot support them. Both shrink over time. Once the providers CloudStack supports offer per-bucket credentials, and versions that cannot are out of support, there is a decision to make: deprecate support for the backends and versions that cannot do this (Ceph before Squid is already end of life), at which point new accounts are always per-bucket, the migration becomes a one-off upgrade concern, and this tab can go. One suggestion I could make is to add the Object Storage provider and version into the anonomised usage telemetry service that has been discussed recently. That way, when we have a view on how much Ceph Reef is out there, we can make an informed decision on when to stop supporting it.

Known limitations / follow-ups

  • bucket.secret_key still holds the mirrored active secret in clear, as it does today; only the new bucket_credential_key table is encrypted. Encrypting the existing column would mean migrating the rows already in it, and a rolling-upgrade window in which management servers disagree about the format, so changing it is left out of this PR. It would also not change what the API hands out, since that secret is already returned to callers who can see the bucket and changing that may be a breaking change.
  • Existing provider account details, such as the account-scoped keys the Ceph and MinIO drivers have always written, are left exactly as they are; only the details this feature adds use the reserved namespace.
  • A rotation interrupted between issuing a key and recording it can leave a key on the account's root user that CloudStack does not track, and which still opens every bucket of the account. Rotation removes only the key it issued, and logs a warning naming any others rather than deleting credentials it did not create, since one of them may be an operator's own. Reviewing them is a manual step on the gateway.
  • Bulk migration of buckets and MinIO/Cloudian implementations are follow-ups. I don't have familiarity with those software packages nor an environment to test them on.
  • Objects written before an account is migrated stay owned by its pre-migration user, and the admin API cannot change that. CloudStack covers them by granting the bucket's credential in the bucket policy as well as its identity policy. A bucket policy written straight to the gateway drops that grant, leaving the credential blind to everything from before the migration; changing the bucket's public/private setting in CloudStack restores it.

Types of changes

  • Breaking change
  • New feature
  • Bug fix
  • Enhancement
  • Cleanup

Feature/Enhancement Scale or Bug Severity

  • Major

Trying it out

Documentation is being raised separately, so here is enough to exercise the feature.

What you need. A Ceph cluster running Squid (v19) or later with RGW. A single container is enough. The RGW admin user whose keys you register the object store with needs two capabilities, and it will look unsupported without both:

radosgw-admin caps add --uid=<admin uid> --caps="info=*;accounts=*"

Register it in CloudStack as a Ceph object store in the normal way. Infrastructure -> Object Storage -> the store's Details tab reports whether it can provide per-bucket credentials, and what to resolve if it cannot. That readiness information is root admin only.

A new account (the short path). An account that has never used this store is set up for per-bucket credentials on its first bucket, with no migration step.

  1. Create a bucket in that account. Storage -> Buckets shows Credential Scope "Per-Bucket".
  2. Open the bucket -> Keys tab. Slot 1 holds a key, slot 2 is empty.
  3. Use the slot 1 key with any S3 client. It reaches this bucket and is denied on another bucket in the same account.
  4. Rotate the key. Slot 2 fills with a different key. Both keys now work.
  5. Revoke slot 1. That key is rejected by the gateway; slot 2 keeps working.
  6. Try to revoke slot 2. It is refused, because a credential keeps at least one active key.
  7. Delete the bucket. radosgw-admin user list shows the bucket's IAM user is gone.

An existing account (the migration path). This is the path that matters for upgrades, and it is what the account's Object Storage tab walks an administrator through. Use an account that already has buckets on the store.

  1. Account -> Object Storage tab. It shows the store, the account's state on it, and the next step.
  2. Migrate the account. This adopts the account's existing RGW user as the root of a new RGW account at the gateway. It cannot be undone (the UI warns about this). The existing keys keep working afterwards: check with an S3 client before and after. Existing buckets still read "Account" for Credential Scope.
  3. Give each bucket its own credential. "Show Buckets" from the tab lists the account's buckets on this store that still share the account key. Each has a Migrate to Per-Bucket Credential action. A migrated bucket's own key can read objects written before the migration.
  4. Rotate the account key. Offered once no bucket is left on the account key, and refused before that with the list of buckets still using it. Afterwards the original account key is rejected by the gateway, and the per-bucket keys are unaffected.

An unsupported store. Register a Reef (18.x) cluster, or drop the accounts capability from the admin user of a Squid one. The account's Object Storage tab says the store cannot be migrated. As root admin, the store's Details tab names the cause; a domain admin is told to contact their platform administrator.

Checking at the gateway.

radosgw-admin account list                          # one account per migrated CS account
radosgw-admin user list                             # an IAM user per migrated bucket
radosgw-admin bucket stats --bucket=<name>          # owner is the account id

Turning it off. object.storage.per.bucket.credentials=false makes new accounts use the shared credential as before. Accounts already migrated stay migrated.

How Has This Been Tested?

  • Unit tests: CephObjectStoreDriverImplTest (37) and BucketApiServiceImplTest (26), plus a new ObjectStoreAccountDetailTest (3) and four cases added to QueryManagerImplTest for the bucket filters.
  • RGW behaviour verified with scripted checks against single-container Ceph 18.2.7 (Reef), 19.2.3 (Squid) and 20.2.4 (Tentacle) clusters: account probe, adoption keeps legacy keys working, IAM user/policy/key lifecycle, policy scoping, public-policy coverage of IAM-written objects, bucket-policy grant for pre-adoption objects, revocation, no two-key limit.
  • End-to-end through a management server (simulator profile) with the Squid and Tentacle containers registered as Ceph object stores:
    • Tentacle, new account: create bucket -> dedicated credential, account-owned bucket, IAM user with one key, key isolated to its bucket; rotate into slot 2; revoke slot 1 (RGW rejects the old key); last-key revoke refused; rotate back into slot 1; delete bucket removes the IAM user. 19/19.
    • Squid, legacy account (seeded with object.storage.per.bucket.credentials=false): migrateObjectStoreAccount adopts the RGW user as account root with its keys still valid; migrateBucketCredential gives the existing bucket a dedicated key that reads the pre-migration object; then the same new-bucket sequence as above. 25/25.
    • Ceph 20.2.2, legacy account with a pre-migration object: account migration, bucket migration (bucket key reads the old object), rotateObjectStoreAccountKey refused while a legacy bucket remains, then succeeds once none remain - old account key rejected by RGW, root user holds one new key, per-bucket keys unaffected, new buckets created with the rotated key are per-bucket, a second rotation succeeds. Per-store scoping verified: the same account reports legacy on the other two stores.
  • Marvin: a test_02_bucket_key_rotation case added to test_bucket.py, with the helpers it needs added to marvin/lib/base.py. It creates a Simulator object store, then checks that a new bucket has one active key slot mirrored onto the bucket, that rotation fills the free slot with a different key and moves the mirror, that revoking leaves the other slot active, and that revoking the last active key is refused. Run against a management server on the simulator profile: both cases in the file pass.
  • That file's existing test_01_create_bucket was failing before this PR, since createBucket has a required quota parameter (from Add Resource Limits to Backups and Object Storage #10017 ?) that wasn't filled. The one-line fix is included here, as our own case was written from the same pattern.

@boring-cyborg

boring-cyborg Bot commented Sep 15, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@MitchDrage MitchDrage mentioned this pull request Sep 17, 2026
Open
@MitchDrage

MitchDrage commented Sep 17, 2026

Copy link
Copy Markdown
Author

PR for documentation has been added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

component:api component:build component:integration-test component:management-server component:marvin component:storage component:UI Python Warning... Python code Ahead!

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

Per-bucket object storage credentials with key rotation

1 participant

@MitchDrage