Skip to content

feat(monitoring): drop the Grafana rebuild for the upstream image + catalog plugins - #3378

Merged
Andrei Kvapil (kvaps) merged 6 commits into
mainfrom
feat/grafana-upstream-no-rebuild
Jul 28, 2026
Merged

Andrei Kvapil (kvaps) merged 6 commits into
mainfrom
feat/grafana-upstream-no-rebuild

Conversation

@kvaps

@kvaps Andrei Kvapil (kvaps) commented Jul 20, 2026 •

Copy link
Copy Markdown
Member

What this PR does

Cozystack builds and publishes ghcr.io/cozystack/cozystack/grafana — a rebuilt Grafana image — and uses it by default in the monitoring stack. Grafana is AGPL-3.0, so publishing a rebuilt binary makes the project a redistributor of an AGPL Grafana build. This is a governance concern for CNCF Incubation (there is no precedent for a CNCF project redistributing an AGPL binary).

The only reason the image was ever rebuilt was to bundle the VictoriaLogs datasource plugin, which was not in the official Grafana plugin catalog at the time. It is now published there as a signed plugin (victoriametrics-logs-datasource), so the rebuild is no longer necessary.

This PR moves the deployment to the upstream grafana/grafana image (digest-pinned, routed through cozy-lib.image so mirrored-registry installs keep working) and installs the plugins at container startup via GF_INSTALL_PLUGINS, with versions pinned:

  • victoriametrics-logs-datasource 0.29.0 — the VictoriaLogs datasource (signed; the reason the image was rebuilt)
  • natel-discrete-panel 0.1.1, grafana-worldmap-panel 1.0.6, marcusolsson-dynamictext-panel 6.2.0 — reproduce the panel set the removed Dockerfile vendored

Air-gapped installs keep working. The plugin zips are not downloaded from the internet at pod startup: they are mirrored from the official catalog into the existing grafana-dashboards image at build time and served by that in-cluster static file server, next to the dashboards it already serves. GF_INSTALL_PLUGINS points at http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…, so the runtime dependency stays inside the cluster and the artifact is mirrored into private registries together with every other first-party image. A new allow-to-grafana-dashboards CiliumNetworkPolicy (mirroring allow-to-keycloak) opens tenant egress to that server, since tenant egress is default-deny.

As a result, ghcr.io/cozystack/cozystack/grafana is no longer built or published: the Dockerfile, the pinned image tag, the image targets in both monitoring Makefiles, and the build step in the root Makefile are removed.

Notes:

  • The AGPL-3.0 Grafana binary is no longer redistributed; the mirrored plugin zips are Apache-2.0 licensed and are served unmodified from the official catalog builds (signatures intact).
  • allow_loading_unsigned_plugins and the custom /var/lib/grafana-plugins path are dropped: the catalog datasource is signed, and plugins install into the writable default GF_PATHS_PLUGINS (/var/lib/grafana/plugins).
  • The VictoriaLogs datasource provisioning (the GrafanaDatasource CRs pointing at vlselect) is unchanged.
  • Plugins are re-installed on every pod start from the in-cluster server (no PVC involved, ~77 MB per start per replica over the cluster network); a failed download fails the pod loudly instead of silently dropping the datasource.
  • natel-discrete-panel and grafana-worldmap-panel are Angular plugins; Angular is disabled by default in Grafana 11, so they do not load — this is unchanged from the current rebuilt image (same Grafana 11.6.15, same config). They are kept for parity and can be dropped as a follow-up.

How this was verified

  • helm unittest across the repo passes, including new regression tests: the monitoring chart asserts the exact upstream image reference (with and without images-registry set) and the exact GF_INSTALL_PLUGINS value; the tenant chart asserts the new egress policy.
  • Local end-to-end of the full chain in Docker: built the modified grafana-dashboards image, served it on a shared network, and started upstream grafana/grafana:11.6.15 with the rendered GF_INSTALL_PLUGINS value. All four plugins download from the in-cluster-style URL (no internet egress from Grafana), and GET /api/plugins reports victoriametrics-logs-datasource v0.29.0 as enabled: true with signature: valid.

Screenshots

Not applicable — no UI change. Grafana renders the same datasources and panels; only the image source and plugin delivery mechanism change.

Downstream repositories

Walked the trigger map against the diff. This is an internal change to how the monitoring stack sources the Grafana image and its plugins (packages/system/monitoring, packages/system/grafana-operator, packages/apps/tenant network policy). No package is added, renamed or removed; no values.schema.json, ApplicationDefinition, platform values, release asset, shared hack/ tooling or package.mk contract changes; the added grafana.image key is an internal system-chart value, not a user-facing app field. No downstream repository is affected.

Release note

feat(monitoring): Grafana now runs on the upstream `grafana/grafana` image; the VictoriaLogs datasource and panel plugins are mirrored from the official Grafana catalog into the in-cluster `grafana-dashboards` server and installed at startup, so air-gapped installs keep working. Cozystack no longer builds or publishes `ghcr.io/cozystack/cozystack/grafana`.

Summary by CodeRabbit

  • Improvements
    • Grafana in monitoring is now based on a digest-pinned upstream image for more consistent deployments.
    • Required Grafana plugins are installed automatically at startup via GF_INSTALL_PLUGINS; unsigned-plugin configuration was removed.
    • Tenant networking now includes a new egress rule specifically for Grafana dashboards.
    • Grafana dashboards image now includes required plugin assets prepared during image build.
  • Tests
    • Added/updated checks covering digest pinning, mirrored-registry behavior, plugin-installation behavior, and the new Grafana dashboards network policy.

@coderabbitai

coderabbitai Bot commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 97386008-9f69-4468-bbdb-26fd72e773d1

📥 Commits

Reviewing files that changed from the base of the PR and between 1c7eb22 and a5a0449.

📒 Files selected for processing (2)
  • hack/promote-retag_test.bats
  • hack/promote-rewrite-tags_test.bats

📝 Walkthrough

Walkthrough

Grafana image construction moves from a custom Dockerfile to a digest-pinned upstream image. Plugins are mirrored in the Grafana dashboards image and installed at startup, with tenant egress permitted to the dashboard service. Promotion tests now reference the dashboards image.

Changes

Grafana plugin migration

Layer / File(s) Summary
Image build ownership and orchestration
Makefile, packages/extra/monitoring/Makefile, packages/system/monitoring/Makefile, packages/system/monitoring/images/*
The custom Grafana image workflow is removed, system monitoring image handling is updated, and the top-level build skips the monitoring image step.
Dashboard plugin mirroring
packages/system/grafana-operator/images/grafana-dashboards/Dockerfile
The dashboard image downloads pinned Grafana plugins at build time and serves them with the dashboard files.
Runtime image and plugin configuration
packages/system/monitoring/templates/grafana/grafana.yaml, packages/system/monitoring/values.yaml, packages/system/monitoring/tests/grafana_upstream_image_plugins_test.yaml
Grafana uses a digest-pinned upstream image and installs pinned plugins through GF_INSTALL_PLUGINS using URLs from the in-cluster dashboard service.
Tenant egress access
packages/apps/tenant/templates/networkpolicy.yaml, packages/apps/tenant/tests/networkpolicy_grafana_dashboards_test.yaml
A tenant CiliumNetworkPolicy permits egress to endpoints in the cozy-grafana-operator namespace and verifies the rendered policy.
Image promotion test updates
hack/promote-retag_test.bats, hack/promote-rewrite-tags_test.bats
Promotion tests now enumerate and retag grafana-dashboards image references.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GrafanaValues
  participant GrafanaTemplate
  participant GrafanaContainer
  participant GrafanaDashboards
  GrafanaValues->>GrafanaTemplate: provide digest-pinned grafana.image
  GrafanaTemplate->>GrafanaContainer: render image and GF_INSTALL_PLUGINS
  GrafanaContainer->>GrafanaDashboards: request pinned plugin ZIPs
  GrafanaDashboards-->>GrafanaContainer: serve mirrored plugin ZIPs
Loading

Possibly related PRs

Suggested labels: area/build

Suggested reviewers: myasnikovdaniil, lexfrei, ivanhunters

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Clearly summarizes the main change: replacing the rebuilt Grafana image with the upstream image and catalog plugin installation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/grafana-upstream-no-rebuild

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request transitions the monitoring stack from using a custom-rebuilt Grafana image to the official upstream image. By leveraging the official Grafana plugin catalog for necessary datasources and panels at startup, the project eliminates the need to maintain and redistribute a custom binary, thereby resolving potential governance and licensing concerns related to AGPL-3.0 compliance.

Highlights

  • Switch to Upstream Grafana Image: Removed the custom-built Grafana image in favor of the official upstream grafana/grafana image to address AGPL-3.0 redistribution concerns.
  • Dynamic Plugin Installation: Implemented plugin installation at container startup using the GF_INSTALL_PLUGINS environment variable, pulling signed plugins directly from the official Grafana catalog.
  • Cleanup of Build Infrastructure: Removed the Dockerfile, build targets, and associated metadata files used for the previously rebuilt Grafana image.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@dosubot dosubot Bot added area/monitoring Issues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor) kind/cleanup Categorizes issue or PR as related to cleanup of code, process, or technical debt labels Jul 20, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the monitoring setup to use the upstream Grafana image directly, installing required plugins at container startup via the GF_INSTALL_PLUGINS environment variable instead of maintaining a custom rebuilt image. The review feedback highlights a missing cleanup of Dockerfiles and tag files in the packages/extra/monitoring package, potential deployment failures in air-gapped environments due to the runtime dependency on the external Grafana catalog, and an opportunity to harden container security by enabling a read-only root filesystem.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines 1 to 3
NAME=monitoring

include ../../../hack/common-envs.mk

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

It appears that packages/extra/monitoring is a separate package from packages/system/monitoring but shares the same structure. While the image target and GRAFANA_TAG were removed from packages/extra/monitoring/Makefile, the corresponding packages/extra/monitoring/images/grafana/Dockerfile and packages/extra/monitoring/images/grafana.tag files were not deleted in this PR. Additionally, the templates and values.yaml under packages/extra/monitoring/ (such as templates/grafana/grafana.yaml) likely still reference images/grafana.tag and need to be updated to use the upstream image and install plugins via GF_INSTALL_PLUGINS to maintain parity with packages/system/monitoring.

Comment on lines +204 to +205
- name: GF_INSTALL_PLUGINS
value: "victoriametrics-logs-datasource 0.29.0,natel-discrete-panel 0.1.1,grafana-worldmap-panel 1.0.6,marcusolsson-dynamictext-panel 6.2.0"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Installing plugins at container startup via GF_INSTALL_PLUGINS introduces a runtime dependency on the external Grafana plugin catalog (grafana.com). This has two main implications:

  1. Air-gapped environments: In offline or highly restricted network environments, the Grafana pods will fail to start because they cannot download the plugins.
  2. Startup latency & reliability: Pod startup time will increase, and any transient network issues or rate limits on the Grafana marketplace will cause pod startup failures.

If Cozystack is intended to support air-gapped/offline installations, consider documenting this limitation or providing a way to pre-cache/mirror these plugins.

Comment on lines 183 to 185
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

To further harden the Grafana container security, consider setting readOnlyRootFilesystem: true. Since Grafana now installs plugins at startup into /var/lib/grafana/plugins and may write temporary files, you can achieve this by mounting emptyDir volumes at /var/lib/grafana and /tmp.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/system/monitoring/templates/grafana/grafana.yaml (1)

188-205: 🩺 Stability & Availability | 🔵 Trivial

Operational advice: Impact on air-gapped environments and startup reliability.

Switching to GF_INSTALL_PLUGINS changes the deployment architecture by dynamically downloading plugins from the public Grafana catalog at pod startup. Please keep the following operational impacts in mind:

  • Air-gapped / Offline environments: The Grafana pod now strictly requires outbound internet access to start. If the platform is deployed in a restricted-egress environment, it will fail to initialize. If offline support is a requirement, consider utilizing an init-container to mirror and copy these plugins from a local registry image into a shared plugins volume.
  • Startup reliability: If the Grafana plugin catalog is temporarily unavailable, or if rate limits are hit, the pod will fail to start and enter a CrashLoopBackOff state, degrading monitoring availability.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/system/monitoring/templates/grafana/grafana.yaml` around lines 188 -
205, Update the Grafana deployment around GF_INSTALL_PLUGINS to support
air-gapped and unreliable-network environments by sourcing the pinned plugins
from a local image or mirrored artifact via an init-container and shared plugins
volume, rather than requiring startup downloads from the public catalog.
Preserve the existing plugin set and versions, and ensure Grafana starts using
the copied plugins without outbound network access.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/system/monitoring/templates/grafana/grafana.yaml`:
- Around line 188-205: Update the Grafana deployment around GF_INSTALL_PLUGINS
to support air-gapped and unreliable-network environments by sourcing the pinned
plugins from a local image or mirrored artifact via an init-container and shared
plugins volume, rather than requiring startup downloads from the public catalog.
Preserve the existing plugin set and versions, and ensure Grafana starts using
the copied plugins without outbound network access.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4bc5be65-f9ca-40c1-ba6b-5dd6dfdce6c2

📥 Commits

Reviewing files that changed from the base of the PR and between f5227ee and 1e2465a.

📒 Files selected for processing (7)
  • Makefile
  • packages/extra/monitoring/Makefile
  • packages/system/monitoring/Makefile
  • packages/system/monitoring/images/grafana.tag
  • packages/system/monitoring/images/grafana/Dockerfile
  • packages/system/monitoring/templates/grafana/grafana.yaml
  • packages/system/monitoring/values.yaml
💤 Files with no reviewable changes (5)
  • packages/system/monitoring/images/grafana.tag
  • packages/system/monitoring/images/grafana/Dockerfile
  • Makefile
  • packages/system/monitoring/Makefile
  • packages/extra/monitoring/Makefile

@github-actions github-actions Bot added kind/feature Categorizes issue or PR as related to a new feature size/M This PR changes 30-99 lines, ignoring generated files labels Jul 20, 2026

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with the cozy-review methodology. Verdict: request changes (posting as a comment).

Blocking (MAJOR) packages/system/monitoring/templates/grafana/grafana.yaml:204

Switching baked-in plugins to a runtime download via GF_INSTALL_PLUGINS introduces an external network dependency on every Grafana pod start. This is a regression for air-gapped and egress-restricted clusters (a supported cozystack scenario), on both fresh install and upgrade. There is no mechanism to mirror the plugin catalog into the repo (cozy-lib.image / images-registry only rewrite image references). Additionally /var/lib/grafana has no PVC, so plugins are re-downloaded on every restart, for each of the two replicas. The trade-off is not mentioned in the PR description.

Minor

  • grafana.yaml:182 / values.yaml:72: the Grafana image does not go through cozy-lib.image, although the same chart documents that convention for the kubectl image (oidc-users-job.yaml:112-116). Breaks mirrored-registry installs.
  • grafana.yaml:182: behavior change without a regression test. The existing 47 tests do not exercise the image / GF_INSTALL_PLUGINS path.

The goal (dropping the custom image rebuild for AGPL compliance) is sound, but the chosen implementation silently breaks offline installs. A safe version needs an offline plugin-delivery path (a mirrorable bundle via images-registry plus an initContainer/volume, or vendoring into the mirror artifact), or an explicitly documented egress requirement with a legitimate failure signal.

Point the Grafana deployment at the upstream grafana/grafana image and
install the VictoriaLogs datasource and panel plugins at startup via
GF_INSTALL_PLUGINS, pulled from the official Grafana plugin catalog
(https://grafana.com/grafana/plugins) instead of being baked into a
custom rebuilt image.

The VictoriaLogs datasource is now published and signed in the catalog
(victoriametrics-logs-datasource), so allow_loading_unsigned_plugins and
the custom /var/lib/grafana-plugins path are no longer needed; plugins
land in the writable default GF_PATHS_PLUGINS. Plugin versions are pinned
to releases compatible with the shipped Grafana version.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <[email protected]>
Remove the custom Grafana image build now that the deployment runs on the
upstream grafana/grafana image with catalog plugins. Drops the Dockerfile,
the pinned image tag, the image targets in both monitoring Makefiles, and
the build step from the root Makefile, so ghcr.io/cozystack/cozystack/grafana
is no longer produced or published.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <[email protected]>
…boards image

Download the pinned Grafana plugin zips (VictoriaLogs datasource + panels)
from the official catalog at image build time and serve them next to the
dashboards. Grafana pods can then install plugins from this in-cluster
server instead of reaching the public catalog at startup, which keeps
air-gapped and egress-restricted installs working.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <[email protected]>
Point GF_INSTALL_PLUGINS at the grafana-dashboards static file server
instead of the public Grafana catalog, so plugin installation needs no
internet egress at pod startup and keeps working on air-gapped installs.
Route the upstream Grafana image reference through cozy-lib.image so
mirrored-registry installs keep working, and add regression tests for
the image reference and the plugin install path.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <[email protected]>
Tenant egress is default-deny plus explicit allow-to-* policies, so the
Grafana pods deployed per tenant could not reach the in-cluster plugin
server in cozy-grafana-operator. Mirror the allow-to-keycloak pattern
and cover the policy with a test.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <[email protected]>
@kvaps
Andrei Kvapil (kvaps) force-pushed the feat/grafana-upstream-no-rebuild branch from 1e2465a to 1c7eb22 Compare July 24, 2026 14:09
@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files and removed size/M This PR changes 30-99 lines, ignoring generated files labels Jul 24, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
packages/system/grafana-operator/images/grafana-dashboards/Dockerfile (1)

11-22: 🗄️ Data Integrity & Integration | 🔵 Trivial | 🏗️ Heavy lift

Keep the mirrored plugin manifest single-source or enforce synchronization.

The Dockerfile and Helm template duplicate the plugin IDs and versions. A drift can build the image successfully but make Grafana request missing archives at startup.

  • packages/system/grafana-operator/images/grafana-dashboards/Dockerfile#L11-L22: derive downloads from a shared manifest or add a comparison check.
  • packages/system/monitoring/tests/grafana_upstream_image_plugins_test.yaml#L49-L63: add or trigger the cross-file validation; the current assertion covers only the Helm-side list.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/system/grafana-operator/images/grafana-dashboards/Dockerfile` around
lines 11 - 22, Keep the Grafana plugin manifest synchronized across
packages/system/grafana-operator/images/grafana-dashboards/Dockerfile lines
11-22 and
packages/system/monitoring/tests/grafana_upstream_image_plugins_test.yaml lines
49-63 by deriving both from a shared manifest or adding validation that compares
plugin IDs and versions. Extend the existing test assertion to cover the
Dockerfile list as well as the Helm-side list; update the Dockerfile download
loop or manifest handling accordingly.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/apps/tenant/templates/networkpolicy.yaml`:
- Around line 227-231: Scope the egress rule in
packages/apps/tenant/templates/networkpolicy.yaml:227-231 to pods backing the
grafana-dashboards service by adding the required dashboard pod labels alongside
the namespace selector, rather than selecting the entire operator namespace.
Update
packages/apps/tenant/tests/networkpolicy_grafana_dashboards_test.yaml:14-32 to
assert those pod labels and prevent the broader namespace-only selector from
returning.

In `@packages/system/grafana-operator/images/grafana-dashboards/Dockerfile`:
- Around line 24-29: Add a dedicated unprivileged user in the final Dockerfile
stage, ensure it can read the copied dashboard and plugin files, and set the
final runtime user with a USER directive before darkhttpd starts. Keep the
server on port 8080, which does not require root.

---

Nitpick comments:
In `@packages/system/grafana-operator/images/grafana-dashboards/Dockerfile`:
- Around line 11-22: Keep the Grafana plugin manifest synchronized across
packages/system/grafana-operator/images/grafana-dashboards/Dockerfile lines
11-22 and
packages/system/monitoring/tests/grafana_upstream_image_plugins_test.yaml lines
49-63 by deriving both from a shared manifest or adding validation that compares
plugin IDs and versions. Extend the existing test assertion to cover the
Dockerfile list as well as the Helm-side list; update the Dockerfile download
loop or manifest handling accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 01b7ed72-75e9-4402-9b7f-3a35ae2b0216

📥 Commits

Reviewing files that changed from the base of the PR and between 1e2465a and 1c7eb22.

📒 Files selected for processing (11)
  • Makefile
  • packages/apps/tenant/templates/networkpolicy.yaml
  • packages/apps/tenant/tests/networkpolicy_grafana_dashboards_test.yaml
  • packages/extra/monitoring/Makefile
  • packages/system/grafana-operator/images/grafana-dashboards/Dockerfile
  • packages/system/monitoring/Makefile
  • packages/system/monitoring/images/grafana.tag
  • packages/system/monitoring/images/grafana/Dockerfile
  • packages/system/monitoring/templates/grafana/grafana.yaml
  • packages/system/monitoring/tests/grafana_upstream_image_plugins_test.yaml
  • packages/system/monitoring/values.yaml
💤 Files with no reviewable changes (5)
  • packages/system/monitoring/images/grafana.tag
  • packages/system/monitoring/images/grafana/Dockerfile
  • Makefile
  • packages/extra/monitoring/Makefile
  • packages/system/monitoring/Makefile
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/system/monitoring/templates/grafana/grafana.yaml

Comment on lines +227 to +231
endpointSelector: {}
egress:
- toEndpoints:
- matchLabels:
"k8s:io.kubernetes.pod.namespace": cozy-grafana-operator

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Scope the network policy to dashboard pods.

The namespace-only selector grants tenant workloads access to all endpoints in the operator namespace, while the runtime dependency is the grafana-dashboards service.

  • packages/apps/tenant/templates/networkpolicy.yaml#L227-L231: select the grafana-dashboards pod labels instead of only the namespace.
  • packages/apps/tenant/tests/networkpolicy_grafana_dashboards_test.yaml#L14-L32: assert the dashboard pod selector so the test prevents the broader rule from returning.
📍 Affects 2 files
  • packages/apps/tenant/templates/networkpolicy.yaml#L227-L231 (this comment)
  • packages/apps/tenant/tests/networkpolicy_grafana_dashboards_test.yaml#L14-L32
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/apps/tenant/templates/networkpolicy.yaml` around lines 227 - 231,
Scope the egress rule in
packages/apps/tenant/templates/networkpolicy.yaml:227-231 to pods backing the
grafana-dashboards service by adding the required dashboard pod labels alongside
the namespace selector, rather than selecting the entire operator namespace.
Update
packages/apps/tenant/tests/networkpolicy_grafana_dashboards_test.yaml:14-32 to
assert those pod labels and prevent the broader namespace-only selector from
returning.

Comment on lines 24 to +29
FROM alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b

RUN apk add --no-cache darkhttpd

COPY dashboards /var/www/dashboards
COPY --from=plugins /plugins /var/www/dashboards/plugins

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Run the final server as non-root.

The final image has no USER directive, so darkhttpd serves the tenant-accessible plugin endpoint as root. Add a dedicated unprivileged user in the final stage and switch to it after copying the static files; port 8080 does not require root. Trivy reports this as DS-0002.

Suggested fix
 FROM alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b
 
-RUN apk add --no-cache darkhttpd
+RUN apk add --no-cache darkhttpd \
+ && addgroup -S grafana \
+ && adduser -S -G grafana grafana
 
 COPY dashboards /var/www/dashboards
 COPY --from=plugins /plugins /var/www/dashboards/plugins
 
+USER grafana
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
FROM alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b
RUN apk add --no-cache darkhttpd
COPY dashboards /var/www/dashboards
COPY --from=plugins /plugins /var/www/dashboards/plugins
FROM alpine:3.24@sha256:28bd5fe8b56d1bd048e5babf5b10710ebe0bae67db86916198a6eec434943f8b
RUN apk add --no-cache darkhttpd \
&& addgroup -S grafana \
&& adduser -S -G grafana grafana
COPY dashboards /var/www/dashboards
COPY --from=plugins /plugins /var/www/dashboards/plugins
USER grafana
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/system/grafana-operator/images/grafana-dashboards/Dockerfile` around
lines 24 - 29, Add a dedicated unprivileged user in the final Dockerfile stage,
ensure it can read the copied dashboard and plugin files, and set the final
runtime user with a USER directive before darkhttpd starts. Keep the server on
port 8080, which does not require root.

Source: Linters/SAST tools

@kvaps

Copy link
Copy Markdown
Member Author

Addressed in the latest push (rebased on main + 3 commits):

Blocking — runtime dependency on the public catalog: plugins are no longer downloaded from the internet. The pinned zips are mirrored from the official catalog into the existing grafana-dashboards image at build time and served by that in-cluster static file server next to the dashboards it already serves; GF_INSTALL_PLUGINS now points at http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…, so the artifact reaches air-gapped clusters the same way every other first-party image does. Tenant egress is default-deny, so a new allow-to-grafana-dashboards CiliumNetworkPolicy (mirroring allow-to-keycloak) opens the path. Plugins are still re-installed on each pod start, but from the in-cluster server (~77 MB over the cluster network), and a failed download fails the pod loudly. The trade-off is now documented in the PR description.

cozy-lib.image: the Grafana image reference now goes through cozy-lib.image (docker.io/… form, matching the kubectl.tag convention in the same chart).

Regression tests: added grafana_upstream_image_plugins_test.yaml (asserts the exact image reference with and without images-registry, and the exact GF_INSTALL_PLUGINS value) and a tenant-chart test for the new egress policy.

Verified locally end-to-end in Docker: built the modified dashboards image, served it on a shared network, and started upstream Grafana 11.6.15 with the rendered env — all four plugins install from the in-cluster URL and /api/plugins reports the VictoriaLogs datasource as enabled with signature: valid.

…ards

The promote-retag and image-refs enumeration tests used the monitoring
grafana.tag as their canonical example of a reference living outside a
values.yaml. That file is gone now that the Grafana rebuild is dropped,
so use the grafana-dashboards tag — the same storage shape, still in
the tree — as the exemplar.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <[email protected]>
@kvaps
Andrei Kvapil (kvaps) merged commit 8b461f0 into main Jul 28, 2026
42 checks passed
@kvaps
Andrei Kvapil (kvaps) deleted the feat/grafana-upstream-no-rebuild branch July 28, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/monitoring Issues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor) kind/cleanup Categorizes issue or PR as related to cleanup of code, process, or technical debt kind/feature Categorizes issue or PR as related to a new feature size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants