A production-inspired, end-to-end DevSecOps project that builds, secures, and deploys a full-stack web application on Kubernetes. It covers the complete lifecycle from source code to a running cluster — including CI pipelines with security scanning, GitOps-based delivery, secrets management, TLS, and database migrations.
The project ships two Kustomize overlays:
overlays/dev— targets an AWS EKS cluster with RDS, AWS Secrets Manager, and Traefikoverlays/local— targets a local Kind cluster with a MySQL pod, HashiCorp Vault, and NGINX Ingress
- What This Project Is
- Software Components
- Repository Structure
- Traffic Flow in the Cluster
- Local Deployment on Kind
- Production Deployment
- CI/CD Pipeline
- Secrets Architecture
A three-tier web application (React → Node.js → MySQL) deployed on Kubernetes with a full DevSecOps pipeline. It demonstrates real-world patterns:
| Concern | Solution |
|---|---|
| Infrastructure as Code | Kustomize (base + overlays) |
| GitOps delivery | Argo CD |
| CI pipeline | Jenkins (pod-based agents on K8s) |
| Secret Scanning | Gitleaks |
| Dependency vulnerabilities | OWASP Dependency-Check |
| Static code analysis | SonarQube |
| Container scanning | Trivy |
| Secrets at runtime | AWS Secrets Manager (prod) / HashiCorp Vault (local) |
| TLS certificates | cert-manager (Let's Encrypt / self-signed) |
| Database migrations | Kubernetes Job (runs on every deploy) |
| Component | Technology | Description |
|---|---|---|
| Frontend | React (nginx) | Single-page app. Pages: Login, Register, Dashboard. Proxies /api/* to the backend. |
| Backend API | Node.js + Express | REST API. Routes: /api/auth, /api/users. Health probes: /live, /health. |
| Database | MySQL 8 | Stores users. Schema applied via Sequelize migrations. |
| Component | Namespace | Purpose |
|---|---|---|
| NGINX Ingress | ingress-nginx |
Routes external HTTP/HTTPS traffic to frontend and API |
| cert-manager | cert-manager |
Issues and renews TLS certificates (Let's Encrypt or self-signed) |
| External Secrets Operator | external-secrets |
Pulls secrets from Vault/AWS SM → creates K8s Secrets |
| HashiCorp Vault | vault |
Secret store for DB credentials and JWT secret (local) |
| Migration Job | local / dev |
Runs sequelize db:migrate once per deploy, then exits |
| Jenkins | jenkins |
CI server with pod-based agents; deployed via jenkins/setup/deploy.sh |
| SonarQube | sonarqube |
SAST server; deployed alongside Jenkins by deploy.sh |
| BuildKit | jenkins |
Rootless image builder used by Jenkins docker-cli agent container |
| Tool | Role |
|---|---|
| Jenkins | Orchestrates the pipeline; runs as pods inside Kubernetes |
| Gitleaks | Scans source code for accidentally committed secrets |
| OWASP Dependency-Check | Audits Node.js dependencies for known CVEs |
| SonarQube | SAST — static code analysis and quality gate |
| Trivy | Scans Docker images for OS and library vulnerabilities |
| Argo CD | GitOps controller — syncs cluster state from Git |
| Kustomize | Manages environment-specific Kubernetes manifests |
.
├── api/ # Node.js backend
│ ├── Dockerfile
│ ├── Jenkinsfile # CI pipeline for the API
│ ├── routes/ # authRoutes.js, userRoutes.js
│ ├── controllers/ # authController.js, userController.js
│ ├── models/ # Sequelize models
│ ├── migrations/ # DB schema migrations
│ └── seeders/ # Initial data (admin user)
│
├── client/ # React frontend
│ ├── Dockerfile # Multi-stage: build React → serve with nginx
│ ├── Jenkinsfile # CI pipeline for the frontend
│ ├── default.conf # nginx config (proxies /api to backend)
│ └── src/
│ ├── pages/ # Login, Register, Dashboard
│ └── context/ # AuthContext (JWT management)
│
├── kubernetes/
│ ├── base/ # Shared K8s manifests (no env-specific values)
│ │ ├── api-manifests/ # API Deployment + ClusterIP Service
│ │ ├── client-manifests/ # Client Deployment + ClusterIP Service
│ │ ├── mysql-db/ # ConfigMap, ExternalName Service, Migration Job
│ │ ├── aws-sm/ # ServiceAccount + SecretProviderClass (AWS SM)
│ │ └── ingress/ # Ingress resource (path-based routing)
│ │
│ ├── overlays/
│ │ ├── dev/ # EKS overlay: RDS, AWS SM, Traefik, Let's Encrypt
│ │ └── local/ # Kind overlay: MySQL pod, Vault, nginx, self-signed TLS
│ │
│ └── infra/ # Cluster-level prereqs (cert-manager issuers, ESO stores)
│
└── jenkins/
├── Jenkins-agent.yaml # CI agent pod template (nodejs, sonar, trivy, buildkitd, etc.)
├── Jenkins-agent-cd.yaml # CD agent pod template (kubectl only, no PVC dependency)
├── Jenkinsfile-cd-local # CD pipeline: deploy CI-built image to local Kind cluster
└── setup/ # One-shot deployment for local CI/CD stack
├── deploy.sh # Idempotent script: deploys SonarQube + Jenkins
├── jenkins-values.yaml # Jenkins Helm values (JCasC, plugins, ingress)
├── sonarqube-values.yaml # SonarQube Community Helm values (ingress, TLS)
├── kustomization.yaml # RBAC, buildkitd, certs stub, NVD PVC
├── rbac.yaml # ClusterRole for Jenkins agent pods + local namespace deploy
├── buildkitd.yaml # BuildKit daemon for rootless image builds
├── dependency-check-pvc.yaml # 5Gi PVC — persists NVD database between builds
└── credentials-template.yaml # Template for jenkins-credentials Secret
User Browser
│
│ HTTPS (port 443)
▼
┌─────────────────────────────────────────────────────┐
│ Ingress Controller (nginx) │
│ cert-manager TLS termination │
│ host: app.local │
└────────────────┬────────────────────────────────────┘
│
┌────────┴────────┐
│ Path routing │
│ │
path: /api/* path: /
│ │
▼ ▼
┌──────────┐ ┌──────────────┐
│api-service│ │client-service│
│ port 5000 │ │ port 80 │
└─────┬─────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────┐ ┌──────────────┐
│ API Pod │ │ Client Pod │
│ (Node.js) │ │ (nginx) │
└─────┬─────┘ └──────────────┘
│
▼
┌───────────┐
│mysql-db │ ← ClusterIP Service (local: selects MySQL pod)
│ Service │ (dev: ExternalName → AWS RDS endpoint)
└─────┬─────┘
│
▼
┌───────────┐
│ MySQL │
│ Pod/RDS │
└───────────┘
HashiCorp Vault
(secret/app: DB_USER, DB_PASSWORD, JWT_SECRET)
│
│ HTTP API (token auth)
▼
External Secrets Operator
(ClusterSecretStore: vault-backend)
│
│ creates / refreshes every 1h
▼
K8s Secret: app-secrets (namespace: local)
│
│ envFrom: secretRef
▼
API Pod + Migration Job
(reads DB_USER, DB_PASSWORD, JWT_SECRET as env vars)
cert-manager ClusterIssuer (selfsigned-issuer / letsencrypt)
│
│ issues certificate for app.local / your-domain.com
▼
K8s Secret: app-tls-local (type: kubernetes.io/tls)
│
│ referenced by Ingress spec.tls
▼
Ingress Controller → terminates TLS for all inbound HTTPS
brew install kind kubectl helm vault podman
podman machine startexport KIND_EXPERIMENTAL_PROVIDER=podman
kind create cluster --name vault --config kubernetes/overlays/local/kind-config.yaml# cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s
# External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
--namespace external-secrets --create-namespace --wait
# NGINX Ingress (kind-specific manifest — uses hostPort 80/443)
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl wait --namespace ingress-nginx --for=condition=ready pod \
--selector=app.kubernetes.io/component=controller --timeout=90s
# HashiCorp Vault
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
--namespace vault --create-namespace \
--set "server.dev.enabled=true" \
--set "server.dev.devRootToken=root" --wait# Open a shell into Vault
kubectl exec -it vault-0 -n vault -- /bin/sh
# Inside the pod:
export VAULT_TOKEN=root
export VAULT_ADDR=http://127.0.0.1:8200
# Store secrets (must match MySQL credentials in mysql-deployment.yaml)
vault kv put secret/app \
DB_USERNAME=appuser \
DB_PASSWORD=apppass123 \
JWT_SECRET=localjwtsecret12345678
exitThis is created imperatively to avoid committing secrets to Git.
kubectl create secret generic vault-token \
--from-literal=token=<your-vault-root-token> \
-n external-secretspodman pull docker.io/amitactive2008/api:latest
podman pull docker.io/amitactive2008/client:latest1
podman save -o /tmp/api.tar docker.io/amitactive2008/api:latest
podman save -o /tmp/client.tar docker.io/amitactive2008/client:latest1
export KIND_EXPERIMENTAL_PROVIDER=podman
kind load image-archive /tmp/api.tar --name vault
kind load image-archive /tmp/client.tar --name vaultecho "127.0.0.1 app.local" | sudo tee -a /etc/hosts
echo "127.0.0.1 jenkins.kind.local" | sudo tee -a /etc/hosts
echo "127.0.0.1 sonarqube.kind.local" | sudo tee -a /etc/hosts# Preview rendered manifests
kubectl kustomize kubernetes/overlays/local
# Apply
kubectl apply -k kubernetes/overlays/local
# Watch pods come up
kubectl get pods -n local -wExpected pod lifecycle:
| Pod | Status | Description |
|---|---|---|
mysql-* |
Running | MySQL starts first, waits for readiness probe |
database-migration-* |
Completed | Runs schema migrations + seeds admin user |
api-deployment-* |
Running | Starts after DB is ready (readiness probe on /health) |
client-deployment-* |
Running | Starts independently |
# All pods healthy
kubectl get pods -n local
# ESO synced the secret from Vault
kubectl get externalsecret app-secrets -n local
# TLS cert issued
kubectl get certificate -n local
# API responding
kubectl exec -n local \
$(kubectl get pod -l app.kubernetes.io/name=api -n local -o jsonpath='{.items[0].metadata.name}') \
-- wget -qO- http://localhost:5000/health
# Expected: OK
# Login via ingress
curl -sk -X POST https://app.local/api/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"[email protected]","password":"admin123"}'
# Expected: {"token":"...","user":{...}}Open https://app.local in your browser (accept the self-signed cert warning).
Default seeded credentials: [email protected] / admin123
A single script deploys both tools, wires the SonarQube token into Jenkins credentials, and configures everything via JCasC — no manual UI clicks required.
# Create the credentials secret first (DockerHub + NVD API key)
cp jenkins/setup/credentials-template.yaml jenkins/setup/credentials.yaml
# Edit credentials.yaml and fill in real values, then:
kubectl apply -f jenkins/setup/credentials.yaml
# Deploy SonarQube + Jenkins
./jenkins/setup/deploy.shWhat deploy.sh does, in order:
| Step | Action |
|---|---|
| 1 | Check prerequisites (kubectl, helm, kind) |
| 2 | Apply namespace, RBAC, BuildKit via Kustomize |
| 3 | Create jenkins-credentials secret (interactive if not present) |
| 4 | Pre-load Jenkins image into Kind node (bypasses DockerHub TLS issues in kind) |
| 5 | Add Helm repos (Jenkins + SonarQube) |
| 6 | Deploy SonarQube Community via Helm with nginx ingress + TLS |
| 7 | Generate SonarQube tokens via API (must run before admin password is changed): GLOBAL_ANALYSIS_TOKEN for Jenkins scanner auth; USER_TOKEN for webhook creation |
| 8 | Register SonarQube→Jenkins webhook (/sonarqube-webhook/) — required for waitForQualityGate() |
| 9 | Deploy Jenkins via Helm with JCasC (clouds, credentials, jobs pre-configured) |
After the script completes:
| Service | URL | Credentials |
|---|---|---|
| Jenkins | https://jenkins.kind.local |
admin / admin |
| SonarQube | https://sonarqube.kind.local |
admin / admin (change on first login) |
Jenkins comes pre-configured with:
- Kubernetes cloud — dynamic agent pods (defined in
jenkins/Jenkins-agent.yaml) - Credentials —
dockerhub,NVD_API_KEY,sonarqube-token - SonarQube server
mysonarqube— authenticated with the auto-generated token - Pipeline jobs —
api-ciandclient-cipre-created via Job DSL
NVD API key is optional. The
NVD_API_KEYcredential is used by Stage 4 (OWASP Dependency-Check) to call the NVD API with higher rate limits. If the key is absent or set toplaceholder, the--nvdApiKeyflag is skipped and dependency-check runs against the public NVD API (rate-limited but functional). Register a free key at https://nvd.nist.gov/developers/request-an-api-key and update the secret:kubectl patch secret jenkins-credentials -n jenkins --type='json' \ -p='[{"op":"replace","path":"/data/NVD_API_KEY","value":"'$(echo -n YOUR_KEY | base64)'"}]' kubectl rollout restart statefulset/jenkins -n jenkins
SonarQube webhook (required for Quality Gate).
deploy.shcreates it automatically on fresh installs. If re-deploying or if it fails, create it manually:https://sonarqube.kind.local/admin/webhooks→ Create → Name:jenkins, URL:http://jenkins.jenkins.svc.cluster.local:8080/sonarqube-webhook/
kubectl delete -k kubernetes/overlays/local
helm uninstall jenkins -n jenkins
helm uninstall sonarqube -n sonarqube
kind delete cluster --name vault- AWS EKS cluster
- AWS RDS MySQL instance
- AWS Secrets Manager secret with
DB_USERNAME,DB_PASSWORD,JWT_SECRET - IAM Role with IRSA for the
app-access-saServiceAccount - A domain with DNS pointing to the cluster LoadBalancer (e.g. Cloudflare)
# 1. Cluster-level infra (cert-manager issuers, ESO SecretStore)
kubectl apply -k kubernetes/infra/
# 2. Application
kubectl apply -k kubernetes/overlays/devInfra must be applied first because the Ingress references a
ClusterIssuer.
# Infra application
argocd app create infra \
--repo https://github.com/<you>/DevSecOps-Mega-Project \
--path kubernetes/infra \
--dest-namespace cert-manager \
--dest-server https://kubernetes.default.svc \
--sync-policy automated
# App deployment
argocd app create devsecops-dev \
--repo https://github.com/<you>/DevSecOps-Mega-Project \
--path kubernetes/overlays/dev \
--dest-namespace dev \
--dest-server https://kubernetes.default.svc \
--sync-policy automatedEach component has its own CI Jenkinsfile. The pipeline stages are:
git push
│
▼
Stage 1: Checkout — pull source from Git (dev branch)
Stage 2: Compilation — syntax check all .js files
Stage 3: Gitleaks — scan for leaked secrets
Stage 4: SCA — OWASP Dependency-Check (CVE audit, cached NVD DB via PVC)
Stage 5: SAST — SonarQube static analysis
Stage 6: Quality Gate — fail build if SonarQube gate fails
Stage 7: Docker Build — buildx multi-arch build via BuildKit, push to DockerHub
Stage 8: Trivy Scan — scan pushed image for OS/library CVEs
Stage 9: Deploy — trigger deploy-to-local with BUILD_NUMBER (wait: false)
Stage 4 optimisation: the NVD vulnerability database (369k records) is stored in a 5Gi PVC (
dependency-check-data). First build downloads the full DB (~20 min); every subsequent build only syncs the delta (~30 seconds).
Triggered automatically by Stage 9 of api-ci, or run manually with a build number parameter.
deploy-to-local(API_IMAGE_TAG=<build_number>)
│
▼
Stage 1: Checkout — pull latest manifests from main branch
Stage 2: Update Image Tags — sed-patch kustomization.yaml with new tag
Stage 3: Deploy — kubectl delete old migration job
kubectl apply -k kubernetes/overlays/local
Stage 4: Verify Rollout — rollout status + API /health + /live check
The CD agent uses a minimal pod spec (jenkins/Jenkins-agent-cd.yaml — kubectl only) so
it is not blocked by the ReadWriteOnce NVD PVC and can schedule on any cluster node.
Jenkins agents run as ephemeral Kubernetes pods (defined in jenkins/Jenkins-agent.yaml), each container providing a specific tool:
| Container | Image | Purpose |
|---|---|---|
jnlp |
jenkins/inbound-agent |
JNLP agent — connects back to Jenkins controller |
nodejs |
node:22-alpine |
JavaScript syntax check |
gitleaks |
zricethezav/gitleaks |
Secret scanning |
dependency-check |
owasp/dependency-check |
SCA — CVE audit of npm dependencies |
sonar |
sonarsource/sonar-scanner-cli |
SAST — sends results to SonarQube |
docker-cli |
docker:cli |
Builds and pushes Docker images via BuildKit |
trivy |
aquasec/trivy |
Container image vulnerability scan |
BuildKit runs as a separate Deployment (buildkitd) in the jenkins namespace, providing rootless image builds without a Docker daemon on the node.
git push (dev branch)
↓
api-ci (Stage 1-8) → amitactive2008/api:<BUILD_NUMBER> pushed to DockerHub
↓ Stage 9 (wait: false)
deploy-to-local (Stage 1-4) → kustomization.yaml patched
kubectl apply -k overlays/local
Pods rolling update in local namespace
Stage 6 (waitForQualityGate) polls SonarQube until the analysis result is returned. The Quality Gate passes if no new bugs, vulnerabilities, or code smells exceed the configured thresholds. A failing gate blocks the Docker build stage.
The SonarQube server is pre-configured in Jenkins via JCasC:
- Server name:
mysonarqube - URL:
http://sonarqube-sonarqube.sonarqube.svc.cluster.local:9000(in-cluster) - Token: auto-generated by
deploy.sh, stored as K8s Secretsonarqube-tokenin thejenkinsnamespace
| Environment | Secret Store | Delivery Method | Secret in Pod |
|---|---|---|---|
| Local (Kind) | HashiCorp Vault | External Secrets Operator → K8s Secret | envFrom: secretRef |
| Production (EKS) | AWS Secrets Manager | Secrets Store CSI Driver + IRSA | Files at /mnt/secrets/ |
The production pattern (CSI driver + IRSA) avoids storing secrets as K8s Secrets at all — secrets are mounted directly as files. The local pattern (ESO → K8s Secret) is simpler to operate without AWS IAM, while still demonstrating a real secret management workflow with Vault.
| Concern | Local (Kind) | Production (EKS) |
|---|---|---|
| Cluster | Kind + Podman | AWS EKS |
| Database | MySQL pod (ClusterIP) | AWS RDS (ExternalName Service) |
| Secret store | HashiCorp Vault | AWS Secrets Manager |
| Secret delivery | ESO → K8s Secret (envFrom) | CSI driver → file mount |
| Ingress controller | NGINX (hostPort 80/443) | Traefik (LoadBalancer) |
| TLS | Self-signed (cert-manager) | Let's Encrypt (cert-manager) |
| Domain | app.local via /etc/hosts |
Real domain via Cloudflare DNS |
| Replicas | 1 | 2 |
| CI server | Jenkins at https://jenkins.kind.local |
Jenkins on cluster |
| SAST server | SonarQube at https://sonarqube.kind.local |
SonarQube on cluster |
| GitOps | Manual apply | Argo CD |
- Argo CD-based GitOps workflow
- TLS with cert-manager
- Database migrations via Kubernetes Job
- Terraform for infrastructure provisioning
- Observability stack (metrics & logging)
- Production environment overlay
- CD pipeline automation (GitOps-driven)
The NVD API key in jenkins-credentials is invalid or a placeholder. The Jenkinsfile guards against this: if NVD_API_KEY is empty or placeholder, --nvdApiKey is omitted and the scan proceeds without it.
# Check current value
kubectl get secret jenkins-credentials -n jenkins \
-o jsonpath='{.data.NVD_API_KEY}' | base64 -d && echo
# Set to placeholder to skip the key
kubectl patch secret jenkins-credentials -n jenkins --type='json' \
-p='[{"op":"replace","path":"/data/NVD_API_KEY","value":"'$(echo -n placeholder | base64)'"}]'
kubectl rollout restart statefulset/jenkins -n jenkinswaitForQualityGate() requires a SonarQube webhook to call back to Jenkins with the analysis result. Without it, Jenkins polls until timeout.
https://sonarqube.kind.local/admin/webhooks → Create
Name: jenkins
URL: http://jenkins.jenkins.svc.cluster.local:8080/sonarqube-webhook/
The Makefile builder target previously passed TLS --driver-opt flags for BuildKit, but buildkitd runs without TLS in this setup. This has been fixed — the TLS flags were removed from Makefile.
Occurs when the Helm chart's built-in jcasc-default-config.yaml and a custom configScript both define the jenkins.clouds key. Fixed by removing the kubernetes-cloud configScript from jenkins-values.yaml and letting the Helm default handle it.
SonarQube 25+ removed basic-auth for admin API endpoints. Use a USER_TOKEN for API calls. deploy.sh generates one automatically at install time (before the admin password is changed).
The vault-token secret in the external-secrets namespace has a stale or incorrect token.
kubectl create secret generic vault-token \
--from-literal=token=<vault-root-token> \
-n external-secrets --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment external-secrets -n external-secretsThis project is designed to reflect how real DevOps teams structure, deploy, and evolve systems in production.