Skip to content
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

63 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DevSecOps Mega Project

A production-inspired, end-to-end DevSecOps project that builds, secures, and deploys a full-stack web application on Kubernetes. It covers the complete lifecycle from source code to a running cluster — including CI pipelines with security scanning, GitOps-based delivery, secrets management, TLS, and database migrations.

The project ships two Kustomize overlays:

  • overlays/dev — targets an AWS EKS cluster with RDS, AWS Secrets Manager, and Traefik
  • overlays/local — targets a local Kind cluster with a MySQL pod, HashiCorp Vault, and NGINX Ingress

Table of Contents


What This Project Is

A three-tier web application (React → Node.js → MySQL) deployed on Kubernetes with a full DevSecOps pipeline. It demonstrates real-world patterns:

Concern Solution
Infrastructure as Code Kustomize (base + overlays)
GitOps delivery Argo CD
CI pipeline Jenkins (pod-based agents on K8s)
Secret Scanning Gitleaks
Dependency vulnerabilities OWASP Dependency-Check
Static code analysis SonarQube
Container scanning Trivy
Secrets at runtime AWS Secrets Manager (prod) / HashiCorp Vault (local)
TLS certificates cert-manager (Let's Encrypt / self-signed)
Database migrations Kubernetes Job (runs on every deploy)

Software Components

Application Layer

Component Technology Description
Frontend React (nginx) Single-page app. Pages: Login, Register, Dashboard. Proxies /api/* to the backend.
Backend API Node.js + Express REST API. Routes: /api/auth, /api/users. Health probes: /live, /health.
Database MySQL 8 Stores users. Schema applied via Sequelize migrations.

Kubernetes Infrastructure

Component Namespace Purpose
NGINX Ingress ingress-nginx Routes external HTTP/HTTPS traffic to frontend and API
cert-manager cert-manager Issues and renews TLS certificates (Let's Encrypt or self-signed)
External Secrets Operator external-secrets Pulls secrets from Vault/AWS SM → creates K8s Secrets
HashiCorp Vault vault Secret store for DB credentials and JWT secret (local)
Migration Job local / dev Runs sequelize db:migrate once per deploy, then exits
Jenkins jenkins CI server with pod-based agents; deployed via jenkins/setup/deploy.sh
SonarQube sonarqube SAST server; deployed alongside Jenkins by deploy.sh
BuildKit jenkins Rootless image builder used by Jenkins docker-cli agent container

CI/CD Tools

Tool Role
Jenkins Orchestrates the pipeline; runs as pods inside Kubernetes
Gitleaks Scans source code for accidentally committed secrets
OWASP Dependency-Check Audits Node.js dependencies for known CVEs
SonarQube SAST — static code analysis and quality gate
Trivy Scans Docker images for OS and library vulnerabilities
Argo CD GitOps controller — syncs cluster state from Git
Kustomize Manages environment-specific Kubernetes manifests

Repository Structure

.
├── api/                        # Node.js backend
│   ├── Dockerfile
│   ├── Jenkinsfile             # CI pipeline for the API
│   ├── routes/                 # authRoutes.js, userRoutes.js
│   ├── controllers/            # authController.js, userController.js
│   ├── models/                 # Sequelize models
│   ├── migrations/             # DB schema migrations
│   └── seeders/                # Initial data (admin user)
│
├── client/                     # React frontend
│   ├── Dockerfile              # Multi-stage: build React → serve with nginx
│   ├── Jenkinsfile             # CI pipeline for the frontend
│   ├── default.conf            # nginx config (proxies /api to backend)
│   └── src/
│       ├── pages/              # Login, Register, Dashboard
│       └── context/            # AuthContext (JWT management)
│
├── kubernetes/
│   ├── base/                   # Shared K8s manifests (no env-specific values)
│   │   ├── api-manifests/      # API Deployment + ClusterIP Service
│   │   ├── client-manifests/   # Client Deployment + ClusterIP Service
│   │   ├── mysql-db/           # ConfigMap, ExternalName Service, Migration Job
│   │   ├── aws-sm/             # ServiceAccount + SecretProviderClass (AWS SM)
│   │   └── ingress/            # Ingress resource (path-based routing)
│   │
│   ├── overlays/
│   │   ├── dev/                # EKS overlay: RDS, AWS SM, Traefik, Let's Encrypt
│   │   └── local/              # Kind overlay: MySQL pod, Vault, nginx, self-signed TLS
│   │
│   └── infra/                  # Cluster-level prereqs (cert-manager issuers, ESO stores)
│
└── jenkins/
    ├── Jenkins-agent.yaml      # CI agent pod template (nodejs, sonar, trivy, buildkitd, etc.)
    ├── Jenkins-agent-cd.yaml   # CD agent pod template (kubectl only, no PVC dependency)
    ├── Jenkinsfile-cd-local    # CD pipeline: deploy CI-built image to local Kind cluster
    └── setup/                  # One-shot deployment for local CI/CD stack
        ├── deploy.sh           # Idempotent script: deploys SonarQube + Jenkins
        ├── jenkins-values.yaml # Jenkins Helm values (JCasC, plugins, ingress)
        ├── sonarqube-values.yaml # SonarQube Community Helm values (ingress, TLS)
        ├── kustomization.yaml  # RBAC, buildkitd, certs stub, NVD PVC
        ├── rbac.yaml           # ClusterRole for Jenkins agent pods + local namespace deploy
        ├── buildkitd.yaml      # BuildKit daemon for rootless image builds
        ├── dependency-check-pvc.yaml  # 5Gi PVC — persists NVD database between builds
        └── credentials-template.yaml  # Template for jenkins-credentials Secret

Traffic Flow in the Cluster

Request Path (Browser → Application)

User Browser
     │
     │  HTTPS  (port 443)
     ▼
┌─────────────────────────────────────────────────────┐
│              Ingress Controller (nginx)              │
│          cert-manager TLS termination                │
│                  host: app.local                     │
└────────────────┬────────────────────────────────────┘
                 │
        ┌────────┴────────┐
        │  Path routing   │
        │                 │
   path: /api/*      path: /
        │                 │
        ▼                 ▼
  ┌──────────┐      ┌──────────────┐
  │api-service│      │client-service│
  │ port 5000 │      │   port 80    │
  └─────┬─────┘      └──────┬───────┘
        │                   │
        ▼                   ▼
  ┌──────────┐      ┌──────────────┐
  │  API Pod  │      │  Client Pod  │
  │ (Node.js) │      │   (nginx)    │
  └─────┬─────┘      └──────────────┘
        │
        ▼
  ┌───────────┐
  │mysql-db   │  ← ClusterIP Service (local: selects MySQL pod)
  │  Service  │    (dev: ExternalName → AWS RDS endpoint)
  └─────┬─────┘
        │
        ▼
  ┌───────────┐
  │  MySQL    │
  │  Pod/RDS  │
  └───────────┘

Secret Delivery Path (Vault → Pod)

HashiCorp Vault
  (secret/app: DB_USER, DB_PASSWORD, JWT_SECRET)
        │
        │  HTTP API (token auth)
        ▼
External Secrets Operator
  (ClusterSecretStore: vault-backend)
        │
        │  creates / refreshes every 1h
        ▼
  K8s Secret: app-secrets  (namespace: local)
        │
        │  envFrom: secretRef
        ▼
  API Pod  +  Migration Job
  (reads DB_USER, DB_PASSWORD, JWT_SECRET as env vars)

TLS Certificate Path

cert-manager ClusterIssuer (selfsigned-issuer / letsencrypt)
        │
        │  issues certificate for app.local / your-domain.com
        ▼
  K8s Secret: app-tls-local  (type: kubernetes.io/tls)
        │
        │  referenced by Ingress spec.tls
        ▼
  Ingress Controller  →  terminates TLS for all inbound HTTPS

Local Deployment on Kind

Prerequisites

brew install kind kubectl helm vault podman
podman machine start

Step 1 — Create the Kind cluster

export KIND_EXPERIMENTAL_PROVIDER=podman
kind create cluster --name vault --config kubernetes/overlays/local/kind-config.yaml

Step 2 — Install cluster infrastructure

# cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl wait --for=condition=Ready pods --all -n cert-manager --timeout=120s

# External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace --wait

# NGINX Ingress (kind-specific manifest — uses hostPort 80/443)
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml
kubectl wait --namespace ingress-nginx --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller --timeout=90s

# HashiCorp Vault
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set "server.dev.enabled=true" \
  --set "server.dev.devRootToken=root" --wait

Step 3 — Configure Vault

# Open a shell into Vault
kubectl exec -it vault-0 -n vault -- /bin/sh

# Inside the pod:
export VAULT_TOKEN=root
export VAULT_ADDR=http://127.0.0.1:8200

# Store secrets (must match MySQL credentials in mysql-deployment.yaml)
vault kv put secret/app \
  DB_USERNAME=appuser \
  DB_PASSWORD=apppass123 \
  JWT_SECRET=localjwtsecret12345678

exit

Step 4 — Create the Vault token Secret for ESO

This is created imperatively to avoid committing secrets to Git.

kubectl create secret generic vault-token \
  --from-literal=token=<your-vault-root-token> \
  -n external-secrets

Step 5 — Pre-load Docker images (bypasses DockerHub TLS issues in kind)

podman pull docker.io/amitactive2008/api:latest
podman pull docker.io/amitactive2008/client:latest1

podman save -o /tmp/api.tar    docker.io/amitactive2008/api:latest
podman save -o /tmp/client.tar docker.io/amitactive2008/client:latest1

export KIND_EXPERIMENTAL_PROVIDER=podman
kind load image-archive /tmp/api.tar    --name vault
kind load image-archive /tmp/client.tar --name vault

Step 6 — Configure /etc/hosts

echo "127.0.0.1 app.local"              | sudo tee -a /etc/hosts
echo "127.0.0.1 jenkins.kind.local"     | sudo tee -a /etc/hosts
echo "127.0.0.1 sonarqube.kind.local"   | sudo tee -a /etc/hosts

Step 7 — Deploy

# Preview rendered manifests
kubectl kustomize kubernetes/overlays/local

# Apply
kubectl apply -k kubernetes/overlays/local

# Watch pods come up
kubectl get pods -n local -w

Expected pod lifecycle:

Pod Status Description
mysql-* Running MySQL starts first, waits for readiness probe
database-migration-* Completed Runs schema migrations + seeds admin user
api-deployment-* Running Starts after DB is ready (readiness probe on /health)
client-deployment-* Running Starts independently

Step 8 — Verify

# All pods healthy
kubectl get pods -n local

# ESO synced the secret from Vault
kubectl get externalsecret app-secrets -n local

# TLS cert issued
kubectl get certificate -n local

# API responding
kubectl exec -n local \
  $(kubectl get pod -l app.kubernetes.io/name=api -n local -o jsonpath='{.items[0].metadata.name}') \
  -- wget -qO- http://localhost:5000/health
# Expected: OK

# Login via ingress
curl -sk -X POST https://app.local/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"[email protected]","password":"admin123"}'
# Expected: {"token":"...","user":{...}}

Open https://app.local in your browser (accept the self-signed cert warning).

Default seeded credentials: [email protected] / admin123

Step 9 — Deploy Jenkins + SonarQube

A single script deploys both tools, wires the SonarQube token into Jenkins credentials, and configures everything via JCasC — no manual UI clicks required.

# Create the credentials secret first (DockerHub + NVD API key)
cp jenkins/setup/credentials-template.yaml jenkins/setup/credentials.yaml
# Edit credentials.yaml and fill in real values, then:
kubectl apply -f jenkins/setup/credentials.yaml

# Deploy SonarQube + Jenkins
./jenkins/setup/deploy.sh

What deploy.sh does, in order:

Step Action
1 Check prerequisites (kubectl, helm, kind)
2 Apply namespace, RBAC, BuildKit via Kustomize
3 Create jenkins-credentials secret (interactive if not present)
4 Pre-load Jenkins image into Kind node (bypasses DockerHub TLS issues in kind)
5 Add Helm repos (Jenkins + SonarQube)
6 Deploy SonarQube Community via Helm with nginx ingress + TLS
7 Generate SonarQube tokens via API (must run before admin password is changed): GLOBAL_ANALYSIS_TOKEN for Jenkins scanner auth; USER_TOKEN for webhook creation
8 Register SonarQube→Jenkins webhook (/sonarqube-webhook/) — required for waitForQualityGate()
9 Deploy Jenkins via Helm with JCasC (clouds, credentials, jobs pre-configured)

After the script completes:

Service URL Credentials
Jenkins https://jenkins.kind.local admin / admin
SonarQube https://sonarqube.kind.local admin / admin (change on first login)

Jenkins comes pre-configured with:

  • Kubernetes cloud — dynamic agent pods (defined in jenkins/Jenkins-agent.yaml)
  • Credentialsdockerhub, NVD_API_KEY, sonarqube-token
  • SonarQube server mysonarqube — authenticated with the auto-generated token
  • Pipeline jobsapi-ci and client-ci pre-created via Job DSL

NVD API key is optional. The NVD_API_KEY credential is used by Stage 4 (OWASP Dependency-Check) to call the NVD API with higher rate limits. If the key is absent or set to placeholder, the --nvdApiKey flag is skipped and dependency-check runs against the public NVD API (rate-limited but functional). Register a free key at https://nvd.nist.gov/developers/request-an-api-key and update the secret:

kubectl patch secret jenkins-credentials -n jenkins --type='json' \
  -p='[{"op":"replace","path":"/data/NVD_API_KEY","value":"'$(echo -n YOUR_KEY | base64)'"}]'
kubectl rollout restart statefulset/jenkins -n jenkins

SonarQube webhook (required for Quality Gate). deploy.sh creates it automatically on fresh installs. If re-deploying or if it fails, create it manually: https://sonarqube.kind.local/admin/webhooksCreate → Name: jenkins, URL: http://jenkins.jenkins.svc.cluster.local:8080/sonarqube-webhook/

Teardown

kubectl delete -k kubernetes/overlays/local
helm uninstall jenkins   -n jenkins
helm uninstall sonarqube -n sonarqube
kind delete cluster --name vault

Production Deployment

Prerequisites

  • AWS EKS cluster
  • AWS RDS MySQL instance
  • AWS Secrets Manager secret with DB_USERNAME, DB_PASSWORD, JWT_SECRET
  • IAM Role with IRSA for the app-access-sa ServiceAccount
  • A domain with DNS pointing to the cluster LoadBalancer (e.g. Cloudflare)

Deployment Order

# 1. Cluster-level infra (cert-manager issuers, ESO SecretStore)
kubectl apply -k kubernetes/infra/

# 2. Application
kubectl apply -k kubernetes/overlays/dev

Infra must be applied first because the Ingress references a ClusterIssuer.

GitOps with Argo CD

# Infra application
argocd app create infra \
  --repo https://github.com/<you>/DevSecOps-Mega-Project \
  --path kubernetes/infra \
  --dest-namespace cert-manager \
  --dest-server https://kubernetes.default.svc \
  --sync-policy automated

# App deployment
argocd app create devsecops-dev \
  --repo https://github.com/<you>/DevSecOps-Mega-Project \
  --path kubernetes/overlays/dev \
  --dest-namespace dev \
  --dest-server https://kubernetes.default.svc \
  --sync-policy automated

CI/CD Pipeline

CI Pipeline (api/Jenkinsfile and client/Jenkinsfile)

Each component has its own CI Jenkinsfile. The pipeline stages are:

git push
    │
    ▼
Stage 1: Checkout          — pull source from Git (dev branch)
Stage 2: Compilation       — syntax check all .js files
Stage 3: Gitleaks          — scan for leaked secrets
Stage 4: SCA               — OWASP Dependency-Check (CVE audit, cached NVD DB via PVC)
Stage 5: SAST              — SonarQube static analysis
Stage 6: Quality Gate      — fail build if SonarQube gate fails
Stage 7: Docker Build      — buildx multi-arch build via BuildKit, push to DockerHub
Stage 8: Trivy Scan        — scan pushed image for OS/library CVEs
Stage 9: Deploy            — trigger deploy-to-local with BUILD_NUMBER (wait: false)

Stage 4 optimisation: the NVD vulnerability database (369k records) is stored in a 5Gi PVC (dependency-check-data). First build downloads the full DB (~20 min); every subsequent build only syncs the delta (~30 seconds).

CD Pipeline (jenkins/Jenkinsfile-cd-local)

Triggered automatically by Stage 9 of api-ci, or run manually with a build number parameter.

deploy-to-local(API_IMAGE_TAG=<build_number>)
    │
    ▼
Stage 1: Checkout           — pull latest manifests from main branch
Stage 2: Update Image Tags  — sed-patch kustomization.yaml with new tag
Stage 3: Deploy             — kubectl delete old migration job
                              kubectl apply -k kubernetes/overlays/local
Stage 4: Verify Rollout     — rollout status + API /health + /live check

The CD agent uses a minimal pod spec (jenkins/Jenkins-agent-cd.yaml — kubectl only) so it is not blocked by the ReadWriteOnce NVD PVC and can schedule on any cluster node.

Jenkins agents run as ephemeral Kubernetes pods (defined in jenkins/Jenkins-agent.yaml), each container providing a specific tool:

Container Image Purpose
jnlp jenkins/inbound-agent JNLP agent — connects back to Jenkins controller
nodejs node:22-alpine JavaScript syntax check
gitleaks zricethezav/gitleaks Secret scanning
dependency-check owasp/dependency-check SCA — CVE audit of npm dependencies
sonar sonarsource/sonar-scanner-cli SAST — sends results to SonarQube
docker-cli docker:cli Builds and pushes Docker images via BuildKit
trivy aquasec/trivy Container image vulnerability scan

BuildKit runs as a separate Deployment (buildkitd) in the jenkins namespace, providing rootless image builds without a Docker daemon on the node.

End-to-End Flow (local Kind cluster)

git push (dev branch)
    ↓
api-ci (Stage 1-8)  →  amitactive2008/api:<BUILD_NUMBER> pushed to DockerHub
    ↓ Stage 9 (wait: false)
deploy-to-local (Stage 1-4)  →  kustomization.yaml patched
                                  kubectl apply -k overlays/local
                                  Pods rolling update in local namespace

SonarQube Quality Gate

Stage 6 (waitForQualityGate) polls SonarQube until the analysis result is returned. The Quality Gate passes if no new bugs, vulnerabilities, or code smells exceed the configured thresholds. A failing gate blocks the Docker build stage.

The SonarQube server is pre-configured in Jenkins via JCasC:

  • Server name: mysonarqube
  • URL: http://sonarqube-sonarqube.sonarqube.svc.cluster.local:9000 (in-cluster)
  • Token: auto-generated by deploy.sh, stored as K8s Secret sonarqube-token in the jenkins namespace

Secrets Architecture

Environment Secret Store Delivery Method Secret in Pod
Local (Kind) HashiCorp Vault External Secrets Operator → K8s Secret envFrom: secretRef
Production (EKS) AWS Secrets Manager Secrets Store CSI Driver + IRSA Files at /mnt/secrets/

Why two approaches?

The production pattern (CSI driver + IRSA) avoids storing secrets as K8s Secrets at all — secrets are mounted directly as files. The local pattern (ESO → K8s Secret) is simpler to operate without AWS IAM, while still demonstrating a real secret management workflow with Vault.


Local vs Production Comparison

Concern Local (Kind) Production (EKS)
Cluster Kind + Podman AWS EKS
Database MySQL pod (ClusterIP) AWS RDS (ExternalName Service)
Secret store HashiCorp Vault AWS Secrets Manager
Secret delivery ESO → K8s Secret (envFrom) CSI driver → file mount
Ingress controller NGINX (hostPort 80/443) Traefik (LoadBalancer)
TLS Self-signed (cert-manager) Let's Encrypt (cert-manager)
Domain app.local via /etc/hosts Real domain via Cloudflare DNS
Replicas 1 2
CI server Jenkins at https://jenkins.kind.local Jenkins on cluster
SAST server SonarQube at https://sonarqube.kind.local SonarQube on cluster
GitOps Manual apply Argo CD
  • Argo CD-based GitOps workflow
  • TLS with cert-manager
  • Database migrations via Kubernetes Job

Planned

  • Terraform for infrastructure provisioning
  • Observability stack (metrics & logging)
  • Production environment overlay
  • CD pipeline automation (GitOps-driven)

Troubleshooting

Stage 4 (SCA) — Invalid API Key error

The NVD API key in jenkins-credentials is invalid or a placeholder. The Jenkinsfile guards against this: if NVD_API_KEY is empty or placeholder, --nvdApiKey is omitted and the scan proceeds without it.

# Check current value
kubectl get secret jenkins-credentials -n jenkins \
  -o jsonpath='{.data.NVD_API_KEY}' | base64 -d && echo

# Set to placeholder to skip the key
kubectl patch secret jenkins-credentials -n jenkins --type='json' \
  -p='[{"op":"replace","path":"/data/NVD_API_KEY","value":"'$(echo -n placeholder | base64)'"}]'
kubectl rollout restart statefulset/jenkins -n jenkins

Stage 6 (Quality Gate) — times out after 1 minute

waitForQualityGate() requires a SonarQube webhook to call back to Jenkins with the analysis result. Without it, Jenkins polls until timeout.

https://sonarqube.kind.local/admin/webhooks → Create
  Name: jenkins
  URL:  http://jenkins.jenkins.svc.cluster.local:8080/sonarqube-webhook/

Stage 7 (Docker Build) — ca.pem: no such file or directory

The Makefile builder target previously passed TLS --driver-opt flags for BuildKit, but buildkitd runs without TLS in this setup. This has been fixed — the TLS flags were removed from Makefile.

Jenkins pod CrashLoopBackOff — JCasC conflict

Occurs when the Helm chart's built-in jcasc-default-config.yaml and a custom configScript both define the jenkins.clouds key. Fixed by removing the kubernetes-cloud configScript from jenkins-values.yaml and letting the Helm default handle it.

SonarQube admin API returns 401 with admin:admin

SonarQube 25+ removed basic-auth for admin API endpoints. Use a USER_TOKEN for API calls. deploy.sh generates one automatically at install time (before the admin password is changed).

ESO SecretSyncedErrorpermission denied on Vault

The vault-token secret in the external-secrets namespace has a stale or incorrect token.

kubectl create secret generic vault-token \
  --from-literal=token=<vault-root-token> \
  -n external-secrets --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment external-secrets -n external-secrets

📌 Final Note

This project is designed to reflect how real DevOps teams structure, deploy, and evolve systems in production.


About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages