Kubernetes Deployment Guide¶

This guide covers deploying Champa Intelligence on Kubernetes with high availability using Patroni for PostgreSQL clustering and Redis Sentinel for cache redundancy.

Architecture Overview¶

The Kubernetes deployment provides a highly available, scalable architecture:

graph TB
    subgraph "Ingress Layer"
        ING[Nginx Ingress Controller]
    end

    subgraph "Application Layer"
        APP1[champa-intelligence-1]
        APP2[champa-intelligence-2]
    end

    subgraph "Caching Layer"
        REDIS1[Redis Master]
        REDIS2[Redis Replica]
        SENT1[Sentinel-1]
        SENT2[Sentinel-2]
        SENT3[Sentinel-3]
    end

    subgraph "Database Layer"
        PAT1[Patroni Master]
        PAT2[Patroni Replica]
        ETCD[etcd Cluster]
    end

    subgraph "Static Files"
        NGINX[Nginx Static Server]
    end

    ING --> NGINX
    ING --> APP1
    ING --> APP2
    APP1 --> PAT1
    APP2 --> PAT1
    APP1 --> REDIS1
    APP2 --> REDIS1
    PAT1 --> ETCD
    PAT2 --> ETCD
    REDIS1 -.-> REDIS2
    SENT1 -.-> REDIS1
    SENT2 -.-> REDIS1
    SENT3 -.-> REDIS1

Hold "Alt" / "Option" to enable pan & zoom

Key Components:

2x Application Pods: Load-balanced Flask/Gunicorn instances
Patroni PostgreSQL Cluster: 2-node HA database with automatic failover
Redis Sentinel: 2 Redis nodes + 3 Sentinel instances for cache HA
etcd: Distributed configuration store for Patroni
Nginx Ingress: L7 load balancing and SSL termination
Nginx Static Server: Efficient static file serving

Prerequisites¶

Required Software¶

Kubernetes Cluster v1.24+
kubectl CLI configured
Docker for building images
Helm (optional, for Nginx Ingress)

Cluster Resources¶

Minimum Requirements:

Component	CPU	Memory	Storage
Application Pods (2x)	400m	512Mi	20Gi
Patroni PostgreSQL (2x)	400m	512Mi	2Gi each
Redis (2x)	200m	256Mi	1Gi each
Redis Sentinel (3x)	100m	128Mi	50Mi each
etcd (1x)	200m	256Mi	500Mi
Nginx Static	100m	64Mi	300Mi
Total	~3.5 cores	~5GB	~30GB

Recommended for Production:

3+ worker nodes for pod distribution
6+ cores total CPU
12GB+ total memory
SSD-backed persistent volumes

External Dependencies¶

Camunda Database: External PostgreSQL with network access from cluster
Google Gemini API Key: For AI analysis features
Domain Name: For ingress (e.g., champa.yourcompany.com)

Deployment Steps¶

Step 1: Prepare Kubernetes Environment¶

# Create namespace (optional)
kubectl create namespace champa-intelligence

# Set default namespace
kubectl config set-context --current --namespace=champa-intelligence

# Verify cluster access
kubectl cluster-info
kubectl get nodes

Step 2: Build and Push Docker Image¶

# Clone repository
git clone https://github.com/your-org/champa-intelligence.git
cd champa-intelligence

# Build frontend assets
npm install
npm run build:prod

# Build Docker image
docker build -t champa-intelligence:k8s .

# Tag and push to registry (if using remote cluster)
docker tag champa-intelligence:k8s your-registry/champa-intelligence:k8s
docker push your-registry/champa-intelligence:k8s

Local Development

For local Kubernetes (Docker Desktop, Minikube, kind), you can use the local image directly:

# For Docker Desktop, images are automatically available
docker build -t champa-intelligence:k8s .

# For Minikube, load the image
minikube image load champa-intelligence:k8s

# For kind, load the image
kind load docker-image champa-intelligence:k8s

Step 3: Configure Secrets¶

Edit k8s/03-config.yaml and encode your secrets:

# Encode secrets with base64
echo -n "your_password" | base64

# Example outputs:
# champa_pass -> Y2hhbXBhX3Bhc3M=
# postgres_pass -> cG9zdGdyZXNfcGFzcw==
# redis_pass -> cmVkaXNfcGFzcw==

Update the secrets section:

apiVersion: v1
kind: Secret
metadata:
  name: champa-secrets
type: Opaque
data:
  SYSTEM_DB_PASSWORD: "Y2hhbXBhX3Bhc3M="           # champa_pass
  POSTGRES_SUPERUSER_PASSWORD: "cG9zdGdyZXNfcGFzcw==" # postgres_pass
  POSTGRES_REPLICATION_PASSWORD: "cmVwX3Bhc3M="    # rep_pass
  REDIS_PASSWORD: "cmVkaXNfcGFzcw=="               # redis_pass
  DB_PASSWORD: "Y2FtdW5kYQ=="                      # your Camunda DB password
  JWT_SECRET: "Z2VuZXJhdGVfMzJfY2hhcl9yYW5kb21fc3RyaW5n"
  APP_SECRET_KEY: "Z2VuZXJhdGVfMzJfY2hhcl9yYW5kb21fc3RyaW5n"
  GOOGLE_API_KEY: "QUl6YVN5RHpVMHFaYUtJTUx4aWZRWUs0bnJqRms1Y1FHX3hPaXR3"
  CAMUNDA_API_PASSWORD: "MCFBQWU1eWZkZ0FzNWFVRA=="

Update ConfigMap with your settings:

apiVersion: v1
kind: ConfigMap
metadata:
  name: champa-config
data:
  # External Camunda Database
  DB_NAME: "camunda"
  DB_USER: "camunda"
  DB_HOST: "10.0.1.50"  # Your Camunda DB host
  DB_PORT: "5432"

  # Camunda REST API
  CAMUNDA_API_USER: "demo"

Step 4: Deploy Infrastructure Components¶

Deploy in order to ensure dependencies are ready:

Deploy Persistent Volumes¶

kubectl apply -f k8s/01-persistent-volumes.yaml

# Verify PVCs
kubectl get pvc

Deploy etcd (for Patroni)¶

kubectl apply -f k8s/01c-etcd.yaml

# Wait for etcd to be ready
kubectl wait --for=condition=ready pod -l app=etcd --timeout=300s

# Verify etcd
kubectl logs -l app=etcd --tail=20

Deploy Patroni RBAC¶

kubectl apply -f k8s/01d-patroni-rbac.yaml

# Verify RBAC
kubectl get serviceaccount patroni-sa
kubectl get role patroni-role
kubectl get rolebinding patroni-rolebinding

Deploy Bootstrap SQL ConfigMap¶

kubectl apply -f k8s/02a3-patroni-bootstrap-sql.yaml

# Verify ConfigMap
kubectl get configmap patroni-bootstrap-sql

Deploy Patroni PostgreSQL Cluster¶

kubectl apply -f k8s/02b-patroni.yaml

# Wait for Patroni pods
kubectl wait --for=condition=ready pod -l app=champa-patroni --timeout=300s

# Check Patroni cluster status
kubectl exec champa-patroni-0 -- patronictl list

# Expected output:
# + Cluster: champa-cluster (7234567890123456789) -----+----+-----------+
# | Member           | Host        | Role    | State   | TL | Lag in MB |
# +------------------+-------------+---------+---------+----+-----------+
# | champa-patroni-0 | 10.1.2.3    | Leader  | running |  1 |           |
# | champa-patroni-1 | 10.1.2.4    | Replica | running |  1 |         0 |
# +------------------+-------------+---------+---------+----+-----------+

Deploy Redis with Sentinel¶

kubectl apply -f k8s/02c-redis-sentinel.yaml

# Wait for Redis
kubectl wait --for=condition=ready pod -l app=champa-redis --timeout=300s
kubectl wait --for=condition=ready pod -l app=redis-sentinel --timeout=300s

# Check Redis replication
kubectl exec champa-redis-0 -- redis-cli -a redis_pass INFO replication

# Check Sentinel status
kubectl exec redis-sentinel-0 -- redis-cli -p 26379 SENTINEL masters

Step 5: Deploy Configuration¶

kubectl apply -f k8s/03-config.yaml

# Verify secrets and configmap
kubectl get secret champa-secrets
kubectl get configmap champa-config

Step 6: Deploy Application¶

kubectl apply -f k8s/04-champa-app.yaml

# Wait for application pods
kubectl wait --for=condition=ready pod -l app=champa-intelligence --timeout=300s

# Check application logs
kubectl logs -l app=champa-intelligence --tail=50 -f

Step 7: Deploy Nginx Static Server¶

kubectl apply -f k8s/05-nginx.yaml

# Wait for Nginx
kubectl wait --for=condition=ready pod -l app=nginx --timeout=120s

# Verify static files were copied
kubectl logs -l app=nginx -c init-static-files

Step 8: Deploy Ingress¶

# First, ensure Nginx Ingress Controller is installed
# If not, install it:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml

# Wait for ingress controller
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s

# Deploy ingress
kubectl apply -f k8s/06-ingress.yaml

# Verify ingress
kubectl get ingress champa-ingress
kubectl describe ingress champa-ingress

Step 9: Configure DNS¶

Add DNS entry pointing to your ingress:

# Get ingress IP
kubectl get ingress champa-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# For local development, add to /etc/hosts:
echo "127.0.0.1 champa.local" | sudo tee -a /etc/hosts

Step 10: Initialize Database¶

# Initialize auth database (run once)
POD_NAME=$(kubectl get pods -l app=champa-intelligence -o jsonpath='{.items[0].metadata.name}')

kubectl exec $POD_NAME -- python -c "
from db import init_auth_db
init_auth_db()
"

Step 11: Access Application¶

Open your browser: http://champa.local (or your configured domain)

Default credentials: - Username: admin - Password: ChampaAdmin135

Change Default Password

Change the default password immediately after first login!

Verification¶

Check All Components¶

# Check all pods
kubectl get pods

# Expected output:
# NAME                                          READY   STATUS    RESTARTS   AGE
# champa-intelligence-deployment-xxx-yyy        1/1     Running   0          5m
# champa-intelligence-deployment-xxx-zzz        1/1     Running   0          5m
# champa-patroni-0                              1/1     Running   0          10m
# champa-patroni-1                              1/1     Running   0          10m
# champa-redis-0                                1/1     Running   0          8m
# champa-redis-1                                1/1     Running   0          8m
# etcd-0                                        1/1     Running   0          15m
# nginx-deployment-xxx-yyy                      1/1     Running   0          3m
# redis-sentinel-0                              1/1     Running   0          8m
# redis-sentinel-1                              1/1     Running   0          8m
# redis-sentinel-2                              1/1     Running   0          8m

# Check services
kubectl get svc

# Check persistent volumes
kubectl get pv,pvc

# Check ingress
kubectl get ingress

Test Health Endpoints¶

# Test application health
curl http://champa.local/health/ping

# Expected: {"status":"OK","timestamp":"..."}

# Test database connectivity
curl http://champa.local/health/db

# Test through ingress
kubectl run -it --rm debug --image=alpine --restart=Never -- sh
apk add curl
curl http://champa-intelligence-svc:8088/health/ping
exit

Verify Database Cluster¶

# Check Patroni cluster
kubectl exec champa-patroni-0 -- patronictl list

# Check which is master
kubectl get pods -l spilo-role=master

# Test database connection
kubectl exec champa-patroni-0 -- psql -U champa_user -d champa_system -c "SELECT version();"

Verify Redis Sentinel¶

# Check Sentinel status
kubectl exec redis-sentinel-0 -- redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster

# Check Redis replication
kubectl exec champa-redis-0 -- redis-cli -a redis_pass INFO replication

# Test from application
kubectl exec -it $(kubectl get pod -l app=champa-intelligence -o jsonpath='{.items[0].metadata.name}') -- python -c "
import redis
from redis.sentinel import Sentinel
sentinel = Sentinel([('redis-sentinel', 26379)], socket_timeout=0.5)
master = sentinel.discover_master('mymaster')
print(f'Master: {master}')
"

Management Operations¶

Scaling Application¶

# Scale application pods
kubectl scale deployment champa-intelligence-deployment --replicas=4

# Verify scaling
kubectl get pods -l app=champa-intelligence

Database Operations¶

Backup PostgreSQL¶

# Create backup
kubectl exec champa-patroni-0 -- pg_dump -U champa_user champa_system > backup.sql

# Or create backup inside pod
kubectl exec champa-patroni-0 -- bash -c "pg_dump -U champa_user champa_system | gzip > /tmp/backup.sql.gz"

# Copy backup out
kubectl cp champa-patroni-0:/tmp/backup.sql.gz ./backup-$(date +%Y%m%d).sql.gz

Restore PostgreSQL¶

# Copy backup to pod
kubectl cp backup.sql.gz champa-patroni-0:/tmp/backup.sql.gz

# Restore
kubectl exec champa-patroni-0 -- bash -c "gunzip < /tmp/backup.sql.gz | psql -U champa_user champa_system"

Trigger Manual Failover¶

# Switch master to replica-1
kubectl exec champa-patroni-0 -- patronictl switchover champa-cluster --master champa-patroni-0 --candidate champa-patroni-1 --force

Redis Operations¶

Backup Redis¶

# Trigger SAVE
kubectl exec champa-redis-0 -- redis-cli -a redis_pass SAVE

# Copy dump file
kubectl cp champa-redis-0:/data/dump.rdb ./redis-backup-$(date +%Y%m%d).rdb

Test Sentinel Failover¶

# Simulate master failure
kubectl exec champa-redis-0 -- redis-cli -a redis_pass DEBUG SLEEP 30

# Watch Sentinel promote replica
kubectl logs -f redis-sentinel-0

View Logs¶

# Application logs
kubectl logs -l app=champa-intelligence --tail=100 -f

# Database logs
kubectl logs champa-patroni-0 --tail=100 -f

# Redis logs
kubectl logs champa-redis-0 --tail=50 -f

# Sentinel logs
kubectl logs redis-sentinel-0 --tail=50 -f

# Ingress logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=100 -f

Troubleshooting¶

Application Won't Start¶

# Check pod status
kubectl describe pod -l app=champa-intelligence

# Check events
kubectl get events --sort-by='.lastTimestamp'

# Check logs
kubectl logs -l app=champa-intelligence --all-containers=true

# Common issues:
# 1. Image pull error
kubectl get pods -l app=champa-intelligence -o jsonpath='{.items[*].status.containerStatuses[*].state}'

# 2. ConfigMap/Secret missing
kubectl get configmap champa-config
kubectl get secret champa-secrets

# 3. Database connection
kubectl exec -it $(kubectl get pod -l app=champa-intelligence -o jsonpath='{.items[0].metadata.name}') -- bash
# Inside pod:
python -c "from config import db_config; print(db_config())"

Patroni Issues¶

# Check Patroni status
kubectl exec champa-patroni-0 -- patronictl list

# Check etcd connectivity
kubectl exec champa-patroni-0 -- patronictl show-config

# Check PostgreSQL logs
kubectl logs champa-patroni-0 --tail=200

# Restart Patroni (will trigger failover if master)
kubectl delete pod champa-patroni-0

Redis Sentinel Issues¶

# Check Sentinel health
for i in 0 1 2; do
  echo "=== Sentinel $i ==="
  kubectl exec redis-sentinel-$i -- redis-cli -p 26379 SENTINEL masters
done

# Check if master is correctly identified
kubectl exec redis-sentinel-0 -- redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster

# Reset Sentinel (if confused)
kubectl exec redis-sentinel-0 -- redis-cli -p 26379 SENTINEL reset mymaster

Ingress Issues¶

# Check ingress controller logs
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller

# Describe ingress
kubectl describe ingress champa-ingress

# Test from inside cluster
kubectl run test --rm -it --image=alpine -- sh
apk add curl
curl http://nginx-svc
curl http://champa-intelligence-svc:8088

Performance Issues¶

# Check resource usage
kubectl top pods

# Check node resources
kubectl top nodes

# Increase resources
kubectl edit deployment champa-intelligence-deployment
# Update resources section

# Check database performance
kubectl exec champa-patroni-0 -- psql -U champa_user -d champa_system -c "
SELECT pid, now() - query_start as duration, query 
FROM pg_stat_activity 
WHERE state = 'active' 
ORDER BY duration DESC;
"

High Availability Testing¶

Test Application HA¶

# Delete one app pod
kubectl delete pod $(kubectl get pod -l app=champa-intelligence -o jsonpath='{.items[0].metadata.name}')

# Application should remain accessible
curl http://champa.local/health/ping

Test Database HA¶

# Find current master
MASTER=$(kubectl get pods -l spilo-role=master -o jsonpath='{.items[0].metadata.name}')
echo "Current master: $MASTER"

# Delete master pod
kubectl delete pod $MASTER

# Wait for failover (15-30 seconds)
sleep 30

# Check new master
kubectl get pods -l spilo-role=master

# Verify application still works
curl http://champa.local/health/db

Test Redis HA¶

# Find current Redis master
kubectl exec redis-sentinel-0 -- redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster

# Simulate master failure
MASTER=$(kubectl exec redis-sentinel-0 -- redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster | head -1)
kubectl exec champa-redis-0 -- redis-cli -a redis_pass DEBUG SLEEP 30

# Watch failover
kubectl logs -f redis-sentinel-0

# Verify new master
kubectl exec redis-sentinel-0 -- redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster

Production Considerations¶

Security¶

Use Secrets Management

# Use Sealed Secrets or external secrets manager
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

Enable TLS

# Update ingress with TLS
spec:
  tls:
  - hosts:
    - champa.yourcompany.com
    secretName: champa-tls-secret

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: champa-network-policy
spec:
  podSelector:
    matchLabels:
      app: champa-intelligence
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: nginx
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: champa-patroni
    ports:
    - protocol: TCP
      port: 5432
  - to:
    - podSelector:
        matchLabels:
          app: champa-redis
    ports:
    - protocol: TCP
      port: 6379

Monitoring¶

Deploy Prometheus

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack

Configure ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: champa-intelligence
spec:
  selector:
    matchLabels:
      app: champa-intelligence
  endpoints:
  - port: http
    path: /health/light/metrics
    interval: 30s
    bearerTokenSecret:
      name: prometheus-token
      key: token

Resource Limits¶

resources:
  requests:
    memory: "512Mi"
    cpu: "200m"
  limits:
    memory: "1Gi"
    cpu: "1000m"

Autoscaling¶

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: champa-intelligence-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: champa-intelligence-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Cleanup¶

Remove Deployment¶

# Delete all resources in order
kubectl delete -f k8s/06-ingress.yaml
kubectl delete -f k8s/05-nginx.yaml
kubectl delete -f k8s/04-champa-app.yaml
kubectl delete -f k8s/03-config.yaml
kubectl delete -f k8s/02c-redis-sentinel.yaml
kubectl delete -f k8s/02b-patroni.yaml
kubectl delete -f k8s/02a3-patroni-bootstrap-sql.yaml
kubectl delete -f k8s/01d-patroni-rbac.yaml
kubectl delete -f k8s/01c-etcd.yaml
kubectl delete -f k8s/01-persistent-volumes.yaml

# Delete persistent volumes (WARNING: DATA LOSS!)
kubectl delete pvc --all

# Delete namespace (if using dedicated namespace)
kubectl delete namespace champa-intelligence

Next Steps¶

Monitoring Setup - Configure Prometheus/Grafana
Backup Strategy - Automated backup procedures
Security Hardening - Production security best practices
Troubleshooting - Common issues and solutions