🚀 Enterprise GitOps on AWS EKS

📋 Project Overview

A complete, production-ready GitOps implementation demonstrating modern cloud-native practices on AWS. This project showcases enterprise-grade infrastructure automation, multi-cluster management, and comprehensive CI/CD pipelines.

🏗️ Architecture

Multi-Cluster Design

Complete environment isolation with separate EKS clusters for DEV and PROD, each with its own ArgoCD instance, VPC, and security controls.

DEV Cluster Architecture - 2 nodes, single NAT gateway for cost optimization

PROD Cluster Architecture - 2 managed + Karpenter SPOT nodes, dual NAT gateways for high availability

🛠️ Technology Stack

🌐 Live Endpoints

All applications are running live on AWS EKS. Click the links to see them in action!

DEV Cluster (us-east-2)

PROD Cluster (us-east-2)

⚡ Advanced Auto-Scaling Architecture

Production cluster implements a sophisticated multi-layer auto-scaling strategy combining Karpenter for node management, KEDA for event-driven scaling, and HPA for application-level scaling.

1. Karpenter v1.0.5 - Node Auto-Scaling

2. KEDA - Event-Driven Autoscaling

3. HPA - Horizontal Pod Autoscaler

Scaling Flow

🌐 Calico CNI - Advanced Networking

Production cluster uses Calico CNI in policy-only mode alongside AWS VPC CNI. Calico is deployed via Terraform using the Tigera Operator Helm chart.

Current Deployment

📦 Kustomize - Configuration Management

Kustomize is the backbone of our GitOps configuration strategy, enabling environment-specific customizations without duplicating YAML files. ArgoCD natively supports Kustomize, making it the perfect pairing for GitOps deployments.

Repository Structure

gitops-deploy/
├── base/                          # Base Kubernetes manifests
│   ├── deployment.yaml            # Common deployment config
│   ├── service.yaml               # Service definition
│   ├── ingress.yaml               # Ingress template
│   └── kustomization.yaml         # Base kustomization
│
├── overlays/                      # Environment-specific overlays
│   ├── api-dev/                   # API app - DEV environment
│   │   ├── kustomization.yaml     # DEV patches (inline)
│   │   └── external-secret.yaml   # DEV secrets (API keys)
│   │
│   ├── api-prod/                  # API app - PROD environment
│   │   ├── kustomization.yaml     # PROD patches (inline)
│   │   ├── ingress.yaml           # HTTPS with ACM certificate
│   │   ├── hpa.yaml               # HPA (3-11 replicas)
│   │   └── external-secret.yaml   # PROD secrets
│   │
│   ├── demo-dev/                  # Demo app - DEV (no secrets needed)
│   │   └── kustomization.yaml
│   ├── demo-prod/                 # Demo app - PROD
│   │   ├── kustomization.yaml
│   │   ├── ingress.yaml           # HTTPS
│   │   ├── hpa.yaml               # HPA (3-9 replicas)
│   │   └── external-secret.yaml
│   ├── java-dev/                  # Java app - DEV
│   │   ├── kustomization.yaml
│   │   └── external-secret.yaml   # DEV Basic Auth
│   └── java-prod/                 # Java app - PROD
│       ├── kustomization.yaml
│       ├── ingress.yaml           # HTTPS
│       ├── hpa.yaml               # HPA (3-10 replicas)
│       └── external-secret.yaml   # PROD Basic Auth
│
├── apps/                          # ArgoCD Application definitions
│   ├── dev/                       # DEV apps
│   └── prod/                      # PROD apps + platform
│
└── platform/                      # Platform components
    ├── karpenter/                 # NodePool, EC2NodeClass
    ├── kyverno/                   # Policies
    └── network-policies/          # Calico NetworkPolicies

How It Works

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-app-prod
spec:
  source:
    repoURL: https://gitlab.com/devop212/demo-task-api-deploy.git
    targetRevision: main
    path: overlays/api-prod    # Points to Kustomize overlay
  destination:
    server: https://kubernetes.default.svc
    namespace: api-app-prod
  syncPolicy:
    # PRODUCTION: Manual sync only - no auto-sync!
    # Never enable automated sync in production
    syncOptions:
      - CreateNamespace=true

Kustomize Transformations

Example: Image Update Workflow

🎯 App of Apps Pattern

Implemented ArgoCD's "App of Apps" pattern for streamlined cluster management and application lifecycle control.

Architecture

Benefits

Bootstrap Script

Application Organization

✅ ArgoCD Application Status

All 9 ArgoCD applications in the PROD cluster are Synced and Healthy. This required careful configuration of ignoreDifferences to handle Kubernetes field defaulting in Helm charts.

Handling Helm Chart Drift with ignoreDifferences

🔄 GitOps Workflow with ArgoCD & Kustomize

Complete automation from code commit to production deployment, powered by ArgoCD for continuous delivery and Kustomize for configuration management:

Key Features

• Automated DEV Deployments: Changes to main branch automatically deploy to DEV
• Kustomize Overlays: Environment-specific configs without YAML duplication
• ArgoCD Native Support: ArgoCD automatically detects and builds Kustomize apps
• Manual PROD Promotion: Requires explicit approval for production changes
• Multi-Platform Builds: Docker images built for AMD64 and ARM64
• Comprehensive Testing: Unit, integration, and property-based tests
• Zero Downtime: Rolling updates with health checks and PodDisruptionBudgets
• Rollback Capability: Git revert for instant rollback

Environment Sync Policies

🔧 GitLab CI/CD Pipeline Stages

Complete 4-stage pipeline with security scanning, multi-platform builds, and automated GitOps deployment updates:

Why 3 Container Tags?

Security Scanning (GitLab Templates)

Pipeline Configuration

stages:
  - test
  - build
  - update-deploy
  - promote-prod

# Stage 1: Test with security scanning
test:
  stage: test
  image: gradle:8.5-jdk21
  script:
    - ./gradlew test

# Stage 2: Build multi-platform Docker image
build:
  stage: build
  script:
    - docker build -t $IMAGE:$CI_COMMIT_SHORT_SHA
    - docker build -t $IMAGE:latest
    - docker build -t $IMAGE:v1.0.$CI_PIPELINE_IID
    - docker push --all-tags

# Stage 3: Update GitOps deploy repo (DEV)
update-deploy:
  stage: update-deploy
  script:
    - git clone gitops-deploy
    - sed -i "s/newTag:.*/newTag: v1.0.$CI_PIPELINE_IID/"
    - git commit -m "Update to v1.0.$CI_PIPELINE_IID"
    - git push  # Triggers ArgoCD auto-sync

# Stage 4: Manual promotion to PROD
promote-to-prod:
  stage: promote-prod
  when: manual  # Requires click in GitLab UI
  script:
    - # Same as update-deploy but for PROD overlay

🔗 Terraform & Kubernetes Integration

Terraform and Kubernetes are intentionally separated in this architecture - a GitOps best practice that provides blast radius isolation and different approval workflows for infrastructure vs. application changes.

Integration Points

Why This Separation?

How Terraform Outputs Feed ArgoCD

# Terraform creates the cluster and IRSA roles
resource "aws_eks_cluster" "main" {
  name     = "gitops-prod"
  role_arn = aws_iam_role.eks_cluster.arn
  # ... creates EKS control plane
}

# Terraform outputs used by ArgoCD/Kustomize
output "cluster_endpoint" {
  value = aws_eks_cluster.main.endpoint
}

output "efs_csi_role_arn" {
  value = aws_iam_role.efs_csi.arn  # Used in K8s ServiceAccount
}

# ArgoCD then deploys apps to this cluster
# using Kustomize overlays from gitops-deploy repo

🏗️ Infrastructure as Code

All infrastructure is defined in Terraform with a modular, reusable design:

Terraform Modules

Note: ArgoCD is deployed via Helm after cluster creation, not as a Terraform module.

Infrastructure Management:

• Run manually via CLI (not GitOps) - infrastructure changes are infrequent and high-risk
• State in S3 + DynamoDB - remote state with locking
• Requires senior review - all changes need approval before apply

State Management

Cost Breakdown

💾 Persistent Storage with EFS

Production-ready shared storage implementation using AWS EFS for multi-pod access patterns.

Implementation Details

• EFS Filesystem: Encrypted filesystem with lifecycle policies (transition to IA after 30 days)
• Multi-AZ Mount Targets: High availability across us-east-2a and us-east-2b
• EFS CSI Driver: Kubernetes CSI driver (v2.0.7) with IRSA authentication
• Dynamic Provisioning: Automatic access point creation per PVC
• Security: NFS traffic restricted to EKS node security group
• VPC Endpoints: Private API access for EFS, STS, and EC2 services

Storage Architecture

Storage Classes

Benefits Demonstrated

• Multi-Pod Access: Multiple replicas share the same data without conflicts
• HPA Compatible: Auto-scaling works seamlessly with shared storage
• GitOps Managed: All storage configuration in Git, deployed via ArgoCD
• Infrastructure as Code: EFS filesystem and CSI driver managed by Terraform
• Security: IRSA for CSI driver, encrypted filesystem, private VPC endpoints
• Cost Effective: ~$1.50/month for 5GB (first 5GB may be free tier)

Real-World Applications

Technical Implementation

🔒 Security & Compliance

Secrets Management

• AWS Secrets Manager: Centralized secret storage with encryption
• External Secrets Operator: Automatic sync to Kubernetes secrets
• IRSA: IAM Roles for Service Accounts - no long-lived credentials
• KMS Encryption: All secrets encrypted at rest
• Rotation Ready: Infrastructure supports automatic secret rotation

Policy Enforcement with Kyverno v3.x

Network Security

• Private Subnets: EKS nodes run in private subnets with no direct internet access
• VPC Endpoints: Direct AWS service connectivity (SSM, SSM Messages, EC2 Messages) - no NAT dependency
• NAT Gateways: Controlled egress for internet-bound traffic
• Security Groups: Least-privilege network access controls
• Network Policies: Pod-to-pod communication restrictions enforced by Calico

Application Security

• Spring Security: Basic authentication on PROD endpoints (Java app only)
• Non-Root Containers: All containers run as non-root user (UID 1001)
• Read-Only Root Filesystem: Immutable container filesystems
• Dropped Capabilities: Minimal Linux capabilities
• Security Context: Proper security contexts in Deployment manifests

Note: Kyverno policies are in Audit mode - they report violations but don't block deployments. The actual security controls are configured in the Deployment manifests themselves.

Container Image CVE Remediation

CVEs Fixed (January 2026):

Accepted Risks (npm internals - awaiting Node.js update):

📊 Monitoring & Observability with Grafana Cloud

Complete observability stack powered by Grafana Cloud for enterprise-grade monitoring, logging, and alerting across both DEV and PROD clusters.

Metrics Collection with Grafana Alloy

• Grafana Alloy Agents: Deployed as DaemonSet on every node for comprehensive metrics collection
• Kube State Metrics: Kubernetes object state metrics (deployments, pods, services)
• Node Exporter: Node-level system metrics (CPU, memory, disk, network)
• Application Metrics: Spring Boot Actuator endpoints with Prometheus format
• Custom Metrics: Application-specific business metrics

Logs Aggregation with Grafana Loki

• Centralized Logging: All container logs forwarded to Grafana Cloud Loki
• Structured Logging: JSON format for easy parsing and querying
• Log Retention: Configurable retention policies per environment
• LogQL Queries: Powerful query language for log analysis
• Log Correlation: Link logs to traces and metrics for full observability

Dashboards & Alerts

Alerting & Notifications

• Proactive Alerts: CPU/memory thresholds, pod crash loops, deployment failures
• Multi-Channel Notifications: Slack, email, PagerDuty integration
• Alert Grouping: Intelligent alert aggregation to reduce noise
• Runbook Links: Alerts include links to troubleshooting documentation

🧪 Testing Strategy

Comprehensive Test Coverage

Test Types

• Unit Tests: Component-level testing with JUnit
• Property-Based Tests: Universal property validation across random inputs
• Integration Tests: End-to-end API testing
• CI/CD Tests: Automated testing on every commit

Running Tests

CI/CD Automated Testing:

• ✅ Unit tests
• ✅ Property-based tests
• ✅ Integration tests
• ✅ Security scans
• ✅ Container vulnerability scans

✅ Production Features - Currently Implemented

The following enterprise-grade features are already deployed and running in production:

Security & Compliance

• ✅ HTTPS/TLS: ACM certificates with automatic renewal (*.thunk-it.com)
• ✅ Network Policies: Calico CNI with zero-trust pod networking
• ✅ Kyverno: Policy-as-code for admission control and security enforcement
• ✅ External Secrets Operator: AWS Secrets Manager integration
• ✅ IRSA: Pod-level IAM permissions for least privilege access

Auto-Scaling & Performance

• ✅ Karpenter: Fast node provisioning (30-60 seconds vs 3-5 minutes)
• ✅ KEDA: Event-driven pod autoscaling with custom metrics
• ✅ HPA: Horizontal Pod Autoscaler (3-11 replicas per app)
• ✅ Multi-AZ: High availability across availability zones
• ✅ Topology Spread: Pods distributed across AZs via topologySpreadConstraints (ScheduleAnyway)
• ✅ Pod Disruption Budgets: minAvailable: 2 on all prod apps — protects during Karpenter consolidation and node drains

Observability & Monitoring

• ✅ Grafana Cloud: Enterprise SaaS observability platform
• ✅ Grafana Alloy: Unified telemetry collection agents
• ✅ Prometheus Metrics: Application and infrastructure metrics
• ✅ Loki Logs: Centralized log aggregation and analysis
• ✅ Kube State Metrics: Kubernetes cluster state monitoring

GitOps & Configuration Management

• ✅ ArgoCD: Declarative GitOps continuous delivery
• ✅ Kustomize: Template-free configuration management
• ✅ GitLab CI/CD: Automated build, test, and deployment pipelines
• ✅ Multi-Environment: DEV (auto-sync) and PROD (manual approval)

🚀 Future Enhancements - Phase 2

Planned enhancements to further scale and harden the infrastructure:

Advanced Security

• ⏳ WAF: AWS WAF for DDoS protection and rate limiting
• ⏳ GuardDuty: Threat detection and monitoring
• ⏳ CloudTrail: Complete audit logging for compliance
• ⏳ Security Hub: Centralized security findings dashboard

Data & Persistence

• ⏳ RDS Aurora: Multi-AZ PostgreSQL for application data
• ⏳ ElastiCache: Redis for session storage and caching
• ⏳ S3 Versioning: Enhanced backup and recovery capabilities

Resilience & DR

• ✅ Pod Disruption Budgets: minAvailable: 2 on all production apps
• ✅ Topology Spread: Pods distributed across AZs via topologySpreadConstraints
• ⏳ Velero: Automated cluster backups to S3
• ⏳ Multi-Region: DR cluster in us-west-2
• ⏳ Route53 Failover: Automatic regional failover
• ⏳ RTO/RPO: Target < 15 minutes RTO, < 5 minutes RPO

Advanced Features

• ⏳ Service Mesh: Istio for mTLS and advanced traffic management
• ⏳ Canary Deployments: Progressive rollouts with Flagger
• ⏳ CDN: CloudFront for global content delivery
• ⏳ Cost Optimization: Spot instances + Reserved Instances (target 30-50% savings)

🚨 Production Lessons Learned (January 2026)

Real-world production incident that revealed critical configuration requirements for Karpenter, Kyverno, and ArgoCD in private VPC environments.

Key Takeaways

💡 Key Learnings & Best Practices

GitOps Principles

• Git as Single Source of Truth: All configuration in Git, no manual changes
• Declarative Configuration: Describe desired state, let ArgoCD reconcile
• Manual Sync for PROD: Never enable auto-sync in production - all changes require approval
• Observability: ArgoCD provides complete deployment visibility

Multi-Cluster Strategy

• Complete Isolation: Separate clusters prevent blast radius issues
• Independent Scaling: DEV and PROD scale independently
• Cost Optimization: DEV uses cheaper configuration (single NAT, fewer nodes)
• Security: PROD has stricter controls and manual approval gates

Infrastructure as Code

• Modular Design: Reusable Terraform modules for consistency
• Remote State: S3 + DynamoDB for team collaboration
• Separate States: Environment isolation at the state level
• Version Control: All infrastructure changes tracked in Git

🎯 Technical Highlights

📦 Repository Structure & Applications

This project is organized across multiple repositories following GitOps best practices. Each repository serves a specific purpose in the overall architecture:

1. gitops-infra - Infrastructure as Code

2. java-springboot-hello-api - Spring Boot REST API

3. gitops-demo-app - Node.js Express Application

4. gitops-demo-api - Task Manager REST API

5. gitops-deploy - Kubernetes Manifests

🤝 Let's Connect

Interested in discussing this project or exploring how these patterns can be applied to your infrastructure?

📚 Additional Resources

Architecture Diagrams

• 🏗️ Complete Production Architecture (Current State) - Shows Calico, Karpenter, KEDA
• DEV Cluster Architecture Diagram
• Phase 2 Complete Architecture

Documentation

• DEV Cluster Architecture Diagram
• PROD Cluster Architecture Diagram
• Work Flow Diagram

Related Articles

• Backstage Inernal Developer Platform
• GitLab Migration Strategy
• GitOps Portfolio (This Page)