--- title: "Architecture Case Study: Migrating a Developer SaaS from Serverless to a $10 VPS with Docker" author: "Sanju Singh" published: "2026-09-26T08:19:40.271Z" updated: "2026-09-26T08:19:40.271Z" url: "https://zyvop.com/architecture-case-study-migrating-a-developer-saas-from-serverless-to-a-10-vps-with-docker-uph37" canonical: "https://zyvop.com/architecture-case-study-migrating-a-developer-saas-from-serverless-to-a-10-vps-with-docker-uph37" tags: ["Docker","nextjs","Architecture","performance","DevOps","self-hosted"] --- # Architecture Case Study: Migrating a Developer SaaS from Serverless to a $10 VPS with Docker How we eliminated cold starts, solved database connection pool exhaustion, reduced P99 latency by 92%, and stabilized infrastructure costs using Docker, Caddy, and a $10 VPS. ![Cover image]() Serverless platforms like Vercel and AWS Lambda are the default choice for modern web applications. For the first few months of building our developer platform, the serverless promise held up: push to `git main`, instant preview branches, and zero server maintenance. Then real traffic arrived. Within weeks, we ran headfirst into **database connection pool exhaustion**, unpredictable **cold-start spikes up to 1.8 seconds**, restrictive execution timeouts, and an unexpected bandwidth billing spike for serving static assets and dynamic OpenGraph images. We decided to migrate our entire production web application to a **$10/month VPS (2 vCPU, 4GB RAM)** running Docker and Caddy. Here is the complete post-mortem, the architecture shift, production Dockerfile configurations, zero-downtime deployment workflows, memory budget allocations, and real benchmark data. * * * ## 1\. The Serverless Wall: Why We Had to Move Serverless is marketed as "infinitely scalable," but it forces applications into an ephemeral, stateless execution model that creates significant hidden friction for content-heavy or data-intensive developer tools. ```mermaid flowchart TD subgraph Serverless Architecture [The Serverless Bottleneck] Req1[Client Request 1] --> F1[Lambda Function 1] Req2[Client Request 2] --> F2[Lambda Function 2] Req3[Client Request 3] --> F3[Lambda Function 3] F1 -- New TCP Handshake --> DB[(PostgreSQL)] F2 -- New TCP Handshake --> DB F3 -- New TCP Handshake --> DB style DB fill:#ff6b6b,stroke:#333,stroke-width:2px,color:#fff end ``` ### The Three Breaking Points: ### A. Connection Pool Exhaustion on Relational Databases Every serverless invocation spins up an isolated Node.js container. Because instances don't share memory, they cannot share a traditional database connection pool. - A burst of 100 concurrent requests created 100 simultaneous TCP + TLS handshakes to our PostgreSQL instance. - Even with external poolers (like PgBouncer or Supabase/Neon connection poolers), HTTP-based pooling added 40–80ms of transaction overhead per request. ### B. The Cold-Start & Memory Penalty When an API route had to parse Markdown into an Abstract Syntax Tree (AST), sanitize HTML, and compute syntax highlighting, the cold start was brutal: - **Warm Serverless Invocation:** ~85ms - **Cold Serverless Invocation (P99):** 1,450ms – 1,820ms - **Memory Constraints:** Increasing function memory from 1024MB to 3008MB trimmed cold-start times, but tripled invocation costs. ### C. Bandwidth & Asset Markup Serverless platforms typically charge **$0.15 to $0.40 per GB** for outbound bandwidth once you exceed initial tiers. For an app generating dynamic OG images (each 200KB–400KB) and syndicating rich technical content, bandwidth quickly became our largest single operational expense. * * * ## 2\. The Target Architecture: The $10 VPS Stack We selected a mid-tier VPS (such as a Hetzner Cloud CX22 or DigitalOcean Droplet: 2 vCPU, 4GB RAM, 40GB NVMe SSD) costing roughly €5 to $10/month. ```mermaid flowchart LR Client([Global Users]) --> CF[Cloudflare CDN / DNS] CF --> Caddy[Caddy Reverse Proxy\nAutomatic TLS + HTTP/3] subgraph Docker Host [Docker Host: $10 VPS with 4GB Swap] Caddy -->|Internal HTTP Proxy| AppA[App Container: Blue\nPort 3001] Caddy -.->|Atomic Config Reload| AppB[App Container: Green\nPort 3002] AppA --> Redis[(Local Redis\nCache & Queues)] AppA --> DB[(PostgreSQL\nPooled Sockets)] end ``` ### The 4GB RAM Budget Allocation Running a full stack on a 4GB VPS requires explicit memory ceilings to prevent the Linux Out-Of-Memory (OOM) killer from terminating your process: | Service | Memory Limit (Hard Cap) | Reserved / Typical Footprint | | --- | --- | --- | | **Next.js Standalone App** | 1,536 MB | 350 MB – 600 MB | | **PostgreSQL (Local / Container)** | 1,024 MB | 250 MB – 450 MB | | **Redis (Cache & Queues)** | 384 MB | 64 MB – 128 MB | | **Caddy Reverse Proxy** | 128 MB | 30 MB – 60 MB | | **Linux OS & System Daemons** | Uncapped | ~350 MB | | **Swap Buffer (Disk NVMe)** | **4,096 MB** | Safety net for build spikes | * * * ## 3\. The Production Multi-Stage Dockerfile The standard `next build` produces large build artifacts with development dependencies. By enabling `output: 'standalone'` in `next.config.js`, Next.js traces imports and bundles only the exact node\_modules needed in production. ### `next.config.js` ```javascript /** @type {import('next').NextConfig} */ const nextConfig = { output: 'standalone', poweredByHeader: false, compress: true, }; module.exports = nextConfig; ``` ### Production `Dockerfile` This Dockerfile solves three common production pitfalls: 1. Missing `libc6-compat` on Alpine (required by `sharp` for image optimization). 2. Unprivileged user permissions for Next.js Incremental Static Regeneration (`.next/cache`). 3. Internal health check probe for zero-downtime rolling deploys. ```dockerfile # Stage 1: Dependency resolution FROM node:20-alpine AS deps RUN apk add --no-cache libc6-compat WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci --ignore-scripts # Stage 2: Application builder FROM node:20-alpine AS builder RUN apk add --no-cache libc6-compat WORKDIR /app COPY --from=deps /app/node_modules ./node_modules COPY . . ENV NEXT_TELEMETRY_DISABLED=1 ENV NODE_ENV=production RUN npm run build # Stage 3: Minimal runner image (~85MB) FROM node:20-alpine AS runner # Install libc6-compat for sharp native bindings on Alpine RUN apk add --no-cache libc6-compat curl WORKDIR /app ENV NODE_ENV=production ENV NEXT_TELEMETRY_DISABLED=1 ENV PORT=3000 ENV HOSTNAME="0.0.0.0" # Create unprivileged system user for security RUN addgroup --system --gid 1001 nodejs && \ adduser --system --uid 1001 nextjs # Copy public static assets and standalone server bundle COPY --from=builder /app/public ./public COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./ COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static # Ensure nextjs user owns the cache directory for ISR updates RUN mkdir -p .next/cache && chown -R nextjs:nodejs .next/cache USER nextjs EXPOSE 3000 # Health check to ensure container is responding before traffic routing HEALTHCHECK --interval=5s --timeout=3s --start-period=5s --retries=3 \ CMD curl -f http://127.0.0.1:3000/api/health || exit 1 CMD ["node", "server.js"] ``` * * * ## 4\. `docker-compose.yml` with Hard Memory Limits ```yaml version: '3.8' services: caddy: image: caddy:2.8-alpine restart: unless-stopped ports: - "80:80" - "443:443" - "443:443/udp" # HTTP/3 QUIC volumes: - ./Caddyfile:/etc/caddy/Caddyfile:ro - caddy_data:/data - caddy_config:/config networks: - internal-net depends_on: - web-blue web-blue: image: app-web:latest build: context: . dockerfile: Dockerfile restart: unless-stopped env_file: - .env.production expose: - "3000" deploy: resources: limits: cpus: '1.50' memory: 1536M reservations: cpus: '0.25' memory: 512M networks: - internal-net web-green: image: app-web:latest restart: "no" env_file: - .env.production expose: - "3000" deploy: resources: limits: cpus: '1.50' memory: 1536M reservations: cpus: '0.25' memory: 512M networks: - internal-net redis: image: redis:7-alpine restart: unless-stopped command: ["redis-server", "--appendonly", "yes", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"] volumes: - redis_data:/data networks: - internal-net deploy: resources: limits: memory: 384M volumes: caddy_data: caddy_config: redis_data: networks: internal-net: driver: bridge ``` * * * ## 5\. Reverse Proxy: Caddyfile vs. Nginx Nginx requires manual Certbot cron jobs, Diffie-Hellman parameter generation, and verbose SSL configuration. Caddy does this automatically out of the box with modern TLS 1.3 defaults and HTTP/3 support. ### `Caddyfile` ```caddy yourdomain.com { # Modern compression encode zstd gzip # Essential Security Headers header { Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" X-Content-Type-Options "nosniff" X-Frame-Options "DENY" Referrer-Policy "strict-origin-when-cross-origin" } # Reverse proxy to the active Next.js container slot reverse_proxy web-blue:3000 { # Active health checks health_uri /api/health health_interval 5s health_timeout 2s # Persistent HTTP keepalive to backend container transport http { keepalive 30s keepalive_idle_conns 100 } } } ``` * * * ## 6\. Zero-Downtime Blue/Green Deployments with Caddy A common mistake with `docker compose --scale web=2` is that Docker Compose scales down from the highest numbered container, which can accidentally terminate the freshly built container rather than the old one. Instead, we use a simple Blue/Green deployment script. It boots the inactive container, waits for its health check to return HTTP 200, updates Caddy's upstream, issues an atomic `caddy reload` (which drops zero active connections), and stops the previous container. ### `scripts/deploy.sh` ```bash #!/usr/bin/env bash set -euo pipefail ACTIVE_TARGET=$(docker ps --format '{{.Names}}' | grep -E 'web-(blue|green)' | head -n 1 || true) if [[ "$ACTIVE_TARGET" == *"web-blue"* ]]; then NEW_TARGET="web-green" OLD_TARGET="web-blue" else NEW_TARGET="web-blue" OLD_TARGET="web-green" fi echo "==> Currently active: ${OLD_TARGET:-none}" echo "==> Deploying new version to: ${NEW_TARGET}" git pull origin main docker compose build "${NEW_TARGET}" docker compose up -d --no-deps "${NEW_TARGET}" echo "==> Waiting for ${NEW_TARGET} to become healthy..." for i in {1..20}; do STATUS=$(docker inspect --format='{{json .State.Health.Status}}' "zyvop-${NEW_TARGET}-1" 2>/dev/null || echo '"starting"') if [ "$STATUS" == '"healthy"' ]; then echo "==> ${NEW_TARGET} is healthy!" break fi if [ "$i" -eq 20 ]; then echo "==> ERROR: Health check timed out on ${NEW_TARGET}. Aborting cutover." docker compose stop "${NEW_TARGET}" exit 1 fi sleep 2 done # Atomically switch Caddy upstream sed -i "s/${OLD_TARGET}:3000/${NEW_TARGET}:3000/g" Caddyfile docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile echo "==> Traffic switched to ${NEW_TARGET}. Stopping ${OLD_TARGET}..." if [ -n "$OLD_TARGET" ]; then docker compose stop "$OLD_TARGET" fi docker image prune -f echo "==> Deployment completed with zero downtime!" ``` * * * ## 7\. The Database Win: From Leased Connections to Persistent Pool Under serverless, every database query required either an HTTP fetch to a serverless driver or opening a new TCP socket: ```typescript // Old Serverless Approach (Ephemeral) // Created on every function cold start import { Pool } from 'pg'; const pool = new Pool({ connectionString: process.env.DATABASE_URL, max: 1, // Kept at 1 to prevent connection pool exhaustion across lambdas }); ``` On our persistent VPS, the Node.js runtime remains alive indefinitely. We configure a **single persistent pool** of 15–20 reusable connections with graceful shutdown handling: ```typescript // New VPS Approach (Persistent Process) import { Pool } from 'pg'; declare global { var __dbPool: Pool | undefined; } export const db = global.__dbPool ?? new Pool({ connectionString: process.env.DATABASE_URL, max: 20, // 20 persistent, pre-warmed connections idleTimeoutMillis: 30000, connectionTimeoutMillis: 2000, }); if (process.env.NODE_ENV !== 'production') { global.__dbPool = db; } // Gracefully drain pool on container shutdown process.on('SIGTERM', async () => { console.log('SIGTERM received: Draining PostgreSQL connection pool...'); await db.end(); process.exit(0); }); ``` ### The Result: - Database connection latency dropped from **~65ms** (TCP + TLS negotiation per request) to **< 1.2ms** (reused local socket connection). * * * ## 8\. Hardening the $10 VPS for Production A common hesitation with self-hosting is maintenance. By following these four steps, our maintenance overhead is less than 15 minutes per month. ### A. The 4GB Swap Buffer (Prevent OOM Spikes) Never run a 4GB VPS without swap. If `npm run build` or traffic spikes exceed 4GB, swap prevents the kernel from panicking: ```bash sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab ``` ### B. Automatic Docker Log Rotation Unbounded container logs will quietly fill your disk. Configure the Docker daemon globally: `/etc/docker/daemon.json`: ```json { "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "3" } } ``` ### C. Firewall (UFW) Only expose ports 80, 443, and your SSH port: ```bash sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow 22/tcp sudo ufw allow 80/tcp sudo ufw allow 443/tcp sudo ufw allow 443/udp sudo ufw enable ``` ### D. Automated Encrypted Backups via S3/R2 A simple cron job runs `pg_dump`, encrypts the archive, and synchronizes to Cloudflare R2 / AWS S3: `scripts/backup.sh`: ```bash #!/usr/bin/env bash set -euo pipefail BACKUP_FILE="/tmp/backup_$(date +%Y%m%d_%H%M%S).sql.gz" docker compose exec -T postgres pg_dump -U postgres app_production | gzip > "$BACKUP_FILE" # Upload to S3-compatible storage (e.g. Cloudflare R2 via rclone) rclone copy "$BACKUP_FILE" remote:backups-bucket/db/ rm -f "$BACKUP_FILE" echo "==> Backup successfully uploaded to offsite storage." ``` * * * ## 9\. Performance & Cost Post-Mortem We ran a load test using `wrk` simulating 50 concurrent connections over 60 seconds hitting an authenticated database-backed endpoint. ### Benchmark Results | Metric | Serverless (Previous Stack) | $10 VPS + Docker (Current Stack) | Improvement | | --- | --- | --- | --- | | **P50 Latency** | 114 ms | **38 ms** | **3.0x faster (66% drop)** | | **P95 Latency** | 412 ms | **82 ms** | **5.0x faster (80% drop)** | | **P99 Latency (Cold Start Spike)** | 1,740 ms | **124 ms** | **14.0x faster (92.8% drop)** | | **Max Throughput** | ~280 req/sec (hit concurrency limits) | **1,150 req/sec** | **4.1x capacity** | | **Failed Requests (504 / 500)** | 1.8% during spikes | **0.00%** | **Zero errors** | ### Monthly Cost Comparison | Expense Item | Serverless Stack | $10 VPS Stack | | --- | --- | --- | | Compute / Invocations | $45.00 (Base Pro + Overage) | $10.00 (Fixed VPS) | | Bandwidth / Egress | $38.50 ($0.15/GB overage) | $0.00 (20TB included) | | External DB Connection Pooler | $15.00 | $0.00 (Native socket pooling) | | Static Asset Hosting / Preview URLs | Included | $0.00 (Cloudflare Free Tier) | | **Total Monthly Spend** | **$98.50+** | **$10.00 flat** | * * * ## 10\. The Decision Matrix: When Should You Stay Serverless? Self-hosting isn't a silver bullet for every project. Here is our practical decision framework: ### Stay on Serverless if: - Your traffic is completely sporadic (e.g., 0 requests for 8 hours, then a 5-minute spike). - You do not have a dedicated engineer with basic Linux terminal comfort. - Your application architecture is entirely stateless (no relational database or using third-party managed HTTP APIs like DynamoDB/Firestore). ### Migrate to a VPS + Docker if: - You use a relational database (PostgreSQL, MySQL) and suffer connection bottlenecks. - You need predictable sub-100ms response times without cold-start jitter. - You run background jobs, WebSockets, or long-running worker processes. - You want a fixed, predictable monthly infrastructure bill regardless of traffic surges. * * * ## Conclusion Serverless got our project off the ground in a weekend. But moving to a modest $10 VPS simplified our database connectivity, eliminated cold starts, and delivered a snappier, more predictable experience for our users. Modern tools like **Docker standalone builds**, **Caddy's automated SSL**, and **Blue/Green atomic config reloads** have removed the traditional friction of managing servers. You don't need Kubernetes to scale a developer SaaS—a single well-tuned Linux box can take you much further than you think.