Architecture Case Study: Migrating a Developer SaaS from Serverless to a $10 VPS with Docker

Serverless platforms like Vercel and AWS Lambda are the default choice for modern web applications. For the first few months of building our developer platform, the serverless promise held up: push to git main, instant preview branches, and zero server maintenance.

Then real traffic arrived.

Within weeks, we ran headfirst into database connection pool exhaustion, unpredictable cold-start spikes up to 1.8 seconds, restrictive execution timeouts, and an unexpected bandwidth billing spike for serving static assets and dynamic OpenGraph images.

We decided to migrate our entire production web application to a $10/month VPS (2 vCPU, 4GB RAM) running Docker and Caddy.

Here is the complete post-mortem, the architecture shift, production Dockerfile configurations, zero-downtime deployment workflows, memory budget allocations, and real benchmark data.

1. The Serverless Wall: Why We Had to Move

Serverless is marketed as "infinitely scalable," but it forces applications into an ephemeral, stateless execution model that creates significant hidden friction for content-heavy or data-intensive developer tools.

flowchart TD
    subgraph Serverless Architecture [The Serverless Bottleneck]
        Req1[Client Request 1] --> F1[Lambda Function 1]
        Req2[Client Request 2] --> F2[Lambda Function 2]
        Req3[Client Request 3] --> F3[Lambda Function 3]
        F1 -- New TCP Handshake --> DB[(PostgreSQL)]
        F2 -- New TCP Handshake --> DB
        F3 -- New TCP Handshake --> DB
        style DB fill:#ff6b6b,stroke:#333,stroke-width:2px,color:#fff
    end

The Three Breaking Points:

A. Connection Pool Exhaustion on Relational Databases

Every serverless invocation spins up an isolated Node.js container. Because instances don't share memory, they cannot share a traditional database connection pool.

• A burst of 100 concurrent requests created 100 simultaneous TCP + TLS handshakes to our PostgreSQL instance.

• Even with external poolers (like PgBouncer or Supabase/Neon connection poolers), HTTP-based pooling added 40–80ms of transaction overhead per request.

B. The Cold-Start & Memory Penalty

When an API route had to parse Markdown into an Abstract Syntax Tree (AST), sanitize HTML, and compute syntax highlighting, the cold start was brutal:

• Warm Serverless Invocation: ~85ms

• Cold Serverless Invocation (P99): 1,450ms – 1,820ms

• Memory Constraints: Increasing function memory from 1024MB to 3008MB trimmed cold-start times, but tripled invocation costs.

C. Bandwidth & Asset Markup

Serverless platforms typically charge $0.15 to $0.40 per GB for outbound bandwidth once you exceed initial tiers. For an app generating dynamic OG images (each 200KB–400KB) and syndicating rich technical content, bandwidth quickly became our largest single operational expense.

2. The Target Architecture: The $10 VPS Stack

We selected a mid-tier VPS (such as a Hetzner Cloud CX22 or DigitalOcean Droplet: 2 vCPU, 4GB RAM, 40GB NVMe SSD) costing roughly €5 to $10/month.

flowchart LR
    Client([Global Users]) --> CF[Cloudflare CDN / DNS]
    CF --> Caddy[Caddy Reverse Proxy\nAutomatic TLS + HTTP/3]
    
    subgraph Docker Host [Docker Host: $10 VPS with 4GB Swap]
        Caddy -->|Internal HTTP Proxy| AppA[App Container: Blue\nPort 3001]
        Caddy -.->|Atomic Config Reload| AppB[App Container: Green\nPort 3002]
        AppA --> Redis[(Local Redis\nCache & Queues)]
        AppA --> DB[(PostgreSQL\nPooled Sockets)]
    end

The 4GB RAM Budget Allocation

Running a full stack on a 4GB VPS requires explicit memory ceilings to prevent the Linux Out-Of-Memory (OOM) killer from terminating your process:

3. The Production Multi-Stage Dockerfile

The standard next build produces large build artifacts with development dependencies. By enabling output: 'standalone' in next.config.js, Next.js traces imports and bundles only the exact node_modules needed in production.

next.config.js

/** @type {import('next').NextConfig} */
const nextConfig = {
  output: 'standalone',
  poweredByHeader: false,
  compress: true,
};

module.exports = nextConfig;

Production Dockerfile

This Dockerfile solves three common production pitfalls:

1. Missing libc6-compat on Alpine (required by sharp for image optimization).

2. Unprivileged user permissions for Next.js Incremental Static Regeneration (.next/cache).

3. Internal health check probe for zero-downtime rolling deploys.

# Stage 1: Dependency resolution
FROM node:20-alpine AS deps
RUN apk add --no-cache libc6-compat
WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci --ignore-scripts

# Stage 2: Application builder
FROM node:20-alpine AS builder
RUN apk add --no-cache libc6-compat
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .

ENV NEXT_TELEMETRY_DISABLED=1
ENV NODE_ENV=production

RUN npm run build

# Stage 3: Minimal runner image (~85MB)
FROM node:20-alpine AS runner
# Install libc6-compat for sharp native bindings on Alpine
RUN apk add --no-cache libc6-compat curl
WORKDIR /app

ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
ENV PORT=3000
ENV HOSTNAME="0.0.0.0"

# Create unprivileged system user for security
RUN addgroup --system --gid 1001 nodejs && \
    adduser --system --uid 1001 nextjs

# Copy public static assets and standalone server bundle
COPY --from=builder /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static

# Ensure nextjs user owns the cache directory for ISR updates
RUN mkdir -p .next/cache && chown -R nextjs:nodejs .next/cache

USER nextjs

EXPOSE 3000

# Health check to ensure container is responding before traffic routing
HEALTHCHECK --interval=5s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://127.0.0.1:3000/api/health || exit 1

CMD ["node", "server.js"]

4. docker-compose.yml with Hard Memory Limits

version: '3.8'

services:
  caddy:
    image: caddy:2.8-alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp" # HTTP/3 QUIC
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - internal-net
    depends_on:
      - web-blue

  web-blue:
    image: app-web:latest
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    env_file:
      - .env.production
    expose:
      - "3000"
    deploy:
      resources:
        limits:
          cpus: '1.50'
          memory: 1536M
        reservations:
          cpus: '0.25'
          memory: 512M
    networks:
      - internal-net

  web-green:
    image: app-web:latest
    restart: "no"
    env_file:
      - .env.production
    expose:
      - "3000"
    deploy:
      resources:
        limits:
          cpus: '1.50'
          memory: 1536M
        reservations:
          cpus: '0.25'
          memory: 512M
    networks:
      - internal-net

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: ["redis-server", "--appendonly", "yes", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru"]
    volumes:
      - redis_data:/data
    networks:
      - internal-net
    deploy:
      resources:
        limits:
          memory: 384M

volumes:
  caddy_data:
  caddy_config:
  redis_data:

networks:
  internal-net:
    driver: bridge

5. Reverse Proxy: Caddyfile vs. Nginx

Nginx requires manual Certbot cron jobs, Diffie-Hellman parameter generation, and verbose SSL configuration. Caddy does this automatically out of the box with modern TLS 1.3 defaults and HTTP/3 support.

Caddyfile

yourdomain.com {
    # Modern compression
    encode zstd gzip

    # Essential Security Headers
    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        Referrer-Policy "strict-origin-when-cross-origin"
    }

    # Reverse proxy to the active Next.js container slot
    reverse_proxy web-blue:3000 {
        # Active health checks
        health_uri /api/health
        health_interval 5s
        health_timeout 2s

        # Persistent HTTP keepalive to backend container
        transport http {
            keepalive 30s
            keepalive_idle_conns 100
        }
    }
}

6. Zero-Downtime Blue/Green Deployments with Caddy

A common mistake with docker compose --scale web=2 is that Docker Compose scales down from the highest numbered container, which can accidentally terminate the freshly built container rather than the old one.

Instead, we use a simple Blue/Green deployment script. It boots the inactive container, waits for its health check to return HTTP 200, updates Caddy's upstream, issues an atomic caddy reload (which drops zero active connections), and stops the previous container.

scripts/deploy.sh

#!/usr/bin/env bash
set -euo pipefail

ACTIVE_TARGET=$(docker ps --format '{{.Names}}' | grep -E 'web-(blue|green)' | head -n 1 || true)

if [[ "$ACTIVE_TARGET" == *"web-blue"* ]]; then
  NEW_TARGET="web-green"
  OLD_TARGET="web-blue"
else
  NEW_TARGET="web-blue"
  OLD_TARGET="web-green"
fi

echo "==> Currently active: ${OLD_TARGET:-none}"
echo "==> Deploying new version to: ${NEW_TARGET}"

git pull origin main
docker compose build "${NEW_TARGET}"
docker compose up -d --no-deps "${NEW_TARGET}"

echo "==> Waiting for ${NEW_TARGET} to become healthy..."
for i in {1..20}; do
  STATUS=$(docker inspect --format='{{json .State.Health.Status}}' "zyvop-${NEW_TARGET}-1" 2>/dev/null || echo '"starting"')
  if [ "$STATUS" == '"healthy"' ]; then
    echo "==> ${NEW_TARGET} is healthy!"
    break
  fi
  if [ "$i" -eq 20 ]; then
    echo "==> ERROR: Health check timed out on ${NEW_TARGET}. Aborting cutover."
    docker compose stop "${NEW_TARGET}"
    exit 1
  fi
  sleep 2
done

# Atomically switch Caddy upstream
sed -i "s/${OLD_TARGET}:3000/${NEW_TARGET}:3000/g" Caddyfile
docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile

echo "==> Traffic switched to ${NEW_TARGET}. Stopping ${OLD_TARGET}..."
if [ -n "$OLD_TARGET" ]; then
  docker compose stop "$OLD_TARGET"
fi

docker image prune -f
echo "==> Deployment completed with zero downtime!"

7. The Database Win: From Leased Connections to Persistent Pool

Under serverless, every database query required either an HTTP fetch to a serverless driver or opening a new TCP socket:

// Old Serverless Approach (Ephemeral)
// Created on every function cold start
import { Pool } from 'pg';

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 1, // Kept at 1 to prevent connection pool exhaustion across lambdas
});

On our persistent VPS, the Node.js runtime remains alive indefinitely. We configure a single persistent pool of 15–20 reusable connections with graceful shutdown handling:

// New VPS Approach (Persistent Process)
import { Pool } from 'pg';

declare global {
  var __dbPool: Pool | undefined;
}

export const db =
  global.__dbPool ??
  new Pool({
    connectionString: process.env.DATABASE_URL,
    max: 20, // 20 persistent, pre-warmed connections
    idleTimeoutMillis: 30000,
    connectionTimeoutMillis: 2000,
  });

if (process.env.NODE_ENV !== 'production') {
  global.__dbPool = db;
}

// Gracefully drain pool on container shutdown
process.on('SIGTERM', async () => {
  console.log('SIGTERM received: Draining PostgreSQL connection pool...');
  await db.end();
  process.exit(0);
});

The Result:

• Database connection latency dropped from ~65ms (TCP + TLS negotiation per request) to < 1.2ms (reused local socket connection).

8. Hardening the $10 VPS for Production

A common hesitation with self-hosting is maintenance. By following these four steps, our maintenance overhead is less than 15 minutes per month.

A. The 4GB Swap Buffer (Prevent OOM Spikes)

Never run a 4GB VPS without swap. If npm run build or traffic spikes exceed 4GB, swap prevents the kernel from panicking:

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

B. Automatic Docker Log Rotation

Unbounded container logs will quietly fill your disk. Configure the Docker daemon globally:

/etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  }
}

C. Firewall (UFW)

Only expose ports 80, 443, and your SSH port:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 443/udp
sudo ufw enable

D. Automated Encrypted Backups via S3/R2

A simple cron job runs pg_dump, encrypts the archive, and synchronizes to Cloudflare R2 / AWS S3:

scripts/backup.sh:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_FILE="/tmp/backup_$(date +%Y%m%d_%H%M%S).sql.gz"
docker compose exec -T postgres pg_dump -U postgres app_production | gzip > "$BACKUP_FILE"

# Upload to S3-compatible storage (e.g. Cloudflare R2 via rclone)
rclone copy "$BACKUP_FILE" remote:backups-bucket/db/
rm -f "$BACKUP_FILE"
echo "==> Backup successfully uploaded to offsite storage."

9. Performance & Cost Post-Mortem

We ran a load test using wrk simulating 50 concurrent connections over 60 seconds hitting an authenticated database-backed endpoint.

Benchmark Results

Monthly Cost Comparison

10. The Decision Matrix: When Should You Stay Serverless?

Self-hosting isn't a silver bullet for every project. Here is our practical decision framework:

Stay on Serverless if:

• Your traffic is completely sporadic (e.g., 0 requests for 8 hours, then a 5-minute spike).

• You do not have a dedicated engineer with basic Linux terminal comfort.

• Your application architecture is entirely stateless (no relational database or using third-party managed HTTP APIs like DynamoDB/Firestore).

Migrate to a VPS + Docker if:

• You use a relational database (PostgreSQL, MySQL) and suffer connection bottlenecks.

• You need predictable sub-100ms response times without cold-start jitter.

• You run background jobs, WebSockets, or long-running worker processes.

• You want a fixed, predictable monthly infrastructure bill regardless of traffic surges.

Conclusion

Serverless got our project off the ground in a weekend. But moving to a modest $10 VPS simplified our database connectivity, eliminated cold starts, and delivered a snappier, more predictable experience for our users.

Modern tools like Docker standalone builds, Caddy's automated SSL, and Blue/Green atomic config reloads have removed the traditional friction of managing servers. You don't need Kubernetes to scale a developer SaaS—a single well-tuned Linux box can take you much further than you think.