Not a linter check. Not a regex-based scanner. An actual review — the kind a senior security engineer would do — pointing out SQL injection, hardcoded secrets, command injection, and path traversal bugs, with inline comments on the exact lines that are broken.

I built that. It took a weekend. It costs about $0.003 per review. And it runs on Cloudflare Workers with zero servers to manage.

Let me show you how.

The Problem Nobody Talks About

Here's a dirty secret about most engineering teams: security reviews don't happen.

Oh sure, there's a quarterly pen test. Maybe a SAST tool that generates 400 findings nobody reads. But per-PR, inline, "hey this specific line has a command injection" review? That requires a security engineer looking at every diff. And most teams don't have one. Even the ones that do — they can't review every PR.

Meanwhile, 80%+ of breaches start with a vulnerability that was visible in the code at commit time.

I got tired of this gap. So I built CodeGuardAI — a GitHub App that hooks into your PRs and posts security-focused reviews using Claude as the analysis engine.

Architecture: Embarrassingly Simple

The entire thing is four TypeScript files running on a single Cloudflare Worker:

src/
├── index.ts      # Webhook handler + routing (Hono)
├── github.ts     # GitHub API client (auth, fetch diffs, post reviews)
├── reviewer.ts   # AI review engine (Claude API + diff parsing)
├── types.ts      # TypeScript interfaces
└── (that's it)

The flow:

1. GitHub sends a webhook when a PR is opened or updated

2. Worker verifies the HMAC signature

3. Fetches the PR diff via GitHub API

4. Sends the diff to Claude with a security-focused system prompt

5. Parses the structured JSON response

6. Posts inline review comments back on the PR

No queues. No containers. No Redis. Just a Worker that wakes up, does the job, and goes back to sleep. Cloudflare's waitUntil() handles the async processing so the webhook returns 202 Accepted immediately.

Data layer? A single D1 (SQLite) database tracking installations, reviews, and usage. Three tables. That's the whole backend.

The Interesting Parts

Let me walk through the pieces that actually matter.

1. Webhook Security

Every GitHub webhook comes with an HMAC-SHA256 signature. You must verify it, and you must do it with constant-time comparison (or you're vulnerable to timing attacks):

export async function verifyWebhookSignature(
  payload: string,
  signature: string | null,
  secret: string
): Promise<boolean> {
  if (!signature) return false;

  const sig = signature.startsWith("sha256=")
    ? signature.slice(7)
    : signature;

  const encoder = new TextEncoder();
  const key = await crypto.subtle.importKey(
    "raw",
    encoder.encode(secret),
    { name: "HMAC", hash: "SHA-256" },
    false,
    ["sign"]
  );

  const signed = await crypto.subtle.sign(
    "HMAC", key, encoder.encode(payload)
  );
  const expected = Array.from(new Uint8Array(signed))
    .map((b) => b.toString(16).padStart(2, "0"))
    .join("");

  // Constant-time comparison
  if (sig.length !== expected.length) return false;
  let result = 0;
  for (let i = 0; i < sig.length; i++) {
    result |= sig.charCodeAt(i) ^ expected.charCodeAt(i);
  }
  return result === 0;
}

This runs on the Web Crypto API — no Node.js crypto module needed. Works perfectly in Workers.

2. The System Prompt (Where the Magic Lives)

This is the part I iterated on the most. The system prompt turns a general-purpose LLM into a focused security reviewer:

const SYSTEM_PROMPT = `You are CodeGuardAI, an expert security-focused 
code reviewer. You analyze pull request diffs and identify issues.

Focus areas (in priority order):
1. Security vulnerabilities: SQL injection, XSS, SSRF, path traversal, 
   command injection, prototype pollution, ReDoS
2. Hardcoded secrets: API keys, passwords, tokens, private keys
3. Authentication/Authorization flaws: Missing auth checks, broken 
   access control
4. Race conditions: TOCTOU bugs, unprotected shared state
5. Error handling: Information leakage, missing validation
6. Performance anti-patterns: N+1 queries, unbounded loops

Rules:
- Only comment on ADDED or MODIFIED lines (lines starting with +)
- Be specific — reference the exact code pattern
- Provide a fix suggestion when possible
- Don't flag style/formatting issues
- If the diff looks clean, say so briefly`;

Key decisions:

• Priority ordering matters. Claude respects the hierarchy — it won't waste comments on style when there's a SQL injection.

• "Only comment on added lines" prevents noise from reviewing unchanged context.

• Structured JSON output makes parsing deterministic. No regex extraction of natural language.

• "If clean, say so briefly" prevents the AI from inventing problems to justify its existence.

3. Diff Context Building

You can't just dump the entire repo into the context window. I build a focused diff payload with smart filtering:

function buildDiffContext(
  files: FileChange[], 
  maxChars: number = 80000
): string {
  let context = "";
  let truncated = false;

  for (const file of files) {
    if (isIgnoredFile(file.filename)) continue;

    const fileBlock = `\n--- \({file.filename} (\){file.status}) ---\n` +
                      `${file.patch}\n`;

    if (context.length + fileBlock.length > maxChars) {
      truncated = true;
      break;
    }
    context += fileBlock;
  }

  if (truncated) {
    context += "\n[... additional files truncated ...]\n";
  }
  return context;
}

The isIgnoredFile() function skips lockfiles, sourcemaps, images, build artifacts, and vendored dependencies. No point burning tokens reviewing package-lock.json.

For large PRs (5000+ changed lines), I truncate to the first ~3000 lines and leave a comment telling the author to break up the PR. This is both a cost safeguard and genuinely good advice.

4. Posting Inline Reviews

GitHub's review API is powerful but finicky. You post a "review" with inline comments attached to specific lines:

export async function postReview(
  token: string,
  owner: string,
  repo: string,
  prNumber: number,
  commitSha: string,
  comments: ReviewComment[],
  summary: string
): Promise<void> {
  const body: Record<string, unknown> = {
    commit_id: commitSha,
    body: summary,
    event: "COMMENT",
  };

  if (comments.length > 0) {
    body.comments = comments;
  }

  const resp = await fetch(
    `https://api.github.com/repos/\({owner}/\){repo}/pulls/${prNumber}/reviews`,
    {
      method: "POST",
      headers: {
        Authorization: `token ${token}`,
        Accept: "application/vnd.github+json",
        "User-Agent": "CodeGuardAI/1.0",
      },
      body: JSON.stringify(body),
    }
  );

  // If inline comments fail (line not in diff), retry summary-only
  if (!resp.ok && comments.length > 0 && resp.status === 422) {
    await postReview(token, owner, repo, prNumber, commitSha, [], summary);
    return;
  }
}

The fallback logic is important — GitHub returns 422 if a comment references a line that's not in the diff context. Rather than losing the entire review, we retry with just the summary.

Each comment gets a severity emoji and a formatted body:

const severityEmoji =
  comment.severity === "critical" ? "🚨"
  : comment.severity === "warning" ? "⚠️"
  : "💡";

reviewComments.push({
  path: comment.file,
  line: comment.line,
  side: "RIGHT",
  body: `\({severityEmoji} **\){comment.severity.toUpperCase()}**\n\n${comment.message}`,
});

What It Actually Finds

I tested this against a deliberately vulnerable PR with common security anti-patterns. Here's what CodeGuardAI flagged:

🚨 CRITICAL — Command Injection:

// The PR had this:
const output = execSync(`git log --author=${req.query.author}`);
// CodeGuardAI flagged it immediately with a fix:
// Use execFileSync with argument array instead

🚨 CRITICAL — Path Traversal:

// The PR had this:
const file = fs.readFileSync(`./uploads/${req.params.filename}`);
// CodeGuardAI: "User input in file path without sanitization. 
// An attacker can use ../../../etc/passwd to read arbitrary files."

⚠️ WARNING — Hardcoded Secret:

// The PR had this:
const API_KEY = "sk-proj-abc123...";
// CodeGuardAI: "Hardcoded API key. Use environment variables."

All three findings appeared as inline comments on the exact lines in the PR diff. The summary at the top rated it CRITICAL risk.

The Economics

Let's talk money, because this is where it gets interesting.

A typical PR diff is 200-500 lines. With Claude Sonnet, that's roughly:

• Input tokens: ~2,000-4,000 (system prompt + diff)

• Output tokens: ~500-1,000 (JSON response)

• Cost per review: ~$0.01-0.03

With Claude Haiku, it's even cheaper — around $0.003 per review.

For a team doing 50 PRs/week, that's about \(0.60/week or \)2.50/month in API costs. The Cloudflare Worker free tier handles up to 100K requests/day. D1 is free for 5M reads/day.

Total infrastructure cost for a small team: basically zero.

Compare that to the cost of one security incident that could have been caught in code review.

What I'd Do Differently

After running this for a few weeks, here's what I've learned:

1. Model choice matters. Sonnet is better at catching subtle logic bugs. Haiku is fine for the obvious stuff (hardcoded secrets, injection patterns). I'm considering a tiered approach — Haiku for initial scan, Sonnet for files touching auth/crypto/networking.

2. False positives are the enemy. If the bot cries wolf too often, developers ignore it. The "don't be pedantic" instruction in the system prompt helps, but I'm still tuning.

3. Context window limits hurt. For massive PRs (1000+ files), you can't review everything. The truncation strategy works, but ideally you'd prioritize high-risk files (auth handlers, API routes, database queries) over UI components.

4. GitHub App auth is painful. JWT generation, installation tokens, the whole dance. Once it works, it works. But expect to spend time debugging RSA key formatting.

Try It

CodeGuardAI is live and free for public repos:

• Install: github.com/apps/codeguard-ai

• Landing page: codeguard-ai.nopkt.com

Install it, open a PR, and watch it work. The whole thing is ~600 lines of TypeScript running on the edge.

If you're building AI-powered developer tools, I'd love to hear what you're working on. The combination of LLMs + GitHub webhooks + edge computing is wildly underexplored. We're just scratching the surface.

Built with Claude, Hono, Cloudflare Workers, and D1. Deployed in under 300ms globally.

They're wrong. RAG isn't dead — naive RAG is dead. The pattern of "chunk documents → embed → cosine similarity → stuff into prompt" was always a prototype, not a production system. In 2026, production RAG looks radically different.

This article covers the patterns that separate toy demos from systems that actually work.

Why Naive RAG Fails

The classic RAG pipeline has predictable failure modes:

1. Chunking destroys context — Splitting at 512 tokens breaks paragraphs, separates questions from answers, and loses document structure

2. Embedding similarity ≠ relevance — "How do I reset my password?" and "Password reset policy" have high similarity but serve different intents

3. Top-K retrieval is crude — The 5 most similar chunks aren't necessarily the 5 most useful

4. No query understanding — The raw user query goes straight to vector search with no transformation

Let's fix each of these.

Pattern 1: Semantic Chunking

Instead of fixed-size chunks, split at semantic boundaries:

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

chunker = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=90,
)

chunks = chunker.split_text(document_text)

The semantic chunker computes embeddings for each sentence, then splits where the cosine distance between consecutive sentences exceeds a threshold. Sentences about the same topic stay together.

Contextual Retrieval (Anthropic's Approach)

Prepend each chunk with context about where it fits in the document:

def add_context(chunk: str, full_document: str) -> str:
    """Use an LLM to generate context for each chunk."""
    prompt = f"""Given this document:
{full_document[:2000]}

And this specific chunk:
{chunk}

Write a 2-3 sentence context that explains where this chunk 
fits within the overall document. Be specific."""
    
    context = llm.invoke(prompt)
    return f"CONTEXT: {context}\n\n{chunk}"

This costs more upfront but dramatically improves retrieval accuracy — Anthropic reported 49% fewer retrieval failures.

Pattern 2: Hybrid Search

Vector search alone misses exact matches. BM25 alone misses semantic similarity. Combine them:

from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever
from langchain_community.vectorstores import Qdrant

# Vector retriever
vector_store = Qdrant.from_documents(
    documents, embeddings,
    url="http://localhost:6333",
    collection_name="docs"
)
vector_retriever = vector_store.as_retriever(search_kwargs={"k": 10})

# BM25 retriever  
bm25_retriever = BM25Retriever.from_documents(documents)
bm25_retriever.k = 10

# Combine with Reciprocal Rank Fusion
ensemble = EnsembleRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    weights=[0.6, 0.4],  # Favor semantic for most use cases
)

The EnsembleRetriever uses Reciprocal Rank Fusion (RRF) to merge results. A document ranking #1 in vector search and #3 in BM25 scores higher than one ranking #2 in both.

Adding Knowledge Graphs

For structured relationships (org charts, product hierarchies, dependency trees), add a graph layer:

from neo4j import GraphDatabase

def graph_enhanced_retrieval(query: str, vector_results: list) -> list:
    """Enrich vector results with graph context."""
    driver = GraphDatabase.driver("bolt://localhost:7687")
    
    enriched = []
    for doc in vector_results:
        # Find related entities in the graph
        with driver.session() as session:
            result = session.run("""
                MATCH (n)-[r]-(related)
                WHERE n.name = $entity
                RETURN related.name, type(r), related.description
                LIMIT 5
            """, entity=doc.metadata.get("entity"))
            
            context = [f"{r['type(r)']}: {r['related.name']}" for r in result]
            doc.page_content += f"\n\nRelated: {', '.join(context)}"
            enriched.append(doc)
    
    return enriched

Pattern 3: Re-Ranking

Initial retrieval casts a wide net. Re-ranking uses a cross-encoder to score each (query, document) pair more accurately:

from langchain.retrievers import ContextualCompressionRetriever
from langchain_cohere import CohereRerank

# Retrieve 20 candidates, re-rank to top 5
reranker = CohereRerank(
    model="rerank-english-v3.0",
    top_n=5,
)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=reranker,
    base_retriever=ensemble,  # From hybrid search above
)

results = compression_retriever.invoke("How do I configure SSO?")

Cross-encoders process the query and document together (unlike bi-encoders which encode them separately), enabling much more nuanced relevance scoring. The tradeoff is speed — which is why we re-rank a pre-filtered set rather than the entire corpus.

ColBERT v2: Best of Both Worlds

ColBERT stores per-token embeddings and uses late interaction for scoring, giving near cross-encoder accuracy at near bi-encoder speed:

from ragatouille import RAGPretrainedModel

rag = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
rag.index(
    collection=[doc.page_content for doc in documents],
    document_metadatas=[doc.metadata for doc in documents],
    index_name="my_index",
)

results = rag.search(query="SSO configuration", k=5)

Pattern 4: Query Transformation

Don't send the raw user query to retrieval. Transform it first.

HyDE (Hypothetical Document Embeddings)

Generate a hypothetical answer, then search for documents similar to that answer:

def hyde_retrieval(query: str) -> list:
    # Generate hypothetical answer
    hypothetical = llm.invoke(
        f"Write a short paragraph that would answer: {query}"
    )
    
    # Search using the hypothetical document's embedding
    # This often matches better than the question embedding
    return vector_store.similarity_search(hypothetical, k=5)

Multi-Query Expansion

Generate multiple perspectives on the same question:

from langchain.retrievers.multi_query import MultiQueryRetriever

multi_retriever = MultiQueryRetriever.from_llm(
    retriever=vector_store.as_retriever(),
    llm=llm,
)

# Internally generates 3+ query variants and deduplicates results
results = multi_retriever.invoke("Why is our API slow?")
# Generates: "API performance issues", "latency root causes", 
# "slow response time debugging"

Step-Back Prompting

For specific questions, first ask a broader question:

def step_back_retrieval(query: str) -> list:
    # Generate a more general question
    broader = llm.invoke(
        f"Given this specific question: '{query}'\n"
        f"What is a more general question that would help answer it?"
    )
    
    # Retrieve for both specific and general queries
    specific_docs = vector_store.similarity_search(query, k=3)
    general_docs = vector_store.similarity_search(broader, k=3)
    
    return deduplicate(specific_docs + general_docs)

Pattern 5: Agentic RAG

The biggest evolution: let the LLM decide what to retrieve and when.

from langgraph.graph import StateGraph, END
from typing import TypedDict, Annotated

class AgentState(TypedDict):
    question: str
    documents: list
    answer: str
    needs_more_info: bool

def retrieve(state: AgentState) -> AgentState:
    docs = retriever.invoke(state["question"])
    return {"documents": docs}

def grade_documents(state: AgentState) -> AgentState:
    """Let the LLM decide if retrieved docs are sufficient."""
    grade = llm.invoke(
        f"Question: {state['question']}\n"
        f"Documents: {state['documents']}\n"
        f"Are these documents sufficient to answer the question? "
        f"Reply YES or NO with a brief reason."
    )
    return {"needs_more_info": "NO" in grade.upper()}

def generate(state: AgentState) -> AgentState:
    answer = llm.invoke(
        f"Answer based on these documents:\n"
        f"{state['documents']}\n\n"
        f"Question: {state['question']}"
    )
    return {"answer": answer}

def rewrite_query(state: AgentState) -> AgentState:
    better_query = llm.invoke(
        f"The following question didn't get good search results: "
        f"'{state['question']}'. Rewrite it for better retrieval."
    )
    return {"question": better_query}

# Build the graph
workflow = StateGraph(AgentState)
workflow.add_node("retrieve", retrieve)
workflow.add_node("grade", grade_documents)
workflow.add_node("generate", generate)
workflow.add_node("rewrite", rewrite_query)

workflow.set_entry_point("retrieve")
workflow.add_edge("retrieve", "grade")
workflow.add_conditional_edges(
    "grade",
    lambda s: "rewrite" if s["needs_more_info"] else "generate",
)
workflow.add_edge("rewrite", "retrieve")
workflow.add_edge("generate", END)

app = workflow.compile()

This agent retrieves, evaluates quality, rewrites the query if needed, and only generates when it has sufficient context.

Pattern 6: Evaluation with RAGAS

You can't improve what you don't measure:

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

result = evaluate(
    dataset=eval_dataset,  # Questions + ground truth answers
    metrics=[
        faithfulness,       # Is the answer grounded in retrieved context?
        answer_relevancy,   # Does the answer address the question?
        context_precision,  # Are the retrieved docs relevant?
        context_recall,     # Did we retrieve all necessary info?
    ],
)

print(result)
# {'faithfulness': 0.92, 'answer_relevancy': 0.88, 
#  'context_precision': 0.85, 'context_recall': 0.79}

Track these metrics over time. When you change chunking strategy, embedding model, or retrieval pipeline, you'll know immediately if it helped.

RAG vs. Fine-Tuning vs. Long Context

When to use each:

The sweet spot for most production systems: RAG + selective fine-tuning. Fine-tune for domain language and response style. Use RAG for up-to-date facts and source attribution.

Production Tips

1. Cache aggressively — Cache embeddings, cache LLM re-ranking calls, cache final answers for repeated queries

2. Stream the answer — Start generating as soon as retrieval completes; don't wait for re-ranking if latency matters

3. Monitor retrieval quality — Log which chunks were retrieved and whether users found answers helpful

4. Use metadata filters — Filter by date, department, document type before vector search to reduce noise

5. Implement fallback — If RAG confidence is low, fall back to a direct LLM response with a disclaimer

RAG in 2026 is a pipeline engineering challenge, not a simple API call. But get it right, and you have a system that's accurate, attributable, and cost-effective at scale.

If this article helped you, consider buying me a coffee on Ko-fi! Follow me for more AI engineering content.

Before MCP, every AI integration was a custom snowflake. Want Claude to query your database? Write a custom tool. Want GPT to search your docs? Build another integration. MCP changes this by providing a standard protocol that any AI host can use to connect to any data source.

In this tutorial, we'll build a fully functional MCP server in TypeScript from scratch.

What Is MCP and Why Should You Care?

MCP follows a client-server architecture:

• Host: The application the user interacts with (Claude Desktop, VS Code, your custom app)

• Client: A protocol client inside the host that maintains a 1:1 connection with a server

• Server: Your code that exposes tools, resources, and prompts to the AI

The protocol defines three core primitives:

1. Tools — Functions the AI can call (like API endpoints)

2. Resources — Data the AI can read (like files or database records)

3. Prompts — Reusable prompt templates

Project Setup

mkdir my-mcp-server && cd my-mcp-server
npm init -y
npm install @modelcontextprotocol/sdk zod
npm install -D typescript @types/node
npx tsc --init

Update tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "Node16",
    "moduleResolution": "Node16",
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "declaration": true
  },
  "include": ["src/**/*"]
}

Add to package.json:

{
  "type": "module",
  "bin": {
    "my-mcp-server": "./dist/index.js"
  },
  "scripts": {
    "build": "tsc",
    "start": "node dist/index.js"
  }
}

Building the Server

Create src/index.ts:

#!/usr/bin/env node

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

const server = new McpServer({
  name: "my-mcp-server",
  version: "1.0.0",
});

// --- TOOL 1: Search files by content ---
server.tool(
  "search_files",
  "Search for files containing a specific text pattern",
  {
    pattern: z.string().describe("The text pattern to search for"),
    directory: z.string().default(".").describe("Directory to search in"),
    fileType: z.string().optional().describe("File extension filter (e.g., 'ts', 'py')"),
  },
  async ({ pattern, directory, fileType }) => {
    const { execSync } = await import("child_process");
    
    let cmd = `grep -rl "\({pattern}" "\){directory}" --include="*.${fileType || '*'}" 2>/dev/null | head -20`;
    
    try {
      const result = execSync(cmd, { encoding: "utf-8", timeout: 10000 });
      const files = result.trim().split("\n").filter(Boolean);
      
      return {
        content: [
          {
            type: "text" as const,
            text: files.length > 0
              ? `Found \({files.length} files matching "\){pattern}":\n\({files.map(f => `  - \){f}`).join("\n")}`
              : `No files found matching "\({pattern}" in \){directory}`,
          },
        ],
      };
    } catch {
      return {
        content: [{ type: "text" as const, text: `No matches found for "${pattern}"` }],
      };
    }
  }
);

// --- TOOL 2: Execute SQL query ---
server.tool(
  "query_database",
  "Execute a read-only SQL query against the SQLite database",
  {
    query: z.string().describe("SQL SELECT query to execute"),
    database: z.string().describe("Path to SQLite database file"),
  },
  async ({ query, database }) => {
    // Safety: only allow SELECT queries
    const normalized = query.trim().toUpperCase();
    if (!normalized.startsWith("SELECT")) {
      return {
        content: [{ type: "text" as const, text: "Error: Only SELECT queries are allowed for safety." }],
        isError: true,
      };
    }

    const { execSync } = await import("child_process");
    
    try {
      const result = execSync(
        `sqlite3 -header -csv "\({database}" "\){query}"`,
        { encoding: "utf-8", timeout: 30000 }
      );
      
      return {
        content: [{ type: "text" as const, text: result || "Query returned no results." }],
      };
    } catch (error: any) {
      return {
        content: [{ type: "text" as const, text: `SQL Error: ${error.message}` }],
        isError: true,
      };
    }
  }
);

// --- TOOL 3: HTTP request tool ---
server.tool(
  "http_request",
  "Make an HTTP GET request to a URL and return the response",
  {
    url: z.string().url().describe("The URL to fetch"),
    headers: z.record(z.string()).optional().describe("Optional headers"),
  },
  async ({ url, headers }) => {
    try {
      const response = await fetch(url, {
        headers: headers || {},
        signal: AbortSignal.timeout(15000),
      });
      
      const contentType = response.headers.get("content-type") || "";
      const body = contentType.includes("json")
        ? JSON.stringify(await response.json(), null, 2)
        : await response.text();
      
      return {
        content: [{
          type: "text" as const,
          text: `HTTP \({response.status} \){response.statusText}\n\n${body.slice(0, 5000)}`,
        }],
      };
    } catch (error: any) {
      return {
        content: [{ type: "text" as const, text: `Request failed: ${error.message}` }],
        isError: true,
      };
    }
  }
);

// --- RESOURCE: Expose project configuration ---
server.resource(
  "config",
  "file:///config/app",
  async (uri) => {
    const fs = await import("fs/promises");
    
    try {
      const pkg = JSON.parse(
        await fs.readFile("package.json", "utf-8")
      );
      
      return {
        contents: [
          {
            uri: uri.href,
            mimeType: "application/json",
            text: JSON.stringify({
              name: pkg.name,
              version: pkg.version,
              dependencies: Object.keys(pkg.dependencies || {}),
              scripts: Object.keys(pkg.scripts || {}),
            }, null, 2),
          },
        ],
      };
    } catch {
      return {
        contents: [{
          uri: uri.href,
          mimeType: "text/plain",
          text: "No package.json found in current directory",
        }],
      };
    }
  }
);

// --- Start the server ---
async function main() {
  const transport = new StdioServerTransport();
  await server.connect(transport);
  console.error("MCP server running on stdio");
}

main().catch(console.error);

Connecting to Claude Desktop

Build your server:

npm run build

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "my-server": {
      "command": "node",
      "args": ["/absolute/path/to/my-mcp-server/dist/index.js"],
      "env": {}
    }
  }
}

Restart Claude Desktop. You should see a hammer icon indicating available tools.

Testing Your Server

You can test without Claude Desktop using the MCP Inspector:

npx @modelcontextprotocol/inspector node dist/index.js

This opens a web UI where you can:

• List all available tools and resources

• Call tools with custom arguments

• View the raw JSON-RPC messages

For programmatic testing:

import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

const transport = new StdioClientTransport({
  command: "node",
  args: ["dist/index.js"],
});

const client = new Client({ name: "test-client", version: "1.0.0" });
await client.connect(transport);

// List tools
const tools = await client.listTools();
console.log("Available tools:", tools);

// Call a tool
const result = await client.callTool({
  name: "search_files",
  arguments: { pattern: "TODO", directory: ".", fileType: "ts" },
});
console.log("Result:", result);

Production Considerations

Authentication

MCP doesn't include authentication in the base protocol. For remote servers, wrap your transport:

// Validate API key from environment
const apiKey = process.env.MCP_API_KEY;
if (!apiKey) {
  throw new Error("MCP_API_KEY required");
}

// For HTTP transport (SSE), add auth middleware
server.tool("protected_tool", "...", {}, async (args, extra) => {
  // The 2026 MCP roadmap includes OAuth 2.1 support
  // For now, use environment-based auth
  // ...
});

Error Handling

Always return structured errors:

server.tool("risky_operation", "...", { input: z.string() }, async ({ input }) => {
  try {
    const result = await doSomethingRisky(input);
    return { content: [{ type: "text", text: result }] };
  } catch (error: any) {
    return {
      content: [{ type: "text", text: `Operation failed: ${error.message}` }],
      isError: true, // This tells the AI the tool call failed
    };
  }
});

Rate Limiting

Protect expensive operations:

const callCounts = new Map<string, { count: number; resetAt: number }>();

function rateLimit(toolName: string, maxPerMinute: number): boolean {
  const now = Date.now();
  const entry = callCounts.get(toolName);
  
  if (!entry || now > entry.resetAt) {
    callCounts.set(toolName, { count: 1, resetAt: now + 60000 });
    return true;
  }
  
  if (entry.count >= maxPerMinute) return false;
  entry.count++;
  return true;
}

What's Next for MCP?

The 2026 MCP roadmap includes exciting developments:

• OAuth 2.1 integration for standardized auth

• Streamable HTTP transport replacing SSE for better reliability

• Agent-to-Agent (A2A) protocol interoperability

• Elicitation — servers can ask users for input mid-operation

• Annotations — rich metadata on tool responses

MCP is rapidly becoming the standard way AI interacts with the world. Building MCP servers now positions you at the forefront of the agentic AI ecosystem.

The complete source code is available as a template you can fork and customize for your own data sources.

If this article helped you, consider buying me a coffee on Ko-fi! Follow me for more AI engineering content.

Database migrations are the most dangerous routine operation in backend engineering. Your code deploys are (hopefully) stateless and reversible. Your database changes are neither. They mutate persistent state that your entire system depends on, and getting them wrong can mean downtime, data loss, or both.

This article covers the patterns, tooling, and discipline required to run migrations in production without breaking things.

Why Migrations Are Scary

The fundamental problem is simple: schema changes and code changes don't deploy atomically. There is always a window where your running application code and your database schema are out of sync. During a typical rolling deployment:

1. Migration runs, changing the schema

2. Old application code is still running against the new schema

3. New application code starts rolling out

4. Both old and new code run simultaneously against the new schema

5. Old code finishes draining

Steps 2-4 are where things break. If your migration renames a column, old code looking for the old name throws errors. If it drops a column, same problem. If it adds a NOT NULL column without a default, old code inserting rows fails.

The second problem is locking. Most ALTER TABLE operations in traditional databases acquire locks that block reads, writes, or both. On a table with millions of rows, a lock held for even a few seconds can cascade into connection pool exhaustion, request timeouts, and a full outage.

The third problem is irreversibility. You can roll back a code deploy in seconds. Rolling back a migration that dropped a column means restoring from backup — if you even have a recent one that's consistent.

Zero-Downtime Migration: The Core Principle

The rule is straightforward: at every point during the migration and deployment process, all running code must be compatible with the current database schema.

This means:

• Never rename a column in a single step

• Never drop a column that running code still references

• Never add a NOT NULL constraint without a default

• Never assume the migration and code deploy happen simultaneously

Every migration must be backward compatible with the currently deployed code and forward compatible with the code about to be deployed.

The Expand-Contract Pattern

This is the most important pattern in zero-downtime migrations. Instead of making a breaking change in one step, you split it into three phases:

Phase 1: Expand

Add the new structure alongside the old one. Both old and new code work.

-- Migration: Add new column (nullable, so old code can still INSERT)
ALTER TABLE orders ADD COLUMN status_v2 VARCHAR(50);

Phase 2: Migrate

Deploy code that writes to both old and new structures. Backfill existing data.

-- Backfill in batches (more on this later)
UPDATE orders SET status_v2 = status WHERE status_v2 IS NULL
  AND id BETWEEN \(start AND \)end;

Phase 3: Contract

Once all code uses the new structure and all data is migrated, remove the old one.

-- Only after all application code stops reading `status`
ALTER TABLE orders DROP COLUMN status;
ALTER TABLE orders RENAME COLUMN status_v2 TO status;

Each phase is a separate migration tied to a separate code deploy. Phase 1 goes out with or before the code that starts using the new column. Phase 3 goes out only after you've verified all services have been updated and the old column is truly unused.

Real-world example: renaming a column

Renaming user_name to display_name on a users table with 10M rows:

Deploy 1: Migration adds `display_name` column
Deploy 2: Code writes to both columns, reads from `display_name` with fallback to `user_name`
Deploy 3: Backfill script copies remaining data
Deploy 4: Code reads only from `display_name`
Deploy 5: Migration drops `user_name`

Yes, that's five deploys for a column rename. That's the cost of zero downtime. In practice, deploys 1-2 often ship together, and 4-5 ship together, so it's usually three deploy cycles.

Handling Large Table Alterations

On PostgreSQL, many ALTER TABLE operations are fast because they only update catalog metadata:

-- These are ~instant in PostgreSQL, regardless of table size
ALTER TABLE orders ADD COLUMN notes TEXT;
ALTER TABLE orders ALTER COLUMN notes SET DEFAULT '';
ALTER TABLE orders DROP COLUMN old_field;

But some operations are not:

-- These rewrite the entire table or scan all rows
ALTER TABLE orders ALTER COLUMN amount TYPE BIGINT;  -- full rewrite
ALTER TABLE orders ADD COLUMN verified BOOLEAN NOT NULL DEFAULT true;
-- (instant in PG 11+ but rewrites in older versions)
CREATE INDEX ON orders (customer_id);  -- full table scan

For type changes on large tables, use the expand-contract pattern: add a new column with the desired type, backfill, switch reads, drop the old column.

For index creation, always use CONCURRENTLY:

-- Blocks writes:
CREATE INDEX idx_orders_customer ON orders (customer_id);

-- Does NOT block writes (takes longer, but safe):
CREATE INDEX CONCURRENTLY idx_orders_customer ON orders (customer_id);

Important caveat: CREATE INDEX CONCURRENTLY cannot run inside a transaction. Most migration tools wrap each migration in a transaction by default. You need to either disable that behavior for this specific migration or use a tool that handles it.

In golang-migrate:

-- +migrate: no-transaction
CREATE INDEX CONCURRENTLY idx_orders_customer ON orders (customer_id);

In Flyway, you would set executeInTransaction=false on the migration.

Advisory Locks and Migration Safety

When running migrations in a horizontally scaled environment, you need to ensure only one instance runs migrations at a time. Most tools handle this with advisory locks:

-- golang-migrate uses PostgreSQL advisory locks
SELECT pg_advisory_lock(12345);
-- ... run migrations ...
SELECT pg_advisory_unlock(12345);

This prevents two pods starting simultaneously from both trying to run the same migration and corrupting the schema history.

Migration Tooling

golang-migrate

Straightforward, language-agnostic, uses plain SQL files:

migrations/
  000001_create_users.up.sql
  000001_create_users.down.sql
  000002_add_orders.up.sql
  000002_add_orders.down.sql

Run from CLI or embedded in your Go application:

import "github.com/golang-migrate/migrate/v4"

m, err := migrate.New(
    "file://migrations",
    "postgres://localhost:5432/mydb?sslmode=disable",
)
if err != nil {
    log.Fatal(err)
}

if err := m.Up(); err != nil && err != migrate.ErrNoChange {
    log.Fatal(err)
}

Strengths: simple, no DSL to learn, works with any language's deployment pipeline. Weaknesses: no built-in support for non-transactional migrations, limited state tracking.

Flyway

JVM-based, more opinionated, supports versioned and repeatable migrations:

-- V1__Create_users.sql
CREATE TABLE users (
    id BIGSERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- V2__Add_orders.sql
CREATE TABLE orders (
    id BIGSERIAL PRIMARY KEY,
    user_id BIGINT NOT NULL REFERENCES users(id),
    total_cents BIGINT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Flyway tracks state in a flyway_schema_history table and supports callbacks, placeholders, and Java-based migrations for complex logic. It's the standard in Java/Kotlin ecosystems.

Other Notable Tools

• Alembic (Python/SQLAlchemy): generates migrations from model diffs, good for Python shops

• Sqitch: dependency-based rather than version-ordered, powerful but steeper learning curve

• Atlas (by Ariga): declarative schema management, computes diffs automatically, gaining traction in Go ecosystems

• pg_partman / pgloader: for partition management and bulk data loading, respectively

Choosing a Tool

Pick based on your ecosystem and complexity:

Data Backfills Done Right

Backfilling data across millions of rows is where most migration disasters happen. The naive approach:

-- DO NOT DO THIS
UPDATE orders SET status_v2 = compute_new_status(status);

This acquires a lock on every row, generates enormous WAL (write-ahead log) volume, and can take hours while blocking other writes.

Instead, backfill in batches with throttling:

import time
import psycopg2

BATCH_SIZE = 5000
SLEEP_SECONDS = 0.1  # throttle to reduce replication lag

conn = psycopg2.connect(dsn)
conn.autocommit = True

cursor = conn.cursor()
cursor.execute("SELECT MIN(id), MAX(id) FROM orders")
min_id, max_id = cursor.fetchone()

current = min_id
while current <= max_id:
    cursor.execute("""
        UPDATE orders
        SET status_v2 = compute_new_status(status)
        WHERE id >= %s AND id < %s
          AND status_v2 IS NULL
    """, (current, current + BATCH_SIZE))

    updated = cursor.rowcount
    print(f"Batch {current}-{current + BATCH_SIZE}: updated {updated} rows")

    current += BATCH_SIZE
    time.sleep(SLEEP_SECONDS)

Key practices:

• Batch by primary key range, not LIMIT/OFFSET (which gets slower as offset grows)

• Sleep between batches to let replicas catch up and avoid saturating I/O

• Make it idempotent (WHERE status_v2 IS NULL) so you can restart safely if it fails midway

• Monitor replication lag during the backfill and pause if it exceeds your threshold

• Run during low-traffic windows when possible, even if it's technically safe at peak

For truly massive tables (billions of rows), consider doing the backfill at the application level: update the new column whenever a row is naturally read or written, and run the batch backfill for the long tail of untouched rows.

Blue-Green Database Deployments

Blue-green deployments for databases are harder than for stateless application servers, but the pattern exists and works well for certain scenarios.

The idea: maintain two database schemas (or databases) — blue (current) and green (next). Migrate green, point traffic to it, keep blue as a rollback target.

Schema-Level Blue-Green

-- Blue schema (current)
CREATE SCHEMA blue;
-- ... tables in blue schema

-- Green schema (next version)
CREATE SCHEMA green;
-- ... migrated tables in green schema

-- Application config points to schema
SET search_path = 'blue';  -- current
SET search_path = 'green'; -- after cutover

This works when your migration is a large structural change that's hard to do incrementally. You build the new schema, backfill it from the old one, then switch the application's search_path.

Limitations

True blue-green for databases requires either:

• A brief write-freeze during cutover (seconds, not minutes)

• Dual-write during the transition period

• Logical replication between old and new schemas

For most teams, the expand-contract pattern is more practical than blue-green for databases. Reserve blue-green for major version upgrades (e.g., PostgreSQL 14 to 16) where you set up logical replication between the old and new clusters.

Rollback Strategies

Down Migrations

Every migration tool supports down migrations. Write them:

-- 000005_add_verified_column.up.sql
ALTER TABLE users ADD COLUMN verified BOOLEAN DEFAULT false;

-- 000005_add_verified_column.down.sql
ALTER TABLE users DROP COLUMN verified;

But understand their limits. A down migration that drops a column destroys data. If you added the column, backfilled it over three days, and then need to roll back, that data is gone.

Forward-Only Rollbacks

A safer pattern: instead of rolling back the migration, roll forward with a new migration that undoes the change:

000005_add_verified_column.up.sql    -- adds column
000006_remove_verified_column.up.sql -- removes it (if needed)

This keeps the migration history linear and auditable. Many teams mandate forward-only migrations in production while keeping down migrations for local development.

Point-In-Time Recovery

For catastrophic failures, you need PITR:

# PostgreSQL continuous archiving
archive_mode = on
archive_command = 'cp %p /archive/%f'

# Restore to a specific timestamp
recovery_target_time = '2025-03-15 14:30:00 UTC'

Test your PITR process regularly. The worst time to discover your backups don't work is during an incident.

The Migration Rollback Matrix

Testing Migrations

Against Production-Like Data

Your test database with 50 rows will not reveal the problems that show up with 50 million rows. Test against a copy of production data:

# Snapshot production (use your cloud provider's snapshot feature)
# Restore to a test instance
# Run the migration
# Measure: time, locks held, WAL generated, replication lag

Schema Diffing

After running migrations on a staging environment, diff the resulting schema against what you expect:

# Using pg_dump to compare schemas
pg_dump --schema-only production_db > prod_schema.sql
pg_dump --schema-only staging_db > staging_schema.sql
diff prod_schema.sql staging_schema.sql

Atlas has this built in:

atlas schema diff \
  --from "postgres://localhost/production" \
  --to "postgres://localhost/staging"

CI Pipeline Integration

Run migrations in CI against a fresh database and a database with the previous version's schema:

# .github/workflows/migrations.yml
migration-test:
  services:
    postgres:
      image: postgres:16
      env:
        POSTGRES_DB: test
        POSTGRES_PASSWORD: test
  steps:
    - uses: actions/checkout@v4
    - name: Run all migrations from scratch
      run: migrate -path ./migrations -database "$DB_URL" up
    - name: Verify schema matches expectations
      run: ./scripts/verify-schema.sh
    - name: Test rollback of latest migration
      run: migrate -path ./migrations -database "$DB_URL" down 1
    - name: Re-apply latest migration
      run: migrate -path ./migrations -database "$DB_URL" up

Statement-Level Analysis

Before running a migration in production, analyze what it will actually do:

-- Check if an ALTER TABLE will rewrite the table
-- (PostgreSQL-specific: check pg_catalog after the change)

-- Estimate lock duration using pg_stat_activity during staging test
SELECT pid, wait_event_type, wait_event, query
FROM pg_stat_activity
WHERE wait_event_type = 'Lock';

A Migration Checklist

Before every production migration:

1. Is it backward compatible? Can the currently deployed code work with the new schema?

2. Is it forward compatible? Can the about-to-be-deployed code work with the old schema (in case you need to roll back the code deploy)?

3. Does it hold locks? If so, for how long? On how many rows?

4. Has it been tested against production-sized data?

5. Is there a rollback plan? Down migration, forward fix, or PITR?

6. Is the backfill batched and idempotent?

7. Is the migration wrapped in a transaction? Should it be? (CREATE INDEX CONCURRENTLY cannot be.)

8. Has someone else reviewed it? Migration PRs deserve as much scrutiny as application code.

Summary

Database migrations will always carry risk. The goal is not to eliminate risk but to reduce it to a level where deploying a migration is routine rather than terrifying. The core practices:

• Use expand-contract for every breaking schema change

• Batch and throttle data backfills

• Test against production-scale data

• Write rollback plans before you need them

• Use CONCURRENTLY for index operations

• Keep migrations small, frequent, and reviewable

The teams that migrate databases confidently are not the ones with the best tools. They are the ones with the most disciplined processes.

Building resilient backend systems? This is article #17 in the Production Backend Patterns series. Follow for the next one.

If you found this useful, consider supporting my work:

Every AI tutorial shows you a chatbot that answers questions. That's not an agent. An agent decides what to do, takes action, observes the result, and adapts. In production, it does all of that reliably, with audit trails, error recovery, and human oversight.

LangGraph — the graph-based orchestration layer from LangChain — has quietly become the framework of choice for teams shipping real agents. Uber routes support workflows through it. LinkedIn uses it for internal knowledge agents. Klarna runs customer-facing agents on it at scale.

This article is the guide I wish I had when I moved from prototype to production. We'll build a Research Assistant agent end-to-end, covering every pattern that matters when uptime counts.

When to Use Agents (and When Not To)

Before writing a single line of agent code, ask yourself: does this task require dynamic decision-making?

Use agents when:

• The number of steps is unknown at design time

• The task requires selecting from multiple tools based on context

• Intermediate results change the execution path

• You need autonomous error recovery

Don't use agents when:

• A fixed pipeline (prompt ‚Üí LLM ‚Üí output) solves the problem

• You can enumerate all paths in advance (use a simple chain)

• Latency budget is under 2 seconds (agents loop; loops are slow)

• The cost of a wrong autonomous action is high and you can't add human checkpoints

Agents add complexity. A well-designed chain with structured outputs will outperform a poorly-designed agent every time. Start with the simplest approach that works, then graduate to agents when you hit the wall.

LangGraph Core Concepts

LangGraph models agent logic as a directed graph where:

• State is a typed dictionary that flows through the graph

• Nodes are functions that read and write state

• Edges connect nodes (static or conditional)

• Conditional edges inspect state and route to different nodes

Here's the minimal mental model:

from langgraph.graph import StateGraph, START, END
from typing import TypedDict, Annotated
from operator import add

class AgentState(TypedDict):
    messages: Annotated[list, add]  # append-only message list
    step_count: int

def process(state: AgentState) -> dict:
    return {"messages": ["processed"], "step_count": state["step_count"] + 1}

def should_continue(state: AgentState) -> str:
    return "end" if state["step_count"] >= 3 else "process"

graph = StateGraph(AgentState)
graph.add_node("process", process)
graph.add_conditional_edges(START, should_continue, {"process": "process", "end": END})
graph.add_conditional_edges("process", should_continue, {"process": "process", "end": END})

app = graph.compile()
result = app.invoke({"messages": [], "step_count": 0})

The Annotated[list, add] is critical ‚Äî it tells LangGraph to merge list returns instead of overwriting. Without it, each node would clobber the previous messages.

Building the Research Assistant

Let's build something real: an agent that takes a research question, searches the web, reads and summarizes relevant pages, and produces a structured report. This is the kind of agent companies actually deploy.

Step 1: Define the State

from typing import TypedDict, Annotated, Literal
from operator import add
from pydantic import BaseModel

class Source(BaseModel):
    url: str
    title: str
    summary: str
    relevance_score: float

class ResearchState(TypedDict):
    question: str
    search_queries: list[str]
    sources: Annotated[list[Source], add]
    draft_report: str
    critique: str
    final_report: str
    iteration: int
    status: str

I'm using Pydantic models for Source ‚Äî this gives you validation and serialization for free, which matters when you're persisting state to a database.

Step 2: Define the Nodes

from langchain_openai import ChatOpenAI
from langchain_core.messages import SystemMessage, HumanMessage
from langchain_community.tools.tavily_search import TavilySearchResults

llm = ChatOpenAI(model="gpt-4o", temperature=0)
search_tool = TavilySearchResults(max_results=5)

async def generate_queries(state: ResearchState) -> dict:
    """Turn the research question into targeted search queries."""
    response = await llm.ainvoke([
        SystemMessage(content="Generate 3 specific search queries to research this topic. Return only the queries, one per line."),
        HumanMessage(content=state["question"])
    ])
    queries = [q.strip() for q in response.content.strip().split("\n") if q.strip()]
    return {"search_queries": queries, "status": "searching"}

async def search_web(state: ResearchState) -> dict:
    """Execute searches and collect sources."""
    all_sources = []
    for query in state["search_queries"]:
        results = await search_tool.ainvoke({"query": query})
        for r in results:
            source = Source(
                url=r["url"],
                title=r.get("title", ""),
                summary=r["content"][:500],
                relevance_score=0.0  # scored in next step
            )
            all_sources.append(source)
    return {"sources": all_sources, "status": "analyzing"}

async def write_report(state: ResearchState) -> dict:
    """Synthesize sources into a structured report."""
    source_text = "\n\n".join(
        f"[{s.title}]({s.url})\n{s.summary}" for s in state["sources"]
    )
    response = await llm.ainvoke([
        SystemMessage(content="""Write a detailed research report based on these sources.
Structure: Executive Summary, Key Findings (numbered), Analysis, Conclusion.
Cite sources inline as [1], [2], etc."""),
        HumanMessage(content=f"Question: {state['question']}\n\nSources:\n{source_text}")
    ])
    return {"draft_report": response.content, "status": "reviewing"}

async def critique_report(state: ResearchState) -> dict:
    """Self-critique the draft for gaps and improvements."""
    response = await llm.ainvoke([
        SystemMessage(content="""Review this research report critically. Identify:
1. Factual gaps or unsupported claims
2. Missing perspectives
3. Areas needing more depth
Be specific and actionable. If the report is solid, say "APPROVED"."""),
        HumanMessage(content=state["draft_report"])
    ])
    return {
        "critique": response.content,
        "iteration": state["iteration"] + 1,
        "status": "critiqued"
    }

async def revise_report(state: ResearchState) -> dict:
    """Revise the report based on critique."""
    response = await llm.ainvoke([
        SystemMessage(content="Revise this report to address the critique. Maintain the same structure."),
        HumanMessage(content=f"Report:\n{state['draft_report']}\n\nCritique:\n{state['critique']}")
    ])
    return {"draft_report": response.content, "status": "revised"}

async def finalize(state: ResearchState) -> dict:
    return {"final_report": state["draft_report"], "status": "complete"}

Step 3: Wire the Graph

from langgraph.graph import StateGraph, START, END

def route_after_critique(state: ResearchState) -> Literal["revise", "finalize"]:
    if "APPROVED" in state["critique"] or state["iteration"] >= 3:
        return "finalize"
    return "revise"

builder = StateGraph(ResearchState)

# Add nodes
builder.add_node("generate_queries", generate_queries)
builder.add_node("search_web", search_web)
builder.add_node("write_report", write_report)
builder.add_node("critique_report", critique_report)
builder.add_node("revise_report", revise_report)
builder.add_node("finalize", finalize)

# Add edges
builder.add_edge(START, "generate_queries")
builder.add_edge("generate_queries", "search_web")
builder.add_edge("search_web", "write_report")
builder.add_edge("write_report", "critique_report")
builder.add_conditional_edges("critique_report", route_after_critique)
builder.add_edge("revise_report", "critique_report")  # loop back
builder.add_edge("finalize", END)

research_agent = builder.compile()

Run it:

result = await research_agent.ainvoke({
    "question": "What are the most effective strategies for reducing LLM hallucinations in production systems?",
    "search_queries": [],
    "sources": [],
    "draft_report": "",
    "critique": "",
    "final_report": "",
    "iteration": 0,
    "status": "starting"
})
print(result["final_report"])

State Management and Persistence

In production, agents crash. Servers restart. Users close browsers. You need checkpointing.

LangGraph has built-in support for persisting state at every step via checkpointers:

from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver

DB_URI = "postgresql://user:pass@localhost:5432/agents"

async with AsyncPostgresSaver.from_conn_string(DB_URI) as checkpointer:
    await checkpointer.setup()  # creates tables on first run

    research_agent = builder.compile(checkpointer=checkpointer)

    # Every invocation now saves state after each node
    config = {"configurable": {"thread_id": "research-001"}}
    result = await research_agent.ainvoke(initial_state, config)

If the process dies mid-execution, restart with the same thread_id and it picks up exactly where it left off:

# Resume from last checkpoint
result = await research_agent.ainvoke(None, config)

Production tip: Use thread_id as your correlation ID across logging, tracing, and customer support. When a user reports a problem, you can replay the exact state transitions.

For high-throughput systems, the Postgres checkpointer supports connection pooling. For simpler setups, SqliteSaver works fine. For serverless, use the MemorySaver during development but always switch to a durable store before deploying.

Human-in-the-Loop Patterns

Fully autonomous agents are a liability in production. The most reliable pattern is human-on-the-loop: the agent runs autonomously but pauses at critical decision points.

LangGraph supports this natively with interrupt:

from langgraph.types import interrupt, Command

async def write_report(state: ResearchState) -> dict:
    # ... generate draft ...

    # Pause and wait for human approval
    approval = interrupt({
        "question": "Review this draft report. Reply 'approved' or provide feedback.",
        "draft": draft_content
    })

    if approval.lower() != "approved":
        # Human provided feedback — use it as critique
        return {"draft_report": draft_content, "critique": approval, "status": "human_feedback"}

    return {"draft_report": draft_content, "status": "approved"}

On the calling side, you handle the interrupt:

config = {"configurable": {"thread_id": "research-001"}}

# First invocation runs until interrupt
result = await research_agent.ainvoke(initial_state, config)

# Agent is now paused. Show draft to user via your UI.
# When user responds:
result = await research_agent.ainvoke(
    Command(resume="approved"),  # or resume="Add more detail about X"
    config
)

This pattern maps cleanly to web UIs (show a review screen), Slack bots (send a message and wait for reply), or email workflows.

Advanced pattern — tiered autonomy:

def route_by_confidence(state: ResearchState) -> str:
    confidence = state.get("confidence_score", 0)
    if confidence > 0.9:
        return "auto_approve"     # agent proceeds
    elif confidence > 0.7:
        return "notify_human"     # agent proceeds but flags for review
    else:
        return "require_approval" # agent pauses

This lets low-risk actions flow through while escalating uncertain ones — the sweet spot for production throughput.

Tool Calling Best Practices

Tools are how agents interact with the real world. Get this wrong and you get agents that burn API credits, leak data, or take destructive actions.

Structured tool definitions

from langchain_core.tools import tool
from pydantic import Field

@tool
def search_knowledge_base(
    query: str = Field(description="Natural language search query"),
    filters: dict | None = Field(default=None, description="Optional metadata filters: {department: str, date_range: str}"),
    max_results: int = Field(default=10, ge=1, le=50, description="Number of results to return")
) -> list[dict]:
    """Search the internal knowledge base for documents matching the query.
    Use this for company-specific information. For general web information, use web_search instead."""
    # implementation
    ...

Key practices:

1. Rich descriptions matter more than you think. The LLM reads the docstring and field descriptions to decide when and how to call the tool. Vague descriptions lead to wrong tool selection.

2. Constrain inputs. Use ge, le, enums, and Pydantic validators. An agent that can pass max_results=10000 will eventually do it.

3. Separate read and write tools. Never have a single database_tool that can both query and delete. Give the agent db_query and db_delete separately, and only bind db_delete when you've added human approval.

4. Tool result formatting. Return structured data, not free text. The LLM processes structured results more reliably:

@tool
def get_order_status(order_id: str) -> dict:
    """Look up the status of a customer order."""
    order = db.get_order(order_id)
    return {
        "order_id": order.id,
        "status": order.status,
        "items_count": len(order.items),
        "estimated_delivery": order.eta.isoformat(),
        "action_available": ["cancel"] if order.status == "processing" else []
    }

1. Bind tools selectively per node. Not every node needs every tool:

research_llm = llm.bind_tools([search_tool, scrape_tool])
writing_llm = llm.bind_tools([])  # no tools during writing

Error Handling and Retry Strategies

Production agents face three categories of failures:

1. Transient failures (API timeouts, rate limits)

Use LangGraph's built-in retry policy:

from langgraph.pregel import RetryPolicy

builder.add_node(
    "search_web",
    search_web,
    retry=RetryPolicy(
        max_attempts=3,
        initial_interval=1.0,  # seconds
        backoff_factor=2.0,
        retry_on=(TimeoutError, RateLimitError)
    )
)

2. LLM failures (malformed output, hallucinated tool calls)

Wrap tool execution with validation:

async def safe_tool_executor(state: AgentState) -> dict:
    last_message = state["messages"][-1]

    for tool_call in last_message.tool_calls:
        try:
            # Validate tool exists
            tool = tool_map.get(tool_call["name"])
            if not tool:
                return {"messages": [ToolMessage(
                    content=f"Tool '{tool_call['name']}' does not exist. Available: {list(tool_map.keys())}",
                    tool_call_id=tool_call["id"]
                )]}

            # Execute with timeout
            result = await asyncio.wait_for(
                tool.ainvoke(tool_call["args"]),
                timeout=30.0
            )
            return {"messages": [ToolMessage(content=str(result), tool_call_id=tool_call["id"])]}

        except ValidationError as e:
            return {"messages": [ToolMessage(
                content=f"Invalid arguments: {e}. Please fix and retry.",
                tool_call_id=tool_call["id"]
            )]}

The agent sees the error message and self-corrects on the next iteration. This works surprisingly well — LLMs are good at fixing their own mistakes when given clear error messages.

3. Logical failures (infinite loops, stuck states)

Guard against these at the graph level:

def route_after_critique(state: ResearchState) -> str:
    # Hard cap on iterations
    if state["iteration"] >= 3:
        return "finalize"

    # Detect stuck state: same critique twice
    if state.get("prev_critique") == state["critique"]:
        return "finalize"

    return "revise"

Also set a global timeout on the entire graph execution:

result = await asyncio.wait_for(
    research_agent.ainvoke(initial_state, config),
    timeout=300.0  # 5 minute hard limit
)

Observability with LangSmith

You cannot operate what you cannot see. LangSmith is the observability layer for LangGraph — think Datadog for agent workflows.

Setup is two environment variables:

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=lsv2_...

Every node execution, tool call, LLM invocation, and state transition is now traced automatically. No code changes required.

What to monitor in production:

# Custom metadata for filtering traces
config = {
    "configurable": {"thread_id": "research-001"},
    "metadata": {
        "user_id": "u_12345",
        "environment": "production",
        "agent_version": "2.1.0"
    },
    "tags": ["research", "priority-high"]
}

Key metrics to track:

• Tokens per task: Set budgets. A research agent shouldn't exceed 50k tokens per run. Alert if it does.

• Iterations per completion: If your average is climbing, your prompts or critique logic are degrading.

• Tool call success rate: Below 95%? Your tool descriptions need work.

• Time to completion: Set SLOs. p50 under 30s, p99 under 120s.

• Human intervention rate: Track how often agents escalate. Trending up = model or prompt regression. Trending down = your agent is learning (or your thresholds are too loose).

LangSmith also supports evaluation datasets — curated input/output pairs that you run nightly to catch regressions:

from langsmith import Client

client = Client()

# Create a dataset of expected research outputs
dataset = client.create_dataset("research-agent-evals")
client.create_example(
    inputs={"question": "What is retrieval augmented generation?"},
    outputs={"expected_sections": ["Executive Summary", "Key Findings"]},
    dataset_id=dataset.id
)

LangGraph vs. CrewAI vs. AutoGen

The framework landscape has matured significantly. Here's when to use what:

Choose LangGraph when:

• You need fine-grained control over execution flow

• Persistence and checkpointing are requirements

• You're building a single agent with complex routing

• You need production-grade observability

Choose CrewAI when:

• Your problem naturally decomposes into roles (researcher, writer, reviewer)

• You want rapid prototyping of multi-agent systems

• Team-based delegation is the core pattern

Choose AutoGen when:

• You're building conversational multi-agent systems

• Agents need to debate or negotiate

• Research and experimentation are the primary goals

Hybrid approach (what I recommend): Use LangGraph as the orchestration layer and implement individual "agents" within it as specialized nodes. You get the reliability of graph-based control flow with the flexibility to swap implementations.

Production Deployment Tips

1. Use LangGraph Platform for managed deployment

// langgraph.json
{
    "graphs": {
        "research_agent": "./agent.py:research_agent"
    },
    "dependencies": ["langchain-openai", "tavily-python"],
    "env": ".env"
}

langgraph dev     # local development server with hot reload
langgraph build   # Docker image for deployment
langgraph deploy  # deploy to LangGraph Cloud

The platform gives you a REST API, WebSocket streaming, cron triggers, and a built-in task queue — eliminating significant infrastructure work.

2. Streaming for UX

Never make users stare at a spinner. Stream intermediate state:

async for event in research_agent.astream_events(initial_state, config, version="v2"):
    if event["event"] == "on_chat_model_stream":
        # Token-level streaming for the writing step
        print(event["data"]["chunk"].content, end="", flush=True)
    elif event["event"] == "on_chain_end":
        # Node completion events
        node_name = event.get("name", "")
        print(f"\n[Completed: {node_name}]")

3. Rate limiting and cost controls

import tiktoken

class TokenBudget:
    def __init__(self, max_tokens: int = 50_000):
        self.max_tokens = max_tokens
        self.used = 0
        self.encoder = tiktoken.encoding_for_model("gpt-4o")

    def check(self, text: str) -> bool:
        tokens = len(self.encoder.encode(text))
        self.used += tokens
        if self.used > self.max_tokens:
            raise TokenBudgetExceeded(f"Used {self.used}/{self.max_tokens} tokens")
        return True

Wire this into your LLM callbacks. When an agent hits its budget, force it to the finalize step with whatever it has.

4. Version your prompts

Never hardcode prompts in your node functions. Use a prompt registry:

from langsmith import Client

client = Client()

# Pull versioned prompts from LangSmith Hub
system_prompt = client.pull_prompt("research-agent/critique:v3")

This lets you A/B test prompts, roll back bad deployments, and track which prompt version produced which outputs.

5. Graceful degradation

Build fallback paths into your graph:

def route_search_results(state: ResearchState) -> str:
    if not state["sources"]:
        return "fallback_generate"  # LLM generates from knowledge
    if len(state["sources"]) < 3:
        return "search_again"       # try different queries
    return "write_report"           # proceed normally

An agent that returns a partial result is infinitely more useful than one that throws a 500.

Wrapping Up

The gap between an agent demo and a production agent is the same gap between a script and a service — error handling, observability, persistence, and operational controls.

LangGraph gives you the primitives to bridge that gap: typed state, persistent checkpoints, conditional routing, human-in-the-loop interrupts, and native observability. It's opinionated enough to prevent common mistakes but flexible enough to model real workflows.

Start with the simplest graph that solves your problem. Add checkpointing on day one — you'll thank yourself the first time a process crashes mid-run. Add human approval gates before any destructive action. Monitor token usage religiously. And version everything: prompts, tools, graph topology.

The agents that succeed in production aren't the cleverest ones — they're the most predictable ones.

If this article helped you, consider buying me a coffee on Ko-fi! Follow me for more AI engineering content.

Database migrations are the most dangerous routine operation in backend engineering. Your code deploys are (hopefully) stateless and reversible. Your database changes are neither. They mutate persistent state that your entire system depends on, and getting them wrong can mean downtime, data loss, or both.

This article covers the patterns, tooling, and discipline required to run migrations in production without breaking things.

Why Migrations Are Scary

The fundamental problem is simple: schema changes and code changes don't deploy atomically. There is always a window where your running application code and your database schema are out of sync. During a typical rolling deployment:

1. Migration runs, changing the schema

2. Old application code is still running against the new schema

3. New application code starts rolling out

4. Both old and new code run simultaneously against the new schema

5. Old code finishes draining

Steps 2-4 are where things break. If your migration renames a column, old code looking for the old name throws errors. If it drops a column, same problem. If it adds a NOT NULL column without a default, old code inserting rows fails.

The second problem is locking. Most ALTER TABLE operations in traditional databases acquire locks that block reads, writes, or both. On a table with millions of rows, a lock held for even a few seconds can cascade into connection pool exhaustion, request timeouts, and a full outage.

The third problem is irreversibility. You can roll back a code deploy in seconds. Rolling back a migration that dropped a column means restoring from backup — if you even have a recent one that's consistent.

Zero-Downtime Migration: The Core Principle

The rule is straightforward: at every point during the migration and deployment process, all running code must be compatible with the current database schema.

This means:

• Never rename a column in a single step

• Never drop a column that running code still references

• Never add a NOT NULL constraint without a default

• Never assume the migration and code deploy happen simultaneously

Every migration must be backward compatible with the currently deployed code and forward compatible with the code about to be deployed.

The Expand-Contract Pattern

This is the most important pattern in zero-downtime migrations. Instead of making a breaking change in one step, you split it into three phases:

Phase 1: Expand

Add the new structure alongside the old one. Both old and new code work.

-- Migration: Add new column (nullable, so old code can still INSERT)
ALTER TABLE orders ADD COLUMN status_v2 VARCHAR(50);

Phase 2: Migrate

Deploy code that writes to both old and new structures. Backfill existing data.

-- Backfill in batches (more on this later)
UPDATE orders SET status_v2 = status WHERE status_v2 IS NULL
  AND id BETWEEN \(start AND \)end;

Phase 3: Contract

Once all code uses the new structure and all data is migrated, remove the old one.

-- Only after all application code stops reading `status`
ALTER TABLE orders DROP COLUMN status;
ALTER TABLE orders RENAME COLUMN status_v2 TO status;

Each phase is a separate migration tied to a separate code deploy. Phase 1 goes out with or before the code that starts using the new column. Phase 3 goes out only after you've verified all services have been updated and the old column is truly unused.

Real-world example: renaming a column

Renaming user_name to display_name on a users table with 10M rows:

Deploy 1: Migration adds `display_name` column
Deploy 2: Code writes to both columns, reads from `display_name` with fallback to `user_name`
Deploy 3: Backfill script copies remaining data
Deploy 4: Code reads only from `display_name`
Deploy 5: Migration drops `user_name`

Yes, that's five deploys for a column rename. That's the cost of zero downtime. In practice, deploys 1-2 often ship together, and 4-5 ship together, so it's usually three deploy cycles.

Handling Large Table Alterations

On PostgreSQL, many ALTER TABLE operations are fast because they only update catalog metadata:

-- These are ~instant in PostgreSQL, regardless of table size
ALTER TABLE orders ADD COLUMN notes TEXT;
ALTER TABLE orders ALTER COLUMN notes SET DEFAULT '';
ALTER TABLE orders DROP COLUMN old_field;

But some operations are not:

-- These rewrite the entire table or scan all rows
ALTER TABLE orders ALTER COLUMN amount TYPE BIGINT;  -- full rewrite
ALTER TABLE orders ADD COLUMN verified BOOLEAN NOT NULL DEFAULT true;
-- (instant in PG 11+ but rewrites in older versions)
CREATE INDEX ON orders (customer_id);  -- full table scan

For type changes on large tables, use the expand-contract pattern: add a new column with the desired type, backfill, switch reads, drop the old column.

For index creation, always use CONCURRENTLY:

-- Blocks writes:
CREATE INDEX idx_orders_customer ON orders (customer_id);

-- Does NOT block writes (takes longer, but safe):
CREATE INDEX CONCURRENTLY idx_orders_customer ON orders (customer_id);

Important caveat: CREATE INDEX CONCURRENTLY cannot run inside a transaction. Most migration tools wrap each migration in a transaction by default. You need to either disable that behavior for this specific migration or use a tool that handles it.

In golang-migrate:

-- +migrate: no-transaction
CREATE INDEX CONCURRENTLY idx_orders_customer ON orders (customer_id);

In Flyway, you would set executeInTransaction=false on the migration.

Advisory Locks and Migration Safety

When running migrations in a horizontally scaled environment, you need to ensure only one instance runs migrations at a time. Most tools handle this with advisory locks:

-- golang-migrate uses PostgreSQL advisory locks
SELECT pg_advisory_lock(12345);
-- ... run migrations ...
SELECT pg_advisory_unlock(12345);

This prevents two pods starting simultaneously from both trying to run the same migration and corrupting the schema history.

Migration Tooling

golang-migrate

Straightforward, language-agnostic, uses plain SQL files:

migrations/
  000001_create_users.up.sql
  000001_create_users.down.sql
  000002_add_orders.up.sql
  000002_add_orders.down.sql

Run from CLI or embedded in your Go application:

import "github.com/golang-migrate/migrate/v4"

m, err := migrate.New(
    "file://migrations",
    "postgres://localhost:5432/mydb?sslmode=disable",
)
if err != nil {
    log.Fatal(err)
}

if err := m.Up(); err != nil && err != migrate.ErrNoChange {
    log.Fatal(err)
}

Strengths: simple, no DSL to learn, works with any language's deployment pipeline. Weaknesses: no built-in support for non-transactional migrations, limited state tracking.

Flyway

JVM-based, more opinionated, supports versioned and repeatable migrations:

-- V1__Create_users.sql
CREATE TABLE users (
    id BIGSERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL UNIQUE,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- V2__Add_orders.sql
CREATE TABLE orders (
    id BIGSERIAL PRIMARY KEY,
    user_id BIGINT NOT NULL REFERENCES users(id),
    total_cents BIGINT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Flyway tracks state in a flyway_schema_history table and supports callbacks, placeholders, and Java-based migrations for complex logic. It's the standard in Java/Kotlin ecosystems.

Other Notable Tools

• Alembic (Python/SQLAlchemy): generates migrations from model diffs, good for Python shops

• Sqitch: dependency-based rather than version-ordered, powerful but steeper learning curve

• Atlas (by Ariga): declarative schema management, computes diffs automatically, gaining traction in Go ecosystems

• pg_partman / pgloader: for partition management and bulk data loading, respectively

Choosing a Tool

Pick based on your ecosystem and complexity:

Data Backfills Done Right

Backfilling data across millions of rows is where most migration disasters happen. The naive approach:

-- DO NOT DO THIS
UPDATE orders SET status_v2 = compute_new_status(status);

This acquires a lock on every row, generates enormous WAL (write-ahead log) volume, and can take hours while blocking other writes.

Instead, backfill in batches with throttling:

import time
import psycopg2

BATCH_SIZE = 5000
SLEEP_SECONDS = 0.1  # throttle to reduce replication lag

conn = psycopg2.connect(dsn)
conn.autocommit = True

cursor = conn.cursor()
cursor.execute("SELECT MIN(id), MAX(id) FROM orders")
min_id, max_id = cursor.fetchone()

current = min_id
while current <= max_id:
    cursor.execute("""
        UPDATE orders
        SET status_v2 = compute_new_status(status)
        WHERE id >= %s AND id < %s
          AND status_v2 IS NULL
    """, (current, current + BATCH_SIZE))

    updated = cursor.rowcount
    print(f"Batch {current}-{current + BATCH_SIZE}: updated {updated} rows")

    current += BATCH_SIZE
    time.sleep(SLEEP_SECONDS)

Key practices:

• Batch by primary key range, not LIMIT/OFFSET (which gets slower as offset grows)

• Sleep between batches to let replicas catch up and avoid saturating I/O

• Make it idempotent (WHERE status_v2 IS NULL) so you can restart safely if it fails midway

• Monitor replication lag during the backfill and pause if it exceeds your threshold

• Run during low-traffic windows when possible, even if it's technically safe at peak

For truly massive tables (billions of rows), consider doing the backfill at the application level: update the new column whenever a row is naturally read or written, and run the batch backfill for the long tail of untouched rows.

Blue-Green Database Deployments

Blue-green deployments for databases are harder than for stateless application servers, but the pattern exists and works well for certain scenarios.

The idea: maintain two database schemas (or databases) — blue (current) and green (next). Migrate green, point traffic to it, keep blue as a rollback target.

Schema-Level Blue-Green

-- Blue schema (current)
CREATE SCHEMA blue;
-- ... tables in blue schema

-- Green schema (next version)
CREATE SCHEMA green;
-- ... migrated tables in green schema

-- Application config points to schema
SET search_path = 'blue';  -- current
SET search_path = 'green'; -- after cutover

This works when your migration is a large structural change that's hard to do incrementally. You build the new schema, backfill it from the old one, then switch the application's search_path.

Limitations

True blue-green for databases requires either:

• A brief write-freeze during cutover (seconds, not minutes)

• Dual-write during the transition period

• Logical replication between old and new schemas

For most teams, the expand-contract pattern is more practical than blue-green for databases. Reserve blue-green for major version upgrades (e.g., PostgreSQL 14 to 16) where you set up logical replication between the old and new clusters.

Rollback Strategies

Down Migrations

Every migration tool supports down migrations. Write them:

-- 000005_add_verified_column.up.sql
ALTER TABLE users ADD COLUMN verified BOOLEAN DEFAULT false;

-- 000005_add_verified_column.down.sql
ALTER TABLE users DROP COLUMN verified;

But understand their limits. A down migration that drops a column destroys data. If you added the column, backfilled it over three days, and then need to roll back, that data is gone.

Forward-Only Rollbacks

A safer pattern: instead of rolling back the migration, roll forward with a new migration that undoes the change:

000005_add_verified_column.up.sql    -- adds column
000006_remove_verified_column.up.sql -- removes it (if needed)

This keeps the migration history linear and auditable. Many teams mandate forward-only migrations in production while keeping down migrations for local development.

Point-In-Time Recovery

For catastrophic failures, you need PITR:

# PostgreSQL continuous archiving
archive_mode = on
archive_command = 'cp %p /archive/%f'

# Restore to a specific timestamp
recovery_target_time = '2025-03-15 14:30:00 UTC'

Test your PITR process regularly. The worst time to discover your backups don't work is during an incident.

The Migration Rollback Matrix

Testing Migrations

Against Production-Like Data

Your test database with 50 rows will not reveal the problems that show up with 50 million rows. Test against a copy of production data:

# Snapshot production (use your cloud provider's snapshot feature)
# Restore to a test instance
# Run the migration
# Measure: time, locks held, WAL generated, replication lag

Schema Diffing

After running migrations on a staging environment, diff the resulting schema against what you expect:

# Using pg_dump to compare schemas
pg_dump --schema-only production_db > prod_schema.sql
pg_dump --schema-only staging_db > staging_schema.sql
diff prod_schema.sql staging_schema.sql

Atlas has this built in:

atlas schema diff \
  --from "postgres://localhost/production" \
  --to "postgres://localhost/staging"

CI Pipeline Integration

Run migrations in CI against a fresh database and a database with the previous version's schema:

# .github/workflows/migrations.yml
migration-test:
  services:
    postgres:
      image: postgres:16
      env:
        POSTGRES_DB: test
        POSTGRES_PASSWORD: test
  steps:
    - uses: actions/checkout@v4
    - name: Run all migrations from scratch
      run: migrate -path ./migrations -database "$DB_URL" up
    - name: Verify schema matches expectations
      run: ./scripts/verify-schema.sh
    - name: Test rollback of latest migration
      run: migrate -path ./migrations -database "$DB_URL" down 1
    - name: Re-apply latest migration
      run: migrate -path ./migrations -database "$DB_URL" up

Statement-Level Analysis

Before running a migration in production, analyze what it will actually do:

-- Check if an ALTER TABLE will rewrite the table
-- (PostgreSQL-specific: check pg_catalog after the change)

-- Estimate lock duration using pg_stat_activity during staging test
SELECT pid, wait_event_type, wait_event, query
FROM pg_stat_activity
WHERE wait_event_type = 'Lock';

A Migration Checklist

Before every production migration:

1. Is it backward compatible? Can the currently deployed code work with the new schema?

2. Is it forward compatible? Can the about-to-be-deployed code work with the old schema (in case you need to roll back the code deploy)?

3. Does it hold locks? If so, for how long? On how many rows?

4. Has it been tested against production-sized data?

5. Is there a rollback plan? Down migration, forward fix, or PITR?

6. Is the backfill batched and idempotent?

7. Is the migration wrapped in a transaction? Should it be? (CREATE INDEX CONCURRENTLY cannot be.)

8. Has someone else reviewed it? Migration PRs deserve as much scrutiny as application code.

Summary

Database migrations will always carry risk. The goal is not to eliminate risk but to reduce it to a level where deploying a migration is routine rather than terrifying. The core practices:

• Use expand-contract for every breaking schema change

• Batch and throttle data backfills

• Test against production-scale data

• Write rollback plans before you need them

• Use CONCURRENTLY for index operations

• Keep migrations small, frequent, and reviewable

The teams that migrate databases confidently are not the ones with the best tools. They are the ones with the most disciplined processes.

Building resilient backend systems? This is article #17 in the Production Backend Patterns series. Follow for the next one.

If you found this useful, consider supporting my work:

Every production system eventually outgrows synchronous request-response. Order processing, email delivery, image resizing, webhook fanout — these all belong in a queue. But the consumer side is where things get ugly. Messages arrive out of order. Workers crash mid-processing. Duplicates sneak through. Your "simple consumer" becomes a tangle of retries, panics, and lost messages.

This article builds a production-grade NATS JetStream consumer in Go, piece by piece, covering the patterns that keep it alive at 3 AM.

Why Message Queues Matter

Synchronous architectures hit a wall when:

• Latency budgets are tight. Your API shouldn't block for 30 seconds while a PDF renders.

• Failure domains bleed. A downstream service outage shouldn't take your entire API down.

• Load is bursty. Black Friday traffic shouldn't require Black Friday compute 365 days a year.

Message queues decouple producers from consumers. The producer writes a message and moves on. The consumer processes it at its own pace, retries on failure, and scales independently. This is table stakes for any system that handles real traffic.

We'll use NATS JetStream as our concrete example. It's operationally simpler than RabbitMQ or Kafka, supports persistent streams with at-least-once delivery, and has an excellent Go client. The patterns apply universally.

The Architecture

Here's what we're building:

Producer ‚Üí NATS JetStream Stream ‚Üí Consumer Group
                                      ├── Worker Pool (N goroutines)
                                      ├── Retry with Exponential Backoff
                                      ├── Dead Letter Queue
                                      ├── Idempotency Check
                                      └── Metrics + Tracing

Let's define the core interfaces first:

package consumer

import (
    "context"
    "time"
)

// Message represents a queue message with metadata.
type Message struct {
    ID        string
    Subject   string
    Data      []byte
    Headers   map[string]string
    Timestamp time.Time
    Attempt   int
}

// Handler processes a single message. Return an error to trigger retry.
type Handler func(ctx context.Context, msg Message) error

// IdempotencyStore tracks processed message IDs.
type IdempotencyStore interface {
    Exists(ctx context.Context, messageID string) (bool, error)
    Mark(ctx context.Context, messageID string, ttl time.Duration) error
}

Designing Idempotent Consumers

At-least-once delivery means your handler will receive duplicates. Network blips, consumer restarts, rebalancing — all cause redelivery. Your handler must be idempotent: processing the same message twice produces the same result.

Two strategies work in practice:

1. Deduplication at the consumer level — check a store before processing:

type RedisIdempotencyStore struct {
    client *redis.Client
}

func (s *RedisIdempotencyStore) Exists(ctx context.Context, id string) (bool, error) {
    val, err := s.client.Exists(ctx, "idem:"+id).Result()
    if err != nil {
        return false, fmt.Errorf("idempotency check failed: %w", err)
    }
    return val > 0, nil
}

func (s *RedisIdempotencyStore) Mark(ctx context.Context, id string, ttl time.Duration) error {
    return s.client.Set(ctx, "idem:"+id, "1", ttl).Err()
}

2. Idempotent operations ‚Äî design your writes so repeats are safe. Use INSERT ... ON CONFLICT DO NOTHING, conditional updates with version checks, or deterministic IDs derived from message content.

The first approach is simpler to bolt on. The second is more robust. Production systems usually combine both: a deduplication layer as a fast path, with idempotent database operations as the safety net.

The TTL on the idempotency key matters. Set it too short, and late redeliveries slip through. Set it too long, and your store grows unbounded. Match it to your stream's max redelivery window — typically 24–72 hours.

The Consumer Core

Now let's build the consumer. The Config struct captures every tunable:

type Config struct {
    // NATS connection
    NATSUrl    string
    StreamName string
    Subject    string
    Durable    string // consumer group name

    // Concurrency
    WorkerCount int

    // Retry
    MaxRetries     int
    InitialBackoff time.Duration
    MaxBackoff     time.Duration
    BackoffFactor  float64

    // Idempotency
    IdempotencyTTL time.Duration

    // Observability
    ServiceName string
}

func DefaultConfig() Config {
    return Config{
        WorkerCount:    10,
        MaxRetries:     5,
        InitialBackoff: 1 * time.Second,
        MaxBackoff:     60 * time.Second,
        BackoffFactor:  2.0,
        IdempotencyTTL: 24 * time.Hour,
        ServiceName:    "queue-consumer",
    }
}

The consumer struct ties everything together:

type Consumer struct {
    cfg        Config
    js         nats.JetStreamContext
    handler    Handler
    idemStore  IdempotencyStore
    metrics    *Metrics
    tracer     trace.Tracer
    logger     *slog.Logger
    sub        *nats.Subscription
    wg         sync.WaitGroup
    msgCh      chan *nats.Msg
    shutdownCh chan struct{}
}

func New(nc *nats.Conn, cfg Config, handler Handler, idemStore IdempotencyStore) (*Consumer, error) {
    js, err := nc.JetStream()
    if err != nil {
        return nil, fmt.Errorf("jetstream init: %w", err)
    }

    tp := otel.GetTracerProvider()
    meter := otel.GetMeterProvider().Meter(cfg.ServiceName)

    return &Consumer{
        cfg:        cfg,
        js:         js,
        handler:    handler,
        idemStore:  idemStore,
        metrics:    NewMetrics(meter),
        tracer:     tp.Tracer(cfg.ServiceName),
        logger:     slog.Default(),
        msgCh:      make(chan *nats.Msg, cfg.WorkerCount*2),
        shutdownCh: make(chan struct{}),
    }, nil
}

Retry with Exponential Backoff

The retry logic lives in the message processing path. NATS JetStream supports server-side NakWithDelay, which tells the server to redeliver after a specified duration. This is vastly superior to client-side retry loops because it survives consumer restarts:

func (c *Consumer) processMessage(natsMsg *nats.Msg) {
    ctx, span := c.tracer.Start(context.Background(), "process_message",
        trace.WithAttributes(
            attribute.String("subject", natsMsg.Subject),
        ),
    )
    defer span.End()

    meta, err := natsMsg.Metadata()
    if err != nil {
        c.logger.Error("failed to read metadata", "error", err)
        natsMsg.Nak()
        return
    }

    attempt := int(meta.NumDelivered)
    msgID := natsMsg.Header.Get("Nats-Msg-Id")
    if msgID == "" {
        msgID = fmt.Sprintf("%s-%d", meta.Stream, meta.Sequence.Stream)
    }

    span.SetAttributes(
        attribute.String("message_id", msgID),
        attribute.Int("attempt", attempt),
    )

    // Idempotency check
    if exists, err := c.idemStore.Exists(ctx, msgID); err != nil {
        c.logger.Error("idempotency check failed", "error", err, "msg_id", msgID)
        span.RecordError(err)
        // Retry — the store might be temporarily down
        c.nakWithBackoff(natsMsg, attempt)
        return
    } else if exists {
        c.logger.Debug("duplicate message, skipping", "msg_id", msgID)
        c.metrics.duplicatesSkipped.Add(ctx, 1)
        natsMsg.Ack()
        return
    }

    msg := Message{
        ID:        msgID,
        Subject:   natsMsg.Subject,
        Data:      natsMsg.Data,
        Headers:   flattenHeaders(natsMsg.Header),
        Timestamp: meta.Timestamp,
        Attempt:   attempt,
    }

    // Process
    start := time.Now()
    err = c.handler(ctx, msg)
    duration := time.Since(start)

    c.metrics.processingDuration.Record(ctx, duration.Seconds(),
        metric.WithAttributes(attribute.Bool("success", err == nil)),
    )

    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        c.metrics.processingErrors.Add(ctx, 1)

        if attempt >= c.cfg.MaxRetries {
            c.logger.Error("max retries exceeded, sending to DLQ",
                "msg_id", msgID, "error", err, "attempts", attempt,
            )
            c.sendToDLQ(ctx, natsMsg, err)
            natsMsg.Ack() // Ack original to stop redelivery
            return
        }

        c.logger.Warn("processing failed, scheduling retry",
            "msg_id", msgID, "error", err, "attempt", attempt,
        )
        c.nakWithBackoff(natsMsg, attempt)
        return
    }

    // Success — mark as processed and ack
    if err := c.idemStore.Mark(ctx, msgID, c.cfg.IdempotencyTTL); err != nil {
        c.logger.Error("failed to mark idempotency", "error", err, "msg_id", msgID)
        // Don't fail the message — it was processed successfully.
        // Worst case: a duplicate gets processed again idempotently.
    }

    c.metrics.messagesProcessed.Add(ctx, 1)
    natsMsg.Ack()
}

func (c *Consumer) nakWithBackoff(msg *nats.Msg, attempt int) {
    delay := c.cfg.InitialBackoff
    for i := 1; i < attempt; i++ {
        delay = time.Duration(float64(delay) * c.cfg.BackoffFactor)
        if delay > c.cfg.MaxBackoff {
            delay = c.cfg.MaxBackoff
            break
        }
    }
    msg.NakWithDelay(delay)
}

Key decisions here:

• Server-side retry via NakWithDelay ‚Äî the consumer doesn't hold the message during backoff. If this consumer dies, another picks it up.

• Ack after DLQ publish ‚Äî we ack the original message to stop the redelivery loop. The DLQ is the new source of truth.

• Idempotency mark happens after success ‚Äî if the mark fails, the message might be reprocessed. That's fine because operations are idempotent.

Dead Letter Queue

Messages that fail all retries go to a separate stream. Your ops team reviews them, fixes the bug, and replays:

func (c *Consumer) sendToDLQ(ctx context.Context, original *nats.Msg, processErr error) {
    _, span := c.tracer.Start(ctx, "send_to_dlq")
    defer span.End()

    headers := nats.Header{}
    // Preserve original headers
    for k, v := range original.Header {
        headers[k] = v
    }
    headers.Set("X-DLQ-Error", processErr.Error())
    headers.Set("X-DLQ-Timestamp", time.Now().UTC().Format(time.RFC3339))
    headers.Set("X-Original-Subject", original.Subject)

    dlqMsg := &nats.Msg{
        Subject: fmt.Sprintf("dlq.%s", c.cfg.Subject),
        Data:    original.Data,
        Header:  headers,
    }

    if _, err := c.js.PublishMsg(dlqMsg); err != nil {
        c.logger.Error("failed to publish to DLQ", "error", err)
        span.RecordError(err)
        c.metrics.dlqFailures.Add(ctx, 1)
        // This is bad. The message will be lost after ack.
        // In practice, also log the full message body for manual recovery.
        c.logger.Error("LOST MESSAGE — manual recovery required",
            "subject", original.Subject,
            "data", string(original.Data),
        )
    }
    c.metrics.dlqMessages.Add(ctx, 1)
}

Create the DLQ stream during setup:

func EnsureStreams(js nats.JetStreamContext, streamName, subject string) error {
    // Main stream
    _, err := js.AddStream(&nats.StreamConfig{
        Name:      streamName,
        Subjects:  []string{subject},
        Retention: nats.WorkQueuePolicy,
        MaxAge:    72 * time.Hour,
    })
    if err != nil {
        return fmt.Errorf("create main stream: %w", err)
    }

    // DLQ stream
    _, err = js.AddStream(&nats.StreamConfig{
        Name:      streamName + "_dlq",
        Subjects:  []string{"dlq." + subject},
        Retention: nats.LimitsPolicy,
        MaxAge:    30 * 24 * time.Hour, // Keep DLQ messages for 30 days
    })
    if err != nil {
        return fmt.Errorf("create DLQ stream: %w", err)
    }
    return nil
}

Concurrency Control with Worker Pools

A single goroutine consuming messages wastes most of its time waiting on I/O. A worker pool lets you process N messages concurrently while keeping backpressure under control:

func (c *Consumer) Start(ctx context.Context) error {
    sub, err := c.js.PullSubscribe(
        c.cfg.Subject,
        c.cfg.Durable,
        nats.ManualAck(),
        nats.AckWait(30*time.Second),
        nats.MaxDeliver(c.cfg.MaxRetries+1),
    )
    if err != nil {
        return fmt.Errorf("subscribe: %w", err)
    }
    c.sub = sub

    // Start workers
    for i := 0; i < c.cfg.WorkerCount; i++ {
        c.wg.Add(1)
        go c.worker(i)
    }

    // Fetch loop — pulls messages from NATS and fans out to workers
    c.wg.Add(1)
    go c.fetchLoop(ctx)

    c.logger.Info("consumer started",
        "workers", c.cfg.WorkerCount,
        "subject", c.cfg.Subject,
        "durable", c.cfg.Durable,
    )
    return nil
}

func (c *Consumer) fetchLoop(ctx context.Context) {
    defer c.wg.Done()
    defer close(c.msgCh)

    for {
        select {
        case <-ctx.Done():
            return
        case <-c.shutdownCh:
            return
        default:
        }

        msgs, err := c.sub.Fetch(c.cfg.WorkerCount, nats.MaxWait(5*time.Second))
        if err != nil {
            if errors.Is(err, nats.ErrTimeout) {
                continue // No messages available, poll again
            }
            c.logger.Error("fetch error", "error", err)
            time.Sleep(time.Second) // Back off on unexpected errors
            continue
        }

        for _, msg := range msgs {
            select {
            case c.msgCh <- msg:
            case <-ctx.Done():
                return
            case <-c.shutdownCh:
                return
            }
        }
    }
}

func (c *Consumer) worker(id int) {
    defer c.wg.Done()

    c.logger.Debug("worker started", "worker_id", id)
    for msg := range c.msgCh {
        c.processMessage(msg)
    }
    c.logger.Debug("worker stopped", "worker_id", id)
}

The Fetch batch size matches the worker count. This keeps the channel buffer from growing unbounded while ensuring every worker stays fed. The MaxWait on Fetch prevents tight-looping when the stream is empty.

Graceful Shutdown Integration

Following the patterns from article #14, we integrate with OS signals to drain in-flight work before exiting:

func (c *Consumer) Shutdown(ctx context.Context) error {
    c.logger.Info("shutting down consumer...")

    // Signal the fetch loop to stop
    close(c.shutdownCh)

    // Drain the subscription — no new messages will be delivered
    if c.sub != nil {
        if err := c.sub.Drain(); err != nil {
            c.logger.Error("subscription drain failed", "error", err)
        }
    }

    // Wait for in-flight messages to complete (or context deadline)
    done := make(chan struct{})
    go func() {
        c.wg.Wait()
        close(done)
    }()

    select {
    case <-done:
        c.logger.Info("consumer shutdown complete")
        return nil
    case <-ctx.Done():
        c.logger.Warn("shutdown timed out, some messages may be redelivered")
        return ctx.Err()
    }
}

Wire it into main:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    nc, _ := nats.Connect(nats.DefaultURL)
    defer nc.Close()

    cfg := consumer.DefaultConfig()
    cfg.NATSUrl = nats.DefaultURL
    cfg.StreamName = "ORDERS"
    cfg.Subject = "orders.process"
    cfg.Durable = "order-processor"

    store := &RedisIdempotencyStore{client: redis.NewClient(&redis.Options{Addr: "localhost:6379"})}

    c, err := consumer.New(nc, cfg, handleOrder, store)
    if err != nil {
        log.Fatal(err)
    }

    if err := c.Start(ctx); err != nil {
        log.Fatal(err)
    }

    <-ctx.Done()

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    c.Shutdown(shutdownCtx)
}

func handleOrder(ctx context.Context, msg consumer.Message) error {
    var order Order
    if err := json.Unmarshal(msg.Data, &order); err != nil {
        return fmt.Errorf("unmarshal order: %w", err) // Will retry, but won't help — consider a non-retryable error type
    }
    // Process the order...
    return nil
}

A subtle point: the handleOrder comment hints at an important production refinement. Some errors are non-retryable ‚Äî bad JSON, invalid business data, schema mismatches. Retrying won't help. A production consumer should distinguish these:

type PermanentError struct {
    Err error
}

func (e *PermanentError) Error() string { return e.Err.Error() }
func (e *PermanentError) Unwrap() error { return e.Err }

// In processMessage, before retry logic:
var permErr *PermanentError
if errors.As(err, &permErr) {
    c.logger.Error("permanent error, sending to DLQ immediately",
        "msg_id", msgID, "error", err,
    )
    c.sendToDLQ(ctx, natsMsg, err)
    natsMsg.Ack()
    return
}

Observability: Metrics and Tracing

The metrics struct captures the four golden signals for a queue consumer:

type Metrics struct {
    messagesProcessed metric.Int64Counter
    processingErrors  metric.Int64Counter
    processingDuration metric.Float64Histogram
    duplicatesSkipped metric.Int64Counter
    dlqMessages       metric.Int64Counter
    dlqFailures       metric.Int64Counter
    inflightMessages  metric.Int64UpDownCounter
}

func NewMetrics(meter metric.Meter) *Metrics {
    m := &Metrics{}
    m.messagesProcessed, _ = meter.Int64Counter("consumer.messages.processed",
        metric.WithDescription("Total messages successfully processed"))
    m.processingErrors, _ = meter.Int64Counter("consumer.messages.errors",
        metric.WithDescription("Total processing errors"))
    m.processingDuration, _ = meter.Float64Histogram("consumer.messages.duration_seconds",
        metric.WithDescription("Message processing duration"),
        metric.WithExplicitBucketBoundaries(0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30))
    m.duplicatesSkipped, _ = meter.Int64Counter("consumer.messages.duplicates",
        metric.WithDescription("Duplicate messages skipped"))
    m.dlqMessages, _ = meter.Int64Counter("consumer.dlq.sent",
        metric.WithDescription("Messages sent to DLQ"))
    m.dlqFailures, _ = meter.Int64Counter("consumer.dlq.failures",
        metric.WithDescription("Failed DLQ publishes — potential message loss"))
    m.inflightMessages, _ = meter.Int64UpDownCounter("consumer.messages.inflight",
        metric.WithDescription("Currently processing messages"))
    return m
}

The dlq.failures counter deserves an alert. If this fires, you're losing messages. Set a PagerDuty threshold at > 0.

For tracing, the spans we set in processMessage create a trace per message. To connect producer and consumer traces, propagate the trace context through NATS headers:

// Producer side
func publishWithTrace(ctx context.Context, js nats.JetStreamContext, subject string, data []byte) error {
    msg := &nats.Msg{
        Subject: subject,
        Data:    data,
        Header:  nats.Header{},
    }
    otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(msg.Header))
    _, err := js.PublishMsg(msg)
    return err
}

// Consumer side — in processMessage, replace context.Background():
ctx = otel.GetTextMapPropagator().Extract(ctx, propagation.HeaderCarrier(natsMsg.Header))
ctx, span := c.tracer.Start(ctx, "process_message", ...)

Now your traces show the full journey: API request ‚Üí publish ‚Üí consume ‚Üí process ‚Üí downstream calls. This is indispensable for debugging latency in async pipelines.

Testing Strategies

Queue consumers are notoriously hard to test. Here's a layered approach.

Unit test the handler in isolation — no queue, no infrastructure:

func TestHandleOrder_Success(t *testing.T) {
    msg := consumer.Message{
        ID:   "test-1",
        Data: []byte(`{"id":"order-123","amount":99.99}`),
    }

    err := handleOrder(context.Background(), msg)
    assert.NoError(t, err)
    // Assert side effects: database writes, API calls, etc.
}

func TestHandleOrder_InvalidJSON(t *testing.T) {
    msg := consumer.Message{
        ID:   "test-2",
        Data: []byte(`not json`),
    }

    err := handleOrder(context.Background(), msg)
    assert.Error(t, err)

    var permErr *consumer.PermanentError
    assert.True(t, errors.As(err, &permErr), "bad JSON should be a permanent error")
}

Integration test with an embedded NATS server — tests the full consumer lifecycle:

func TestConsumer_ProcessAndAck(t *testing.T) {
    // Start embedded NATS
    srv, _ := server.NewServer(&server.Options{
        Port:      -1,
        JetStream: true,
        StoreDir:  t.TempDir(),
    })
    go srv.Start()
    defer srv.Shutdown()
    srv.ReadyForConnections(5 * time.Second)

    nc, _ := nats.Connect(srv.ClientURL())
    defer nc.Close()

    js, _ := nc.JetStream()
    consumer.EnsureStreams(js, "TEST", "test.subject")

    processed := make(chan string, 1)
    handler := func(ctx context.Context, msg consumer.Message) error {
        processed <- msg.ID
        return nil
    }

    store := &InMemoryIdempotencyStore{seen: map[string]bool{}}
    cfg := consumer.DefaultConfig()
    cfg.StreamName = "TEST"
    cfg.Subject = "test.subject"
    cfg.Durable = "test-consumer"
    cfg.WorkerCount = 1

    c, _ := consumer.New(nc, cfg, handler, store)

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    c.Start(ctx)

    // Publish a message
    js.Publish("test.subject", []byte(`{"key":"value"}`),
        nats.MsgId("msg-001"))

    select {
    case id := <-processed:
        assert.Equal(t, "msg-001", id)
    case <-time.After(5 * time.Second):
        t.Fatal("message not processed within timeout")
    }
}

func TestConsumer_DuplicateSkipped(t *testing.T) {
    // Same setup as above, but pre-mark the message ID
    store := &InMemoryIdempotencyStore{seen: map[string]bool{"msg-001": true}}
    // ... publish msg-001, assert it gets acked but handler is NOT called
}

func TestConsumer_RetryThenDLQ(t *testing.T) {
    // Handler returns error every time.
    // Assert message appears in DLQ stream after MaxRetries.
}

Chaos test — kill the consumer mid-processing, restart, and verify no messages are lost:

func TestConsumer_CrashRecovery(t *testing.T) {
    // 1. Publish 100 messages
    // 2. Start consumer, let it process ~50
    // 3. Cancel context (simulates crash)
    // 4. Start a new consumer with the same durable name
    // 5. Wait for all 100 to be processed (some may be processed twice)
    // 6. Assert: handler was called >= 100 times, all message IDs covered
}

The in-memory idempotency store for tests:

type InMemoryIdempotencyStore struct {
    mu   sync.Mutex
    seen map[string]bool
}

func (s *InMemoryIdempotencyStore) Exists(_ context.Context, id string) (bool, error) {
    s.mu.Lock()
    defer s.mu.Unlock()
    return s.seen[id], nil
}

func (s *InMemoryIdempotencyStore) Mark(_ context.Context, id string, _ time.Duration) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.seen[id] = true
    return nil
}

Production Checklist

Before shipping your consumer:

• [ ] Idempotency ‚Äî handler is safe to run twice on the same input

• [ ] Permanent vs transient errors ‚Äî bad data goes straight to DLQ, not through the retry loop

• [ ] Backoff bounds ‚Äî MaxBackoff prevents retry storms; jitter is even better

• [ ] DLQ monitoring ‚Äî alert on dlq.sent > 0 and page on dlq.failures > 0

• [ ] Graceful shutdown ‚Äî drain subscription, wait for in-flight, then exit

• [ ] Health check ‚Äî expose /healthz that checks NATS and idempotency store connectivity

• [ ] Consumer lag metric ‚Äî monitor consumer.messages.pending via NATS admin API

• [ ] Resource limits ‚Äî bound worker count, channel buffer, and message size

• [ ] Trace propagation ‚Äî connect producer and consumer spans for end-to-end visibility

Wrapping Up

A message queue consumer is deceptively simple to prototype and deceptively hard to get right in production. The patterns in this article — idempotent processing, server-side retry with backoff, dead letter queues, bounded worker pools, graceful shutdown, and full observability — form a foundation that handles the failures real systems encounter.

The complete consumer is around 300 lines of focused Go code. No frameworks, no magic. Just clear concurrency patterns and deliberate error handling. That's the kind of code you want running at 3 AM.

Next in the series, we'll look at building a production-ready rate limiter using Redis and sliding window counters. Stay tuned.

If this article helped you, consider buying me a coffee on Ko-fi! Follow me for more production backend patterns.

Distributed tracing eliminates that pain. It gives you a single, end-to-end view of a request as it flows through every service in your architecture. And with OpenTelemetry becoming the industry standard, there has never been a better time to wire it in.

This article walks through instrumenting Go services with OpenTelemetry from scratch. No toy examples — everything here is production-grade code you can drop into a real system.

Why Distributed Tracing Matters

In a monolith, a stack trace tells you everything. In a distributed system, a single user action might touch an API gateway, an auth service, an order service, a payment provider, a notification queue, and a database. When something goes wrong — or just gets slow — you need to answer: which service, which call, which dependency?

Distributed tracing answers this by assigning a trace ID to each incoming request and propagating it through every downstream call. Each unit of work within a service becomes a span, and spans nest to form a tree that represents the full request lifecycle.

The payoff is immediate:

• Latency diagnosis: See exactly which service or database call is the bottleneck.

• Error attribution: Know that the 500 came from the payment service's connection pool, not your code.

• Dependency mapping: Visualize how services actually communicate at runtime, not how the architecture diagram says they should.

• SLA tracking: Measure per-endpoint latency distributions across the entire call chain.

OpenTelemetry in 60 Seconds

OpenTelemetry (OTel) is a CNCF project that provides a vendor-neutral API, SDK, and set of tools for generating telemetry data. For tracing, the key concepts are:

• TracerProvider: Factory that creates tracers and manages span processors.

• Tracer: Creates spans within a specific instrumentation scope (usually one per package).

• Span: Represents a unit of work. Has a name, start/end time, attributes, events, and status.

• SpanProcessor: Handles completed spans (batching, exporting).

• Exporter: Sends span data to a backend (Jaeger, Zipkin, OTLP collector).

• Propagator: Serializes/deserializes trace context across process boundaries (HTTP headers, message queues).

Setting Up the OTel SDK

Start with the dependencies. We will use OTLP over gRPC as the export protocol, which works with Jaeger, Tempo, Datadog, and any OTLP-compatible backend.

go get go.opentelemetry.io/otel \
       go.opentelemetry.io/otel/sdk \
       go.opentelemetry.io/otel/sdk/trace \
       go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc \
       go.opentelemetry.io/otel/propagation \
       go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Now build the tracer provider. This is your application's tracing backbone — initialize it once at startup and shut it down cleanly on exit.

package telemetry

import (
    "context"
    "fmt"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

type Config struct {
    ServiceName    string
    ServiceVersion string
    Environment    string
    OTLPEndpoint   string // e.g., "otel-collector:4317"
    SampleRate     float64
}

func InitTracer(ctx context.Context, cfg Config) (shutdown func(context.Context) error, err error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint(cfg.OTLPEndpoint),
        otlptracegrpc.WithInsecure(), // Use WithTLSCredentials in production
    )
    if err != nil {
        return nil, fmt.Errorf("creating OTLP exporter: %w", err)
    }

    res, err := resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName(cfg.ServiceName),
            semconv.ServiceVersion(cfg.ServiceVersion),
            semconv.DeploymentEnvironment(cfg.Environment),
        ),
    )
    if err != nil {
        return nil, fmt.Errorf("creating resource: %w", err)
    }

    sampler := sdktrace.ParentBased(
        sdktrace.TraceIDRatioBased(cfg.SampleRate),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter,
            sdktrace.WithMaxQueueSize(2048),
            sdktrace.WithMaxExportBatchSize(512),
            sdktrace.WithBatchTimeout(5*time.Second),
        ),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sampler),
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    return tp.Shutdown, nil
}

A few deliberate decisions here worth explaining:

ParentBased sampler: If an incoming request already carries a sampling decision (from an upstream service), we honor it. This prevents broken traces where a parent is sampled but a child is not. The TraceIDRatioBased sampler only applies to root spans ‚Äî requests that originate at this service.

Batch processor tuning: The defaults are conservative. For high-throughput services (10k+ requests/sec), increase MaxQueueSize and MaxExportBatchSize. The BatchTimeout controls the maximum delay before a batch is flushed, even if it is not full.

Resource attributes: These tag every span with service identity. Backend UIs use them for filtering and grouping. Always include service name, version, and environment at minimum.

Wiring It Into main()

func main() {
    ctx := context.Background()

    shutdown, err := telemetry.InitTracer(ctx, telemetry.Config{
        ServiceName:    "order-service",
        ServiceVersion: "1.4.2",
        Environment:    os.Getenv("APP_ENV"),
        OTLPEndpoint:   os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),
        SampleRate:     0.1, // Sample 10% of new traces
    })
    if err != nil {
        log.Fatalf("init tracer: %v", err)
    }
    defer func() {
        // Give the exporter 10 seconds to flush remaining spans
        ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        defer cancel()
        if err := shutdown(ctx); err != nil {
            log.Printf("tracer shutdown: %v", err)
        }
    }()

    // ... start HTTP server
}

The deferred shutdown is critical. Without it, spans buffered in the batch processor are lost when the process exits. This pairs well with the graceful shutdown pattern from article #14 in this series.

Instrumenting HTTP Handlers

OpenTelemetry provides otelhttp, a middleware that automatically creates spans for incoming HTTP requests and extracts trace context from headers.

package main

import (
    "net/http"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func newRouter() http.Handler {
    mux := http.NewServeMux()
    mux.HandleFunc("/orders", handleListOrders)
    mux.HandleFunc("/orders/create", handleCreateOrder)

    // Wrap the entire mux with OTel instrumentation
    return otelhttp.NewHandler(mux, "http-server",
        otelhttp.WithMessageEvents(otelhttp.ReadEvents, otelhttp.WriteEvents),
    )
}

Every incoming request now generates a span named after the HTTP route, with attributes for method, status code, URL scheme, and request/response size. The middleware also extracts traceparent and tracestate headers automatically ‚Äî this is how trace context arrives from upstream services.

For more granular route naming (avoiding high-cardinality span names like /orders/abc123), use a custom span name formatter:

otelhttp.NewHandler(mux, "http-server",
    otelhttp.WithSpanNameFormatter(func(operation string, r *http.Request) string {
        // Use the route pattern, not the actual URL
        return fmt.Sprintf("%s %s", r.Method, r.Pattern)
    }),
)

Propagating Context Across Services

Trace context propagation is where distributed tracing earns its name. When service A calls service B, it must inject the current trace context into the outgoing request headers. Service B then extracts it and continues the same trace.

Wrap your HTTP client with otelhttp.NewTransport:

package httpclient

import (
    "net/http"
    "time"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

// New returns an HTTP client that propagates trace context
// and creates client spans for every outgoing request.
func New() *http.Client {
    return &http.Client{
        Timeout: 30 * time.Second,
        Transport: otelhttp.NewTransport(
            &http.Transport{
                MaxIdleConns:        100,
                MaxIdleConnsPerHost: 10,
                IdleConnTimeout:     90 * time.Second,
            },
        ),
    }
}

Now every outgoing HTTP call automatically injects traceparent headers and creates a client-side span. Here is what it looks like in a handler that calls a downstream service:

func handleCreateOrder(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()

    // This span is a child of the HTTP server span created by otelhttp
    order, err := processOrder(ctx, r)
    if err != nil {
        http.Error(w, "failed to process order", http.StatusInternalServerError)
        return
    }

    // The client automatically propagates trace context
    client := httpclient.New()
    req, _ := http.NewRequestWithContext(ctx, "POST",
        "http://payment-service/charge", orderToJSON(order))
    resp, err := client.Do(req)
    // ...
}

The key detail: always pass ctx through. The context carries the current span. If you use context.Background() instead of the request context, you break the trace chain.

Custom Spans and Attributes

Auto-instrumentation covers HTTP boundaries, but the most valuable tracing data comes from custom spans around your business logic.

package order

import (
    "context"
    "fmt"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

var tracer = otel.Tracer("github.com/yourorg/order-service/internal/order")

func ProcessOrder(ctx context.Context, req CreateOrderRequest) (*Order, error) {
    ctx, span := tracer.Start(ctx, "order.Process",
        trace.WithAttributes(
            attribute.String("order.customer_id", req.CustomerID),
            attribute.Int("order.item_count", len(req.Items)),
            attribute.String("order.currency", req.Currency),
        ),
    )
    defer span.End()

    // Validate inventory
    if err := validateInventory(ctx, req.Items); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "inventory validation failed")
        return nil, fmt.Errorf("validate inventory: %w", err)
    }

    // Calculate pricing
    total, err := calculateTotal(ctx, req.Items, req.Currency)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "pricing calculation failed")
        return nil, fmt.Errorf("calculate total: %w", err)
    }
    span.SetAttributes(attribute.Float64("order.total", total))

    // Persist
    order, err := saveOrder(ctx, req, total)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "persistence failed")
        return nil, fmt.Errorf("save order: %w", err)
    }

    span.SetAttributes(attribute.String("order.id", order.ID))
    return order, nil
}

Several patterns at work here:

1. Tracer per package: otel.Tracer("...") creates an instrumentation scope. Use the fully qualified package path. This shows up in tracing UIs and helps identify which code produced a span.

2. Record errors AND set status: RecordError adds an exception event to the span (with stack trace if available). SetStatus marks the span as failed. Do both ‚Äî some backends use one, some the other.

3. Add attributes progressively: You do not need to know all attributes upfront. Add them as the function progresses. The order.id is only available after the database write, so we set it then.

4. Return the enriched ctx: tracer.Start returns a new context with the span. Pass this to downstream functions so their spans nest correctly.

Instrumenting Database Calls

Database calls are almost always where latency hides. Create spans around them:

func saveOrder(ctx context.Context, req CreateOrderRequest, total float64) (*Order, error) {
    ctx, span := tracer.Start(ctx, "order.SaveToDB",
        trace.WithAttributes(
            attribute.String("db.system", "postgresql"),
            attribute.String("db.operation", "INSERT"),
            attribute.String("db.sql.table", "orders"),
        ),
    )
    defer span.End()

    query := `INSERT INTO orders (customer_id, total, currency, status)
              VALUES (\(1, \)2, $3, 'pending') RETURNING id, created_at`

    var order Order
    err := db.QueryRowContext(ctx, query, req.CustomerID, total, req.Currency).
        Scan(&order.ID, &order.CreatedAt)
    if err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "db insert failed")
        return nil, err
    }

    return &order, nil
}

Use the OpenTelemetry semantic conventions for databases (db.system, db.operation, db.sql.table) so your tracing backend can render database-specific views. Do not put the full SQL query in a span attribute in production ‚Äî it may contain PII.

Connecting to Jaeger or Zipkin

Option 1: OTLP-native (Recommended)

Modern Jaeger (v1.35+) natively supports OTLP over gRPC on port 4317. Our setup already works ‚Äî just point OTEL_EXPORTER_OTLP_ENDPOINT at jaeger:4317.

For Grafana Tempo, it is the same: OTLP on port 4317.

Option 2: OpenTelemetry Collector

For production, run an OTel Collector as a sidecar or daemonset. This decouples your services from the tracing backend and adds capabilities like tail-based sampling, attribute filtering, and multi-backend fan-out.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
    send_batch_size: 512
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger, otlp/tempo]

The collector's memory_limiter processor is essential. Without it, a traffic spike can OOM the collector and you lose all buffered spans.

Option 3: Zipkin Exporter

If you are locked into Zipkin, use the dedicated exporter:

import "go.opentelemetry.io/otel/exporters/zipkin"

exporter, err := zipkin.New("http://zipkin:9411/api/v2/spans")

Trace Context in Async Workflows

HTTP propagation covers synchronous calls, but many production systems use message queues. You need to manually inject and extract trace context.

Producer (publishing to a queue):

func publishOrderEvent(ctx context.Context, event OrderCreatedEvent) error {
    carrier := make(propagation.MapCarrier)
    otel.GetTextMapPropagator().Inject(ctx, carrier)

    // Attach trace context as message headers
    headers := make(map[string]string)
    for _, key := range []string{"traceparent", "tracestate"} {
        if val := carrier.Get(key); val != "" {
            headers[key] = val
        }
    }

    msg := &queue.Message{
        Body:    encodeEvent(event),
        Headers: headers,
    }
    return queue.Publish(ctx, "order.created", msg)
}

Consumer (reading from a queue):

func handleOrderCreatedMessage(msg *queue.Message) error {
    carrier := propagation.MapCarrier(msg.Headers)
    ctx := otel.GetTextMapPropagator().Extract(context.Background(), carrier)

    ctx, span := tracer.Start(ctx, "queue.ProcessOrderCreated",
        trace.WithSpanKind(trace.SpanKindConsumer),
        trace.WithAttributes(
            attribute.String("messaging.system", "rabbitmq"),
            attribute.String("messaging.operation", "process"),
        ),
    )
    defer span.End()

    // Process with the restored trace context
    return processOrderCreated(ctx, msg.Body)
}

The consumer's span becomes a child of the producer's span, even though they run in different processes and potentially on different machines. This is the real power of distributed tracing.

Production Best Practices

1. Sample Intelligently

Tracing 100% of traffic is expensive. Use ParentBased(TraceIDRatioBased(0.1)) to sample 10% of new traces while always honoring upstream sampling decisions. For error investigation, implement tail-based sampling at the collector level ‚Äî it captures all traces that contain an error span.

2. Control Span Cardinality

Every unique span name creates a series in your backend. Avoid dynamic values in span names:

// BAD: creates thousands of unique span names
tracer.Start(ctx, fmt.Sprintf("getUser-%s", userID))

// GOOD: use attributes for dynamic values
tracer.Start(ctx, "user.Get",
    trace.WithAttributes(attribute.String("user.id", userID)),
)

3. Keep Attribute Values Bounded

High-cardinality attributes (full URLs, raw SQL, request bodies) bloat storage and slow down queries. Stick to IDs, enums, and short strings. Use span events for detailed debugging data that you only need occasionally.

4. Set Span Kind Correctly

Span kind tells the backend how to render the span:

• SpanKindServer ‚Äî incoming RPC/HTTP (set by otelhttp middleware)

• SpanKindClient ‚Äî outgoing RPC/HTTP (set by otelhttp transport)

• SpanKindProducer ‚Äî message queue publish

• SpanKindConsumer ‚Äî message queue consume

• SpanKindInternal ‚Äî default, local operations

5. Use Span Links for Fan-Out

When one request triggers multiple async operations, use span links instead of parent-child relationships. A batch job that processes 1000 messages should link to each producer span, not parent them — otherwise you get an unreadable trace tree.

links := make([]trace.Link, len(messages))
for i, msg := range messages {
    carrier := propagation.MapCarrier(msg.Headers)
    remoteCtx := otel.GetTextMapPropagator().Extract(context.Background(), carrier)
    links[i] = trace.LinkFromContext(remoteCtx)
}

ctx, span := tracer.Start(ctx, "batch.ProcessMessages",
    trace.WithLinks(links...),
)

6. Correlate Traces with Logs

Inject the trace ID into your structured logger so log lines are searchable by trace:

func loggerFromContext(ctx context.Context) *slog.Logger {
    spanCtx := trace.SpanContextFromContext(ctx)
    return slog.Default().With(
        slog.String("trace_id", spanCtx.TraceID().String()),
        slog.String("span_id", spanCtx.SpanID().String()),
    )
}

Grafana, Datadog, and most observability platforms can jump from a log line to the full trace when trace_id is present.

7. Flush on Shutdown

We covered this in the setup, but it bears repeating: TracerProvider.Shutdown() must be called with a generous timeout. The batch processor may have hundreds of spans queued. Pair it with your graceful shutdown handler.

What a Full Trace Looks Like

After instrumenting two services, here is what a trace looks like in Jaeger:

order-service: POST /orders/create         [============]  320ms
  ├─ order.Process                         [==========]    280ms
  │   ├─ order.ValidateInventory           [===]            85ms
  │   ├─ order.CalculateTotal              [=]              12ms
  │   └─ order.SaveToDB                    [==]             45ms
  └─ HTTP POST payment-service/charge      [=====]        130ms
      payment-service: POST /charge        [====]         125ms
        ├─ payment.ValidateCard            [=]              15ms
        └─ payment.ProcessCharge           [===]           105ms

At a glance: 320ms total, and the payment service accounts for 130ms of that. The database insert is 45ms — reasonable for a write with index updates. If this endpoint starts breaching its latency SLO, you know exactly where to look.

Wrapping Up

The initial investment in distributed tracing pays for itself the first time you debug a cross-service latency issue without manually correlating logs. The setup is:

1. Initialize a TracerProvider with OTLP export and sensible batching.

2. Wrap HTTP servers with otelhttp.NewHandler for automatic span creation.

3. Wrap HTTP clients with otelhttp.NewTransport for automatic context propagation.

4. Add custom spans around business logic and database calls.

5. Manually propagate context through message queues.

6. Sample, control cardinality, correlate with logs.

All the code in this article uses the stable OpenTelemetry Go SDK. It works with any OTLP-compatible backend — Jaeger, Tempo, Datadog, Honeycomb, or the OTel Collector. Pick one, point the exporter at it, and start seeing your system for what it really is.

If this article helped you, consider buying me a coffee on Ko-fi! Follow me for more production backend patterns.

I was unable to save this to a file because both Write and Bash permissions were denied. The complete article is above -- you can copy it or grant file write permission so I can save it to /Users/miniclaw/workspace/article-15-distributed-tracing-otel.md.

Get it wrong and you get data loss, broken client connections, and 502s during every deploy. Get it right and your deployments become invisible to users.

This article walks through the patterns that make graceful shutdown reliable in production Go services.

Why Graceful Shutdown Matters

Three things go wrong when a service dies abruptly:

Data loss. Buffered writes — log batches, metrics, queue messages — vanish. Database transactions in progress get rolled back (best case) or leave inconsistent state (worst case).

Connection drops. In-flight HTTP requests get TCP RST. gRPC streams break mid-message. WebSocket clients see unexpected disconnections. Depending on client retry logic, this can cascade.

Load balancer draining failures. Kubernetes removes the pod from the Service endpoints, but if your process exits before the kubelet propagates that change, the load balancer still sends traffic to a dead pod. The standard fix is to keep serving for a few seconds after receiving SIGTERM — but only if your shutdown sequence is correct.

The Foundation: os.Signal + context.Context

Every graceful shutdown in Go starts with the same two primitives: signal notification and context cancellation.

func main() {
    // Create a context that cancels on SIGINT or SIGTERM
    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    // Pass ctx to everything that needs to know about shutdown
    if err := run(ctx); err != nil {
        log.Fatalf("service exited with error: %v", err)
    }
}

signal.NotifyContext (added in Go 1.16) is the cleanest way to wire OS signals into the context tree. When the signal arrives, the context's Done() channel closes, and every goroutine watching that context knows it's time to wrap up.

The key principle: propagate context everywhere. Every HTTP handler, every database query, every background worker should accept a context.Context. This is how you make shutdown cooperative rather than forceful.

HTTP Server Graceful Shutdown

http.Server.Shutdown() does exactly what you want: it stops accepting new connections, waits for in-flight requests to complete, and then returns.

func runHTTPServer(ctx context.Context) error {
    mux := http.NewServeMux()
    mux.HandleFunc("/api/data", handleData)

    srv := &http.Server{
        Addr:    ":8080",
        Handler: mux,
    }

    // Start server in a goroutine
    errCh := make(chan error, 1)
    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            errCh <- err
        }
        close(errCh)
    }()

    // Wait for shutdown signal
    <-ctx.Done()
    log.Println("shutting down HTTP server...")

    // Give in-flight requests a deadline to finish
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        return fmt.Errorf("http server shutdown: %w", err)
    }

    // Check if server goroutine hit an error before shutdown
    return <-errCh
}

Notice the shutdown context uses context.Background(), not the already-cancelled parent context. This is a common mistake — if you derive the shutdown timeout from the cancelled context, the timeout is already expired and Shutdown() returns immediately, dropping in-flight requests.

gRPC Graceful Stop

gRPC has its own graceful shutdown method that mirrors the HTTP pattern:

func runGRPCServer(ctx context.Context) error {
    lis, err := net.Listen("tcp", ":9090")
    if err != nil {
        return fmt.Errorf("listen: %w", err)
    }

    grpcServer := grpc.NewServer()
    pb.RegisterMyServiceServer(grpcServer, &myService{})

    // Enable reflection for debugging
    reflection.Register(grpcServer)

    errCh := make(chan error, 1)
    go func() {
        if err := grpcServer.Serve(lis); err != nil {
            errCh <- err
        }
        close(errCh)
    }()

    <-ctx.Done()
    log.Println("shutting down gRPC server...")

    // GracefulStop waits for active RPCs to finish
    // but we add a hard deadline as a safety net
    stopped := make(chan struct{})
    go func() {
        grpcServer.GracefulStop()
        close(stopped)
    }()

    select {
    case <-stopped:
        log.Println("gRPC server stopped gracefully")
    case <-time.After(10 * time.Second):
        log.Println("gRPC server force stop (timeout)")
        grpcServer.Stop()
    }

    return <-errCh
}

GracefulStop() has no built-in timeout, so you must wrap it with a deadline yourself. Without this, a single slow-draining stream can hang your shutdown indefinitely.

Worker Pool Draining with sync.WaitGroup

Background workers — queue consumers, cron jobs, batch processors — need a different pattern. The worker checks ctx.Done() between units of work, and a WaitGroup lets the main goroutine wait for all workers to finish their current task.

func runWorkerPool(ctx context.Context, queue <-chan Job) error {
    const numWorkers = 8
    var wg sync.WaitGroup

    for i := 0; i < numWorkers; i++ {
        wg.Add(1)
        go func(id int) {
            defer wg.Done()
            for {
                select {
                case <-ctx.Done():
                    log.Printf("worker %d: shutting down", id)
                    return
                case job, ok := <-queue:
                    if !ok {
                        return // channel closed
                    }
                    if err := processJob(ctx, job); err != nil {
                        log.Printf("worker %d: job failed: %v", id, err)
                    }
                }
            }
        }(i)
    }

    // Wait for all workers to finish their current job
    wg.Wait()
    return nil
}

The critical detail: processJob receives the context so it can abandon long-running work. If a job takes 5 minutes and your shutdown budget is 30 seconds, the job needs to respect context cancellation or you'll hit SIGKILL.

Database Connection Cleanup

Database connections are the easiest to get right and the most painful to get wrong. Leaked connections exhaust the connection pool on the database side, causing failures across all services.

func setupDatabase(ctx context.Context) (*sql.DB, error) {
    db, err := sql.Open("postgres", os.Getenv("DATABASE_URL"))
    if err != nil {
        return nil, err
    }

    db.SetMaxOpenConns(25)
    db.SetMaxIdleConns(5)
    db.SetConnMaxLifetime(5 * time.Minute)

    if err := db.PingContext(ctx); err != nil {
        db.Close()
        return nil, fmt.Errorf("database ping: %w", err)
    }

    return db, nil
}

The cleanup is simple — db.Close() — but the ordering matters. Close the database after servers and workers have stopped, since they may still be executing queries during their drain period.

Composing Everything with errgroup

Now let's stitch it all together. errgroup from golang.org/x/sync gives us structured concurrency: run components in parallel, shut them all down if any one fails, and collect errors.

func run(ctx context.Context) error {
    db, err := setupDatabase(ctx)
    if err != nil {
        return fmt.Errorf("database: %w", err)
    }
    defer db.Close() // Always last to close

    jobQueue := make(chan Job, 100)

    g, ctx := errgroup.WithContext(ctx)

    // HTTP server
    g.Go(func() error {
        return runHTTPServer(ctx)
    })

    // gRPC server
    g.Go(func() error {
        return runGRPCServer(ctx)
    })

    // Worker pool
    g.Go(func() error {
        return runWorkerPool(ctx, jobQueue)
    })

    // Health monitor (optional: triggers shutdown on critical failures)
    g.Go(func() error {
        return runHealthMonitor(ctx, db)
    })

    log.Println("service started")
    err = g.Wait()
    log.Println("all components stopped")

    return err
}

errgroup.WithContext creates a derived context that cancels when any goroutine in the group returns an error. This means if the HTTP server fails to bind its port, the context cancels, and the gRPC server and workers also begin shutting down. Unified lifecycle management.

Note that db.Close() runs via defer — it executes after g.Wait() returns, guaranteeing all servers and workers have fully drained before we close database connections. Dependency ordering through deferred cleanup is the simplest correct pattern.

Health Check Integration with Kubernetes

Kubernetes needs to know when your pod is ready to receive traffic and when it should stop sending traffic. The readiness probe is your coordination point.

type healthHandler struct {
    ready atomic.Bool
}

func (h *healthHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if h.ready.Load() {
        w.WriteHeader(http.StatusOK)
        w.Write([]byte("ok"))
    } else {
        w.WriteHeader(http.StatusServiceUnavailable)
        w.Write([]byte("shutting down"))
    }
}

func runHTTPServer(ctx context.Context) error {
    health := &healthHandler{}
    health.ready.Store(true)

    mux := http.NewServeMux()
    mux.Handle("/healthz", health)
    mux.HandleFunc("/api/data", handleData)

    srv := &http.Server{Addr: ":8080", Handler: mux}

    errCh := make(chan error, 1)
    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            errCh <- err
        }
        close(errCh)
    }()

    <-ctx.Done()

    // Step 1: Mark as not ready — Kubernetes stops sending traffic
    health.ready.Store(false)

    // Step 2: Wait for load balancer to propagate the change
    time.Sleep(5 * time.Second)

    // Step 3: Now shut down the server
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        return fmt.Errorf("http server shutdown: %w", err)
    }

    return <-errCh
}

That time.Sleep(5 * time.Second) looks wrong, but it's essential. Kubernetes endpoint propagation is eventually consistent. After your readiness probe starts failing, it takes a few seconds for kube-proxy/iptables rules to update across nodes. During that window, traffic still arrives. The sleep keeps your server alive to handle it.

In your pod spec:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 2
  failureThreshold: 1
terminationGracePeriodSeconds: 45

Set terminationGracePeriodSeconds to comfortably exceed your total shutdown budget (sleep + drain timeout + cleanup).

Common Mistakes

1. Deriving shutdown context from the cancelled parent.

// WRONG: ctx is already cancelled, timeout has no effect
shutdownCtx, cancel := context.WithTimeout(ctx, 10*time.Second)

// RIGHT: fresh context with its own deadline
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)

2. No shutdown timeout at all. GracefulStop() and custom drain loops can block forever. Always wrap with a deadline and fall back to a forced stop.

3. Wrong cleanup ordering. If you close the database before the HTTP server finishes draining, in-flight request handlers get connection errors. Rule of thumb: shut down in reverse order of initialization. Servers first, then workers, then infrastructure (DB, caches, message brokers).

4. Forgetting to stop timers and tickers.

ticker := time.NewTicker(30 * time.Second)
defer ticker.Stop() // Don't leak this

for {
    select {
    case <-ctx.Done():
        return nil
    case <-ticker.C:
        doPeriodicWork()
    }
}

5. Not testing shutdown. Send SIGTERM to your service in integration tests. Verify that in-flight requests complete, no errors appear in logs, and the process exits with code 0.

Putting It All Together

The complete pattern:

1. Trap SIGINT/SIGTERM with signal.NotifyContext

2. Start all components under an errgroup with the signal-aware context

3. Each component watches ctx.Done() and begins draining

4. Mark readiness probe as unhealthy, sleep for LB propagation

5. Shut down servers (HTTP, gRPC) with a timeout

6. Wait for workers to finish current tasks

7. Close infrastructure connections (DB, caches) last via defer

This sequence handles the normal deploy case (Kubernetes rolling update), the crash case (a component fails and triggers group shutdown), and the manual stop case (Ctrl+C in development).

Graceful shutdown isn't glamorous, but it's the difference between deploys that are invisible to users and deploys that generate a spike of errors in your dashboard. Build it in from the start — retrofitting it later is always harder.

This is part of the Production Backend Patterns series. Next up: structured logging and distributed tracing in Go.

If this article helped you, consider buying me a coffee on Ko-fi! Follow me for more production backend patterns.

Beyond if err != nil - techniques that work.

Always Wrap

return fmt.Errorf("get order %s: %w", id, err)

Sentinels

var ErrNotFound = errors.New("not found")

HTTP Pattern

type HandlerFunc func(w http.ResponseWriter, r *http.Request) error

errgroup

g, ctx := errgroup.WithContext(ctx)
g.Go(func() error { return fn() })

If this was helpful, you can support my work at ko-fi.com/nopkt

1. Using SELECT * Instead of Specific Columns

Bad SQL:

SELECT * FROM orders WHERE customer_id = 42;

Good SQL:

SELECT order_id, total_amount, created_at
FROM orders
WHERE customer_id = 42;

Why: SELECT * bloats I/O, defeats index-only scans, and wastes bandwidth. Selecting specific columns lets PostgreSQL use an index-only scan without touching the heap.

Typical speedup: 2-10x

2. Missing Indexes on WHERE and JOIN Columns

Bad SQL:

-- No index on orders.customer_id
SELECT order_id, total_amount
FROM orders
WHERE customer_id = 42;

Good SQL:

CREATE INDEX idx_orders_customer_id ON orders (customer_id);

SELECT order_id, total_amount
FROM orders
WHERE customer_id = 42;

Why: Without an index, PostgreSQL scans every row in the table. On a 10M-row table, that means scanning all rows to find maybe 50 matches. An index turns this into a B-tree lookup touching a handful of pages.

Typical speedup: 100-10,000x on large tables. The single biggest performance win.

3. OFFSET Pagination vs Keyset Pagination

Bad SQL:

SELECT id, title, created_at
FROM articles
ORDER BY created_at DESC
LIMIT 20 OFFSET 9980;

Good SQL:

SELECT id, title, created_at
FROM articles
WHERE created_at < '2024-01-15 10:30:00'
ORDER BY created_at DESC
LIMIT 20;

Why: OFFSET forces PostgreSQL to fetch and discard 9,980 rows before returning your 20. It is O(n) where n is the offset. Keyset pagination uses a WHERE clause to jump directly to the right position - O(1) regardless of depth.

Typical speedup: 10-1000x on deep pages

4. N+1 Queries vs JOINs

Bad SQL (application code pattern):

SELECT id, name FROM customers LIMIT 100;
-- Then 100 separate queries:
SELECT SUM(total_amount) FROM orders WHERE customer_id = 1;
SELECT SUM(total_amount) FROM orders WHERE customer_id = 2;
-- ... 98 more

Good SQL:

SELECT c.id, c.name, COALESCE(SUM(o.total_amount), 0) AS total_spent
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id
GROUP BY c.id, c.name
LIMIT 100;

Why: The N+1 pattern sends 101 separate queries with individual round-trips. A single JOIN lets PostgreSQL optimize the entire operation with hash joins, merge joins, or nested loops.

Typical speedup: 5-50x

5. Not Using EXPLAIN ANALYZE

Bad approach:

-- Guessing which indexes to add
CREATE INDEX idx_whatever ON orders (status);
CREATE INDEX idx_whatever2 ON orders (status, customer_id);

Good approach:

EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT o.order_id, c.name
FROM orders o
JOIN customers c ON c.id = o.customer_id
WHERE o.status = 'pending'
  AND o.created_at > NOW() - INTERVAL '7 days';

Why: EXPLAIN ANALYZE executes the query and shows exactly what PostgreSQL did: which indexes it used, row estimate accuracy, and where time was spent. Look for Seq Scans on large tables, estimate vs actual row divergence, and expensive sorts. Stop guessing, start measuring.

Typical speedup: Tells you exactly what to fix.

6. LIKE with Leading Wildcards vs pg_trgm

Bad SQL:

SELECT id, name FROM products
WHERE name LIKE '%%wireless%%';

Good SQL:

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX idx_products_name_trgm
  ON products USING gin (name gin_trgm_ops);

SELECT id, name FROM products
WHERE name LIKE '%%wireless%%';

Why: B-tree indexes only help with prefix matching (LIKE 'wireless%%'). A leading wildcard forces a sequential scan. The pg_trgm extension builds a GIN index over trigrams, enabling index usage for substring, ILIKE, and regex matching.

Typical speedup: 50-500x on large text columns

7. COUNT(*) for Existence Checks vs EXISTS

Bad SQL:

SELECT CASE WHEN COUNT(*) > 0 THEN true ELSE false END
FROM orders
WHERE customer_id = 42 AND status = 'pending';

Good SQL:

SELECT EXISTS (
  SELECT 1 FROM orders
  WHERE customer_id = 42 AND status = 'pending'
);

Why: COUNT(*) scans and counts every matching row. If there are 10,000 matches, it counts all 10,000 just to check if any exist. EXISTS returns true on the first match and stops - a short-circuit operation.

Typical speedup: 10-10,000x depending on match count

8. Single-Row INSERTs vs Batching / COPY

Bad SQL:

INSERT INTO events (type, data, created_at) VALUES ('click', '{}', NOW());
INSERT INTO events (type, data, created_at) VALUES ('view', '{}', NOW());
-- 9,998 more individual statements

Good SQL:

-- Multi-value INSERT
INSERT INTO events (type, data, created_at) VALUES
  ('click', '{}', NOW()),
  ('view', '{}', NOW()),
  ('scroll', '{}', NOW());

-- Or COPY for bulk loads
COPY events (type, data, created_at)
FROM STDIN WITH (FORMAT csv);

Why: Each INSERT has overhead: network round-trip, SQL parsing, query planning, WAL logging, and transaction commit. Batching and COPY amortize this dramatically.

Typical speedup: 10-100x for batching, 100-1000x for COPY

9. No Connection Pooling (Use PgBouncer)

Bad approach: Opening a new database connection per request. Each connection forks a new PostgreSQL backend process and allocates ~10MB of memory.

Good approach: Use PgBouncer with transaction-mode pooling:

[databases]
mydb = host=localhost port=5432 dbname=mydb

[pgbouncer]
listen_port = 6432
pool_mode = transaction
max_client_conn = 1000
default_pool_size = 20

Why: Connection creation is expensive. Under load, hundreds of connections create memory pressure and context-switching overhead. PgBouncer maintains a small pool of actual connections serving thousands of clients, and protects against connection storms during traffic spikes.

Typical speedup: 2-20x under concurrent load

10. Not Using Partial Indexes

Bad SQL:

CREATE INDEX idx_orders_status ON orders (status);
SELECT * FROM orders WHERE status = 'failed';

Good SQL:

CREATE INDEX idx_orders_failed ON orders (created_at)
WHERE status = 'failed';

SELECT * FROM orders
WHERE status = 'failed'
ORDER BY created_at DESC LIMIT 10;

Why: If 95%% of orders are completed and only 1%% are failed, a full index on status is massive but mostly useless. A partial index with WHERE status = 'failed' indexes only the 1%% you query. It is dramatically smaller, fits in cache, and is faster to scan and maintain.

Use partial indexes for: status flags, soft deletes, queue tables, and any skewed distribution column.

Typical speedup: 3-10x with significantly less disk space and write overhead

Summary

Every optimization follows one principle: give PostgreSQL less work to do. Start with EXPLAIN ANALYZE on your slowest queries today.

Why Build a CLI Tool in Rust?

Rust is an excellent language for CLI tools. It compiles to a single static binary, starts instantly, and gives you memory safety without a garbage collector.

Many beloved tools like ripgrep, fd, bat, and exa are written in Rust.

In this tutorial, we will build rsearch -- a fast file content searcher with colored output. By the end, you will have a tool published on crates.io that others can install with cargo install.

Step 1: Project Setup

cargo new rsearch
cd rsearch

This gives you:

rsearch/
  Cargo.toml
  src/
    main.rs

Step 2: Dependencies in Cargo.toml

Open Cargo.toml and add the crates we need:

[package]
name = "rsearch"
version = "0.1.0"
edition = "2021"
description = "A fast file content searcher with colored output"
license = "MIT"

[dependencies]
clap = { version = "4", features = ["derive"] }
regex = "1"
colored = "2"
anyhow = "1"
ignore = "0.4"

[dev-dependencies]
assert_cmd = "2"
predicates = "3"
tempfile = "3"

[profile.release]
opt-level = "z"
lto = true
codegen-units = 1
strip = true
panic = "abort"

Here is what each crate does:

The [profile.release] section enables aggressive binary size optimization -- we will revisit this at the end.

Step 3: Argument Parsing with Clap Derive API

Clap derive API lets you define your CLI interface as a struct. Replace the contents of src/main.rs:

use anyhow::{Context, Result};
use clap::Parser;
use colored::Colorize;
use ignore::WalkBuilder;
use regex::Regex;
use std::fs;
use std::path::PathBuf;

/// rsearch - A fast file content searcher
#[derive(Parser, Debug)]
#[command(name = "rsearch", version, about)]
struct Args {
    /// The regex pattern to search for
    pattern: String,

    /// Directory to search in
    #[arg(default_value = ".")]
    path: PathBuf,

    /// File extension filter
    #[arg(short, long)]
    ext: Option<String>,

    /// Show line numbers
    #[arg(short = 'n', long, default_value_t = true)]
    line_numbers: bool,

    /// Maximum search depth
    #[arg(short, long)]
    depth: Option<usize>,

    /// Include hidden files
    #[arg(long)]
    hidden: bool,

    /// Case-insensitive search
    #[arg(short = 'i', long)]
    ignore_case: bool,
}

Every field becomes a CLI flag. Clap auto-generates --help and --version for free.

Running rsearch --help will output:

rsearch - A fast file content searcher

Usage: rsearch [OPTIONS] <PATTERN> [PATH]

Arguments:
  <PATTERN>  The regex pattern to search for
  [PATH]     Directory to search in [default: .]

Options:
  -e, --ext <EXT>      File extension filter
  -n, --line-numbers   Show line numbers
  -d, --depth <DEPTH>  Maximum search depth
      --hidden         Include hidden files
  -i, --ignore-case    Case-insensitive search
  -h, --help           Print help
  -V, --version        Print version

Step 4: File Walking with the ignore Crate

The ignore crate (from the ripgrep family) walks directories while automatically respecting .gitignore rules.

fn walk_files(args: &Args) -> impl Iterator<Item = PathBuf> + '_ {
    let mut builder = WalkBuilder::new(&args.path);
    builder.hidden(!args.hidden);

    if let Some(depth) = args.depth {
        builder.max_depth(Some(depth));
    }

    let ext_filter = args.ext.clone();

    builder.build().filter_map(move |entry| {
        let entry = entry.ok()?;
        let path = entry.path();

        if !path.is_file() {
            return None;
        }

        if let Some(ref ext) = ext_filter {
            if path.extension()?.to_str()? != ext.as_str() {
                return None;
            }
        }

        Some(path.to_path_buf())
    })
}

Key points:

• WalkBuilder handles .gitignore parsing automatically

• We filter by extension if the user passed --ext

• Hidden files are skipped by default unless --hidden is set

• The iterator is lazy -- files are yielded on demand

Step 5: Regex Search with Colored Output

Now the core search logic. For each file, we read its contents, search line by line, and print matches with color highlighting:

fn search_file(
    path: &PathBuf,
    re: &Regex,
    show_line_nums: bool,
) -> Result<Vec<String>> {
    let content = fs::read_to_string(path)
        .with_context(|| format!("Failed to read {}", path.display()))?;

    let mut matches = Vec::new();

    for (line_num, line) in content.lines().enumerate() {
        if re.is_match(line) {
            let highlighted = re.replace_all(
                line,
                |caps: &regex::Captures| {
                    caps[0].red().bold().to_string()
                },
            );

            let formatted = if show_line_nums {
                format!(
                    "{}:{} {}",
                    path.display().to_string().green(),
                    (line_num + 1).to_string().yellow(),
                    highlighted
                )
            } else {
                format!(
                    "{}: {}",
                    path.display().to_string().green(),
                    highlighted
                )
            };

            matches.push(formatted);
        }
    }

    Ok(matches)
}

The output looks like: src/main.rs:42 fn search_file(...) where the filename is green, the line number is yellow, and the matched text is bold red.

Step 6

test

Main fn uses anyhow Result for error handling.

Step 7: Unit Tests

Add a tests module at the bottom of main.rs:

#[cfg(test)]
mod tests {
    use super::*;
    use std::io::Write;
    use tempfile::NamedTempFile;

    #[test]
    fn test_search_finds_match() {
        let mut file = NamedTempFile::new().unwrap();
        writeln!(file, "hello world").unwrap();
        writeln!(file, "foo bar").unwrap();
        writeln!(file, "hello rust").unwrap();
        let re = Regex::new("hello").unwrap();
        let path = file.path().to_path_buf();
        let results = search_file(&path, &re, false).unwrap();
        assert_eq!(results.len(), 2);
    }

    #[test]
    fn test_search_no_match() {
        let mut file = NamedTempFile::new().unwrap();
        writeln!(file, "nothing here").unwrap();
        let re = Regex::new("xyz").unwrap();
        let results = search_file(&file.path().to_path_buf(), &re, false).unwrap();
        assert!(results.is_empty());
    }

    #[test]
    fn test_case_insensitive() {
        let mut file = NamedTempFile::new().unwrap();
        writeln!(file, "Hello World").unwrap();
        let re = Regex::new("(?i)hello").unwrap();
        let results = search_file(&file.path().to_path_buf(), &re, false).unwrap();
        assert_eq!(results.len(), 1);
    }
}

Run them with cargo test. Each test creates a temp file and verifies search behavior.

Step 8: Integration Tests with assert_cmd

Create tests/integration.rs for end-to-end testing:

use assert_cmd::Command;
use predicates::prelude::*;
use std::fs;
use tempfile::TempDir;

#[test]
fn test_search_in_directory() {
    let dir = TempDir::new().unwrap();
    fs::write(dir.path().join("test.txt"), "hello world
foo bar
hello rust
").unwrap();
    Command::cargo_bin("rsearch").unwrap()
        .args(&["hello", dir.path().to_str().unwrap()])
        .assert().success()
        .stdout(predicate::str::contains("hello"));
}

assert_cmd tests the real compiled binary.

Step 9: Publishing to crates.io

Binary Size Optimization

A typical Rust CLI drops from 4MB to 1.5MB with these settings.

Publishing Steps

1. Create account on crates.io (GitHub login)

2. cargo login your-api-token

3. cargo publish --dry-run

4. cargo publish

Now anyone can install: cargo install rsearch

Wrapping Up

We built a real CLI tool in Rust from scratch:

• Clap derive gave us a polished CLI interface

• ignore handles .gitignore rules automatically

• regex + colored produce scannable colored output

• anyhow made error handling clean

• assert_cmd let us write integration tests

• Release profile cut binary size significantly

The resulting binary starts in under 5ms and can be installed with a single command. Happy hacking!

Deploying without downtime isn't just about setting strategy: RollingUpdate. It's about health checks that actually verify readiness, connection draining that doesn't drop requests, and rollback triggers that catch problems before users do.

Here's how to set up each deployment strategy correctly on Kubernetes.

The Basics: Why Deployments Fail

Most "zero-downtime" deployments still drop requests because of three mistakes:

1. Readiness probes that lie — returning 200 before the app can serve traffic

2. No graceful shutdown — pods killed mid-request

3. Missing preStop hooks — pod removed from service before in-flight requests complete

Rolling Update (Default)

Rolling updates replace pods incrementally. Kubernetes creates new pods before terminating old ones.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Create 1 extra pod during update
      maxUnavailable: 0   # Never reduce below desired count
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: api
          image: myregistry/api-server:v2.1.0
          ports:
            - containerPort: 8080

          # Readiness: "Can this pod serve traffic?"
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3

          # Liveness: "Is this pod stuck/deadlocked?"
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
            failureThreshold: 3

          # Startup: "Is the app still starting up?"
          startupProbe:
            httpGet:
              path: /health
              port: 8080
            failureThreshold: 30
            periodSeconds: 2
            # Gives app up to 60s to start before liveness kicks in

          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]
                # Wait for endpoints controller to remove pod from Service

Why preStop: sleep 10?

When Kubernetes terminates a pod, two things happen concurrently:

1. The Endpoints controller removes the pod from the Service

2. The container receives SIGTERM

If SIGTERM arrives before the Endpoints update propagates, the load balancer still sends traffic to a pod that's shutting down. The preStop sleep gives time for the Endpoints change to propagate.

Readiness vs Liveness vs Startup

// In your Go server:
func main() {
    ready := false

    // Startup: load config, warm caches, connect to DB
    go func() {
        connectDB()
        warmCache()
        ready = true // Only now accept traffic
    }()

    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        // Liveness: am I alive and not deadlocked?
        w.WriteHeader(http.StatusOK)
    })

    http.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
        // Readiness: can I serve production traffic right now?
        if !ready {
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        // Optionally check DB connection
        if err := db.Ping(); err != nil {
            w.WriteHeader(http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })

    http.ListenAndServe(":8080", nil)
}

Never make liveness probes depend on external services. If your database goes down, liveness fails, Kubernetes restarts your pod, the new pod also can't reach the database, it restarts again — crash loop. Use readiness to stop traffic; use liveness only for detecting internal deadlocks.

Blue-Green Deployment

Run two identical environments. Switch traffic from blue (current) to green (new) atomically.

# blue-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server-blue
spec:
  replicas: 4
  selector:
    matchLabels:
      app: api-server
      version: blue
  template:
    metadata:
      labels:
        app: api-server
        version: blue
    spec:
      containers:
        - name: api
          image: myregistry/api-server:v2.0.0
---
# green-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server-green
spec:
  replicas: 4
  selector:
    matchLabels:
      app: api-server
      version: green
  template:
    metadata:
      labels:
        app: api-server
        version: green
    spec:
      containers:
        - name: api
          image: myregistry/api-server:v2.1.0
---
# service.yaml — switch by changing selector
apiVersion: v1
kind: Service
metadata:
  name: api-server
spec:
  selector:
    app: api-server
    version: blue   # Change to "green" to switch
  ports:
    - port: 80
      targetPort: 8080

Switch traffic:

# Deploy green with new version
kubectl apply -f green-deployment.yaml

# Wait for all green pods to be ready
kubectl rollout status deployment/api-server-green

# Switch traffic (atomic — one API call)
kubectl patch service api-server -p '{"spec":{"selector":{"version":"green"}}}'

# Verify, then scale down blue
kubectl scale deployment api-server-blue --replicas=0

Advantage: Instant rollback — just switch the Service selector back to "blue".

Disadvantage: Requires 2x resources during deployment.

Canary Deployment

Route a small percentage of traffic to the new version. If metrics look good, gradually increase.

Simple Canary with Replica Ratios

# Stable: 9 replicas of v2.0.0
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server-stable
spec:
  replicas: 9
  selector:
    matchLabels:
      app: api-server
      track: stable
  template:
    metadata:
      labels:
        app: api-server
        track: stable
    spec:
      containers:
        - name: api
          image: myregistry/api-server:v2.0.0
---
# Canary: 1 replica of v2.1.0 (gets ~10% of traffic)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-server
      track: canary
  template:
    metadata:
      labels:
        app: api-server
        track: canary
    spec:
      containers:
        - name: api
          image: myregistry/api-server:v2.1.0
---
# Service selects both — traffic split by replica count
apiVersion: v1
kind: Service
metadata:
  name: api-server
spec:
  selector:
    app: api-server    # Matches both stable and canary
  ports:
    - port: 80
      targetPort: 8080

Scale up canary gradually:

# Start: 10% canary
kubectl scale deployment api-server-canary --replicas=1
kubectl scale deployment api-server-stable --replicas=9

# 50% canary
kubectl scale deployment api-server-canary --replicas=5
kubectl scale deployment api-server-stable --replicas=5

# 100% canary (promote)
kubectl scale deployment api-server-canary --replicas=10
kubectl scale deployment api-server-stable --replicas=0

Automated Canary with Metrics

Use a shell script (or Flagger/Argo Rollouts in production):

#!/bin/bash
# canary-promote.sh

CANARY_DEPLOY="api-server-canary"
STABLE_DEPLOY="api-server-stable"
TOTAL_REPLICAS=10
ERROR_THRESHOLD=1  # percent

for pct in 10 25 50 75 100; do
    canary_replicas=$((TOTAL_REPLICAS * pct / 100))
    stable_replicas=$((TOTAL_REPLICAS - canary_replicas))

    echo "Setting canary to \({pct}% (\){canary_replicas} replicas)"
    kubectl scale deployment \(CANARY_DEPLOY --replicas=\)canary_replicas
    kubectl scale deployment \(STABLE_DEPLOY --replicas=\)stable_replicas

    # Wait and check error rate
    sleep 60

    # Query Prometheus for error rate (adjust query for your setup)
    error_rate=$(curl -s "http://prometheus:9090/api/v1/query?query=\
        rate(http_requests_total{deployment=\"${CANARY_DEPLOY}\",code=~\"5..\"}[1m])\
        /rate(http_requests_total{deployment=\"${CANARY_DEPLOY}\"}[1m])*100" \
        | jq '.data.result[0].value[1] // "0"' -r)

    if (( \((echo "\)error_rate > $ERROR_THRESHOLD" | bc -l) )); then
        echo "Error rate ${error_rate}% exceeds threshold. Rolling back."
        kubectl scale deployment $CANARY_DEPLOY --replicas=0
        kubectl scale deployment \(STABLE_DEPLOY --replicas=\)TOTAL_REPLICAS
        exit 1
    fi

    echo "Error rate ${error_rate}% — looks good"
done

echo "Canary promoted to 100%"

Graceful Shutdown

Your application must handle SIGTERM correctly:

func main() {
    srv := &http.Server{Addr: ":8080", Handler: mux}

    // Start server
    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            log.Fatal(err)
        }
    }()

    // Wait for SIGTERM
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
    <-quit

    log.Println("Shutting down — finishing in-flight requests...")

    // Give in-flight requests up to 30s to complete
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Printf("Forced shutdown: %v", err)
    }

    log.Println("Server stopped")
}

The shutdown sequence:

1. Kubernetes sends SIGTERM

2. preStop hook runs (sleep 10 — lets Endpoints update propagate)

3. App receives SIGTERM, stops accepting new connections

4. App finishes in-flight requests (up to 30s)

5. App exits cleanly

6. Kubernetes waits up to terminationGracePeriodSeconds (60s), then SIGKILL

Pod Disruption Budgets

Prevent too many pods from going down simultaneously (especially during node maintenance):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-server-pdb
spec:
  minAvailable: 3    # Always keep at least 3 pods running
  selector:
    matchLabels:
      app: api-server

Quick Reference

Start with rolling updates (with proper probes and preStop hooks). Move to canary when you have metrics/monitoring in place. Use blue-green for databases or stateful services where you need instant rollback.

Conclusion

Zero-downtime deployments require getting three things right: readiness probes that genuinely verify readiness, graceful shutdown that drains connections, and preStop hooks that account for Endpoint propagation delay. The deployment strategy (rolling, blue-green, canary) is secondary — if your health checks lie, every strategy will drop requests.

If this was helpful, you can support my work at ko-fi.com/nopkt ☕

Every API needs rate limiting. Most developers reach for a library or cloud service, but understanding the algorithms lets you pick the right one and debug it when things go wrong. Here's how to build four different rate limiters in Go — from simple to production-grade.

The Four Algorithms

Shared Interface

// limiter.go
package ratelimit

import "time"

type Result struct {
    Allowed   bool
    Remaining int
    ResetAt   time.Time
    RetryAt   time.Time // Zero if allowed
}

type Limiter interface {
    Allow(key string) Result
}

1. Fixed Window Counter

Simplest approach: count requests in fixed time intervals (e.g., 100 requests per minute). At the boundary of each window, the count resets.

// fixed_window.go
package ratelimit

import (
    "sync"
    "time"
)

type FixedWindow struct {
    limit    int
    window   time.Duration
    counters map[string]*counter
    mu       sync.Mutex
}

type counter struct {
    count    int
    windowStart time.Time
}

func NewFixedWindow(limit int, window time.Duration) *FixedWindow {
    return &FixedWindow{
        limit:    limit,
        window:   window,
        counters: make(map[string]*counter),
    }
}

func (fw *FixedWindow) Allow(key string) Result {
    fw.mu.Lock()
    defer fw.mu.Unlock()

    now := time.Now()
    windowStart := now.Truncate(fw.window)

    c, exists := fw.counters[key]
    if !exists || c.windowStart != windowStart {
        c = &counter{count: 0, windowStart: windowStart}
        fw.counters[key] = c
    }

    resetAt := windowStart.Add(fw.window)

    if c.count >= fw.limit {
        return Result{
            Allowed:   false,
            Remaining: 0,
            ResetAt:   resetAt,
            RetryAt:   resetAt,
        }
    }

    c.count++
    return Result{
        Allowed:   true,
        Remaining: fw.limit - c.count,
        ResetAt:   resetAt,
    }
}

Problem: At the boundary between two windows, a client can send limit * 2 requests (e.g., 100 at 12:00:59 and 100 at 12:01:00). The sliding window fixes this.

2. Sliding Window Counter

Approximates a sliding window by weighting the previous window's count based on how far into the current window we are.

// sliding_window.go
package ratelimit

import (
    "sync"
    "time"
)

type SlidingWindow struct {
    limit    int
    window   time.Duration
    counters map[string]*windowPair
    mu       sync.Mutex
}

type windowPair struct {
    prevCount    int
    prevStart    time.Time
    currentCount int
    currentStart time.Time
}

func NewSlidingWindow(limit int, window time.Duration) *SlidingWindow {
    return &SlidingWindow{
        limit:    limit,
        window:   window,
        counters: make(map[string]*windowPair),
    }
}

func (sw *SlidingWindow) Allow(key string) Result {
    sw.mu.Lock()
    defer sw.mu.Unlock()

    now := time.Now()
    currentWindowStart := now.Truncate(sw.window)

    wp, exists := sw.counters[key]
    if !exists {
        wp = &windowPair{currentStart: currentWindowStart}
        sw.counters[key] = wp
    }

    // Rotate windows if needed
    if currentWindowStart != wp.currentStart {
        if currentWindowStart.Sub(wp.currentStart) == sw.window {
            wp.prevCount = wp.currentCount
            wp.prevStart = wp.currentStart
        } else {
            wp.prevCount = 0
        }
        wp.currentCount = 0
        wp.currentStart = currentWindowStart
    }

    // Weight: how far through the current window are we?
    elapsed := now.Sub(currentWindowStart)
    weight := float64(sw.window-elapsed) / float64(sw.window)
    estimated := int(float64(wp.prevCount)*weight) + wp.currentCount

    resetAt := currentWindowStart.Add(sw.window)

    if estimated >= sw.limit {
        return Result{
            Allowed:   false,
            Remaining: 0,
            ResetAt:   resetAt,
            RetryAt:   now.Add(time.Duration(float64(sw.window) * (1 - weight))),
        }
    }

    wp.currentCount++
    return Result{
        Allowed:   true,
        Remaining: sw.limit - estimated - 1,
        ResetAt:   resetAt,
    }
}

This uses only two counters per key — O(1) memory regardless of request volume.

3. Token Bucket

Tokens accumulate at a fixed rate up to a maximum. Each request consumes one token. Allows bursts up to the bucket size while maintaining a long-term average rate.

// token_bucket.go
package ratelimit

import (
    "sync"
    "time"
)

type TokenBucket struct {
    rate       float64 // tokens per second
    bucketSize int
    buckets    map[string]*bucket
    mu         sync.Mutex
}

type bucket struct {
    tokens   float64
    lastFill time.Time
}

func NewTokenBucket(ratePerSecond float64, bucketSize int) *TokenBucket {
    return &TokenBucket{
        rate:       ratePerSecond,
        bucketSize: bucketSize,
        buckets:    make(map[string]*bucket),
    }
}

func (tb *TokenBucket) Allow(key string) Result {
    tb.mu.Lock()
    defer tb.mu.Unlock()

    now := time.Now()

    b, exists := tb.buckets[key]
    if !exists {
        b = &bucket{tokens: float64(tb.bucketSize), lastFill: now}
        tb.buckets[key] = b
    }

    // Add tokens based on elapsed time
    elapsed := now.Sub(b.lastFill).Seconds()
    b.tokens += elapsed * tb.rate
    if b.tokens > float64(tb.bucketSize) {
        b.tokens = float64(tb.bucketSize)
    }
    b.lastFill = now

    if b.tokens < 1 {
        // Calculate when next token arrives
        waitSeconds := (1 - b.tokens) / tb.rate
        return Result{
            Allowed:   false,
            Remaining: 0,
            RetryAt:   now.Add(time.Duration(waitSeconds * float64(time.Second))),
        }
    }

    b.tokens--
    return Result{
        Allowed:   true,
        Remaining: int(b.tokens),
    }
}

Token bucket is what most cloud APIs use (AWS, Stripe, GitHub). It naturally handles bursty traffic.

4. HTTP Middleware

// middleware.go
package ratelimit

import (
    "encoding/json"
    "fmt"
    "net/http"
    "strconv"
    "time"
)

func Middleware(limiter Limiter, keyFunc func(r *http.Request) string) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            key := keyFunc(r)
            result := limiter.Allow(key)

            // Always set rate limit headers (draft RFC 7231 extension)
            w.Header().Set("X-RateLimit-Remaining", strconv.Itoa(result.Remaining))
            if !result.ResetAt.IsZero() {
                w.Header().Set("X-RateLimit-Reset", strconv.FormatInt(result.ResetAt.Unix(), 10))
            }

            if !result.Allowed {
                w.Header().Set("Retry-After", fmt.Sprintf("%.0f", time.Until(result.RetryAt).Seconds()))
                w.Header().Set("Content-Type", "application/json")
                w.WriteHeader(http.StatusTooManyRequests)
                json.NewEncoder(w).Encode(map[string]any{
                    "error":      "rate limit exceeded",
                    "retry_after": time.Until(result.RetryAt).Seconds(),
                })
                return
            }

            next.ServeHTTP(w, r)
        })
    }
}

// Common key functions
func ByIP(r *http.Request) string {
    // Check X-Forwarded-For first (when behind a reverse proxy)
    if xff := r.Header.Get("X-Forwarded-For"); xff != "" {
        return "ip:" + xff
    }
    return "ip:" + r.RemoteAddr
}

func ByAPIKey(r *http.Request) string {
    key := r.Header.Get("X-API-Key")
    if key == "" {
        key = r.URL.Query().Get("api_key")
    }
    if key == "" {
        return ByIP(r) // Fallback to IP
    }
    return "key:" + key
}

Usage

// main.go
package main

import (
    "fmt"
    "log"
    "net/http"

    "yourmod/ratelimit"
)

func main() {
    // 100 requests per minute per IP
    limiter := ratelimit.NewSlidingWindow(100, time.Minute)

    mux := http.NewServeMux()
    mux.HandleFunc("/api/data", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, `{"status": "ok"}`)
    })

    // Apply rate limiting
    handler := ratelimit.Middleware(limiter, ratelimit.ByIP)(mux)

    log.Println("Server starting on :8080")
    log.Fatal(http.ListenAndServe(":8080", handler))
}

Distributed Rate Limiting with Redis

The in-memory implementations above work for single servers. For distributed systems, use Redis:

// redis_limiter.go
package ratelimit

import (
    "context"
    "time"

    "github.com/redis/go-redis/v9"
)

type RedisLimiter struct {
    client *redis.Client
    limit  int
    window time.Duration
}

func NewRedisLimiter(client *redis.Client, limit int, window time.Duration) *RedisLimiter {
    return &RedisLimiter{client: client, limit: limit, window: window}
}

// Lua script ensures atomic increment + expiry check
var slidingWindowScript = redis.NewScript(`
    local key = KEYS[1]
    local limit = tonumber(ARGV[1])
    local window = tonumber(ARGV[2])
    local now = tonumber(ARGV[3])

    -- Remove expired entries
    redis.call('ZREMRANGEBYSCORE', key, 0, now - window)

    -- Count current entries
    local count = redis.call('ZCARD', key)

    if count < limit then
        -- Add this request
        redis.call('ZADD', key, now, now .. '-' .. math.random(1000000))
        redis.call('PEXPIRE', key, window)
        return {1, limit - count - 1}
    end

    return {0, 0}
`)

func (rl *RedisLimiter) Allow(key string) Result {
    ctx := context.Background()
    now := time.Now()

    result, err := slidingWindowScript.Run(ctx, rl.client,
        []string{"ratelimit:" + key},
        rl.limit,
        rl.window.Milliseconds(),
        now.UnixMilli(),
    ).Int64Slice()

    if err != nil {
        // On Redis failure, allow the request (fail open)
        return Result{Allowed: true, Remaining: rl.limit}
    }

    return Result{
        Allowed:   result[0] == 1,
        Remaining: int(result[1]),
        RetryAt:   now.Add(rl.window),
    }
}

Using a Lua script makes the check-and-increment atomic — no race conditions even under high concurrency.

Choosing the Right Algorithm

• Fixed Window: Simplest. Use when exact precision at boundaries doesn't matter (internal APIs, development).

• Sliding Window Counter: Best default. Low memory, smooth limiting, good enough for most production APIs.

• Token Bucket: When you want to allow bursts. Good for user-facing APIs where occasional spikes are expected.

• Sliding Window Log (Redis sorted set): When you need exact counting. Higher memory usage but zero approximation error.

Memory Cleanup

For in-memory limiters, run periodic cleanup:

func (fw *FixedWindow) Cleanup() {
    fw.mu.Lock()
    defer fw.mu.Unlock()

    cutoff := time.Now().Add(-fw.window * 2)
    for key, c := range fw.counters {
        if c.windowStart.Before(cutoff) {
            delete(fw.counters, key)
        }
    }
}

// Run every minute
go func() {
    ticker := time.NewTicker(time.Minute)
    for range ticker.C {
        limiter.Cleanup()
    }
}()

Conclusion

Rate limiting isn't complex, but picking the wrong algorithm causes either false rejections (lost revenue) or ineffective protection (outages). Start with sliding window counters for single servers, move to Redis when you scale horizontally. Always set Retry-After headers — good clients will back off automatically.

If this was helpful, you can support my work at ko-fi.com/nopkt ☕

WebSockets are the default answer for real-time features. But for most notification systems — activity feeds, live dashboards, deployment status updates — they're overkill. Server-Sent Events (SSE) give you real-time push over plain HTTP with zero client-side libraries, automatic reconnection, and drastically simpler server code.

Here's how to build a production notification system using SSE with Node.js and TypeScript.

Why SSE Over WebSockets

If your client only needs to receive updates (notifications, live feeds, progress bars), SSE is the simpler, more reliable choice.

The Protocol in 30 Seconds

SSE is just an HTTP response with Content-Type: text/event-stream that stays open. The server writes lines in this format:

event: notification
data: {"id": 1, "message": "Deploy succeeded"}
id: 1

event: notification
data: {"id": 2, "message": "New comment on PR #42"}
id: 2

Each message is separated by a blank line. That's it. The browser handles parsing, reconnection, and event dispatching.

Project Setup

mkdir sse-notifications && cd sse-notifications
npm init -y
npm install express cors
npm install -D typescript @types/express @types/cors tsx

// tsconfig.json
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "esModuleInterop": true,
    "outDir": "dist"
  }
}

Step 1: The SSE Connection Manager

// src/sse.ts
import { Response } from 'express';

interface Client {
  id: string;
  userId: string;
  res: Response;
  connectedAt: Date;
  lastEventId: number;
}

class SSEManager {
  private clients = new Map<string, Client>();
  private eventCounter = 0;
  private eventBuffer: Array<{
    id: number;
    event: string;
    data: string;
    userId: string | null; // null = broadcast
    timestamp: number;
  }> = [];
  private readonly BUFFER_SIZE = 1000;
  private readonly BUFFER_TTL = 5 * 60 * 1000; // 5 minutes

  addClient(userId: string, res: Response, lastEventId?: string): string {
    const clientId = `\({userId}-\){Date.now()}-${Math.random().toString(36).slice(2, 8)}`;

    // Set SSE headers
    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      Connection: 'keep-alive',
      'X-Accel-Buffering': 'no', // Disable nginx buffering
    });

    // Send initial connection event
    res.write(`event: connected\ndata: ${JSON.stringify({ clientId })}\n\n`);

    const client: Client = {
      id: clientId,
      userId,
      res,
      connectedAt: new Date(),
      lastEventId: lastEventId ? parseInt(lastEventId) : 0,
    };

    this.clients.set(clientId, client);

    // Replay missed events if client reconnected
    if (client.lastEventId > 0) {
      this.replayEvents(client);
    }

    // Send heartbeat every 30s to keep connection alive
    const heartbeat = setInterval(() => {
      try {
        res.write(': heartbeat\n\n');
      } catch {
        clearInterval(heartbeat);
      }
    }, 30000);

    // Clean up on disconnect
    res.on('close', () => {
      clearInterval(heartbeat);
      this.clients.delete(clientId);
    });

    return clientId;
  }

  // Send event to a specific user (all their connected devices)
  sendToUser(userId: string, event: string, data: unknown): void {
    const id = ++this.eventCounter;
    const payload = JSON.stringify(data);

    this.bufferEvent(id, event, payload, userId);

    for (const client of this.clients.values()) {
      if (client.userId === userId) {
        this.writeEvent(client.res, id, event, payload);
      }
    }
  }

  // Broadcast to all connected clients
  broadcast(event: string, data: unknown): void {
    const id = ++this.eventCounter;
    const payload = JSON.stringify(data);

    this.bufferEvent(id, event, payload, null);

    for (const client of this.clients.values()) {
      this.writeEvent(client.res, id, event, payload);
    }
  }

  getStats() {
    const userCounts = new Map<string, number>();
    for (const client of this.clients.values()) {
      userCounts.set(client.userId, (userCounts.get(client.userId) || 0) + 1);
    }
    return {
      totalConnections: this.clients.size,
      uniqueUsers: userCounts.size,
      eventsBuffered: this.eventBuffer.length,
      totalEventsSent: this.eventCounter,
    };
  }

  private writeEvent(res: Response, id: number, event: string, data: string): void {
    try {
      res.write(`id: \({id}\nevent: \){event}\ndata: ${data}\n\n`);
    } catch {
      // Client disconnected — cleanup happens via the 'close' handler
    }
  }

  private bufferEvent(id: number, event: string, data: string, userId: string | null): void {
    this.eventBuffer.push({ id, event, data, userId, timestamp: Date.now() });

    // Prune old events
    const cutoff = Date.now() - this.BUFFER_TTL;
    while (this.eventBuffer.length > this.BUFFER_SIZE ||
           (this.eventBuffer.length > 0 && this.eventBuffer[0].timestamp < cutoff)) {
      this.eventBuffer.shift();
    }
  }

  private replayEvents(client: Client): void {
    const missed = this.eventBuffer.filter(
      (e) => e.id > client.lastEventId && (e.userId === null || e.userId === client.userId)
    );

    for (const event of missed) {
      this.writeEvent(client.res, event.id, event.event, event.data);
    }
  }
}

export const sseManager = new SSEManager();

The key features:

• Event buffering — when a client reconnects, Last-Event-ID header tells us where they left off, and we replay missed events

• Heartbeats — keep the connection alive through proxies and load balancers

• X-Accel-Buffering: no — prevents nginx from buffering the stream (a common gotcha)

• Per-user targeting — send notifications to specific users across all their devices

Step 2: Notification Service

// src/notifications.ts
import { sseManager } from './sse.js';

interface Notification {
  id: string;
  type: 'info' | 'success' | 'warning' | 'error';
  title: string;
  message: string;
  link?: string;
  timestamp: string;
}

// In-memory store (use Redis or a database in production)
const notifications = new Map<string, Notification[]>();

export function createNotification(
  userId: string,
  type: Notification['type'],
  title: string,
  message: string,
  link?: string
): Notification {
  const notification: Notification = {
    id: crypto.randomUUID(),
    type,
    title,
    message,
    link,
    timestamp: new Date().toISOString(),
  };

  // Store
  const userNotifs = notifications.get(userId) || [];
  userNotifs.unshift(notification);
  if (userNotifs.length > 100) userNotifs.pop(); // Keep last 100
  notifications.set(userId, userNotifs);

  // Push via SSE
  sseManager.sendToUser(userId, 'notification', notification);

  return notification;
}

export function getNotifications(userId: string, limit = 20): Notification[] {
  return (notifications.get(userId) || []).slice(0, limit);
}

export function broadcastAnnouncement(title: string, message: string): void {
  sseManager.broadcast('announcement', {
    id: crypto.randomUUID(),
    type: 'info',
    title,
    message,
    timestamp: new Date().toISOString(),
  });
}

Step 3: Express Routes

// src/server.ts
import express from 'express';
import cors from 'cors';
import { sseManager } from './sse.js';
import { createNotification, getNotifications, broadcastAnnouncement } from './notifications.js';

const app = express();
app.use(cors());
app.use(express.json());

// SSE endpoint — client connects here
app.get('/events/:userId', (req, res) => {
  const { userId } = req.params;
  const lastEventId = req.headers['last-event-id'] as string | undefined;

  const clientId = sseManager.addClient(userId, res, lastEventId);
  console.log(`Client connected: \({clientId} (user: \){userId})`);
});

// REST: Get notification history
app.get('/api/notifications/:userId', (req, res) => {
  const limit = parseInt(req.query.limit as string) || 20;
  const notifs = getNotifications(req.params.userId, limit);
  res.json({ notifications: notifs, count: notifs.length });
});

// REST: Send notification to user
app.post('/api/notifications/:userId', (req, res) => {
  const { type = 'info', title, message, link } = req.body;

  if (!title || !message) {
    return res.status(400).json({ error: 'title and message are required' });
  }

  const notification = createNotification(req.params.userId, type, title, message, link);
  res.status(201).json(notification);
});

// REST: Broadcast to all
app.post('/api/broadcast', (req, res) => {
  const { title, message } = req.body;

  if (!title || !message) {
    return res.status(400).json({ error: 'title and message are required' });
  }

  broadcastAnnouncement(title, message);
  res.json({ sent: true, connections: sseManager.getStats().totalConnections });
});

// Stats
app.get('/api/stats', (_, res) => {
  res.json(sseManager.getStats());
});

// Demo page
app.get('/', (_, res) => {
  res.send(`<!DOCTYPE html>
<html>
<head><title>SSE Notifications Demo</title></head>
<body>
  <h1>Notifications</h1>
  <div id="notifs"></div>
  <script>
    const userId = 'demo-user';
    const events = new EventSource('/events/' + userId);

    events.addEventListener('connected', (e) => {
      console.log('Connected:', JSON.parse(e.data));
    });

    events.addEventListener('notification', (e) => {
      const n = JSON.parse(e.data);
      const div = document.createElement('div');
      div.style.cssText = 'padding:12px;margin:8px 0;border-radius:8px;background:#f0f4ff;border-left:4px solid #4f46e5';
      div.innerHTML = '<strong>' + n.title + '</strong><br>' + n.message + '<br><small>' + n.timestamp + '</small>';
      document.getElementById('notifs').prepend(div);
    });

    events.addEventListener('announcement', (e) => {
      const n = JSON.parse(e.data);
      const div = document.createElement('div');
      div.style.cssText = 'padding:12px;margin:8px 0;border-radius:8px;background:#fef3c7;border-left:4px solid #f59e0b';
      div.innerHTML = '📢 <strong>' + n.title + '</strong><br>' + n.message;
      document.getElementById('notifs').prepend(div);
    });

    events.onerror = () => console.log('Reconnecting...');
  </script>
</body>
</html>`);
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Server running on http://localhost:${PORT}`));

Step 4: Testing It

npx tsx src/server.ts

In another terminal:

# Send a notification
curl -X POST http://localhost:3000/api/notifications/demo-user \
  -H "Content-Type: application/json" \
  -d '{"type": "success", "title": "Deploy Complete", "message": "v2.3.1 deployed to production"}'

# Broadcast to all
curl -X POST http://localhost:3000/api/broadcast \
  -H "Content-Type: application/json" \
  -d '{"title": "Scheduled Maintenance", "message": "Database maintenance at 2am UTC"}'

# Check stats
curl http://localhost:3000/api/stats

Open http://localhost:3000 in your browser — notifications appear instantly without polling.

Production Considerations

Scaling Beyond One Server

SSE connections are stateful — each lives on a specific server. For multiple servers, use Redis pub/sub:

// src/sse-redis.ts
import { createClient } from 'redis';
import { sseManager } from './sse.js';

const pub = createClient({ url: process.env.REDIS_URL });
const sub = pub.duplicate();

await pub.connect();
await sub.connect();

// Publish events to Redis (instead of calling sseManager directly)
export async function publishNotification(userId: string, event: string, data: unknown) {
  await pub.publish('sse:events', JSON.stringify({ userId, event, data }));
}

// Each server subscribes and delivers to its local clients
await sub.subscribe('sse:events', (message) => {
  const { userId, event, data } = JSON.parse(message);
  if (userId) {
    sseManager.sendToUser(userId, event, data);
  } else {
    sseManager.broadcast(event, data);
  }
});

Connection Limits

Each SSE connection holds an open HTTP connection. Default Node.js limit is ~1000 concurrent sockets. Increase it:

import http from 'http';
const server = http.createServer(app);
server.maxConnections = 10000;
server.listen(PORT);

For higher numbers, use uWebSockets.js or put a reverse proxy (nginx with proxy_buffering off) in front.

Authentication

SSE uses standard HTTP, so auth works normally:

app.get('/events/:userId', authenticate, (req, res) => {
  // authenticate middleware validates JWT from cookie or Authorization header
  if (req.user.id !== req.params.userId) {
    return res.status(403).json({ error: 'Forbidden' });
  }
  sseManager.addClient(req.user.id, res);
});

The EventSource API doesn't support custom headers, so use cookies or query params for auth:

// Client-side with auth token
const events = new EventSource('/events/me?token=' + authToken);

Or use the fetch-based polyfill for custom headers:

// Using fetch for SSE with headers
const response = await fetch('/events/me', {
  headers: { Authorization: `Bearer ${token}` },
});
const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  const text = decoder.decode(value);
  // Parse SSE format manually
  for (const line of text.split('\n')) {
    if (line.startsWith('data: ')) {
      const data = JSON.parse(line.slice(6));
      handleEvent(data);
    }
  }
}

Graceful Shutdown

process.on('SIGTERM', () => {
  // Close all SSE connections — clients will auto-reconnect to another server
  for (const client of sseManager.clients.values()) {
    client.res.end();
  }
  server.close();
});

When to Use WebSockets Instead

SSE is wrong when:

• Clients send frequent messages (chat, collaborative editing, gaming)

• Binary data streaming (SSE is text-only)

• You need sub-10ms latency (SSE adds HTTP overhead)

SSE is right when:

• Server pushes updates to clients (notifications, feeds, dashboards)

• You want simplicity (no library needed, standard HTTP)

• Reliability matters (auto-reconnect with event replay)

Conclusion

SSE gives you real-time server-to-client push with less code, better reliability, and simpler infrastructure than WebSockets. The built-in reconnection with Last-Event-ID means clients never miss events, even across network drops. For notification systems, live dashboards, and status feeds, it's the pragmatic choice.

If this was helpful, you can support my work at ko-fi.com/nopkt

Most API test suites are theater. They test that 200 OK is returned for valid input, but miss the edge cases that break production — race conditions, partial failures, invalid state transitions, and subtle serialization bugs.

Here's a testing strategy built from real production incidents, organized by what each test type catches and how much it costs to maintain.

The Testing Pyramid for APIs

        /  E2E  \        — 5% of tests, catch integration failures
       / Contract \       — 15%, catch API compatibility breaks
      / Integration \     — 30%, catch database + service bugs
     /    Unit Tests  \   — 50%, catch logic bugs fast

Project Setup

mkdir api-testing && cd api-testing
npm init -y
npm install express zod drizzle-orm better-sqlite3
npm install -D vitest supertest @types/supertest testcontainers msw

// vitest.config.ts
import { defineConfig } from 'vitest/config';

export default defineConfig({
  test: {
    globals: true,
    environment: 'node',
    coverage: {
      provider: 'v8',
      reporter: ['text', 'lcov'],
      exclude: ['**/test/**', '**/*.test.ts'],
    },
    // Run tests in sequence for integration tests
    pool: 'forks',
    poolOptions: {
      forks: { singleFork: true },
    },
  },
});

Unit Tests: Pure Logic, No I/O

Unit tests should cover business logic — validation, transformations, calculations. No database, no HTTP, no file system.

// src/services/pricing.ts
export function calculateDiscount(
  basePrice: number,
  quantity: number,
  customerTier: 'standard' | 'premium' | 'enterprise'
): { price: number; discount: number; total: number } {
  if (basePrice < 0 || quantity < 1) {
    throw new Error('Invalid input');
  }

  const tierMultipliers = { standard: 0, premium: 0.1, enterprise: 0.2 };
  const volumeDiscount = quantity >= 100 ? 0.15 : quantity >= 10 ? 0.05 : 0;
  const discount = Math.min(tierMultipliers[customerTier] + volumeDiscount, 0.35);
  const total = basePrice * quantity * (1 - discount);

  return { price: basePrice, discount, total: Math.round(total * 100) / 100 };
}

// src/services/pricing.test.ts
import { describe, it, expect } from 'vitest';
import { calculateDiscount } from './pricing';

describe('calculateDiscount', () => {
  it('applies no discount for standard tier, small quantity', () => {
    const result = calculateDiscount(100, 5, 'standard');
    expect(result).toEqual({ price: 100, discount: 0, total: 500 });
  });

  it('caps combined discount at 35%', () => {
    // Enterprise (20%) + volume 100+ (15%) = 35% cap
    const result = calculateDiscount(100, 100, 'enterprise');
    expect(result.discount).toBe(0.35);
    expect(result.total).toBe(6500);
  });

  it('stacks tier and volume discounts', () => {
    // Premium (10%) + volume 10+ (5%) = 15%
    const result = calculateDiscount(50, 20, 'premium');
    expect(result.discount).toBe(0.15);
    expect(result.total).toBe(850);
  });

  it('rejects negative prices', () => {
    expect(() => calculateDiscount(-10, 1, 'standard')).toThrow('Invalid input');
  });

  it('rejects zero quantity', () => {
    expect(() => calculateDiscount(100, 0, 'standard')).toThrow('Invalid input');
  });

  // This test caught a real bug — floating point precision
  it('rounds total to 2 decimal places', () => {
    const result = calculateDiscount(19.99, 3, 'premium');
    // Without rounding: 19.99 * 3 * 0.9 = 53.973
    expect(result.total).toBe(53.97);
  });
});

What unit tests catch: Logic errors, edge cases, floating point bugs, off-by-one errors. They're fast (run in <1ms each) and stable (no external dependencies).

Integration Tests: Real Database, Real Queries

Integration tests hit the actual database. Mocking SQL queries is worse than useless — it tests that your mock matches your assumptions, not that your query works.

// src/repositories/user.integration.test.ts
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest';
import { drizzle } from 'drizzle-orm/better-sqlite3';
import Database from 'better-sqlite3';
import { migrate } from 'drizzle-orm/better-sqlite3/migrator';
import { users } from '../schema';
import { UserRepository } from './user';

describe('UserRepository', () => {
  let db: ReturnType<typeof drizzle>;
  let sqlite: Database.Database;
  let repo: UserRepository;

  beforeAll(() => {
    sqlite = new Database(':memory:');
    db = drizzle(sqlite);
    migrate(db, { migrationsFolder: './drizzle' });
    repo = new UserRepository(db);
  });

  afterAll(() => {
    sqlite.close();
  });

  beforeEach(() => {
    // Clean slate for each test
    db.delete(users).run();
  });

  it('creates a user and retrieves by ID', async () => {
    const created = await repo.create({
      email: 'test@example.com',
      name: 'Test User',
    });

    expect(created.id).toBeDefined();
    expect(created.email).toBe('test@example.com');

    const found = await repo.findById(created.id);
    expect(found).toEqual(created);
  });

  it('enforces unique email constraint', async () => {
    await repo.create({ email: 'dup@example.com', name: 'User 1' });

    await expect(
      repo.create({ email: 'dup@example.com', name: 'User 2' })
    ).rejects.toThrow(/UNIQUE constraint/);
  });

  it('handles pagination correctly at boundaries', async () => {
    // Create exactly 25 users
    for (let i = 0; i < 25; i++) {
      await repo.create({ email: `user\({i}@test.com`, name: `User \){i}` });
    }

    const page1 = await repo.list({ page: 1, limit: 10 });
    const page3 = await repo.list({ page: 3, limit: 10 });

    expect(page1.items).toHaveLength(10);
    expect(page3.items).toHaveLength(5);
    expect(page3.totalPages).toBe(3);
  });

  it('soft-deletes without removing from database', async () => {
    const user = await repo.create({ email: 'del@test.com', name: 'Delete Me' });
    await repo.softDelete(user.id);

    // findById should not return soft-deleted users
    const found = await repo.findById(user.id);
    expect(found).toBeNull();

    // But the row still exists (for audit/recovery)
    const raw = sqlite.prepare('SELECT * FROM users WHERE id = ?').get(user.id);
    expect(raw).toBeDefined();
    expect((raw as any).deleted_at).not.toBeNull();
  });
});

What integration tests catch: SQL bugs, constraint violations, migration issues, ORM misuse, query performance regressions. Using SQLite in-memory is fast enough for most tests (sub-second for hundreds of queries).

API Tests: HTTP Layer + Middleware

Test the full HTTP stack — routing, middleware, serialization, error handling.

// src/app.test.ts
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import request from 'supertest';
import { createApp } from './app';

describe('API endpoints', () => {
  let app: ReturnType<typeof createApp>;

  beforeAll(() => {
    app = createApp({ database: ':memory:' });
  });

  describe('POST /api/users', () => {
    it('creates user with valid input', async () => {
      const res = await request(app)
        .post('/api/users')
        .send({ email: 'new@test.com', name: 'New User' })
        .expect(201);

      expect(res.body).toMatchObject({
        email: 'new@test.com',
        name: 'New User',
      });
      expect(res.body.id).toBeDefined();
      expect(res.body.createdAt).toBeDefined();
    });

    it('returns 400 with validation details for bad input', async () => {
      const res = await request(app)
        .post('/api/users')
        .send({ email: 'not-an-email', name: '' })
        .expect(400);

      expect(res.body.error).toBe('Validation failed');
      expect(res.body.details).toHaveLength(2);
      expect(res.body.details[0].path).toContain('email');
      expect(res.body.details[1].path).toContain('name');
    });

    it('returns 409 for duplicate email', async () => {
      await request(app)
        .post('/api/users')
        .send({ email: 'dup@test.com', name: 'First' });

      const res = await request(app)
        .post('/api/users')
        .send({ email: 'dup@test.com', name: 'Second' })
        .expect(409);

      expect(res.body.error).toContain('already exists');
    });
  });

  describe('GET /api/users', () => {
    it('returns paginated results with correct headers', async () => {
      // Seed some data
      for (let i = 0; i < 15; i++) {
        await request(app)
          .post('/api/users')
          .send({ email: `page\({i}@test.com`, name: `User \){i}` });
      }

      const res = await request(app)
        .get('/api/users?page=2&limit=10')
        .expect(200);

      expect(res.body.items).toHaveLength(5);
      expect(res.body.pagination.page).toBe(2);
      expect(res.body.pagination.totalPages).toBe(2);
    });

    it('ignores unknown query parameters', async () => {
      await request(app)
        .get('/api/users?foo=bar&page=1')
        .expect(200);
    });
  });

  describe('Authentication', () => {
    it('returns 401 for missing token', async () => {
      await request(app).get('/api/admin/stats').expect(401);
    });

    it('returns 401 for expired token', async () => {
      const expiredToken = createExpiredJWT();
      await request(app)
        .get('/api/admin/stats')
        .set('Authorization', `Bearer ${expiredToken}`)
        .expect(401);
    });
  });
});

What API tests catch: Routing bugs, middleware ordering issues, status code mistakes, response format errors, authentication bypass.

Contract Tests: API Compatibility

When other services depend on your API, contract tests prevent breaking changes.

// src/contracts/user-api.contract.test.ts
import { describe, it, expect } from 'vitest';
import { z } from 'zod';
import request from 'supertest';
import { createApp } from '../app';

// This schema represents what consumers expect
const UserResponseContract = z.object({
  id: z.string().uuid(),
  email: z.string().email(),
  name: z.string(),
  createdAt: z.string().datetime(),
  updatedAt: z.string().datetime(),
});

const PaginatedUsersContract = z.object({
  items: z.array(UserResponseContract),
  pagination: z.object({
    page: z.number(),
    limit: z.number(),
    total: z.number(),
    totalPages: z.number(),
  }),
});

describe('User API contract', () => {
  const app = createApp({ database: ':memory:' });

  it('POST /api/users response matches contract', async () => {
    const res = await request(app)
      .post('/api/users')
      .send({ email: 'contract@test.com', name: 'Contract Test' });

    const parsed = UserResponseContract.safeParse(res.body);
    expect(parsed.success).toBe(true);
  });

  it('GET /api/users response matches paginated contract', async () => {
    const res = await request(app).get('/api/users');

    const parsed = PaginatedUsersContract.safeParse(res.body);
    expect(parsed.success).toBe(true);
  });

  it('error responses always include error field', async () => {
    const res = await request(app)
      .post('/api/users')
      .send({ email: 'bad' });

    expect(res.body).toHaveProperty('error');
    expect(typeof res.body.error).toBe('string');
  });
});

What contract tests catch: Accidental field renames, type changes, missing fields. They fail when you'd break a consumer.

Testing External Service Calls with MSW

Mock Service Worker intercepts HTTP calls at the network level — no mocking fetch or axios.

// src/services/payment.test.ts
import { describe, it, expect, beforeAll, afterAll, afterEach } from 'vitest';
import { setupServer } from 'msw/node';
import { http, HttpResponse } from 'msw';
import { PaymentService } from './payment';

const server = setupServer();

beforeAll(() => server.listen());
afterEach(() => server.resetHandlers());
afterAll(() => server.close());

describe('PaymentService', () => {
  const service = new PaymentService('https://api.stripe.test');

  it('creates a payment intent', async () => {
    server.use(
      http.post('https://api.stripe.test/v1/payment_intents', () => {
        return HttpResponse.json({
          id: 'pi_test_123',
          status: 'requires_payment_method',
          amount: 1000,
        });
      })
    );

    const result = await service.createPaymentIntent(1000, 'usd');
    expect(result.id).toBe('pi_test_123');
    expect(result.amount).toBe(1000);
  });

  it('retries on 503 then succeeds', async () => {
    let attempts = 0;
    server.use(
      http.post('https://api.stripe.test/v1/payment_intents', () => {
        attempts++;
        if (attempts < 3) {
          return new HttpResponse(null, { status: 503 });
        }
        return HttpResponse.json({ id: 'pi_retry', status: 'created', amount: 500 });
      })
    );

    const result = await service.createPaymentIntent(500, 'usd');
    expect(result.id).toBe('pi_retry');
    expect(attempts).toBe(3);
  });

  it('throws after max retries exhausted', async () => {
    server.use(
      http.post('https://api.stripe.test/v1/payment_intents', () => {
        return new HttpResponse(null, { status: 503 });
      })
    );

    await expect(service.createPaymentIntent(500, 'usd')).rejects.toThrow(
      'Payment service unavailable'
    );
  });

  it('handles malformed JSON response', async () => {
    server.use(
      http.post('https://api.stripe.test/v1/payment_intents', () => {
        return new HttpResponse('not json', {
          headers: { 'Content-Type': 'application/json' },
        });
      })
    );

    await expect(service.createPaymentIntent(500, 'usd')).rejects.toThrow();
  });
});

What MSW tests catch: Retry logic bugs, timeout handling, error mapping, response parsing failures.

Test Organization

src/
├── services/
│   ├── pricing.ts
│   └── pricing.test.ts          # Unit tests: co-located
├── repositories/
│   ├── user.ts
│   └── user.integration.test.ts # Integration tests: co-located
├── app.test.ts                  # API tests: top-level
└── contracts/
    └── user-api.contract.test.ts # Contract tests: separate dir

Run them separately:

{
  "scripts": {
    "test": "vitest run",
    "test:unit": "vitest run --testPathPattern='.test.ts$' --testPathIgnorePatterns='integration|contract'",
    "test:integration": "vitest run --testPathPattern='integration'",
    "test:contract": "vitest run --testPathPattern='contract'",
    "test:watch": "vitest watch"
  }
}

What Not to Test

• Framework behavior: Don't test that Express returns 404 for unregistered routes. That's Express's job.

• Simple getters/setters: If a function just passes through data, the integration test covers it.

• Third-party library internals: Don't test that Zod validates emails correctly.

• Implementation details: Test the output, not how a function internally computes it.

Conclusion

The strategy is straightforward:

• Unit tests for business logic and calculations (fast, numerous)

• Integration tests for database operations (use real databases, not mocks)

• API tests for HTTP behavior and middleware (use supertest)

• Contract tests for API stability (use Zod schemas)

• MSW for external service interactions (network-level mocking)

Each layer catches different bugs. Skip a layer and those bugs reach production.

If this was helpful, you can support my work at ko-fi.com/nopkt

Most Go Docker images are built wrong. A typical FROM golang:1.22 image weighs 800MB+. Your compiled Go binary is probably 10-15MB. Here's how to use multi-stage builds to ship minimal, secure images — and avoid the common pitfalls.

The Problem

# DON'T do this
FROM golang:1.22
WORKDIR /app
COPY . .
RUN go build -o server .
CMD ["./server"]

This image includes the entire Go toolchain, build cache, and source code. Result: ~850MB image containing ~840MB of stuff you don't need in production.

The Basic Multi-Stage Fix

# Stage 1: Build
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o server .

# Stage 2: Run
FROM alpine:3.19
RUN apk --no-cache add ca-certificates
COPY --from=builder /app/server /server
CMD ["/server"]

This gets you to ~20MB. But we can do better, and there are production concerns to address.

The Production Dockerfile

# syntax=docker/dockerfile:1

# ============ Stage 1: Dependencies ============
FROM golang:1.22-alpine AS deps
WORKDIR /app
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download && go mod verify

# ============ Stage 2: Build ============
FROM golang:1.22-alpine AS builder
WORKDIR /app

# Copy cached modules
COPY --from=deps /go/pkg/mod /go/pkg/mod

# Copy source
COPY . .

# Build with optimizations
ARG VERSION=dev
ARG COMMIT=unknown
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
    go build \
      -ldflags="-s -w -X main.version=\({VERSION} -X main.commit=\){COMMIT}" \
      -trimpath \
      -o /server \
      ./cmd/server

# ============ Stage 3: Production ============
FROM gcr.io/distroless/static-debian12:nonroot

COPY --from=builder /server /server

EXPOSE 8080
USER nonroot:nonroot

ENTRYPOINT ["/server"]

Let's break down what each piece does and why.

Key Decisions Explained

CGO_ENABLED=0

Go can link against C libraries (CGo). Disabling it produces a fully static binary — no libc dependency, no shared library issues, runs on any Linux. If you need CGo (SQLite, certain crypto), you'll need a different base image.

-ldflags="-s -w"

-s strips the symbol table, -w strips DWARF debug information. Together they reduce binary size by 20-30%. You lose go tool pprof symbol names, but production binaries shouldn't need them.

-trimpath

Removes local file paths from the binary. Without this, panic stack traces reveal your build directory structure — a minor security concern.

--mount=type=cache

BuildKit cache mounts persist the Go module cache and build cache across builds. First build downloads all modules; subsequent builds reuse the cache. On a project with 200 dependencies, this cuts build time from 90s to 15s.

Distroless vs Alpine vs Scratch

# Option 1: scratch (smallest, ~12MB total)
FROM scratch
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /server /server

# Option 2: distroless (recommended, ~14MB total)
FROM gcr.io/distroless/static-debian12:nonroot

# Option 3: alpine (~22MB total)
FROM alpine:3.19
RUN apk --no-cache add ca-certificates

Scratch has no shell, no tools, no users — just your binary. Debugging is hard (can't exec into the container).

Distroless adds CA certificates, timezone data, a nonroot user, and /tmp. Still no shell or package manager. This is the sweet spot for most production services.

Alpine adds a shell, package manager, and musl libc. Use it when you need to debug containers in production or when you have CGo dependencies.

Non-Root User

USER nonroot:nonroot

Never run containers as root. Distroless provides a nonroot user (UID 65532). If using scratch:

FROM scratch
COPY --from=builder /etc/passwd /etc/passwd
USER 65532:65532

Handling Static Files and Config

If your Go binary needs to serve static files or read config:

FROM gcr.io/distroless/static-debian12:nonroot

# Copy binary
COPY --from=builder /server /server

# Copy static files (embedded in binary is better, but sometimes you need files)
COPY --from=builder /app/static /static
COPY --from=builder /app/migrations /migrations

EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/server"]

Better approach — embed files in the binary:

//go:embed static/*
var staticFiles embed.FS

//go:embed migrations/*.sql
var migrations embed.FS

Now your binary is self-contained. No files to copy, no paths to get wrong.

CI/CD Integration

Build with version info from Git:

docker build \
  --build-arg VERSION=$(git describe --tags --always) \
  --build-arg COMMIT=$(git rev-parse --short HEAD) \
  -t myapp:$(git describe --tags --always) \
  .

In your main.go:

var (
    version = "dev"
    commit  = "unknown"
)

func main() {
    slog.Info("starting server", "version", version, "commit", commit)
    // ...
}

Health Check for Orchestrators

# If using alpine (has wget)
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s \
  CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1

For distroless (no wget), implement the health check in your Go binary:

http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ok"))
})

And let Kubernetes/ECS probe the endpoint directly.

Multi-Architecture Builds

Ship for both AMD64 and ARM64:

docker buildx create --name multiarch --use
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --build-arg VERSION=$(git describe --tags) \
  -t myregistry/myapp:latest \
  --push \
  .

Update your Dockerfile to use the build arg:

ARG TARGETOS TARGETARCH
RUN CGO_ENABLED=0 GOOS=\({TARGETOS} GOARCH=\){TARGETARCH} \
    go build -ldflags="-s -w" -trimpath -o /server ./cmd/server

Docker sets TARGETOS and TARGETARCH automatically during multi-platform builds.

Security Scanning

# Scan with trivy
trivy image myapp:latest

# Scan with grype
grype myapp:latest

Distroless and scratch images typically have zero CVEs because there's nothing to scan — no OS packages, no libraries. Alpine images usually have a few low-severity findings in musl or busybox.

Size Comparison

The binary is smaller with -ldflags="-s -w" (15MB → 12MB).

Common Mistakes

1. Copying go.sum but not go.mod — Docker layer caching works on go.mod and go.sum together. Copy both before go mod download.

2. Not separating dependency download from build — If you copy all source before downloading modules, every code change invalidates the module cache layer.

3. Forgetting CA certificates — TLS connections fail silently or with cryptic errors. Distroless includes them. Scratch and alpine need manual handling.

4. Building for the wrong architecture — Always set GOOS and GOARCH explicitly. Your CI might be ARM but your target is AMD64.

5. Including .git in the build context — Add .git to .dockerignore. A large git history can add hundreds of MB to the build context.

# .dockerignore
.git
*.md
docs/
**/*_test.go

Conclusion

A production Go Docker image should be under 15MB, run as non-root, and contain nothing except your binary and its required certificates. Multi-stage builds make this straightforward — one stage for building, one for running. The build cache flags (--mount=type=cache) keep iteration fast.

Start with distroless for most services. Drop to scratch if you need the absolute minimum. Use alpine only when you need a shell for debugging.

If this was helpful, you can support my work at ko-fi.com/nopkt

What We'''re Building

A complete REST API for a task management system with:

• Request/response validation using Zod schemas

• Automatic OpenAPI 3.1 spec generation from those schemas

• Type-safe request handlers (no any types)

• Proper error responses with structured validation messages

Prerequisites

• Node.js 20+ and npm

• Basic TypeScript and Express knowledge

• Familiarity with REST API concepts

Project Setup

mkdir task-api && cd task-api
npm init -y
npm install express zod zod-to-openapi uuid
npm install -D typescript @types/express @types/uuid tsx

Create tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "esModuleInterop": true,
    "outDir": "dist",
    "rootDir": "src"
  },
  "include": ["src/**/*"]
}

Step 1: Define Schemas as the Source of Truth

// src/schemas/task.ts
import { z } from '''zod''';
import { extendZodWithOpenApi } from '''zod-to-openapi''';

extendZodWithOpenApi(z);

// Base task schema — represents what'''s stored
export const TaskSchema = z.object({
  id: z.string().uuid().openapi({ example: '''550e8400-e29b-41d4-a716-446655440000''' }),
  title: z.string().min(1).max(200).openapi({ example: '''Ship v2.0 release''' }),
  description: z.string().max(2000).optional().openapi({ example: '''Final QA pass and deploy''' }),
  status: z.enum(['''todo''', '''in_progress''', '''done''']).openapi({ example: '''todo''' }),
  priority: z.enum(['''low''', '''medium''', '''high''', '''critical''']).openapi({ example: '''high''' }),
  assignee: z.string().email().optional().openapi({ example: '''dev@company.com''' }),
  dueDate: z.string().datetime().optional().openapi({ example: '''2025-06-15T00:00:00Z''' }),
  tags: z.array(z.string().max(50)).max(10).default([]).openapi({ example: ['''backend''', '''release'''] }),
  createdAt: z.string().datetime(),
  updatedAt: z.string().datetime(),
}).openapi('''Task''');

// Create request — subset of fields
export const CreateTaskSchema = TaskSchema.pick({
  title: true,
  description: true,
  priority: true,
  assignee: true,
  dueDate: true,
  tags: true,
}).extend({
  title: z.string().min(1).max(200),
}).openapi('''CreateTask''');

// Update request — all fields optional
export const UpdateTaskSchema = CreateTaskSchema.partial().openapi('''UpdateTask''');

// Query parameters for listing
export const TaskQuerySchema = z.object({
  status: z.enum(['''todo''', '''in_progress''', '''done''']).optional(),
  priority: z.enum(['''low''', '''medium''', '''high''', '''critical''']).optional(),
  assignee: z.string().email().optional(),
  page: z.coerce.number().int().min(1).default(1),
  limit: z.coerce.number().int().min(1).max(100).default(20),
  sortBy: z.enum(['''createdAt''', '''updatedAt''', '''dueDate''', '''priority''']).default('''createdAt'''),
  order: z.enum(['''asc''', '''desc''']).default('''desc'''),
}).openapi('''TaskQuery''');

// Derive TypeScript types directly from schemas
export type Task = z.infer<typeof TaskSchema>;
export type CreateTask = z.infer<typeof CreateTaskSchema>;
export type UpdateTask = z.infer<typeof UpdateTaskSchema>;
export type TaskQuery = z.infer<typeof TaskQuerySchema>;

Every type, every validation rule, every OpenAPI example lives in one place. Change the schema, and validation, types, and docs all update together.

Step 2: Type-Safe Validation Middleware

// src/middleware/validate.ts
import { Request, Response, NextFunction } from '''express''';
import { z, ZodError, ZodSchema } from '''zod''';

type ValidationTarget = '''body''' | '''query''' | '''params''';

interface ValidationSchema {
  body?: ZodSchema;
  query?: ZodSchema;
  params?: ZodSchema;
}

function formatZodError(error: ZodError) {
  return {
    error: '''Validation failed''',
    details: error.issues.map((issue) => ({
      path: issue.path.join('''.'''),
      message: issue.message,
      code: issue.code,
    })),
  };
}

export function validate(schemas: ValidationSchema) {
  return (req: Request, res: Response, next: NextFunction) => {
    const targets: ValidationTarget[] = ['''body''', '''query''', '''params'''];
    const errors: z.ZodIssue[] = [];

    for (const target of targets) {
      const schema = schemas[target];
      if (!schema) continue;

      const result = schema.safeParse(req[target]);
      if (!result.success) {
        for (const issue of result.error.issues) {
          issue.path = [target, ...issue.path];
          errors.push(issue);
        }
      } else {
        (req as any)[target] = result.data;
      }
    }

    if (errors.length > 0) {
      const zodError = new ZodError(errors);
      return res.status(400).json(formatZodError(zodError));
    }

    next();
  };
}

Two details matter here. First, safeParse returns the transformed data (coerced numbers, defaults applied), and we replace req.body/req.query with it. Downstream handlers get clean, typed data. Second, we validate all targets before responding, so the client sees every error at once.

Step 3: Request Handlers with Full Type Safety

// src/routes/tasks.ts
import { Router, Request, Response } from '''express''';
import { v4 as uuidv4 } from '''uuid''';
import { validate } from '''../middleware/validate.js''';
import {
  CreateTaskSchema,
  UpdateTaskSchema,
  TaskQuerySchema,
  type Task,
  type CreateTask,
  type TaskQuery,
} from '''../schemas/task.js''';
import { z } from '''zod''';

const router = Router();
const tasks = new Map<string, Task>();

const IdParamSchema = z.object({ id: z.string().uuid() });

// LIST tasks with filtering, pagination, sorting
router.get('''/''',
  validate({ query: TaskQuerySchema }),
  (req: Request, res: Response) => {
    const query = req.query as unknown as TaskQuery;
    let results = Array.from(tasks.values());

    if (query.status) results = results.filter((t) => t.status === query.status);
    if (query.priority) results = results.filter((t) => t.priority === query.priority);
    if (query.assignee) results = results.filter((t) => t.assignee === query.assignee);

    results.sort((a, b) => {
      const aVal = a[query.sortBy] ?? '''''';
      const bVal = b[query.sortBy] ?? '''''';
      const cmp = String(aVal).localeCompare(String(bVal));
      return query.order === '''asc''' ? cmp : -cmp;
    });

    const total = results.length;
    const start = (query.page - 1) * query.limit;
    const items = results.slice(start, start + query.limit);

    res.json({
      items,
      pagination: { page: query.page, limit: query.limit, total, totalPages: Math.ceil(total / query.limit) },
    });
  }
);

// CREATE task
router.post('''/''',
  validate({ body: CreateTaskSchema }),
  (req: Request, res: Response) => {
    const input = req.body as CreateTask;
    const now = new Date().toISOString();
    const task: Task = {
      id: uuidv4(),
      status: '''todo''',
      ...input,
      tags: input.tags ?? [],
      createdAt: now,
      updatedAt: now,
    };
    tasks.set(task.id, task);
    res.status(201).json(task);
  }
);

// GET single task
router.get('''/:id''',
  validate({ params: IdParamSchema }),
  (req: Request, res: Response) => {
    const task = tasks.get(req.params.id);
    if (!task) return res.status(404).json({ error: '''Task not found''' });
    res.json(task);
  }
);

// UPDATE task
router.patch('''/:id''',
  validate({ params: IdParamSchema, body: UpdateTaskSchema }),
  (req: Request, res: Response) => {
    const existing = tasks.get(req.params.id);
    if (!existing) return res.status(404).json({ error: '''Task not found''' });
    const updated: Task = {
      ...existing,
      ...req.body,
      id: existing.id,
      createdAt: existing.createdAt,
      updatedAt: new Date().toISOString(),
    };
    tasks.set(updated.id, updated);
    res.json(updated);
  }
);

// DELETE task
router.delete('''/:id''',
  validate({ params: IdParamSchema }),
  (req: Request, res: Response) => {
    if (!tasks.delete(req.params.id)) return res.status(404).json({ error: '''Task not found''' });
    res.status(204).send();
  }
);

export default router;

Notice there are zero manual type assertions beyond the initial cast from req.query. The validation middleware guarantees the shape, and TypeScript carries it forward.

Step 4: Auto-Generate OpenAPI Spec

// src/openapi.ts
import { OpenAPIRegistry, OpenApiGeneratorV31 } from '''zod-to-openapi''';
import { TaskSchema, CreateTaskSchema, UpdateTaskSchema, TaskQuerySchema } from '''./schemas/task.js''';
import { z } from '''zod''';

const registry = new OpenAPIRegistry();

registry.register('''Task''', TaskSchema);
registry.register('''CreateTask''', CreateTaskSchema);
registry.register('''UpdateTask''', UpdateTaskSchema);

const IdParam = registry.registerParameter('''taskId''',
  z.string().uuid().openapi({ param: { name: '''id''', in: '''path''' } })
);

registry.registerPath({
  method: '''get''',
  path: '''/api/tasks''',
  summary: '''List tasks with filtering and pagination''',
  request: { query: TaskQuerySchema },
  responses: {
    200: {
      description: '''Paginated task list''',
      content: {
        '''application/json''': {
          schema: z.object({
            items: z.array(TaskSchema),
            pagination: z.object({
              page: z.number(), limit: z.number(),
              total: z.number(), totalPages: z.number(),
            }),
          }),
        },
      },
    },
  },
});

registry.registerPath({
  method: '''post''',
  path: '''/api/tasks''',
  summary: '''Create a new task''',
  request: { body: { content: { '''application/json''': { schema: CreateTaskSchema } } } },
  responses: {
    201: { description: '''Task created''', content: { '''application/json''': { schema: TaskSchema } } },
    400: { description: '''Validation error''' },
  },
});

registry.registerPath({
  method: '''get''',
  path: '''/api/tasks/{id}''',
  summary: '''Get a task by ID''',
  request: { params: z.object({ id: IdParam }) },
  responses: {
    200: { description: '''Task found''', content: { '''application/json''': { schema: TaskSchema } } },
    404: { description: '''Task not found''' },
  },
});

registry.registerPath({
  method: '''patch''',
  path: '''/api/tasks/{id}''',
  summary: '''Update a task''',
  request: {
    params: z.object({ id: IdParam }),
    body: { content: { '''application/json''': { schema: UpdateTaskSchema } } },
  },
  responses: {
    200: { description: '''Task updated''', content: { '''application/json''': { schema: TaskSchema } } },
    404: { description: '''Task not found''' },
  },
});

registry.registerPath({
  method: '''delete''',
  path: '''/api/tasks/{id}''',
  summary: '''Delete a task''',
  request: { params: z.object({ id: IdParam }) },
  responses: { 204: { description: '''Task deleted''' }, 404: { description: '''Task not found''' } },
});

const generator = new OpenApiGeneratorV31(registry.definitions);
export const openApiSpec = generator.generateDocument({
  openapi: '''3.1.0''',
  info: { title: '''Task Management API''', version: '''1.0.0''', description: '''Type-safe REST API with automatic validation and documentation''' },
  servers: [{ url: '''http://localhost:3000''' }],
});

Step 5: Wire Everything Together

// src/index.ts
import express from '''express''';
import taskRoutes from '''./routes/tasks.js''';
import { openApiSpec } from '''./openapi.js''';

const app = express();
app.use(express.json());
app.get('''/openapi.json''', (_, res) => res.json(openApiSpec));
app.use('''/api/tasks''', taskRoutes);

app.use((err: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => {
  console.error(err);
  res.status(500).json({ error: '''Internal server error''' });
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
  console.log(`OpenAPI spec: http://localhost:${PORT}/openapi.json`);
});

Testing It

npx tsx src/index.ts

Create a task:

curl -X POST http://localhost:3000/api/tasks \
  -H "Content-Type: application/json" \
  -d '''{"title": "Write API docs", "priority": "high", "tags": ["docs"]}'''

Hit the validation:

curl -X POST http://localhost:3000/api/tasks \
  -H "Content-Type: application/json" \
  -d '''{"title": "", "priority": "extreme"}'''

Response:

{
  "error": "Validation failed",
  "details": [
    { "path": "body.title", "message": "String must contain at least 1 character(s)", "code": "too_small" },
    { "path": "body.priority", "message": "Invalid enum value", "code": "invalid_enum_value" }
  ]
}

Why This Works Better Than Alternatives

vs. Joi + Swagger JSDoc: Joi schemas can'''t generate TypeScript types. You maintain types, Joi schemas, and JSDoc annotations separately. Three sources of truth that drift apart.

vs. class-validator + class-transformer: Decorator-based approaches require classes, which conflicts with functional TypeScript. Zod works with plain objects.

vs. tRPC: tRPC is excellent for TypeScript-to-TypeScript communication. But if non-TypeScript clients need your API, you need OpenAPI — and zod-to-openapi gives you that.

Going Further

Add Swagger UI:

npm install swagger-ui-express @types/swagger-ui-express

import swaggerUi from '''swagger-ui-express''';
app.use('''/docs''', swaggerUi.serve, swaggerUi.setup(openApiSpec));

Now http://localhost:3000/docs serves interactive API documentation generated entirely from your Zod schemas.

Conclusion

Zod as the single source of truth eliminates an entire class of bugs — the ones where validation accepts something the type system doesn'''t expect, or the API docs describe a field that the code ignores. Define once, validate everywhere, document automatically.

The full source is about 250 lines of actual logic. The OpenAPI spec it generates is production-ready — import it into Postman, generate client SDKs, or feed it to API gateways.

If this was helpful, you can support my work at ko-fi.com/nopkt

In this guide, we'll build a production-ready API gateway using Hono (a lightweight web framework), Durable Objects (for distributed rate limiting), and Workers' built-in Cache API.

Prerequisites

• Node.js 18+ and npm

• A Cloudflare account (free tier works)

• Basic familiarity with TypeScript and REST APIs

Project Setup

npm create cloudflare@latest api-gateway -- --template hono
cd api-gateway
npm install hono jose

Your wrangler.toml needs Durable Object bindings:

name = "api-gateway"
main = "src/index.ts"
compatibility_date = "2024-01-01"

[durable_objects]
bindings = [
  { name = "RATE_LIMITER", class_name = "RateLimiter" }
]

[[migrations]]
tag = "v1"
new_classes = ["RateLimiter"]

[vars]
UPSTREAM_URL = "https://api.example.com"
RATE_LIMIT_RPM = "60"

Gateway Architecture

Client -> [Auth] -> [Rate Limit] -> [Cache Check] -> [Proxy] -> [Log] -> Response

Each step is a Hono middleware. If any step fails, the request short-circuits with an error.

Step 1: The Hono Application Shell

// src/index.ts
import { Hono } from 'hono';
import { cors } from 'hono/cors';
import { RateLimiter } from './rate-limiter';
import { authMiddleware } from './middleware/auth';
import { rateLimitMiddleware } from './middleware/rate-limit';
import { cacheMiddleware } from './middleware/cache';
import { loggingMiddleware } from './middleware/logging';
import { proxyHandler } from './handlers/proxy';

type Bindings = {
  RATE_LIMITER: DurableObjectNamespace;
  UPSTREAM_URL: string;
  RATE_LIMIT_RPM: string;
  JWT_SECRET: string;
};

const app = new Hono<{ Bindings: Bindings }>();

app.use('*', cors());
app.use('*', loggingMiddleware);

app.get('/health', (c) => c.json({ status: 'ok', edge: c.req.header('cf-ray') }));

app.use('/api/*', authMiddleware);
app.use('/api/*', rateLimitMiddleware);
app.get('/api/*', cacheMiddleware);
app.all('/api/*', proxyHandler);

export default app;
export { RateLimiter };

Step 2: JWT Authentication

// src/middleware/auth.ts
import { createMiddleware } from 'hono/factory';
import * as jose from 'jose';

export const authMiddleware = createMiddleware(async (c, next) => {
  const authHeader = c.req.header('Authorization');
  if (!authHeader?.startsWith('Bearer ')) {
    return c.json({ error: 'Missing or invalid Authorization header' }, 401);
  }

  const token = authHeader.slice(7);
  try {
    const secret = new TextEncoder().encode(c.env.JWT_SECRET);
    const { payload } = await jose.jwtVerify(token, secret, {
      algorithms: ['HS256'],
    });
    c.set('userId', payload.sub as string);
    c.set('scopes', (payload.scopes as string[]) || []);
    await next();
  } catch (err) {
    if (err instanceof jose.errors.JWTExpired) {
      return c.json({ error: 'Token expired' }, 401);
    }
    return c.json({ error: 'Invalid token' }, 401);
  }
});

Why jose over jsonwebtoken? jose uses the Web Crypto API natively -- perfect for edge runtimes without Node.js polyfills.

Step 3: Distributed Rate Limiting with Durable Objects

// src/rate-limiter.ts
export class RateLimiter {
  private state: DurableObjectState;
  private requests: number[] = [];

  constructor(state: DurableObjectState) {
    this.state = state;
  }

  async fetch(request: Request): Promise<Response> {
    const url = new URL(request.url);
    const limit = parseInt(url.searchParams.get('limit') || '60');
    const windowMs = parseInt(url.searchParams.get('window') || '60000');
    const now = Date.now();

    const stored = await this.state.storage.get<number[]>('requests');
    if (stored) this.requests = stored;

    this.requests = this.requests.filter((ts) => now - ts < windowMs);

    if (this.requests.length >= limit) {
      const oldestInWindow = Math.min(...this.requests);
      const retryAfter = Math.ceil((oldestInWindow + windowMs - now) / 1000);
      return new Response(
        JSON.stringify({ error: 'Rate limit exceeded', retryAfter, limit, remaining: 0 }),
        {
          status: 429,
          headers: {
            'Content-Type': 'application/json',
            'Retry-After': retryAfter.toString(),
            'X-RateLimit-Limit': limit.toString(),
            'X-RateLimit-Remaining': '0',
          },
        }
      );
    }

    this.requests.push(now);
    await this.state.storage.put('requests', this.requests);

    const remaining = limit - this.requests.length;
    return new Response(
      JSON.stringify({ allowed: true, remaining, limit }),
      {
        headers: {
          'X-RateLimit-Limit': limit.toString(),
          'X-RateLimit-Remaining': remaining.toString(),
        },
      }
    );
  }
}

The middleware:

// src/middleware/rate-limit.ts
import { createMiddleware } from 'hono/factory';

export const rateLimitMiddleware = createMiddleware(async (c, next) => {
  const userId = c.get('userId') || c.req.header('cf-connecting-ip') || 'anonymous';
  const id = c.env.RATE_LIMITER.idFromName(userId);
  const limiter = c.env.RATE_LIMITER.get(id);

  const limit = parseInt(c.env.RATE_LIMIT_RPM || '60');
  const resp = await limiter.fetch(
    new Request(`https://limiter/?limit=${limit}&window=60000`)
  );

  const result = await resp.json<{ allowed?: boolean; remaining: number; limit: number }>();

  c.header('X-RateLimit-Limit', result.limit.toString());
  c.header('X-RateLimit-Remaining', result.remaining.toString());

  if (!result.allowed) {
    return c.json({ error: 'Rate limit exceeded' }, 429);
  }
  await next();
});

Each user gets their own Durable Object instance -- rate limits are per-user and globally consistent across all edge locations.

Step 4: Response Caching

// src/middleware/cache.ts
import { createMiddleware } from 'hono/factory';

export const cacheMiddleware = createMiddleware(async (c, next) => {
  if (c.req.method !== 'GET') { await next(); return; }

  const cache = caches.default;
  const cacheKey = new Request(c.req.url, { method: 'GET' });

  const cached = await cache.match(cacheKey);
  if (cached) {
    c.header('X-Cache', 'HIT');
    const body = await cached.text();
    const headers = Object.fromEntries(cached.headers.entries());
    return c.body(body, 200, headers);
  }

  c.header('X-Cache', 'MISS');
  await next();

  if (c.res.status === 200) {
    const response = c.res.clone();
    const cacheResponse = new Response(response.body, {
      headers: {
        ...Object.fromEntries(response.headers.entries()),
        'Cache-Control': 'public, max-age=60',
      },
    });
    c.executionCtx.waitUntil(cache.put(cacheKey, cacheResponse));
  }
});

Step 5: Request Proxying

// src/handlers/proxy.ts
import { createMiddleware } from 'hono/factory';

export const proxyHandler = createMiddleware(async (c) => {
  const upstreamUrl = new URL(c.req.path.replace('/api', ''), c.env.UPSTREAM_URL);

  const requestUrl = new URL(c.req.url);
  requestUrl.searchParams.forEach((value, key) => {
    upstreamUrl.searchParams.set(key, value);
  });

  const headers = new Headers(c.req.raw.headers);
  headers.delete('Authorization');
  headers.set('X-Forwarded-For', c.req.header('cf-connecting-ip') || '');
  headers.set('X-Request-ID', crypto.randomUUID());

  const upstreamReq = new Request(upstreamUrl.toString(), {
    method: c.req.method,
    headers,
    body: ['GET', 'HEAD'].includes(c.req.method) ? null : c.req.raw.body,
  });

  const startTime = Date.now();
  const response = await fetch(upstreamReq);
  const duration = Date.now() - startTime;

  const responseHeaders = new Headers(response.headers);
  responseHeaders.set('X-Gateway-Duration', `${duration}ms`);
  responseHeaders.set('X-Request-ID', headers.get('X-Request-ID')!);

  return new Response(response.body, {
    status: response.status,
    headers: responseHeaders,
  });
});

Step 6: Structured Logging

// src/middleware/logging.ts
import { createMiddleware } from 'hono/factory';

export const loggingMiddleware = createMiddleware(async (c, next) => {
  const requestId = crypto.randomUUID();
  const startTime = Date.now();
  c.header('X-Request-ID', requestId);

  await next();

  const logEntry = {
    timestamp: new Date().toISOString(),
    requestId,
    method: c.req.method,
    path: c.req.path,
    status: c.res.status,
    duration: Date.now() - startTime,
    ip: c.req.header('cf-connecting-ip'),
    userAgent: c.req.header('user-agent'),
    country: c.req.header('cf-ipcountry'),
    userId: c.get('userId') || null,
    cacheStatus: c.res.headers.get('X-Cache') || 'N/A',
  };

  console.log(JSON.stringify(logEntry));
});

Performance

Total overhead for cached responses: under 5ms. For uncached with rate limiting: 10-20ms.

Production Hardening

app.onError((err, c) => {
  console.error(JSON.stringify({
    error: err.message,
    stack: err.stack,
    path: c.req.path,
  }));
  return c.json({ error: 'Internal gateway error' }, 500);
});

// Request size limit
app.use('/api/*', async (c, next) => {
  const contentLength = parseInt(c.req.header('content-length') || '0');
  if (contentLength > 10 * 1024 * 1024) {
    return c.json({ error: 'Request too large' }, 413);
  }
  await next();
});

Conclusion

Under 300 lines of TypeScript gives you authentication, distributed rate limiting, caching, and structured logging at the edge. Key advantages:

• Zero cold starts and global distribution across 300+ cities

• Pay per request ($0.50/million on paid plan)

• Strongly consistent rate limiting via Durable Objects

Next steps: API key management (KV), request transformation, A/B routing, WebSocket proxying.

If this was helpful, you can support my work at ko-fi.com/nopkt