feat: adding new skills, including testing patterns and methodologies, along with bundled resources for better usability.

This commit is contained in:
duthaho
2026-03-30 12:18:00 +07:00
parent 0ff5ae4082
commit 7fa9a48c6c
89 changed files with 25808 additions and 923 deletions
+606 -41
View File
@@ -1,70 +1,216 @@
---
name: docker
description: >
Use this skill whenever containerizing applications, writing Dockerfiles, configuring Docker Compose, or optimizing container images. Trigger on keywords like Docker, Dockerfile, container, docker-compose, multi-stage build, image, or container registry. Also applies when setting up local development environments with containers, debugging container networking, or preparing applications for container-based deployment in CI/CD pipelines.
---
# Docker
## Description
Docker containerization including Dockerfiles, compose, and best practices.
## When to Use
- Containerizing applications
- Local development environments
- CI/CD pipelines
## When NOT to Use
- Serverless-only deployments where containers are not part of the architecture (e.g., pure AWS Lambda, Cloudflare Workers)
- Local development without containers where native tooling is preferred
- Simple scripts or utilities that do not need isolation or reproducible environments
---
## Core Patterns
### Multi-stage Dockerfile (Node.js)
### 1. Multi-Stage Builds
Multi-stage builds separate build-time dependencies from the runtime image, producing
smaller, more secure containers.
#### Python (builder + slim runtime)
```dockerfile
# Build stage
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
# ---- Build stage ----
FROM python:3.12-slim AS builder
# Production stage
FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
CMD ["node", "dist/index.js"]
```
WORKDIR /build
### Python Dockerfile
# Install build-only dependencies (gcc, etc.) needed by some wheels
RUN apt-get update && \
apt-get install -y --no-install-recommends gcc libpq-dev && \
rm -rf /var/lib/apt/lists/*
```dockerfile
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
# ---- Runtime stage ----
FROM python:3.12-slim
WORKDIR /app
# Install dependencies first (caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy only the installed packages from the builder
COPY --from=builder /install /usr/local
COPY . .
# Copy application code
COPY src/ ./src/
COPY main.py .
# Run as non-root
RUN addgroup --system app && adduser --system --ingroup app app
USER app
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0"]
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
```
### Docker Compose
#### Node.js (build + nginx/alpine)
```dockerfile
# ---- Build stage ----
FROM node:20-alpine AS builder
WORKDIR /app
# Install dependencies first for layer caching
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile
# Copy source and build
COPY tsconfig.json ./
COPY src/ ./src/
COPY public/ ./public/
RUN pnpm build
# ---- Runtime stage (static site served by nginx) ----
FROM nginx:1.27-alpine
# Copy custom nginx config
COPY nginx.conf /etc/nginx/conf.d/default.conf
# Copy built assets from builder
COPY --from=builder /app/dist /usr/share/nginx/html
# Run as non-root
RUN chown -R nginx:nginx /usr/share/nginx/html && \
chown -R nginx:nginx /var/cache/nginx && \
chown -R nginx:nginx /var/log/nginx && \
touch /var/run/nginx.pid && \
chown -R nginx:nginx /var/run/nginx.pid
USER nginx
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/ || exit 1
CMD ["nginx", "-g", "daemon off;"]
```
#### Node.js (API server with alpine runtime)
```dockerfile
# ---- Build stage ----
FROM node:20-alpine AS builder
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile
COPY tsconfig.json ./
COPY src/ ./src/
RUN pnpm build
# Prune dev dependencies for a lighter production node_modules
RUN pnpm prune --prod
# ---- Runtime stage ----
FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./
RUN addgroup -S app && adduser -S app -G app
USER app
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1
CMD ["node", "dist/index.js"]
```
#### Go (build + scratch)
```dockerfile
# ---- Build stage ----
FROM golang:1.22-alpine AS builder
WORKDIR /build
# Download dependencies first for caching
COPY go.mod go.sum ./
RUN go mod download
# Copy source and build a static binary
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /app/server ./cmd/server
# ---- Runtime stage (scratch = empty image) ----
FROM scratch
# Copy CA certificates for HTTPS calls
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# Copy the static binary
COPY --from=builder /app/server /server
EXPOSE 8080
ENTRYPOINT ["/server"]
```
---
### 2. Docker Compose for Development
A full-featured Compose file with services, volumes, networks, healthchecks, and
environment variable management.
```yaml
version: '3.8'
services:
app:
build: .
build:
context: .
dockerfile: Dockerfile
target: builder # Use builder stage for dev with hot-reload
ports:
- "3000:3000"
environment:
- DATABASE_URL=postgresql://user:pass@db:5432/app
NODE_ENV: development
DATABASE_URL: postgresql://user:pass@db:5432/app
REDIS_URL: redis://redis:6379
env_file:
- .env.local # Local overrides (gitignored)
volumes:
- .:/app # Bind-mount source for hot-reload
- /app/node_modules # Anonymous volume to preserve node_modules
depends_on:
- db
db:
condition: service_healthy
redis:
condition: service_started
networks:
- backend
restart: unless-stopped
db:
image: postgres:16-alpine
@@ -72,23 +218,442 @@ services:
POSTGRES_USER: user
POSTGRES_PASSWORD: pass
POSTGRES_DB: app
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
- ./scripts/init.sql:/docker-entrypoint-initdb.d/init.sql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U user -d app"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
networks:
- backend
redis:
image: redis:7-alpine
ports:
- "6379:6379"
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 3
networks:
- backend
worker:
build:
context: .
dockerfile: Dockerfile.worker
environment:
DATABASE_URL: postgresql://user:pass@db:5432/app
REDIS_URL: redis://redis:6379
depends_on:
db:
condition: service_healthy
redis:
condition: service_started
networks:
- backend
restart: unless-stopped
volumes:
postgres_data:
redis_data:
networks:
backend:
driver: bridge
```
---
### 3. Layer Caching
Docker caches each layer. If a layer has not changed, every layer after it is also
cached. Order instructions from least-frequently-changed to most-frequently-changed.
#### Optimal instruction order
```dockerfile
FROM python:3.12-slim
WORKDIR /app
# 1. System dependencies (rarely change)
RUN apt-get update && apt-get install -y --no-install-recommends curl && \
rm -rf /var/lib/apt/lists/*
# 2. Dependency manifests (change when adding packages)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# 3. Application code (changes most often)
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0"]
```
#### .dockerignore patterns
Always include a `.dockerignore` to keep the build context small and avoid leaking
secrets into layers.
```
# Version control
.git
.gitignore
# Dependencies (rebuilt inside container)
node_modules
__pycache__
*.pyc
.venv
venv
# Build output
dist
build
*.egg-info
# IDE and editor files
.vscode
.idea
*.swp
*.swo
# Environment and secrets
.env
.env.*
*.pem
*.key
# Docker files (not needed in context)
Dockerfile*
docker-compose*
.dockerignore
# Documentation and misc
README.md
CHANGELOG.md
LICENSE
docs/
```
---
### 4. Health Checks
Health checks let Docker (and orchestrators like Compose/Swarm/K8s) know when a
container is actually ready to serve traffic.
#### HTTP health check with curl
```dockerfile
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
```
#### HTTP health check with wget (alpine images without curl)
```dockerfile
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1
```
#### TCP port check (for non-HTTP services)
```dockerfile
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD nc -z localhost 5432 || exit 1
```
#### Python-native check (no extra binaries needed)
```dockerfile
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
```
**Parameter reference:**
| Parameter | Description | Default |
|------------------|--------------------------------------------------|---------|
| `--interval` | Time between checks | 30s |
| `--timeout` | Max time for a single check | 30s |
| `--start-period` | Grace period before checks count as failures | 0s |
| `--retries` | Consecutive failures before marking unhealthy | 3 |
---
### 5. Security Hardening
#### Run as non-root user
```dockerfile
# Debian/Ubuntu based images
RUN addgroup --system app && adduser --system --ingroup app app
USER app
# Alpine based images
RUN addgroup -S app && adduser -S app -G app
USER app
```
#### Use minimal base images
| Base Image | Size | Use Case |
|--------------------|---------|---------------------------------------|
| `alpine` | ~5 MB | General minimal base |
| `*-slim` | ~50 MB | Debian-based with fewer packages |
| `distroless` | ~20 MB | Google's no-shell, no-package-manager |
| `scratch` | 0 MB | Static binaries only (Go, Rust) |
```dockerfile
# Distroless for Python
FROM gcr.io/distroless/python3-debian12
COPY --from=builder /app /app
CMD ["main.py"]
```
#### Never put secrets in image layers
```dockerfile
# BAD - secret is baked into image history
COPY .env /app/.env
RUN echo "API_KEY=secret123" >> /app/.env
# GOOD - pass secrets at runtime
CMD ["python", "main.py"]
# docker run -e API_KEY=secret123 myapp
# or docker run --env-file .env myapp
```
#### Multi-stage to exclude build tools
Build tools (compilers, package managers, source code) stay in the builder stage
and never reach the runtime image. This reduces attack surface and image size.
```dockerfile
FROM node:20-alpine AS builder
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN corepack enable && pnpm install --frozen-lockfile
COPY . .
RUN pnpm build && pnpm prune --prod
FROM node:20-alpine
WORKDIR /app
# Only the built output and production deps are copied
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
USER node
CMD ["node", "dist/index.js"]
```
---
### 6. Environment Configuration
#### ARG vs ENV
| Directive | Available at | Persists in image | Use for |
|-----------|-------------|-------------------|-----------------------------|
| `ARG` | Build time | No | Build-time variables |
| `ENV` | Build + run | Yes | Runtime configuration |
```dockerfile
# ARG - only available during build
ARG NODE_ENV=production
ARG BUILD_VERSION=unknown
# ENV - available at build and runtime
ENV NODE_ENV=${NODE_ENV}
ENV APP_VERSION=${BUILD_VERSION}
# Build with: docker build --build-arg BUILD_VERSION=1.2.3 .
```
#### .env files with Compose
```yaml
services:
app:
build: .
# Single .env file
env_file:
- .env
# Multiple files (later files override earlier ones)
env_file:
- .env.defaults
- .env.local
# Inline environment variables (override env_file)
environment:
LOG_LEVEL: debug
DEBUG: "true"
```
#### Secrets management with Docker Compose
```yaml
services:
app:
build: .
secrets:
- db_password
- api_key
environment:
DB_PASSWORD_FILE: /run/secrets/db_password
secrets:
db_password:
file: ./secrets/db_password.txt
api_key:
environment: API_KEY # Read from host environment
```
Inside the container, secrets are mounted at `/run/secrets/<name>` as files.
---
### 7. Networking
#### Bridge networks for service isolation
```yaml
services:
frontend:
build: ./frontend
ports:
- "3000:3000"
networks:
- frontend-net
- backend-net # Can reach the API
api:
build: ./api
ports:
- "8000:8000"
networks:
- backend-net # Reachable by frontend and workers
db:
image: postgres:16-alpine
networks:
- backend-net # Only reachable by api and workers
# No ports exposed to host
worker:
build: ./worker
networks:
- backend-net
networks:
frontend-net:
driver: bridge
backend-net:
driver: bridge
```
#### Service discovery
Within a Docker Compose network, services reach each other by **service name**
as the hostname.
```python
# In the api service, connect to db using its service name
DATABASE_URL = "postgresql://user:pass@db:5432/app"
# In the frontend service, call the api by service name
API_URL = "http://api:8000"
```
#### Exposing ports
```yaml
services:
app:
ports:
- "3000:3000" # host:container, binds to 0.0.0.0
- "127.0.0.1:3000:3000" # bind to localhost only (more secure)
expose:
- "3000" # expose to other containers only, not host
```
---
## Best Practices
1. Use multi-stage builds
2. Order commands for cache efficiency
3. Use .dockerignore
4. Run as non-root user
5. Use specific image tags
1. **Use multi-stage builds** -- Separate build dependencies from the runtime
image. The final image should contain only what is needed to run the
application.
2. **Pin image tags** -- Use `node:20.11-alpine` or a digest instead of
`node:latest` or `node:20`. Floating tags lead to unpredictable builds.
3. **Order instructions for cache efficiency** -- Copy dependency manifests and
install dependencies before copying application code. This ensures that code
changes do not invalidate the dependency layer cache.
4. **Use .dockerignore** -- Exclude `.git`, `node_modules`, `__pycache__`, `.env`
files, and anything not needed inside the container to keep the build context
small and avoid leaking secrets.
5. **Run as non-root** -- Add a `USER` instruction to run the process as an
unprivileged user. Never run production containers as root.
6. **Combine RUN commands** -- Merge related `RUN` instructions with `&&` to
reduce layers and always clean up apt/apk caches in the same layer that
installs packages.
7. **Use COPY instead of ADD** -- `COPY` is explicit and predictable. `ADD` has
implicit behaviors (tar extraction, URL fetching) that can surprise you.
8. **Set explicit HEALTHCHECK** -- Define health checks in the Dockerfile so
orchestrators know when the container is ready. This prevents routing traffic
to containers that are still starting up.
---
## Common Pitfalls
- **Large images**: Use slim/alpine bases
- **Cache busting**: Order COPY commands properly
- **Root user**: Add USER instruction
1. **Bloated images** -- Using full base images like `python:3.12` instead of
`python:3.12-slim` adds hundreds of megabytes. Always prefer slim or alpine
variants. Use multi-stage builds to exclude build tools.
2. **Cache invalidation by COPY order** -- Placing `COPY . .` before
`RUN pip install` means every code change reinstalls all dependencies. Always
copy the dependency manifest first, install, then copy the rest of the code.
3. **Running as root** -- Forgetting the `USER` instruction means the container
process runs as root. If the application is compromised, the attacker has full
control of the container filesystem.
4. **Secrets baked into layers** -- Using `COPY .env .` or `ARG` for secrets
embeds them in the image layer history. Anyone with access to the image can
extract them with `docker history`. Pass secrets at runtime via environment
variables or Docker secrets.
5. **Missing .dockerignore** -- Without a `.dockerignore`, the entire directory
(including `.git`, `node_modules`, `.env` files) is sent as build context.
This slows builds, increases image size, and risks leaking credentials.
6. **Ignoring healthchecks in Compose** -- Using `depends_on` without
`condition: service_healthy` means the dependent service starts as soon as
the database container starts, not when the database is actually ready to
accept connections. Always pair `depends_on` with healthchecks.
---
## Related Skills
- `devops/github-actions` - CI/CD workflows for building and deploying Docker containers
- `security/owasp` - Security best practices for container hardening and vulnerability scanning
- `patterns/logging` — Container logging and log aggregation
@@ -0,0 +1,196 @@
# Dockerfile Best Practices Reference
Quick reference for writing efficient, secure, and maintainable Dockerfiles.
## Layer Ordering for Cache Optimization
Order instructions from least-frequently-changed to most-frequently-changed:
```dockerfile
# 1. Base image (changes: rarely)
FROM node:22-slim
# 2. System dependencies (changes: rarely)
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
&& rm -rf /var/lib/apt/lists/*
# 3. App dependency manifest (changes: sometimes)
COPY package.json pnpm-lock.yaml ./
# 4. Install dependencies (changes: sometimes, cached if manifests unchanged)
RUN pnpm install --frozen-lockfile
# 5. Copy source code (changes: frequently)
COPY . .
# 6. Build step (changes: frequently)
RUN pnpm build
# 7. Runtime config (changes: rarely, but placed last for clarity)
EXPOSE 3000
CMD ["node", "dist/server.js"]
```
**Key rule**: If a layer changes, all subsequent layers are rebuilt. Separate dependency installation from source code copying.
## Multi-Stage Builds
Reduce final image size by separating build and runtime stages.
```
+-------------------+ +-------------------+
| Build Stage | | Runtime Stage |
| | | |
| - Full toolchain | ---> | - Minimal base |
| - Dev deps | | - Only artifacts |
| - Source code | | - No build tools |
| - Build output | | - No source code |
+-------------------+ +-------------------+
~800 MB ~80 MB
```
**Benefits**: Smaller images, faster deploys, reduced attack surface, no build tools in production.
## Base Image Selection
| Image | Size | Use Case | Security | Package Manager |
|-------|------|----------|----------|-----------------|
| **alpine** | ~5 MB | Small images, CLI tools | Good (small surface) | apk |
| **slim** (Debian) | ~80 MB | Most apps (Python, Node) | Good | apt |
| **distroless** | ~20 MB | Production, no shell needed | Excellent (no shell) | None |
| **scratch** | 0 MB | Static Go/Rust binaries | Excellent (nothing) | None |
| **full** (Debian) | ~300 MB | Build stages, debugging | Fair (large surface) | apt |
### Recommendations by Language
| Language | Build Stage | Runtime Stage |
|----------|-------------|---------------|
| **Python** | `python:3.12-slim` | `python:3.12-slim` or `distroless/python3` |
| **Node.js** | `node:22-slim` | `node:22-slim` or `distroless/nodejs22` |
| **Go** | `golang:1.23` | `scratch` or `distroless/static` |
| **Rust** | `rust:1.83` | `scratch` or `distroless/cc` |
| **Java** | `eclipse-temurin:21-jdk` | `eclipse-temurin:21-jre-alpine` |
## Instruction Best Practices
### RUN: Combine and Clean Up
```dockerfile
# BAD: Multiple layers, leftover cache
RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y git
# GOOD: Single layer, cache cleaned
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
git \
&& rm -rf /var/lib/apt/lists/*
```
### COPY: Be Specific
```dockerfile
# BAD: Copies everything, including .git, node_modules, etc.
COPY . .
# GOOD: Copy only what's needed (use .dockerignore too)
COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
COPY src/ ./src/
COPY tsconfig.json ./
```
### .dockerignore Essentials
```
.git
node_modules
__pycache__
.env
*.log
dist
.venv
.pytest_cache
coverage
.DS_Store
```
### USER: Don't Run as Root
```dockerfile
# Create non-root user
RUN groupadd -r appuser && useradd -r -g appuser -s /bin/false appuser
USER appuser
```
### HEALTHCHECK
```dockerfile
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
CMD curl -f http://localhost:3000/health || exit 1
```
### ARG vs ENV
| Directive | Available at | Persists in image | Use for |
|-----------|-------------|-------------------|---------|
| `ARG` | Build time only | No | Build-time toggles, versions |
| `ENV` | Build + runtime | Yes | App configuration |
```dockerfile
ARG PYTHON_VERSION=3.12
FROM python:${PYTHON_VERSION}-slim
ENV APP_ENV=production
ENV PORT=8000
```
## Security Checklist
| Practice | Command/Example |
|----------|----------------|
| Pin base image digests | `FROM node:22-slim@sha256:abc123...` |
| Run as non-root | `USER appuser` |
| No secrets in layers | Use `--mount=type=secret` or build args |
| Scan for vulnerabilities | `docker scout cves`, `trivy image` |
| Read-only filesystem | `docker run --read-only` |
| Drop capabilities | `docker run --cap-drop ALL` |
| Use `.dockerignore` | Exclude `.env`, `.git`, credentials |
| Minimal base image | Use slim/distroless/scratch |
### Secrets at Build Time (BuildKit)
```dockerfile
# Mount a secret file without baking it into a layer
RUN --mount=type=secret,id=npm_token \
NPM_TOKEN=$(cat /run/secrets/npm_token) \
npm install
# Build command:
# docker build --secret id=npm_token,src=.npmrc .
```
## Image Size Reduction Checklist
1. Use multi-stage builds
2. Choose slim/alpine/distroless base
3. Combine RUN commands
4. Remove package manager caches (`rm -rf /var/lib/apt/lists/*`)
5. Use `.dockerignore`
6. Don't install dev dependencies in runtime stage
7. Remove unnecessary files after build
8. Use `--no-install-recommends` with apt
## Common Pitfalls
| Pitfall | Impact | Fix |
|---------|--------|-----|
| `COPY . .` before `npm install` | No dependency caching | Copy lockfile first, install, then copy source |
| Using `latest` tag | Non-reproducible builds | Pin specific version tags or digests |
| Secrets in `ENV` or `COPY` | Leaked in image layers | Use BuildKit secrets mount |
| Running as root | Security vulnerability | Add `USER` directive |
| No `.dockerignore` | Bloated context, slow builds | Add and maintain `.dockerignore` |
| Installing build tools in final stage | Bloated image | Use multi-stage; build in first stage |
| Not using `--frozen-lockfile` | Non-deterministic installs | Always use lockfile flags |
@@ -0,0 +1,93 @@
# =============================================================================
# Multi-Stage Node.js Dockerfile
# Usage:
# docker build -t myapp .
# docker run -p 3000:3000 myapp
# =============================================================================
# ---------------------------------------------------------------------------
# Stage 1: Install dependencies
# ---------------------------------------------------------------------------
FROM node:22-slim AS deps
# Enable corepack for pnpm support.
RUN corepack enable
WORKDIR /app
# Copy only package manifests first for dependency layer caching.
# Dependencies are only reinstalled when these files change.
COPY package.json pnpm-lock.yaml ./
# Install production and dev dependencies (dev deps needed for build step).
RUN pnpm install --frozen-lockfile
# ---------------------------------------------------------------------------
# Stage 2: Build the application
# ---------------------------------------------------------------------------
FROM node:22-slim AS builder
RUN corepack enable
WORKDIR /app
# Copy dependencies from the deps stage.
COPY --from=deps /app/node_modules ./node_modules
COPY --from=deps /app/package.json /app/pnpm-lock.yaml ./
# Copy source code and config files needed for the build.
COPY tsconfig.json ./
COPY src/ ./src/
# COPY public/ ./public/ # Uncomment for Next.js or static assets
# Build the application.
RUN pnpm build
# Remove dev dependencies after build to reduce size.
RUN pnpm prune --prod
# ---------------------------------------------------------------------------
# Stage 3: Production runtime
# ---------------------------------------------------------------------------
FROM node:22-slim AS runtime
# Run as non-root for security.
RUN groupadd -r appuser && useradd -r -g appuser -d /app -s /bin/false appuser
WORKDIR /app
# Set production environment.
ENV NODE_ENV=production \
PORT=3000
# Copy only production artifacts from the builder stage.
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package.json ./
# For Next.js standalone output, use instead:
# COPY --from=builder /app/.next/standalone ./
# COPY --from=builder /app/.next/static ./.next/static
# COPY --from=builder /app/public ./public
# Switch to non-root user.
USER appuser
# Expose the application port.
EXPOSE 3000
# Health check -- adjust the endpoint to match your app.
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD node -e "fetch('http://localhost:3000/health').then(r => { if (!r.ok) process.exit(1) })" || exit 1
# Run the application.
CMD ["node", "dist/server.js"]
# For Next.js standalone:
# CMD ["node", "server.js"]
# For NestJS:
# CMD ["node", "dist/main.js"]
# For Express with ts-node (dev only, not recommended for production):
# CMD ["npx", "ts-node", "src/server.ts"]
@@ -0,0 +1,78 @@
# =============================================================================
# Multi-Stage Python Dockerfile
# Usage:
# docker build -t myapp .
# docker run -p 8000:8000 myapp
# =============================================================================
# ---------------------------------------------------------------------------
# Stage 1: Build dependencies
# ---------------------------------------------------------------------------
# Use slim for building - it has gcc and headers available via apt.
FROM python:3.12-slim AS builder
# Prevent Python from writing .pyc files and enable unbuffered output.
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1
# Install build-time system dependencies (if any compiled packages need them).
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Install Python dependencies into a virtual environment.
# Copying requirements first enables Docker layer caching --
# dependencies are only reinstalled when requirements.txt changes.
COPY requirements.txt .
RUN python -m venv /app/.venv \
&& /app/.venv/bin/pip install --no-cache-dir --upgrade pip \
&& /app/.venv/bin/pip install --no-cache-dir -r requirements.txt
# ---------------------------------------------------------------------------
# Stage 2: Runtime
# ---------------------------------------------------------------------------
FROM python:3.12-slim AS runtime
# Prevent .pyc files and enable unbuffered output for logging.
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PATH="/app/.venv/bin:$PATH"
# Install only runtime system dependencies (no build tools).
# Add packages here if your app needs them at runtime (e.g., libpq for psycopg).
# RUN apt-get update && apt-get install -y --no-install-recommends \
# libpq5 \
# && rm -rf /var/lib/apt/lists/*
# Create a non-root user for security.
RUN groupadd -r appuser && useradd -r -g appuser -d /app -s /bin/false appuser
WORKDIR /app
# Copy the virtual environment from the builder stage.
COPY --from=builder /app/.venv /app/.venv
# Copy application source code.
COPY src/ ./src/
# Switch to non-root user.
USER appuser
# Expose the application port.
EXPOSE 8000
# Health check -- adjust the endpoint to match your app.
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1
# Run the application.
# For FastAPI/Uvicorn:
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]
# For Django/Gunicorn:
# CMD ["gunicorn", "src.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "4"]
# For a plain script:
# CMD ["python", "-m", "src.main"]
@@ -0,0 +1,100 @@
# =============================================================================
# Development Docker Compose
# Usage:
# docker compose -f docker-compose.dev.yaml up
# docker compose -f docker-compose.dev.yaml down -v # remove volumes too
# =============================================================================
services:
# ---------------------------------------------------------------------------
# Application
# ---------------------------------------------------------------------------
app:
build:
context: .
dockerfile: Dockerfile
target: builder # Use the build stage for dev (includes dev deps)
ports:
- "${APP_PORT:-3000}:3000"
environment:
NODE_ENV: development
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/app_dev
REDIS_URL: redis://redis:6379/0
volumes:
# Mount source code for hot-reload. Exclude node_modules.
- .:/app
- /app/node_modules
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
# Override CMD for development (hot-reload).
command: ["pnpm", "dev"]
# For Python:
# command: ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "3000", "--reload"]
# ---------------------------------------------------------------------------
# PostgreSQL
# ---------------------------------------------------------------------------
postgres:
image: postgres:17-alpine
ports:
- "${POSTGRES_PORT:-5432}:5432"
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: app_dev
volumes:
# Persist data across restarts.
- postgres_data:/var/lib/postgresql/data
# Run init scripts on first start.
# - ./scripts/init-db.sql:/docker-entrypoint-initdb.d/init.sql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 3s
retries: 5
# ---------------------------------------------------------------------------
# Redis
# ---------------------------------------------------------------------------
redis:
image: redis:7-alpine
ports:
- "${REDIS_PORT:-6379}:6379"
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 5
# Persist data to disk every 60 seconds if at least 1 key changed.
command: ["redis-server", "--save", "60", "1", "--loglevel", "warning"]
# ---------------------------------------------------------------------------
# Optional: pgAdmin (database GUI)
# ---------------------------------------------------------------------------
# pgadmin:
# image: dpage/pgadmin4:latest
# ports:
# - "5050:80"
# environment:
# PGADMIN_DEFAULT_EMAIL: admin@local.dev
# PGADMIN_DEFAULT_PASSWORD: admin
# depends_on:
# - postgres
# ---------------------------------------------------------------------------
# Optional: Mailpit (email testing)
# ---------------------------------------------------------------------------
# mailpit:
# image: axllent/mailpit:latest
# ports:
# - "8025:8025" # Web UI
# - "1025:1025" # SMTP
volumes:
postgres_data:
redis_data:
+745 -29
View File
@@ -1,20 +1,33 @@
---
name: github-actions
description: >
Use this skill whenever setting up or modifying GitHub Actions CI/CD workflows, automating tests, builds, or deployments on GitHub. Trigger on keywords like GitHub Actions, workflow YAML, CI/CD pipeline, actions/checkout, matrix builds, workflow_dispatch, or .github/workflows. Also applies when configuring caching in workflows, managing GitHub secrets, or troubleshooting failed workflow runs.
---
# GitHub Actions
## Description
GitHub Actions CI/CD workflows including testing, building, and deployment.
## When to Use
- Setting up CI/CD pipelines
- Automating tests and builds
- Deployment automation
## When NOT to Use
- GitLab CI projects using `.gitlab-ci.yml` configuration
- Jenkins pipelines using Jenkinsfile or Groovy-based configuration
- CircleCI, Travis CI, or other non-GitHub CI/CD systems
---
## Core Patterns
### Basic CI Workflow
### 1. CI Pipeline
Complete CI workflow covering checkout, setup, install, lint, test, and build for
both Python and Node.js projects.
#### Node.js CI Pipeline
```yaml
name: CI
@@ -25,64 +38,767 @@ on:
pull_request:
branches: [main]
permissions:
contents: read
jobs:
test:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
node-version: "20"
cache: "pnpm"
- run: corepack enable
- run: pnpm install --frozen-lockfile
- run: pnpm lint
- run: pnpm typecheck
test:
name: Test
runs-on: ubuntu-latest
needs: lint
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: "pnpm"
- run: corepack enable
- run: pnpm install --frozen-lockfile
- run: pnpm test -- --coverage
- name: Upload coverage
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: coverage/
retention-days: 7
build:
name: Build
runs-on: ubuntu-latest
needs: test
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: "pnpm"
- run: corepack enable
- run: pnpm install --frozen-lockfile
- run: pnpm build
- name: Upload build artifact
uses: actions/upload-artifact@v4
with:
name: build-output
path: dist/
retention-days: 5
```
#### Python CI Pipeline
```yaml
name: CI - Python
on:
push:
branches: [main]
pull_request:
branches: [main]
permissions:
contents: read
jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"
- run: pip install -r requirements-dev.txt
- run: ruff check .
- run: ruff format --check .
- run: mypy src/
test:
name: Test
runs-on: ubuntu-latest
needs: lint
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: test
POSTGRES_PASSWORD: test
POSTGRES_DB: testdb
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U test"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"
- run: pip install -r requirements.txt -r requirements-dev.txt
- name: Run tests
env:
DATABASE_URL: postgresql://test:test@localhost:5432/testdb
run: pytest -v --cov=src --cov-report=xml
- name: Upload coverage
uses: actions/upload-artifact@v4
with:
name: coverage-xml
path: coverage.xml
retention-days: 7
```
---
### 2. Matrix Strategy
Matrix builds run the same job across multiple combinations of OS, language
version, or other variables.
#### OS and version matrix
```yaml
jobs:
test:
name: Test (${{ matrix.os }}, Node ${{ matrix.node }})
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
node: [18, 20, 22]
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
cache: "npm"
- run: npm ci
- run: npm test
```
### Matrix Builds
#### Include and exclude
```yaml
jobs:
test:
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
node: [18, 20]
os: [ubuntu-latest, macos-latest, windows-latest]
python: ["3.11", "3.12"]
exclude:
# Skip Python 3.11 on Windows
- os: windows-latest
python: "3.11"
include:
# Add a specific combination with extra env
- os: ubuntu-latest
python: "3.13"
experimental: true
runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.experimental || false }}
steps:
- uses: actions/setup-node@v4
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
node-version: ${{ matrix.node }}
python-version: ${{ matrix.python }}
- run: pip install -r requirements.txt
- run: pytest
```
### Caching
---
### 3. Caching
Caching avoids re-downloading dependencies on every run. Use `hashFiles` to
generate cache keys from lockfiles so the cache invalidates when dependencies
change.
#### npm cache
```yaml
- uses: actions/cache@v4
with:
path: ~/.npm
key: npm-${{ hashFiles('**/package-lock.json') }}
restore-keys: npm-
key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
npm-${{ runner.os }}-
```
### Secrets
#### pnpm cache
```yaml
- name: Deploy
env:
API_KEY: ${{ secrets.API_KEY }}
run: deploy --key "$API_KEY"
- name: Get pnpm store directory
id: pnpm-cache
shell: bash
run: echo "store=$(pnpm store path)" >> "$GITHUB_OUTPUT"
- uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.store }}
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
pnpm-${{ runner.os }}-
```
#### pip cache
```yaml
- uses: actions/cache@v4
with:
path: ~/.cache/pip
key: pip-${{ runner.os }}-${{ hashFiles('**/requirements*.txt') }}
restore-keys: |
pip-${{ runner.os }}-
```
#### Docker layer cache
```yaml
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: myapp:latest
cache-from: type=gha
cache-to: type=gha,mode=max
```
---
### 4. Reusable Workflows
Reusable workflows let you define a workflow once and call it from other
workflows, reducing duplication across repositories.
#### Defining a reusable workflow (`.github/workflows/reusable-test.yml`)
```yaml
name: Reusable Test Workflow
on:
workflow_call:
inputs:
node-version:
description: "Node.js version to use"
required: false
type: string
default: "20"
working-directory:
description: "Directory to run commands in"
required: false
type: string
default: "."
secrets:
NPM_TOKEN:
required: false
jobs:
test:
runs-on: ubuntu-latest
defaults:
run:
working-directory: ${{ inputs.working-directory }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node-version }}
cache: "npm"
registry-url: "https://registry.npmjs.org"
- run: npm ci
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- run: npm test
```
#### Calling a reusable workflow
```yaml
name: CI
on:
push:
branches: [main]
jobs:
test-app:
uses: ./.github/workflows/reusable-test.yml
with:
node-version: "20"
working-directory: "packages/app"
secrets: inherit # Pass all secrets to the called workflow
test-lib:
uses: ./.github/workflows/reusable-test.yml
with:
node-version: "20"
working-directory: "packages/lib"
secrets: inherit
```
---
### 5. Composite Actions
Composite actions package multiple steps into a single reusable action. Unlike
reusable workflows, they run inline within the calling job.
#### Action definition (`.github/actions/setup-project/action.yml`)
```yaml
name: "Setup Project"
description: "Install Node.js, enable corepack, and install dependencies"
inputs:
node-version:
description: "Node.js version"
required: false
default: "20"
install-command:
description: "Command to install dependencies"
required: false
default: "pnpm install --frozen-lockfile"
runs:
using: "composite"
steps:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ inputs.node-version }}
- name: Enable corepack
shell: bash
run: corepack enable
- name: Get pnpm store directory
id: pnpm-cache
shell: bash
run: echo "store=$(pnpm store path)" >> "$GITHUB_OUTPUT"
- name: Cache pnpm store
uses: actions/cache@v4
with:
path: ${{ steps.pnpm-cache.outputs.store }}
key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }}
restore-keys: |
pnpm-${{ runner.os }}-
- name: Install dependencies
shell: bash
run: ${{ inputs.install-command }}
```
#### Using the composite action
```yaml
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ./.github/actions/setup-project
with:
node-version: "20"
- run: pnpm build
```
---
### 6. Deployment
Deployment workflows with environment protection rules, manual approval gates,
and multi-stage promotion.
```yaml
name: Deploy
on:
push:
branches: [main]
workflow_dispatch:
inputs:
environment:
description: "Target environment"
required: true
type: choice
options:
- staging
- production
permissions:
contents: read
deployments: write
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
cache: "pnpm"
- run: corepack enable && pnpm install --frozen-lockfile
- run: pnpm build
- uses: actions/upload-artifact@v4
with:
name: build-output
path: dist/
deploy-staging:
name: Deploy to Staging
runs-on: ubuntu-latest
needs: build
environment:
name: staging
url: https://staging.example.com
steps:
- uses: actions/download-artifact@v4
with:
name: build-output
path: dist/
- name: Deploy to staging
env:
DEPLOY_TOKEN: ${{ secrets.STAGING_DEPLOY_TOKEN }}
run: |
echo "Deploying to staging..."
# Replace with your actual deploy command
# e.g., aws s3 sync, rsync, wrangler publish, etc.
deploy-production:
name: Deploy to Production
runs-on: ubuntu-latest
needs: deploy-staging
if: github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'production'
environment:
name: production
url: https://example.com
# Production environment should have required reviewers configured
# in GitHub Settings > Environments > production > Protection rules
steps:
- uses: actions/download-artifact@v4
with:
name: build-output
path: dist/
- name: Deploy to production
env:
DEPLOY_TOKEN: ${{ secrets.PRODUCTION_DEPLOY_TOKEN }}
run: |
echo "Deploying to production..."
```
---
### 7. Artifacts
Artifacts let you share data between jobs in the same workflow or persist build
outputs for later download.
#### Upload artifact
```yaml
- name: Upload test results
uses: actions/upload-artifact@v4
if: always() # Upload even if tests fail
with:
name: test-results-${{ matrix.os }}-${{ matrix.node }}
path: |
test-results/
coverage/
retention-days: 14
if-no-files-found: warn # warn, error, or ignore
```
#### Download artifact in another job
```yaml
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci && npm run build
- uses: actions/upload-artifact@v4
with:
name: dist
path: dist/
deploy:
runs-on: ubuntu-latest
needs: build
steps:
- uses: actions/download-artifact@v4
with:
name: dist
path: dist/
- run: ls -la dist/
```
#### Download all artifacts
```yaml
- uses: actions/download-artifact@v4
with:
path: all-artifacts/
# Each artifact is placed in a subdirectory named after the artifact
```
---
### 8. Conditional Execution
Control when jobs and steps run using `if` expressions, job dependencies, and
path filters.
#### Path filters on triggers
```yaml
on:
push:
branches: [main]
paths:
- "src/**"
- "package.json"
- "pnpm-lock.yaml"
paths-ignore:
- "docs/**"
- "*.md"
```
#### Conditional jobs
```yaml
jobs:
changes:
runs-on: ubuntu-latest
outputs:
backend: ${{ steps.filter.outputs.backend }}
frontend: ${{ steps.filter.outputs.frontend }}
steps:
- uses: actions/checkout@v4
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
backend:
- 'src/api/**'
- 'requirements*.txt'
frontend:
- 'src/web/**'
- 'package.json'
test-backend:
needs: changes
if: needs.changes.outputs.backend == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements.txt && pytest
test-frontend:
needs: changes
if: needs.changes.outputs.frontend == 'true'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci && npm test
```
#### Conditional steps with if expressions
```yaml
steps:
- name: Run only on main branch
if: github.ref == 'refs/heads/main'
run: echo "On main"
- name: Run only on pull requests
if: github.event_name == 'pull_request'
run: echo "PR event"
- name: Run only when previous step failed
if: failure()
run: echo "Something failed"
- name: Always run (cleanup)
if: always()
run: echo "Cleanup"
- name: Run only when a label is present
if: contains(github.event.pull_request.labels.*.name, 'deploy')
run: echo "Deploy label found"
- name: Skip for dependabot
if: github.actor != 'dependabot[bot]'
run: npm test
```
#### Job dependencies
```yaml
jobs:
lint:
runs-on: ubuntu-latest
steps:
- run: echo "Linting..."
test:
runs-on: ubuntu-latest
steps:
- run: echo "Testing..."
# Runs after both lint and test succeed
deploy:
runs-on: ubuntu-latest
needs: [lint, test]
steps:
- run: echo "Deploying..."
# Runs even if test fails, but only after it completes
notify:
runs-on: ubuntu-latest
needs: [test]
if: always()
steps:
- run: echo "Test job status: ${{ needs.test.result }}"
```
---
## Best Practices
1. Use caching for dependencies
2. Run jobs in parallel when possible
3. Use environment secrets
4. Pin action versions
5. Add proper triggers
1. **Pin action versions with SHA** -- Use the full commit SHA instead of a
mutable tag: `actions/checkout@b4ffde65f...` (or at minimum a major version
tag like `@v4`). This prevents supply-chain attacks where a tag is moved.
2. **Use caching aggressively** -- Cache package manager stores (`~/.npm`,
pnpm store, `~/.cache/pip`) and Docker layers. A well-cached pipeline can
cut run times by 50-80%.
3. **Set minimal permissions** -- Add a top-level `permissions` block and grant
only what is needed. Default permissions are overly broad and pose a security
risk, especially for pull requests from forks.
4. **Run jobs in parallel** -- Structure independent jobs (lint, test, typecheck)
to run concurrently. Use `needs` only when there is a real dependency between
jobs.
5. **Use `fail-fast: false` in matrix builds** -- By default a failing matrix
combination cancels all others. Setting `fail-fast: false` lets all
combinations complete so you get the full picture of what is broken.
6. **Use environment protection rules** -- Configure required reviewers and wait
timers on production environments in GitHub Settings. This adds a human gate
before production deploys.
7. **Extract reusable workflows and composite actions** -- If the same steps
appear in multiple workflows, factor them into a reusable workflow
(`workflow_call`) or composite action to keep things DRY.
8. **Keep secrets out of logs** -- Never `echo` a secret. GitHub masks known
secrets, but dynamically constructed values may leak. Use `::add-mask::` for
runtime values that should be hidden.
---
## Common Pitfalls
- **Slow pipelines**: Add caching
- **Secret exposure**: Never echo secrets
- **Unpinned versions**: Use @v4 not @main
1. **Unpinned action versions** -- Using `actions/checkout@main` means your
workflow pulls whatever is on main today. A bad push to that action
repository could break or compromise your builds. Pin to a tag (`@v4`) or
SHA.
2. **Missing caching** -- Running `npm ci` or `pip install` from scratch on
every run wastes minutes. Always configure caching for your package manager,
or use the built-in `cache` option in setup actions (e.g.,
`actions/setup-node` has a `cache` input).
3. **Overly broad triggers** -- Triggering on every push to every branch floods
the queue. Restrict triggers to `main` and pull requests. Use `paths` or
`paths-ignore` to skip runs when only docs or unrelated files change.
4. **Secret exposure in pull requests from forks** -- Secrets are NOT available
in workflows triggered by `pull_request` from forks (by design). If your
workflow needs secrets for fork PRs, use `pull_request_target` carefully and
never check out untrusted code in that context.
5. **Large artifacts without retention limits** -- Uploading artifacts without
setting `retention-days` uses the repository default (90 days), consuming
storage quota. Set short retention for transient artifacts like test results
and coverage reports.
6. **Ignoring `if: always()` for cleanup** -- Steps after a failure are skipped
by default. If you need to upload test results, send notifications, or run
cleanup regardless of prior step results, use `if: always()` or
`if: failure()`.
---
## Related Skills
- `devops/docker` - Container patterns for building and deploying Dockerized applications in workflows
- `testing/pytest` - Python test configuration for CI pipeline integration
- `testing/vitest` - TypeScript/JavaScript test configuration for CI pipeline integration
@@ -0,0 +1,250 @@
# GitHub Actions Syntax Quick Reference
## Workflow File Structure
```yaml
name: CI # Workflow name (shown in GitHub UI)
on: # Triggers
push:
branches: [main]
pull_request:
branches: [main]
permissions: # Workflow-level permissions
contents: read
env: # Workflow-level environment variables
NODE_ENV: test
jobs:
build: # Job ID
runs-on: ubuntu-latest # Runner
steps:
- uses: actions/checkout@v4 # Action step
- run: echo "Hello" # Shell step
```
## Triggers (on:)
### Common Events
```yaml
on:
push:
branches: [main, "release/**"]
paths: ["src/**", "!src/**/*.test.ts"] # Path filtering
tags: ["v*"]
pull_request:
branches: [main]
types: [opened, synchronize, reopened]
workflow_dispatch: # Manual trigger
inputs:
environment:
type: choice
options: [staging, production]
schedule:
- cron: "0 6 * * 1" # Every Monday at 6am UTC
release:
types: [published]
workflow_call: # Reusable workflow
inputs:
node-version: { type: string, default: "22" }
secrets:
NPM_TOKEN: { required: true }
```
## Jobs
```yaml
jobs:
test:
name: Run Tests
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- run: npm test
deploy:
needs: [lint, test] # Runs after lint AND test succeed
runs-on: ubuntu-latest
steps: [...]
```
### Matrix Strategy
```yaml
jobs:
test:
strategy:
fail-fast: false # Don't cancel other jobs on failure
matrix:
os: [ubuntu-latest, macos-latest]
node: [20, 22]
exclude:
- os: macos-latest
node: 20
include:
- os: ubuntu-latest
node: 22
coverage: true
runs-on: ${{ matrix.os }}
steps:
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
```
## Steps
### Action Step
```yaml
- uses: actions/checkout@v4
with:
fetch-depth: 0 # Full history (needed for some tools)
- uses: actions/setup-node@v4
with:
node-version-file: ".nvmrc"
cache: "pnpm"
```
### Shell Step
```yaml
- name: Run tests
run: npm test
working-directory: ./packages/api
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
shell: bash
continue-on-error: true # Don't fail the job
timeout-minutes: 10
```
### Multi-line Commands
```yaml
- run: |
echo "Line 1"
echo "Line 2"
npm run build
```
## Conditionals (if:)
```yaml
# Run only on main branch
- if: github.ref == 'refs/heads/main'
# Run only on pull requests
- if: github.event_name == 'pull_request'
# Run only when previous step failed
- if: failure()
# Always run (even if previous steps failed)
- if: always()
# Run only when a matrix variable is set
- if: matrix.coverage == true
# Run based on changed files (requires dorny/paths-filter or similar)
- if: steps.filter.outputs.backend == 'true'
# Run on specific actor
- if: github.actor != 'dependabot[bot]'
```
## Environment and Secrets
```yaml
jobs:
deploy:
environment:
name: production
url: https://example.com
env:
APP_VERSION: ${{ github.sha }}
steps:
- run: deploy.sh
env:
API_KEY: ${{ secrets.API_KEY }} # Repository secret
DEPLOY_TOKEN: ${{ vars.DEPLOY_TOKEN }} # Repository variable
```
## Caching
### Built-in Cache (setup actions)
```yaml
- uses: actions/setup-node@v4
with:
node-version: 22
cache: "pnpm" # Automatic pnpm cache
```
### Manual Cache
```yaml
- uses: actions/cache@v4
with:
path: |
~/.cache/pip
.mypy_cache
key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
```
## Artifacts
### Upload
```yaml
- uses: actions/upload-artifact@v4
with:
name: coverage-report
path: coverage/
retention-days: 7
```
### Download (in another job)
```yaml
- uses: actions/download-artifact@v4
with:
name: coverage-report
path: ./coverage
```
## Services (Containers)
Define `services:` under a job with `image`, `env`, `ports`, and `options` (for health checks). Common: postgres, redis, mysql.
## Outputs (Passing Data Between Steps/Jobs)
```yaml
# Step output: echo "key=value" >> "$GITHUB_OUTPUT"
# Read in later step: ${{ steps.<step-id>.outputs.key }}
# Job output: declare under jobs.<job>.outputs, read via needs.<job>.outputs.key
```
## Permissions
Common values: `contents: read`, `pull-requests: write`, `issues: write`, `packages: write`, `id-token: write` (OIDC).
## Reusable Workflows
Caller uses `uses: ./.github/workflows/reusable.yaml` with `with:` and `secrets: inherit`. Callee triggers on `workflow_call:` with `inputs:` and `secrets:` definitions.
## Useful Expressions
| Expression | Result |
|-----------|--------|
| `${{ github.sha }}` | Full commit SHA |
| `${{ github.ref_name }}` | Branch or tag name |
| `${{ github.event.pull_request.number }}` | PR number |
| `${{ runner.os }}` | `Linux`, `macOS`, `Windows` |
| `${{ hashFiles('**/lockfile') }}` | SHA256 of files |
| `${{ contains(github.event.head_commit.message, '[skip ci]') }}` | Check commit message |
@@ -0,0 +1,176 @@
# =============================================================================
# Node.js CI Pipeline
# Runs: lint (eslint), type check (tsc), test (vitest), build
# =============================================================================
name: Node CI
on:
push:
branches: [main]
paths:
- "**.ts"
- "**.tsx"
- "**.js"
- "**.jsx"
- "package.json"
- "pnpm-lock.yaml"
- "tsconfig.json"
- ".github/workflows/ci-node.yaml"
pull_request:
branches: [main]
paths:
- "**.ts"
- "**.tsx"
- "**.js"
- "**.jsx"
- "package.json"
- "pnpm-lock.yaml"
- "tsconfig.json"
- ".github/workflows/ci-node.yaml"
permissions:
contents: read
env:
NODE_VERSION: "22"
jobs:
# ---------------------------------------------------------------------------
# Install dependencies (shared across jobs via cache)
# ---------------------------------------------------------------------------
install:
name: Install
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: latest
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "pnpm"
- name: Install dependencies
run: pnpm install --frozen-lockfile
# ---------------------------------------------------------------------------
# Lint with ESLint
# ---------------------------------------------------------------------------
lint:
name: Lint
needs: install
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: latest
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "pnpm"
- run: pnpm install --frozen-lockfile
- run: pnpm lint
# ---------------------------------------------------------------------------
# Type check with TypeScript compiler
# ---------------------------------------------------------------------------
type-check:
name: Type Check
needs: install
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: latest
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "pnpm"
- run: pnpm install --frozen-lockfile
- run: pnpm tsc --noEmit
# ---------------------------------------------------------------------------
# Test with Vitest
# ---------------------------------------------------------------------------
test:
name: Test
needs: install
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: latest
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "pnpm"
- run: pnpm install --frozen-lockfile
- name: Run tests with coverage
run: pnpm vitest run --coverage --reporter=junit --outputFile=junit.xml
- name: Upload coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: coverage/
retention-days: 7
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results
path: junit.xml
retention-days: 7
# ---------------------------------------------------------------------------
# Build
# ---------------------------------------------------------------------------
build:
name: Build
needs: [lint, type-check, test]
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
with:
version: latest
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
cache: "pnpm"
- run: pnpm install --frozen-lockfile
- run: pnpm build
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: build-output
path: dist/
retention-days: 7
@@ -0,0 +1,164 @@
# =============================================================================
# Python CI Pipeline
# Runs: lint (ruff), type check (mypy), test (pytest), coverage upload
# =============================================================================
name: Python CI
on:
push:
branches: [main]
paths:
- "**.py"
- "requirements*.txt"
- "pyproject.toml"
- ".github/workflows/ci-python.yaml"
pull_request:
branches: [main]
paths:
- "**.py"
- "requirements*.txt"
- "pyproject.toml"
- ".github/workflows/ci-python.yaml"
permissions:
contents: read
env:
PYTHON_VERSION: "3.12"
jobs:
# ---------------------------------------------------------------------------
# Lint with Ruff
# ---------------------------------------------------------------------------
lint:
name: Lint
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Install ruff
run: pip install ruff
- name: Ruff check (lint)
run: ruff check .
- name: Ruff format (formatting)
run: ruff format --check .
# ---------------------------------------------------------------------------
# Type check with mypy
# ---------------------------------------------------------------------------
type-check:
name: Type Check
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Cache pip dependencies
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install dependencies
run: |
pip install --upgrade pip
pip install -r requirements.txt
pip install mypy
- name: Run mypy
run: mypy src/ --ignore-missing-imports
# ---------------------------------------------------------------------------
# Test with pytest
# ---------------------------------------------------------------------------
test:
name: Test
runs-on: ubuntu-latest
timeout-minutes: 15
# Uncomment to add a PostgreSQL service for integration tests.
# services:
# postgres:
# image: postgres:17-alpine
# env:
# POSTGRES_USER: test
# POSTGRES_PASSWORD: test
# POSTGRES_DB: test_db
# ports:
# - 5432:5432
# options: >-
# --health-cmd pg_isready
# --health-interval 10s
# --health-timeout 5s
# --health-retries 5
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Cache pip dependencies
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install dependencies
run: |
pip install --upgrade pip
pip install -r requirements.txt
pip install -r requirements-dev.txt
- name: Run tests with coverage
run: |
pytest \
--cov=src \
--cov-report=xml:coverage.xml \
--cov-report=term-missing \
--junitxml=junit.xml \
-v
env:
PYTHONPATH: ${{ github.workspace }}
# DATABASE_URL: postgresql://test:test@localhost:5432/test_db
- name: Upload coverage report
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-report
path: coverage.xml
retention-days: 7
- name: Upload test results
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results
path: junit.xml
retention-days: 7
# Uncomment to upload coverage to Codecov.
# - name: Upload to Codecov
# if: github.event_name == 'push' && github.ref == 'refs/heads/main'
# uses: codecov/codecov-action@v4
# with:
# files: coverage.xml
# token: ${{ secrets.CODECOV_TOKEN }}