diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index e6420ba..7fdc6ae 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -335,16 +335,34 @@ For strict TDD enforcement (no production code without failing test): Enable mandatory verification before completion claims: - Reference: `.claude/skills/methodology/verification-before-completion/SKILL.md` -### Available Methodology Skills +### Available Skills | Category | Skills | |----------|--------| -| Planning | brainstorming, writing-plans, executing-plans | -| Testing | test-driven-development, verification-before-completion, testing-anti-patterns | -| Debugging | systematic-debugging, root-cause-tracing, defense-in-depth | -| Collaboration | dispatching-parallel-agents, requesting-code-review, receiving-code-review, finishing-development-branch | +| **Languages** | python, typescript, javascript | +| **Frameworks** | fastapi, django, nextjs, react | +| **Databases** | postgresql, mongodb | +| **DevOps** | docker, github-actions | +| **Frontend** | tailwind, shadcn-ui | +| **Security** | owasp | +| **API** | openapi | +| **Testing** | pytest, vitest | +| **Optimization** | token-efficient | +| **Developer Patterns** | error-handling, state-management, logging, caching, api-client, authentication | +| **Methodology - Planning** | brainstorming, writing-plans, executing-plans | +| **Methodology - Testing** | test-driven-development, verification-before-completion, testing-anti-patterns | +| **Methodology - Debugging** | systematic-debugging, root-cause-tracing, defense-in-depth | +| **Methodology - Collaboration** | dispatching-parallel-agents, requesting-code-review, receiving-code-review, finishing-development-branch | +| **Methodology - Reasoning** | sequential-thinking | -Skills location: `.claude/skills/methodology/` +Skills location: `.claude/skills/` + +Each skill includes: +- YAML frontmatter with trigger description +- "When to Use" / "When NOT to Use" sections +- Core patterns with code examples +- Best practices and common pitfalls +- Bundled reference docs, templates, and scripts ### Sequential Thinking @@ -436,6 +454,10 @@ pnpm install ## Kit Version -- **Claude Kit Version**: 2.0.0 -- **Last Updated**: 2025-01-29 +- **Claude Kit Version**: 3.0.0 +- **Last Updated**: 2026-03-30 - **Compatible with**: Claude Code 1.0+ +- **Total Skills**: 38 (with YAML frontmatter, bundled resources) +- **Total Commands**: 27+ +- **Total Agents**: 20 +- **Behavioral Modes**: 7 diff --git a/.claude/skills/api/openapi/SKILL.md b/.claude/skills/api/openapi/SKILL.md index b2adf24..575b069 100644 --- a/.claude/skills/api/openapi/SKILL.md +++ b/.claude/skills/api/openapi/SKILL.md @@ -1,92 +1,837 @@ -# OpenAPI +--- +name: openapi +description: > + Use this skill when designing, documenting, or generating REST API specifications using OpenAPI/Swagger. Trigger on keywords like OpenAPI, Swagger, API spec, REST documentation, API schema, request body, response schema, and API client generation. Also apply when adopting design-first API development, validating API contracts, or setting up auto-generated API documentation for FastAPI, Express, or NestJS endpoints. +--- -## Description - -OpenAPI/Swagger specification patterns for REST API documentation. +# OpenAPI & REST API Design ## When to Use - Documenting REST APIs - Generating API clients - API design-first development +- Defining webhook contracts +- Establishing pagination, versioning, or auth patterns for a new service + +## When NOT to Use + +- Internal-only scripts or automation that do not expose HTTP endpoints +- CLI tools and command-line utilities without a REST interface +- GraphQL APIs where a different specification format applies --- ## Core Patterns -### Basic Specification +### 1. OpenAPI 3.1 Specification Structure + +A complete spec skeleton showing every top-level section. Use `$ref` to split +large specs into per-resource files. ```yaml -openapi: 3.0.3 +openapi: 3.1.0 info: - title: My API - version: 1.0.0 + title: Acme API + version: 2.0.0 + description: Public API for the Acme platform. + contact: + name: API Support + email: api@acme.dev + license: + name: MIT + url: https://opensource.org/licenses/MIT + +servers: + - url: https://api.acme.dev/v2 + description: Production + - url: https://staging-api.acme.dev/v2 + description: Staging + +tags: + - name: Users + description: User management operations + - name: Orders + description: Order lifecycle operations + +paths: + /users: + $ref: './paths/users.yaml' + /users/{userId}: + $ref: './paths/users-by-id.yaml' + /orders: + $ref: './paths/orders.yaml' + +components: + schemas: + $ref: './components/schemas/_index.yaml' + securitySchemes: + BearerAuth: + type: http + scheme: bearer + bearerFormat: JWT + ApiKeyAuth: + type: apiKey + in: header + name: X-API-Key + +security: + - BearerAuth: [] + +webhooks: + orderCompleted: + $ref: './webhooks/order-completed.yaml' +``` + +**Organizing with `$ref`** -- keep one file per resource under `paths/` and +shared schemas under `components/schemas/`. A bundler such as +`@redocly/cli bundle` resolves references into a single file for tooling. + +``` +spec/ +├── openapi.yaml # Root document +├── paths/ +│ ├── users.yaml +│ ├── users-by-id.yaml +│ └── orders.yaml +├── components/ +│ └── schemas/ +│ ├── _index.yaml +│ ├── User.yaml +│ ├── Order.yaml +│ └── ProblemDetail.yaml +└── webhooks/ + └── order-completed.yaml +``` + +--- + +### 2. Path & Operation Patterns + +#### RESTful URL Naming Conventions + +- Use **plural nouns** for collections: `/users`, `/orders`. +- Use **path parameters** for single-resource access: `/users/{userId}`. +- Nest only one level deep: `/users/{userId}/orders` (not deeper). +- Use **query parameters** for filtering, sorting, and pagination. +- Avoid verbs in paths -- let HTTP methods convey the action. + +#### CRUD Operations + +```yaml +paths: + /users: + get: + operationId: listUsers + tags: [Users] + summary: List users + parameters: + - $ref: '#/components/parameters/PageCursor' + - $ref: '#/components/parameters/PageSize' + - name: status + in: query + schema: + type: string + enum: [active, inactive] + responses: + '200': + description: Paginated list of users + content: + application/json: + schema: + $ref: '#/components/schemas/UserListResponse' + + post: + operationId: createUser + tags: [Users] + summary: Create a user + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/CreateUserRequest' + responses: + '201': + description: User created + headers: + Location: + schema: + type: string + description: URL of the new resource + content: + application/json: + schema: + $ref: '#/components/schemas/User' + '409': + $ref: '#/components/responses/Conflict' + '422': + $ref: '#/components/responses/ValidationError' + + /users/{userId}: + parameters: + - name: userId + in: path + required: true + schema: + type: string + format: uuid + + get: + operationId: getUser + tags: [Users] + summary: Get a single user + responses: + '200': + description: User found + content: + application/json: + schema: + $ref: '#/components/schemas/User' + '404': + $ref: '#/components/responses/NotFound' + + patch: + operationId: updateUser + tags: [Users] + summary: Partially update a user + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateUserRequest' + responses: + '200': + description: User updated + content: + application/json: + schema: + $ref: '#/components/schemas/User' + '404': + $ref: '#/components/responses/NotFound' + '422': + $ref: '#/components/responses/ValidationError' + + delete: + operationId: deleteUser + tags: [Users] + summary: Delete a user + responses: + '204': + description: User deleted + '404': + $ref: '#/components/responses/NotFound' +``` + +#### Path Parameters vs Query Parameters + +| Use case | Mechanism | Example | +|----------|-----------|---------| +| Identify a specific resource | Path parameter | `/orders/{orderId}` | +| Filter a collection | Query parameter | `/orders?status=shipped` | +| Sort a collection | Query parameter | `/orders?sort=-createdAt` | +| Paginate | Query parameter | `/orders?cursor=abc&limit=20` | +| Expand nested data | Query parameter | `/orders?expand=items,customer` | + +--- + +### 3. Request Body Patterns + +#### JSON Request Body with Validation + +```yaml +components: + schemas: + CreateUserRequest: + type: object + required: + - email + - name + properties: + email: + type: string + format: email + maxLength: 254 + name: + type: string + minLength: 1 + maxLength: 100 + role: + type: string + enum: [admin, member, viewer] + default: member + additionalProperties: false +``` + +Implementation in **FastAPI** (Python): + +```python +from pydantic import BaseModel, EmailStr, Field + +class CreateUserRequest(BaseModel): + email: EmailStr + name: str = Field(min_length=1, max_length=100) + role: str = Field(default="member", pattern="^(admin|member|viewer)$") + + model_config = {"extra": "forbid"} +``` + +Implementation in **Express** (TypeScript with Zod): + +```typescript +import { z } from "zod"; + +const CreateUserRequest = z.object({ + email: z.string().email().max(254), + name: z.string().min(1).max(100), + role: z.enum(["admin", "member", "viewer"]).default("member"), +}).strict(); + +type CreateUserRequest = z.infer; +``` + +#### Multipart Form Data (File Uploads) + +```yaml +/users/{userId}/avatar: + put: + operationId: uploadAvatar + tags: [Users] + summary: Upload user avatar + parameters: + - name: userId + in: path + required: true + schema: + type: string + format: uuid + requestBody: + required: true + content: + multipart/form-data: + schema: + type: object + required: + - file + properties: + file: + type: string + format: binary + description: Image file (JPEG or PNG, max 5 MB) + caption: + type: string + maxLength: 200 + encoding: + file: + contentType: image/jpeg, image/png + responses: + '200': + description: Avatar updated + content: + application/json: + schema: + type: object + properties: + url: + type: string + format: uri + '413': + $ref: '#/components/responses/PayloadTooLarge' +``` + +#### Content Negotiation + +Support multiple response formats by listing them under `content`: + +```yaml +responses: + '200': + description: Export data + content: + application/json: + schema: + $ref: '#/components/schemas/ExportData' + text/csv: + schema: + type: string + application/pdf: + schema: + type: string + format: binary +``` + +Clients select a format with the `Accept` header. Document which formats your +API actually supports so consumers do not have to guess. + +--- + +### 4. Response Patterns + +#### Success Responses + +| Code | Meaning | Typical use | +|------|---------|-------------| +| `200` | OK | GET, PATCH, general success | +| `201` | Created | POST that creates a resource | +| `202` | Accepted | Async operation started | +| `204` | No Content | DELETE, or PUT with no body returned | + +Always return a `Location` header with `201` pointing to the new resource. + +#### Error Responses -- RFC 7807 Problem Details + +Define a single reusable error schema based on RFC 7807: + +```yaml +components: + schemas: + ProblemDetail: + type: object + required: + - type + - title + - status + properties: + type: + type: string + format: uri + description: URI reference identifying the problem type. + example: https://api.acme.dev/problems/validation-error + title: + type: string + description: Short human-readable summary. + example: Validation Error + status: + type: integer + description: HTTP status code. + example: 422 + detail: + type: string + description: Human-readable explanation specific to this occurrence. + example: "Field 'email' must be a valid email address." + instance: + type: string + format: uri + description: URI identifying this specific occurrence. + errors: + type: array + description: Field-level validation errors (optional extension). + items: + type: object + properties: + field: + type: string + example: email + message: + type: string + example: Must be a valid email address. + code: + type: string + example: invalid_format + + responses: + NotFound: + description: Resource not found + content: + application/problem+json: + schema: + $ref: '#/components/schemas/ProblemDetail' + example: + type: https://api.acme.dev/problems/not-found + title: Not Found + status: 404 + detail: User with ID '550e8400' was not found. + + ValidationError: + description: Request validation failed + content: + application/problem+json: + schema: + $ref: '#/components/schemas/ProblemDetail' + example: + type: https://api.acme.dev/problems/validation-error + title: Validation Error + status: 422 + errors: + - field: email + message: Must be a valid email address. + code: invalid_format + + Conflict: + description: Resource conflict + content: + application/problem+json: + schema: + $ref: '#/components/schemas/ProblemDetail' + + PayloadTooLarge: + description: Request payload exceeds limit + content: + application/problem+json: + schema: + $ref: '#/components/schemas/ProblemDetail' +``` + +Use the `application/problem+json` media type for all error responses to signal +RFC 7807 compliance. + +--- + +### 5. Authentication Schemes + +#### Bearer Token (JWT) + +```yaml +components: + securitySchemes: + BearerAuth: + type: http + scheme: bearer + bearerFormat: JWT + +security: + - BearerAuth: [] +``` + +Override per-operation to allow unauthenticated access: + +```yaml +paths: + /health: + get: + security: [] # No auth required + responses: + '200': + description: Healthy +``` + +#### API Key + +```yaml +components: + securitySchemes: + ApiKeyHeader: + type: apiKey + in: header + name: X-API-Key + ApiKeyQuery: + type: apiKey + in: query + name: api_key +``` + +#### OAuth2 Flows + +```yaml +components: + securitySchemes: + OAuth2: + type: oauth2 + flows: + authorizationCode: + authorizationUrl: https://auth.acme.dev/authorize + tokenUrl: https://auth.acme.dev/token + refreshUrl: https://auth.acme.dev/token + scopes: + users:read: Read user profiles + users:write: Create and update users + orders:read: Read orders paths: /users: get: - summary: List users + security: + - OAuth2: [users:read] +``` + +--- + +### 6. Pagination Patterns + +#### Cursor-Based Pagination (Recommended) + +Best for large, real-time datasets where rows may be inserted or deleted +between pages. + +```yaml +components: + parameters: + PageCursor: + name: cursor + in: query + description: Opaque cursor returned by a previous response. + schema: + type: string + PageSize: + name: limit + in: query + description: Maximum items per page. + schema: + type: integer + minimum: 1 + maximum: 100 + default: 20 + + schemas: + UserListResponse: + type: object + required: + - data + - pagination + properties: + data: + type: array + items: + $ref: '#/components/schemas/User' + pagination: + type: object + required: + - hasMore + properties: + nextCursor: + type: string + nullable: true + hasMore: + type: boolean +``` + +#### Offset-Based Pagination + +Simpler but less efficient for large tables and susceptible to drift when data +changes between requests. + +```yaml +components: + parameters: + PageOffset: + name: offset + in: query + schema: + type: integer + minimum: 0 + default: 0 + PageLimit: + name: limit + in: query + schema: + type: integer + minimum: 1 + maximum: 100 + default: 20 + + schemas: + PaginatedResponse: + type: object + required: + - data + - total + - offset + - limit + properties: + data: + type: array + items: {} + total: + type: integer + description: Total number of matching records. + offset: + type: integer + limit: + type: integer +``` + +#### Response Envelope Pattern + +Wrap every collection in a consistent envelope so clients always know where to +find the data and metadata: + +```json +{ + "data": [ ... ], + "pagination": { "nextCursor": "abc123", "hasMore": true }, + "meta": { "requestId": "req_xyz", "timestamp": "2026-03-29T12:00:00Z" } +} +``` + +--- + +### 7. API Versioning + +#### URL Versioning + +```yaml +servers: + - url: https://api.acme.dev/v1 + description: Version 1 (deprecated) + - url: https://api.acme.dev/v2 + description: Version 2 (current) +``` + +Pros: explicit, easy to route, cache-friendly. +Cons: duplicates paths across versions, harder to share schemas. + +#### Header Versioning + +```yaml +parameters: + - name: X-API-Version + in: header + required: false + schema: + type: string + enum: ['2024-01-15', '2025-06-01'] + default: '2025-06-01' + description: Date-based API version. Defaults to latest stable. +``` + +Pros: clean URLs, fine-grained control. +Cons: less discoverable, harder to test in a browser. + +#### Trade-offs Summary + +| Approach | Discoverability | URL cleanliness | Caching | Migration effort | +|----------|----------------|-----------------|---------|-----------------| +| URL path | High | Lower | Easy | Higher (path changes) | +| Header | Lower | High | Needs Vary header | Lower | +| Query param | Medium | Medium | Easy | Lower | + +Pick one approach and use it consistently. URL versioning is the most common +choice for public APIs; header versioning suits internal services. + +--- + +### 8. Webhook Specifications + +OpenAPI 3.1 supports a top-level `webhooks` key for documenting outbound +event payloads your API will send to consumer-registered URLs. + +```yaml +webhooks: + orderCompleted: + post: + operationId: onOrderCompleted + summary: Fired when an order reaches "completed" status. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/OrderCompletedEvent' responses: '200': - description: Success - content: - application/json: - schema: - type: array - items: - $ref: '#/components/schemas/User' + description: Webhook received successfully. components: schemas: - User: + WebhookEventBase: type: object + required: + - id + - type + - createdAt properties: id: type: string - email: + format: uuid + type: type: string - format: email - required: - - id - - email + createdAt: + type: string + format: date-time + + OrderCompletedEvent: + allOf: + - $ref: '#/components/schemas/WebhookEventBase' + - type: object + required: + - data + properties: + type: + type: string + const: order.completed + data: + type: object + properties: + orderId: + type: string + format: uuid + total: + type: number + format: double + currency: + type: string + example: USD ``` -### Request Body +Document a shared `WebhookEventBase` so all event payloads have a consistent +envelope with `id`, `type`, and `createdAt`. -```yaml -requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/CreateUser' - example: - email: user@example.com - name: John -``` - -### Error Responses - -```yaml -responses: - '400': - description: Bad request - content: - application/json: - schema: - $ref: '#/components/schemas/Error' -``` +--- ## Best Practices -1. Use $ref for reusable schemas -2. Include examples -3. Document all error responses -4. Use proper HTTP status codes -5. Add security schemes +1. **Use consistent, plural resource names.** `/users`, `/orders`, `/invoices` + -- never mix singular and plural within the same API. + +2. **Make mutating operations idempotent.** Accept an `Idempotency-Key` header + on POST endpoints so clients can safely retry without creating duplicates. + +3. **Return rate-limit headers on every response.** Include `X-RateLimit-Limit`, + `X-RateLimit-Remaining`, and `X-RateLimit-Reset` so clients can self-throttle. + +4. **Provide `operationId` for every operation.** Code generators use this as + the method name; without it, generated clients have meaningless names. + +5. **Include realistic examples in the spec.** Examples power documentation UIs, + mock servers, and contract tests. Add them at both the schema and operation + level. + +6. **Use `additionalProperties: false` on request schemas.** This catches typos + in client payloads early and prevents silently ignored fields. + +7. **Document hypermedia links (HATEOAS basics).** Even a minimal `_links` + object with `self` and `next` URIs helps clients navigate without hardcoding + paths. + +8. **Version your spec file alongside code.** Store the OpenAPI document in the + same repository as the implementation. Run a CI check (e.g., `redocly lint`) + to validate the spec on every pull request. + +--- ## Common Pitfalls -- **Missing examples**: Add realistic examples -- **No error docs**: Document all errors -- **Inconsistent naming**: Use consistent conventions +1. **Missing error documentation.** Every operation should list its possible + `4xx` and `5xx` responses. Consumers cannot handle errors they do not know + about. At minimum document `400`, `401`, `403`, `404`, and `500`. + +2. **Overusing `200 OK` for everything.** Return `201` for resource creation, + `204` for deletion, and `202` for asynchronous actions. Correct status codes + let generic HTTP clients behave properly (e.g., following `Location` headers). + +3. **Deeply nested resource URLs.** `/users/{uid}/orders/{oid}/items/{iid}/notes` + is fragile and hard to cache. Flatten to `/order-items/{iid}/notes` once the + relationship is established. + +4. **Inconsistent naming conventions.** Mixing `camelCase` and `snake_case` + within the same API confuses consumers. Pick one JSON field casing and enforce + it with a linter rule. + +5. **Ignoring `nullable` vs optional.** In OpenAPI 3.1, `nullable` is gone; + use `type: ["string", "null"]` instead. A field that is not in `required` + may be absent, but that is different from being explicitly `null`. Be precise + about which you intend. + +6. **No pagination on list endpoints.** Returning unbounded arrays will + eventually cause timeouts or OOM errors. Every collection endpoint should + accept `limit` and either `cursor` or `offset` from day one, even if the + dataset is currently small. + +--- + +## Related Skills + +- `patterns/api-client` - Patterns for consuming and generating API clients from specs +- `patterns/error-handling` - Consistent error response structures and handling +- `frameworks/fastapi` - FastAPI framework with built-in OpenAPI generation diff --git a/.claude/skills/api/openapi/references/http-status-codes.md b/.claude/skills/api/openapi/references/http-status-codes.md new file mode 100644 index 0000000..589bf2b --- /dev/null +++ b/.claude/skills/api/openapi/references/http-status-codes.md @@ -0,0 +1,175 @@ +# HTTP Status Codes for REST APIs + +Quick reference for selecting the correct HTTP status code in REST API responses. + +--- + +## 2xx Success + +| Code | Name | When to Use | +|------|------|-------------| +| `200` | OK | General success. GET returns data, PUT/PATCH returns updated resource. | +| `201` | Created | POST successfully created a resource. Include `Location` header. | +| `202` | Accepted | Request accepted for async processing. Return a job/task ID. | +| `204` | No Content | DELETE success or PUT/PATCH with no response body needed. | + +**Guidelines:** +- `200` is the default success response for GET, PUT, PATCH +- `201` must be used when a new resource is created (POST) +- `204` is preferred for DELETE (no body to return) +- `202` signals "we got it, processing later" -- return a status URL + +```json +// 201 Created response +{ + "id": "usr_abc123", + "name": "Jane Doe", + "created_at": "2025-01-15T10:30:00Z" +} +// Header: Location: /api/v1/users/usr_abc123 +``` + +--- + +## 3xx Redirection + +| Code | Name | When to Use | +|------|------|-------------| +| `301` | Moved Permanently | Resource URL changed permanently. Clients should update bookmarks. | +| `302` | Found | Temporary redirect. Original URL still valid. | +| `304` | Not Modified | Conditional GET -- resource unchanged since `If-None-Match`/`If-Modified-Since`. | +| `307` | Temporary Redirect | Like 302 but preserves HTTP method. Use for API redirects. | +| `308` | Permanent Redirect | Like 301 but preserves HTTP method. | + +**Guidelines:** +- Prefer `307`/`308` over `302`/`301` in APIs (method preservation) +- `304` reduces bandwidth when clients cache responses +- Always include `Location` header with redirect responses + +--- + +## 4xx Client Errors + +| Code | Name | When to Use | +|------|------|-------------| +| `400` | Bad Request | Malformed syntax, invalid JSON, failed validation. | +| `401` | Unauthorized | Missing or invalid authentication credentials. | +| `403` | Forbidden | Authenticated but lacks permission for this resource. | +| `404` | Not Found | Resource does not exist at this URL. | +| `405` | Method Not Allowed | HTTP method not supported on this endpoint. | +| `409` | Conflict | Request conflicts with current state (duplicate, version mismatch). | +| `410` | Gone | Resource existed but has been permanently deleted. | +| `415` | Unsupported Media Type | Content-Type header not supported. | +| `422` | Unprocessable Entity | Valid JSON but semantically invalid (business rule violation). | +| `429` | Too Many Requests | Rate limit exceeded. Include `Retry-After` header. | + +**Guidelines:** +- `400` for structural issues (bad JSON, missing required fields) +- `422` for business logic failures (email already taken, invalid state transition) +- `401` means "who are you?" -- `403` means "I know who you are, but no" +- `409` for optimistic locking failures and unique constraint violations +- `429` must include `Retry-After` header with seconds until retry + +```json +// 422 Unprocessable Entity +{ + "error": { + "code": "VALIDATION_ERROR", + "message": "Request validation failed", + "details": [ + { "field": "email", "message": "Email already registered" }, + { "field": "age", "message": "Must be 18 or older" } + ] + } +} +``` + +```json +// 429 Too Many Requests +// Header: Retry-After: 60 +{ + "error": { + "code": "RATE_LIMITED", + "message": "Rate limit exceeded. Try again in 60 seconds." + } +} +``` + +--- + +## 5xx Server Errors + +| Code | Name | When to Use | +|------|------|-------------| +| `500` | Internal Server Error | Unhandled exception. Generic server failure. | +| `501` | Not Implemented | Endpoint exists but functionality not built yet. | +| `502` | Bad Gateway | Upstream service returned invalid response. | +| `503` | Service Unavailable | Server overloaded or in maintenance. Include `Retry-After`. | +| `504` | Gateway Timeout | Upstream service did not respond in time. | + +**Guidelines:** +- `500` should never expose stack traces in production +- `503` should include `Retry-After` header and a maintenance message +- Log all 5xx errors with request context for debugging +- Return a consistent error body format for all 5xx responses + +```json +// 500 Internal Server Error (production) +{ + "error": { + "code": "INTERNAL_ERROR", + "message": "An unexpected error occurred. Please try again.", + "request_id": "req_7f3a9b2c" + } +} +``` + +--- + +## Decision Flowchart + +``` +Request received + | + +-- Is it valid syntax? -- NO --> 400 Bad Request + | + +-- Is caller authenticated? -- NO --> 401 Unauthorized + | + +-- Is caller authorized? -- NO --> 403 Forbidden + | + +-- Does resource exist? -- NO --> 404 Not Found + | + +-- Is it rate-limited? -- YES --> 429 Too Many Requests + | + +-- Does it pass business rules? -- NO --> 422 Unprocessable Entity + | + +-- Any conflicts? -- YES --> 409 Conflict + | + +-- Server error? -- YES --> 500 Internal Server Error + | + +-- Success! + GET --> 200 OK + POST --> 201 Created + PUT --> 200 OK + PATCH --> 200 OK + DELETE --> 204 No Content +``` + +--- + +## Standard Error Response Format + +Use a consistent structure across all error responses: + +```json +{ + "error": { + "code": "MACHINE_READABLE_CODE", + "message": "Human-readable description", + "details": [], + "request_id": "req_..." + } +} +``` + +*Reference: [RFC 9110 - HTTP Semantics](https://httpwg.org/specs/rfc9110.html), [RFC 9457 - Problem Details](https://www.rfc-editor.org/rfc/rfc9457)* diff --git a/.claude/skills/api/openapi/references/rest-naming.md b/.claude/skills/api/openapi/references/rest-naming.md new file mode 100644 index 0000000..90b647f --- /dev/null +++ b/.claude/skills/api/openapi/references/rest-naming.md @@ -0,0 +1,196 @@ +# REST API Naming Conventions + +Guidelines for consistent, predictable REST endpoint design. + +--- + +## Core Rules + +1. **Use plural nouns** for resource collections +2. **Use kebab-case** for multi-word resources +3. **Use path parameters** for identity, query parameters for filtering +4. **Never use verbs** in URLs (HTTP methods convey the action) +5. **Use lowercase** exclusively + +--- + +## Resource Naming + +| Pattern | Example | Notes | +|---------|---------|-------| +| Collection | `/users` | Plural noun | +| Single resource | `/users/{id}` | Path parameter | +| Multi-word resource | `/order-items` | Kebab-case | +| Nested resource | `/users/{id}/orders` | Parent-child relationship | +| Deep nesting (avoid) | `/users/{id}/orders/{oid}/items` | Max 2 levels deep | +| Singleton sub-resource | `/users/{id}/profile` | One-to-one relationship | + +### Good + +``` +GET /users +GET /users/123 +POST /users +PUT /users/123 +DELETE /users/123 +GET /users/123/orders +GET /order-items +GET /user-profiles/123 +``` + +### Bad + +``` +GET /getUsers # verb in URL +GET /user/123 # singular collection +GET /Users # uppercase +POST /users/create # redundant verb +GET /user_profiles # snake_case +DELETE /users/123/delete # verb in URL +``` + +--- + +## CRUD Mapping + +| Action | Method | Endpoint | Request Body | Response | +|--------|--------|----------|-------------|----------| +| List | `GET` | `/resources` | None | `200` + array | +| Create | `POST` | `/resources` | Resource data | `201` + created | +| Read | `GET` | `/resources/{id}` | None | `200` + object | +| Update (full) | `PUT` | `/resources/{id}` | Full resource | `200` + updated | +| Update (partial) | `PATCH` | `/resources/{id}` | Partial data | `200` + updated | +| Delete | `DELETE` | `/resources/{id}` | None | `204` | + +--- + +## Nested Resources + +Use nesting to express clear parent-child relationships. + +``` +# User's orders (user owns orders) +GET /users/{userId}/orders +POST /users/{userId}/orders + +# Order's line items +GET /orders/{orderId}/items +``` + +**When to nest vs. top-level:** + +| Scenario | Approach | Example | +|----------|----------|---------| +| Resource only exists under parent | Nest | `/users/{id}/sessions` | +| Resource is independently accessible | Top-level with filter | `/orders?user_id=123` | +| Shallow relationship | Top-level | `/comments?post_id=456` | + +**Rule of thumb:** Never nest more than 2 levels. Use query parameters or top-level endpoints instead. + +``` +# Too deep -- avoid +GET /users/{id}/orders/{oid}/items/{iid}/reviews + +# Better alternatives +GET /order-items/{iid}/reviews +GET /reviews?order_item_id={iid} +``` + +--- + +## Query Parameters + +### Filtering + +``` +GET /products?category=electronics&brand=acme +GET /users?status=active&role=admin +GET /orders?created_after=2025-01-01&created_before=2025-02-01 +``` + +| Convention | Example | +|-----------|---------| +| Exact match | `?status=active` | +| Date range | `?created_after=2025-01-01` | +| Multiple values | `?status=active,pending` | +| Search | `?q=search+term` | + +### Sorting + +``` +GET /products?sort=price # ascending (default) +GET /products?sort=-price # descending (prefix -) +GET /products?sort=-created_at,name # multi-field +``` + +### Pagination + +``` +# Offset-based (simple, common) +GET /products?page=2&per_page=25 + +# Cursor-based (better for large datasets) +GET /products?cursor=eyJpZCI6MTAwfQ&limit=25 +``` + +**Response envelope for paginated results:** + +```json +{ + "data": [...], + "pagination": { + "page": 2, + "per_page": 25, + "total": 150, + "total_pages": 6 + } +} +``` + +### Field Selection + +``` +GET /users/123?fields=id,name,email +``` + +--- + +## Non-CRUD Actions + +Some operations do not map cleanly to CRUD. Use sub-resources with a noun or POST with an action resource. + +| Action | Approach | Example | +|--------|----------|---------| +| Send an email | POST to action resource | `POST /users/{id}/verification-email` | +| Archive | PATCH with status | `PATCH /orders/{id} { "status": "archived" }` | +| Bulk delete | POST to action | `POST /users/bulk-delete { "ids": [...] }` | +| Export | GET with format | `GET /reports/sales?format=csv` | +| Search (complex) | POST with body | `POST /products/search { "filters": {...} }` | + +--- + +## Versioning + +| Strategy | Example | Pros | Cons | +|----------|---------|------|------| +| URL path | `/api/v1/users` | Simple, explicit | URL pollution | +| Header | `Accept: application/vnd.api.v1+json` | Clean URLs | Hidden | +| Query param | `/users?version=1` | Easy to test | Caching issues | + +**Recommended:** URL path versioning (`/api/v1/`) for public APIs due to simplicity. + +--- + +## Summary Checklist + +- [ ] Resources are plural nouns (`/users` not `/user`) +- [ ] URLs are kebab-case and lowercase +- [ ] No verbs in URLs +- [ ] Nesting limited to 2 levels +- [ ] Filtering uses query parameters +- [ ] Sorting supports `-field` for descending +- [ ] Pagination included on all list endpoints +- [ ] API version in URL path for public APIs +- [ ] Consistent error response format + +*Reference: [Google API Design Guide](https://cloud.google.com/apis/design), [Microsoft REST Guidelines](https://github.com/microsoft/api-guidelines)* diff --git a/.claude/skills/api/openapi/templates/openapi-3.1-starter.yaml b/.claude/skills/api/openapi/templates/openapi-3.1-starter.yaml new file mode 100644 index 0000000..f85df7e --- /dev/null +++ b/.claude/skills/api/openapi/templates/openapi-3.1-starter.yaml @@ -0,0 +1,240 @@ +openapi: "3.1.0" +info: + title: My API + description: Starter API specification. Replace with your project details. + version: "1.0.0" + contact: + name: API Support + email: support@example.com + +servers: + - url: http://localhost:3000/api/v1 + description: Local development + - url: https://api.example.com/v1 + description: Production + +tags: + - name: Users + description: User management + - name: Health + description: Service health checks + +paths: + /health: + get: + tags: [Health] + summary: Health check + operationId: getHealth + responses: + "200": + description: Service is healthy + content: + application/json: + schema: + type: object + properties: + status: { type: string, example: ok } + timestamp: { type: string, format: date-time } + + /users: + get: + tags: [Users] + summary: List users + operationId: listUsers + security: [{ bearerAuth: [] }] + parameters: + - $ref: "#/components/parameters/PageParam" + - $ref: "#/components/parameters/PerPageParam" + - $ref: "#/components/parameters/SortParam" + - name: status + in: query + schema: { type: string, enum: [active, inactive] } + responses: + "200": + description: Paginated list of users + content: + application/json: + schema: + type: object + properties: + data: + type: array + items: { $ref: "#/components/schemas/User" } + pagination: { $ref: "#/components/schemas/Pagination" } + "401": { $ref: "#/components/responses/Unauthorized" } + + post: + tags: [Users] + summary: Create a user + operationId: createUser + security: [{ bearerAuth: [] }] + requestBody: + required: true + content: + application/json: + schema: { $ref: "#/components/schemas/CreateUserRequest" } + responses: + "201": + description: User created + headers: + Location: { schema: { type: string }, description: URL of created user } + content: + application/json: + schema: { $ref: "#/components/schemas/User" } + "400": { $ref: "#/components/responses/BadRequest" } + "401": { $ref: "#/components/responses/Unauthorized" } + "422": { $ref: "#/components/responses/ValidationError" } + + /users/{userId}: + parameters: + - name: userId + in: path + required: true + schema: { type: string } + + get: + tags: [Users] + summary: Get a user by ID + operationId: getUser + security: [{ bearerAuth: [] }] + responses: + "200": + description: User details + content: + application/json: + schema: { $ref: "#/components/schemas/User" } + "401": { $ref: "#/components/responses/Unauthorized" } + "404": { $ref: "#/components/responses/NotFound" } + + patch: + tags: [Users] + summary: Update a user + operationId: updateUser + security: [{ bearerAuth: [] }] + requestBody: + required: true + content: + application/json: + schema: { $ref: "#/components/schemas/UpdateUserRequest" } + responses: + "200": + description: User updated + content: + application/json: + schema: { $ref: "#/components/schemas/User" } + "401": { $ref: "#/components/responses/Unauthorized" } + "404": { $ref: "#/components/responses/NotFound" } + "422": { $ref: "#/components/responses/ValidationError" } + + delete: + tags: [Users] + summary: Delete a user + operationId: deleteUser + security: [{ bearerAuth: [] }] + responses: + "204": { description: User deleted } + "401": { $ref: "#/components/responses/Unauthorized" } + "404": { $ref: "#/components/responses/NotFound" } + +components: + securitySchemes: + bearerAuth: { type: http, scheme: bearer, bearerFormat: JWT } + apiKeyAuth: { type: apiKey, in: header, name: X-API-Key } + + parameters: + PageParam: + name: page + in: query + schema: { type: integer, minimum: 1, default: 1 } + PerPageParam: + name: per_page + in: query + schema: { type: integer, minimum: 1, maximum: 100, default: 25 } + SortParam: + name: sort + in: query + description: "Field to sort by. Prefix with - for descending." + schema: { type: string, example: "-created_at" } + + schemas: + User: + type: object + required: [id, email, name, status, created_at] + properties: + id: { type: string, example: usr_abc123 } + email: { type: string, format: email, example: jane@example.com } + name: { type: string, example: Jane Doe } + status: { type: string, enum: [active, inactive] } + created_at: { type: string, format: date-time } + updated_at: { type: string, format: date-time } + + CreateUserRequest: + type: object + required: [email, name] + properties: + email: { type: string, format: email } + name: { type: string, minLength: 1, maxLength: 100 } + role: { type: string, enum: [user, admin], default: user } + + UpdateUserRequest: + type: object + properties: + name: { type: string, minLength: 1, maxLength: 100 } + status: { type: string, enum: [active, inactive] } + + Pagination: + type: object + properties: + page: { type: integer } + per_page: { type: integer } + total: { type: integer } + total_pages: { type: integer } + + Error: + type: object + required: [error] + properties: + error: + type: object + required: [code, message] + properties: + code: { type: string } + message: { type: string } + details: + type: array + items: + type: object + properties: + field: { type: string } + message: { type: string } + request_id: { type: string } + + responses: + BadRequest: + description: Bad request + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + example: { error: { code: BAD_REQUEST, message: Malformed request body } } + Unauthorized: + description: Authentication required + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + example: { error: { code: UNAUTHORIZED, message: Missing or invalid token } } + NotFound: + description: Resource not found + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + example: { error: { code: NOT_FOUND, message: Resource not found } } + ValidationError: + description: Validation failed + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + example: + error: + code: VALIDATION_ERROR + message: Request validation failed + details: [{ field: email, message: Email already registered }] diff --git a/.claude/skills/databases/mongodb/SKILL.md b/.claude/skills/databases/mongodb/SKILL.md index 867f23a..bc736ad 100644 --- a/.claude/skills/databases/mongodb/SKILL.md +++ b/.claude/skills/databases/mongodb/SKILL.md @@ -1,73 +1,579 @@ +--- +name: mongodb +description: > + Use this skill whenever working with MongoDB, document databases, or NoSQL data modeling. Trigger on keywords like MongoDB, Mongo, document database, aggregation pipeline, collection, embedded documents, or BSON. Also applies when designing document schemas, building aggregation queries, handling unstructured or semi-structured data, or migrating from relational to document-based storage. +--- + # MongoDB -## Description - -MongoDB patterns including document design, queries, and aggregation. - ## When to Use - MongoDB database operations - Document-based data modeling - Aggregation pipelines +- Semi-structured or polymorphic data that varies per record +- Rapid prototyping where schema flexibility accelerates iteration +- Event logging, IoT telemetry, or content management systems + +## When NOT to Use + +- Relational-heavy data models with complex joins and foreign key constraints +- SQL-only projects where the entire stack is built around relational databases +- Simple key-value storage where Redis or a lightweight store is more appropriate +- Financial systems requiring multi-table ACID transactions as the norm --- ## Core Patterns -### Document Operations +### 1. Schema Design -```javascript -// Insert -db.users.insertOne({ - email: 'user@example.com', - name: 'John', - createdAt: new Date() -}); +The central decision in MongoDB modeling is **embed vs. reference**. -// Find -db.users.find({ active: true }).sort({ createdAt: -1 }).limit(20); +**Decision tree:** -// Update -db.users.updateOne( - { _id: ObjectId('...') }, - { $set: { name: 'Jane' } } -); +``` +Does the child data belong to exactly one parent? + YES --> Is the child array unbounded (could grow to thousands)? + YES --> Reference (separate collection) + NO --> Embed + NO --> Is it a many-to-many relationship? + YES --> Reference (with array of ObjectIds on one or both sides) + NO --> Reference ``` -### Aggregation +**Embedding pattern -- best for data that is read together:** ```javascript +// User with embedded address and preferences +// Good: one read fetches everything the profile page needs +db.users.insertOne({ + email: "user@example.com", + name: "Alice Chen", + address: { + street: "123 Main St", + city: "Portland", + state: "OR", + zip: "97201" + }, + preferences: { + theme: "dark", + language: "en", + notifications: { email: true, push: false } + }, + createdAt: new Date() +}); +``` + +**Referencing pattern -- best for independent or unbounded data:** + +```javascript +// Orders reference the user by ID +// Good: orders grow unboundedly, accessed independently +db.orders.insertOne({ + userId: ObjectId("6651a..."), + status: "shipped", + totalCents: 4999, + items: [ + { sku: "WIDGET-001", name: "Blue Widget", qty: 2, priceCents: 1999 }, + { sku: "GADGET-010", name: "Mini Gadget", qty: 1, priceCents: 1001 } + ], + placedAt: new Date() +}); +``` + +**Denormalization pattern -- duplicate data to avoid frequent lookups:** + +```javascript +// Store author name directly on the post (denormalized from users) +// Trade-off: faster reads, but updates to user name require updating all posts +db.posts.insertOne({ + title: "Getting Started with MongoDB", + body: "...", + author: { + _id: ObjectId("6651a..."), + name: "Alice Chen" // denormalized -- must be updated if name changes + }, + tags: ["mongodb", "tutorial"], + publishedAt: new Date() +}); +``` + +**Polymorphic pattern -- different shapes in one collection:** + +```javascript +// Events collection stores different event types +db.events.insertMany([ + { + type: "page_view", + userId: ObjectId("6651a..."), + url: "/products/widget", + timestamp: new Date() + }, + { + type: "purchase", + userId: ObjectId("6651a..."), + orderId: ObjectId("6651b..."), + totalCents: 4999, + timestamp: new Date() + } +]); +// Use a discriminator field (type) and query by it +``` + +**Schema validation -- enforce structure at the database level:** + +```javascript +db.createCollection("users", { + validator: { + $jsonSchema: { + bsonType: "object", + required: ["email", "name", "createdAt"], + properties: { + email: { + bsonType: "string", + pattern: "^.+@.+\\..+$", + description: "Must be a valid email" + }, + name: { + bsonType: "string", + minLength: 1 + }, + role: { + enum: ["admin", "editor", "viewer"], + description: "Must be a valid role" + }, + createdAt: { bsonType: "date" } + } + } + }, + validationLevel: "strict", + validationAction: "error" +}); +``` + +--- + +### 2. Aggregation Pipeline + +Build complex data transformations as a sequence of stages. + +```javascript +// Revenue report: total and average spend per user, last 30 days db.orders.aggregate([ - { $match: { status: 'completed' } }, - { $group: { - _id: '$userId', - totalSpent: { $sum: '$amount' }, - orderCount: { $count: {} } + // Stage 1: filter to recent delivered orders + { $match: { + status: "delivered", + placedAt: { $gte: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000) } }}, - { $sort: { totalSpent: -1 } } + + // Stage 2: group by user + { $group: { + _id: "$userId", + totalSpent: { $sum: "$totalCents" }, + orderCount: { $sum: 1 }, + avgOrderValue: { $avg: "$totalCents" } + }}, + + // Stage 3: sort by spend + { $sort: { totalSpent: -1 } }, + + // Stage 4: limit to top 10 + { $limit: 10 }, + + // Stage 5: join user details + { $lookup: { + from: "users", + localField: "_id", + foreignField: "_id", + as: "user" + }}, + + // Stage 6: flatten the joined array + { $unwind: "$user" }, + + // Stage 7: reshape output + { $project: { + _id: 0, + userName: "$user.name", + email: "$user.email", + totalSpent: 1, + orderCount: 1, + avgOrderValue: { $round: ["$avgOrderValue", 0] } + }} ]); ``` -### Indexes +**$unwind -- flatten arrays into individual documents:** ```javascript -// Single field +// Expand order items to analyze product-level metrics +db.orders.aggregate([ + { $unwind: "$items" }, + { $group: { + _id: "$items.sku", + totalQty: { $sum: "$items.qty" }, + totalRevenue: { $sum: { $multiply: ["$items.qty", "$items.priceCents"] } } + }}, + { $sort: { totalRevenue: -1 } } +]); +``` + +**$lookup with pipeline -- filtered/correlated joins:** + +```javascript +// For each user, get their 3 most recent orders +db.users.aggregate([ + { $lookup: { + from: "orders", + let: { uid: "$_id" }, + pipeline: [ + { $match: { $expr: { $eq: ["$userId", "$$uid"] } } }, + { $sort: { placedAt: -1 } }, + { $limit: 3 }, + { $project: { status: 1, totalCents: 1, placedAt: 1 } } + ], + as: "recentOrders" + }} +]); +``` + +**$facet -- run multiple aggregations in parallel:** + +```javascript +// Dashboard: get summary stats and top products in one query +db.orders.aggregate([ + { $match: { status: "delivered" } }, + { $facet: { + summary: [ + { $group: { + _id: null, + totalRevenue: { $sum: "$totalCents" }, + totalOrders: { $sum: 1 } + }} + ], + topProducts: [ + { $unwind: "$items" }, + { $group: { _id: "$items.sku", sold: { $sum: "$items.qty" } } }, + { $sort: { sold: -1 } }, + { $limit: 5 } + ], + monthlyTrend: [ + { $group: { + _id: { $dateToString: { format: "%Y-%m", date: "$placedAt" } }, + revenue: { $sum: "$totalCents" } + }}, + { $sort: { _id: 1 } } + ] + }} +]); +``` + +--- + +### 3. Index Strategies + +```javascript +// Single field index -- most common db.users.createIndex({ email: 1 }, { unique: true }); -// Compound -db.posts.createIndex({ userId: 1, createdAt: -1 }); +// Compound index -- order matters, follows the ESR rule: +// Equality fields first, Sort fields next, Range fields last +db.orders.createIndex({ status: 1, placedAt: -1 }); +// Supports: find({status: "pending"}).sort({placedAt: -1}) +// Also supports: find({status: "pending"}) alone (prefix) + +// Multikey index -- automatically indexes each array element +db.posts.createIndex({ tags: 1 }); +// Supports: find({ tags: "mongodb" }) + +// Text index -- basic full-text search +db.posts.createIndex( + { title: "text", body: "text" }, + { weights: { title: 10, body: 1 }, name: "posts_text_search" } +); +// Usage: +db.posts.find( + { $text: { $search: "mongodb aggregation" } }, + { score: { $meta: "textScore" } } +).sort({ score: { $meta: "textScore" } }); + +// TTL index -- auto-delete documents after expiry +db.sessions.createIndex( + { expiresAt: 1 }, + { expireAfterSeconds: 0 } // delete when expiresAt is in the past +); +// Documents must have a Date field; they are removed by a background task ~every 60s + +// Partial index -- only index documents matching a filter +db.orders.createIndex( + { placedAt: -1 }, + { partialFilterExpression: { status: "pending" } } +); +// Smaller index; only used when the query includes the filter condition + +// Wildcard index -- for querying arbitrary keys in a sub-document +db.products.createIndex({ "attributes.$**": 1 }); +// Supports: find({ "attributes.color": "red" }) without knowing keys in advance + +// Collation -- case-insensitive sorting and matching +db.users.createIndex( + { name: 1 }, + { collation: { locale: "en", strength: 2 } } +); ``` +**The ESR rule for compound indexes:** order fields by **E**quality, **S**ort, **R**ange. This produces the most efficient index scans. + +```javascript +// Query: find active orders for a user, sorted by date, in a date range +// Equality: userId, status +// Sort: placedAt +// Range: placedAt (but sort and range on same field -- sort wins) +db.orders.createIndex({ userId: 1, status: 1, placedAt: -1 }); +``` + +--- + +### 4. Transactions + +Multi-document transactions work across collections (requires replica set or sharded cluster). + +```javascript +const session = client.startSession(); + +try { + session.startTransaction({ + readConcern: { level: "snapshot" }, + writeConcern: { w: "majority" }, + readPreference: "primary" + }); + + const accounts = client.db("bank").collection("accounts"); + + // Transfer $50 from account A to account B + const fromAccount = await accounts.findOne( + { _id: "account-A" }, + { session } + ); + + if (fromAccount.balanceCents < 5000) { + await session.abortTransaction(); + throw new Error("Insufficient funds"); + } + + await accounts.updateOne( + { _id: "account-A" }, + { $inc: { balanceCents: -5000 } }, + { session } + ); + + await accounts.updateOne( + { _id: "account-B" }, + { $inc: { balanceCents: 5000 } }, + { session } + ); + + // Record the transfer in a separate collection -- still in the same tx + await client.db("bank").collection("transfers").insertOne({ + from: "account-A", + to: "account-B", + amountCents: 5000, + timestamp: new Date() + }, { session }); + + await session.commitTransaction(); +} catch (error) { + await session.abortTransaction(); + throw error; +} finally { + await session.endSession(); +} +``` + +**Guidelines:** +- Keep transactions short -- they hold locks and consume resources +- Design your schema to minimize the need for multi-document transactions +- Transactions have a default 60-second timeout (`maxTimeMS`) +- Retryable writes (`retryWrites=true` in connection string) handle transient errors automatically + +--- + +### 5. Change Streams + +Watch for real-time changes to collections, databases, or the entire deployment. + +```javascript +// Watch a single collection for inserts and updates +const pipeline = [ + { $match: { + operationType: { $in: ["insert", "update"] }, + "fullDocument.status": "urgent" + }} +]; + +const changeStream = db.collection("tickets").watch(pipeline, { + fullDocument: "updateLookup" // include the full document on updates +}); + +changeStream.on("change", (change) => { + console.log("Change detected:", change.operationType); + console.log("Document:", change.fullDocument); + console.log("Resume token:", change.resumeToken); + + // Process the change (e.g., send notification, update cache) + notifyTeam(change.fullDocument); +}); + +// Handle errors and resume from last known position +changeStream.on("error", (error) => { + console.error("Change stream error:", error); + // Reconnect using the stored resume token +}); +``` + +**Resumable pattern for production:** + +```javascript +let resumeToken = await loadResumeTokenFromStorage(); + +async function watchWithResume(collection) { + const options = { fullDocument: "updateLookup" }; + if (resumeToken) { + options.resumeAfter = resumeToken; + } + + const stream = collection.watch([], options); + + stream.on("change", async (change) => { + // Process change + await handleChange(change); + + // Persist resume token so we can recover after restart + resumeToken = change._id; + await saveResumeTokenToStorage(resumeToken); + }); + + stream.on("error", async () => { + // Wait and reconnect + await new Promise(r => setTimeout(r, 5000)); + watchWithResume(collection); + }); +} +``` + +**Use cases:** real-time dashboards, cache invalidation, event-driven architectures, syncing data to search indexes (e.g., Elasticsearch). + +--- + +### 6. Performance + +#### Reading explain() output + +```javascript +// Run explain to see the query plan +db.orders.find({ + userId: ObjectId("6651a..."), + status: "pending" +}).sort({ placedAt: -1 }).explain("executionStats"); +``` + +**Key fields in executionStats:** + +| Field | What to look for | +|-------|-----------------| +| `winningPlan.stage` | `IXSCAN` good, `COLLSCAN` bad (full collection scan) | +| `totalKeysExamined` | Should be close to `nReturned` (no wasted index scans) | +| `totalDocsExamined` | Should be close to `nReturned` (no wasted document reads) | +| `executionTimeMillis` | Overall query time | +| `rejectedPlans` | Shows alternatives the optimizer considered | + +**Covered queries -- answered entirely from the index:** + +```javascript +// Create an index that covers the query +db.orders.createIndex({ userId: 1, status: 1, totalCents: 1 }); + +// This query only needs fields in the index -- no document fetch +db.orders.find( + { userId: ObjectId("6651a..."), status: "delivered" }, + { _id: 0, totalCents: 1 } // projection must exclude _id and only include indexed fields +); +// explain() will show: "totalDocsExamined": 0 +``` + +**Projection optimization -- fetch only what you need:** + +```javascript +// BAD: fetches entire document including large body field +const posts = await db.posts.find({ author: userId }).toArray(); + +// GOOD: only fetch fields needed for the list view +const posts = await db.posts.find( + { author: userId }, + { projection: { title: 1, publishedAt: 1, tags: 1 } } +).toArray(); +``` + +**Bulk operations for write-heavy workloads:** + +```javascript +const bulk = db.products.initializeUnorderedBulkOp(); + +for (const update of priceUpdates) { + bulk.find({ sku: update.sku }) + .updateOne({ $set: { priceCents: update.newPrice, updatedAt: new Date() } }); +} + +const result = await bulk.execute(); +console.log(`Modified: ${result.nModified}, Errors: ${result.getWriteErrorCount()}`); +``` + +--- + ## Best Practices -1. Embed frequently accessed data -2. Use references for large/independent data -3. Create indexes for query patterns -4. Use aggregation for complex queries -5. Avoid unbounded arrays +1. **Design schema around query patterns, not data relationships.** Ask "how will I read this data?" before "how does this data relate?" Embed data that is always fetched together; reference data accessed independently. + +2. **Use the ESR rule for compound indexes.** Order index fields by Equality, Sort, Range. This maximizes the index's usefulness and minimizes keys examined. + +3. **Set read/write concerns appropriately.** Use `w: "majority"` and `readConcern: "majority"` for data that must survive failovers. Use `w: 1` for non-critical writes where speed matters more than durability. + +4. **Use projection to limit returned fields.** Transferring large documents over the network when you only need two fields wastes bandwidth and memory. Always project. + +5. **Avoid unbounded array growth.** An embedded array that can grow to thousands of elements bloats the document (16 MB max) and degrades performance. Move to a separate collection with a reference when the array exceeds ~100 elements. + +6. **Use bulk operations for batch writes.** Individual `insertOne` or `updateOne` calls in a loop are slow. Batch them with `bulkWrite` or `initializeUnorderedBulkOp` for 10-50x throughput improvement. + +7. **Enable retryable writes.** Add `retryWrites=true` to your connection string. This handles transient network errors and primary elections automatically without application-level retry logic. + +8. **Monitor with database profiler and serverStatus.** Use `db.setProfilingLevel(1, { slowms: 100 })` to log slow queries. Check `db.serverStatus().opcounters` and `db.serverStatus().connections` for overall health. ## Common Pitfalls -- **Unbounded arrays**: Limit array size -- **Missing indexes**: Analyze query patterns -- **Over-embedding**: Consider data access patterns +1. **Treating MongoDB like a relational database.** Normalizing everything into separate collections and using `$lookup` for every query defeats the purpose. If you need heavy joins, PostgreSQL is likely a better fit. Design for embedding first. + +2. **Missing indexes on query fields.** Every `find()`, `$match`, and `sort()` should be backed by an index. Use `db.collection.getIndexes()` and `explain()` to verify. A `COLLSCAN` on a large collection is almost always a bug. + +3. **Ignoring the 16 MB document size limit.** Embedding unbounded arrays (comments, logs, events) will eventually hit this wall, crashing writes. Use the bucket pattern (fixed-size sub-documents) or reference a separate collection. + +4. **Not using readPreference for read-heavy workloads.** By default all reads go to the primary. For analytics or non-critical reads, use `readPreference: "secondaryPreferred"` to distribute load across replicas. + +5. **Forgetting that updates replace matched array elements, not all of them.** Using `$set` on a matched array element with positional `$` only updates the first match. Use `$[]` for all elements or `$[]` with `arrayFilters` for conditional updates: + +```javascript +// Update price for a specific item in all orders +db.orders.updateMany( + { "items.sku": "WIDGET-001" }, + { $set: { "items.$[item].priceCents": 2499 } }, + { arrayFilters: [{ "item.sku": "WIDGET-001" }] } +); +``` + +6. **Running aggregation pipelines without early $match.** Always filter as early as possible in the pipeline. A `$group` or `$unwind` before `$match` processes the entire collection unnecessarily. Put `$match` first to leverage indexes and reduce documents flowing through subsequent stages. + +## Related Skills + +- `databases/postgresql` - Relational database patterns for structured data with complex relationships +- `patterns/caching` - Caching strategies to reduce database load +- `patterns/logging` - Logging patterns for query debugging and monitoring diff --git a/.claude/skills/databases/mongodb/references/schema-patterns.md b/.claude/skills/databases/mongodb/references/schema-patterns.md new file mode 100644 index 0000000..c488607 --- /dev/null +++ b/.claude/skills/databases/mongodb/references/schema-patterns.md @@ -0,0 +1,237 @@ +# MongoDB Schema Design Patterns + +Quick reference for embedding vs referencing decisions and common schema patterns. + +## Embedding vs Referencing Decision Tree + +``` +What is the relationship cardinality? +| ++-- One-to-Few (< 50 items)? +| --> EMBED in parent document +| Example: user.addresses, post.tags +| ++-- One-to-Many (50 - 1000s)? +| | +| +-- Child data always accessed with parent? +| | --> EMBED (but watch 16 MB doc limit) +| | +| +-- Child data accessed independently? +| | --> REFERENCE (store child _id in parent array) +| | +| +-- Need atomic updates across parent + children? +| --> EMBED +| ++-- One-to-Millions? +| --> REFERENCE from child to parent +| Example: log_entry.host_id (not host.log_entry_ids) +| ++-- Many-to-Many? + --> REFERENCE with array of _ids on one or both sides + Example: student.course_ids[], course.student_ids[] +``` + +## Decision Factors + +| Factor | Favor Embedding | Favor Referencing | +|--------|----------------|-------------------| +| **Read pattern** | Always read together | Read independently | +| **Write pattern** | Infrequent child updates | Frequent child updates | +| **Data size** | Small, bounded children | Large or growing children | +| **Atomicity** | Need single-doc transactions | Can tolerate multi-doc txn | +| **Duplication** | OK to denormalize | Must avoid duplication | +| **Cardinality** | Few items | Many/unbounded items | +| **Document size** | Well under 16 MB limit | Approaching 16 MB | + +## Pattern Catalog + +### 1. Subset Pattern + +**Problem**: Document is large but reads only need a few fields from embedded data. + +**Solution**: Embed a subset; keep full data in a separate collection. + +```javascript +// products collection - fast reads for listing pages +{ + _id: ObjectId("..."), + name: "Widget", + price: 29.99, + // Only the 10 most recent reviews (subset) + recent_reviews: [ + { user: "alice", rating: 5, text: "Great!", date: ISODate("...") } + ], + review_count: 247 +} + +// reviews collection - full review data +{ + _id: ObjectId("..."), + product_id: ObjectId("..."), + user: "alice", + rating: 5, + text: "Great!", + date: ISODate("..."), + helpful_votes: 12 +} +``` + +**When to use**: Product pages, user profiles, any "preview + detail" pattern. + +### 2. Computed Pattern + +**Problem**: Expensive aggregation queries run repeatedly on the same data. + +**Solution**: Pre-compute and store the result, update on write. + +```javascript +// movies collection +{ + _id: ObjectId("..."), + title: "Example Movie", + // Pre-computed from screenings collection + computed: { + total_revenue: 1250000, + avg_rating: 4.2, + rating_count: 843, + last_computed: ISODate("2025-01-15T00:00:00Z") + } +} +``` + +**Update strategy**: On each new rating, increment count and recalculate average. Or use a background job for less time-sensitive data. + +**When to use**: Dashboards, leaderboards, summary statistics. + +### 3. Bucket Pattern + +**Problem**: Many small, time-series documents create overhead (indexes, storage per doc). + +**Solution**: Group related data into fixed-size buckets. + +```javascript +// sensor_readings collection - one doc per sensor per hour +{ + sensor_id: "sensor-42", + bucket_start: ISODate("2025-01-15T14:00:00Z"), + bucket_end: ISODate("2025-01-15T14:59:59Z"), + count: 60, + readings: [ + { ts: ISODate("2025-01-15T14:00:00Z"), temp: 22.1, humidity: 45 }, + { ts: ISODate("2025-01-15T14:01:00Z"), temp: 22.3, humidity: 44 } + // ... up to 60 readings per bucket + ], + // Pre-computed aggregates for the bucket + summary: { + avg_temp: 22.4, + min_temp: 21.8, + max_temp: 23.1 + } +} +``` + +**Bucket sizing**: Choose a size that balances doc count reduction vs update frequency. Common choices: 1 hour, 1 day, 100 events. + +**When to use**: IoT, time-series, event logging, analytics. + +### 4. Outlier Pattern + +**Problem**: A few documents have vastly more data than the norm (e.g., a viral post with millions of likes). + +**Solution**: Flag outliers and overflow into separate documents. + +```javascript +// books collection - normal case +{ + _id: ObjectId("..."), + title: "Normal Book", + customers_purchased: ["user1", "user2", "user3"], + has_overflow: false +} + +// books collection - outlier (bestseller) +{ + _id: ObjectId("..."), + title: "Bestseller", + customers_purchased: ["user1", "user2", /* ... first 1000 */], + has_overflow: true +} + +// book_purchases_overflow collection +{ + book_id: ObjectId("..."), + page: 2, + customers_purchased: ["user1001", "user1002", /* ... next 1000 */] +} +``` + +**When to use**: Social media (viral posts), e-commerce (bestsellers), any data with power-law distribution. + +### 5. Extended Reference Pattern + +**Problem**: Frequent joins (lookups) to get a few fields from a referenced document. + +**Solution**: Copy the most-accessed fields into the referencing document. + +```javascript +// orders collection +{ + _id: ObjectId("..."), + date: ISODate("..."), + customer_id: ObjectId("..."), + // Extended reference - copied fields for fast reads + customer_name: "Alice Smith", + customer_email: "alice@example.com", + items: [ + { + product_id: ObjectId("..."), + product_name: "Widget", // copied + price: 29.99, // copied (snapshot at time of order) + quantity: 2 + } + ] +} +``` + +**Trade-off**: Stale data is acceptable (order snapshots price at purchase time). For data that must be current, keep only the reference. + +**When to use**: Orders (snapshot pricing), notifications (snapshot user name), audit logs. + +### 6. Polymorphic Pattern + +**Problem**: Objects share some fields but differ in others (e.g., different product types). + +**Solution**: Store in a single collection with a type discriminator. + +```javascript +// vehicles collection +{ type: "car", make: "Toyota", doors: 4, trunk_size_liters: 450 } +{ type: "truck", make: "Ford", doors: 2, payload_kg: 5000 } +{ type: "motorcycle", make: "Harley", engine_cc: 1200 } +``` + +**Index strategy**: Index common fields. Use partial indexes for type-specific fields. + +```javascript +db.vehicles.createIndex( + { payload_kg: 1 }, + { partialFilterExpression: { type: "truck" } } +); +``` + +**When to use**: Product catalogs, content management (articles, videos, images), mixed event streams. + +## Anti-Patterns + +| Mistake | Problem | Fix | +|---------|---------|-----| +| Unbounded array growth | Document exceeds 16 MB | Use bucket or outlier pattern | +| Deep nesting (> 3 levels) | Hard to query and index | Flatten or reference | +| Normalizing everything | Too many lookups, slow reads | Embed when read together | +| Embedding large blobs | Wastes RAM in working set | Store in GridFS or S3 | +| No schema validation | Inconsistent data over time | Use JSON Schema validation | +| Indexing every field | Slow writes, wasted space | Index based on query patterns | + +## Schema Validation + +Use `db.createCollection()` with `$jsonSchema` validator to enforce structure. Set `validationLevel: "moderate"` to apply only on inserts and updates (not existing docs). diff --git a/.claude/skills/databases/postgresql/SKILL.md b/.claude/skills/databases/postgresql/SKILL.md index 756aeb2..ae8b6c6 100644 --- a/.claude/skills/databases/postgresql/SKILL.md +++ b/.claude/skills/databases/postgresql/SKILL.md @@ -1,69 +1,612 @@ +--- +name: postgresql +description: > + Use this skill whenever working with PostgreSQL databases, writing SQL queries, designing schemas, or optimizing database performance. Trigger on keywords like PostgreSQL, Postgres, SQL query, schema design, indexing, migrations, EXPLAIN ANALYZE, connection pooling, or any relational database operation. Also applies when debugging slow queries, setting up database tables, or working with ORMs that target PostgreSQL. +--- + # PostgreSQL -## Description - -PostgreSQL database patterns including queries, indexing, and optimization. - ## When to Use - PostgreSQL database operations - SQL query optimization -- Schema design +- Schema design and migrations +- JSONB document storage within a relational model +- Full-text search without a dedicated search engine +- Complex analytical queries with window functions and CTEs + +## When NOT to Use + +- NoSQL-only projects where no relational database is involved +- In-memory databases like Redis or SQLite used purely for caching or ephemeral storage +- File-based storage scenarios that do not require a database engine --- ## Core Patterns -### Basic Queries +### 1. Schema Design + +Design tables with explicit constraints, proper types, and clear relationships. ```sql --- Select with filtering -SELECT id, name, email -FROM users -WHERE active = true -ORDER BY created_at DESC -LIMIT 20 OFFSET 0; +-- Enums for constrained value sets +CREATE TYPE user_role AS ENUM ('admin', 'editor', 'viewer'); +CREATE TYPE order_status AS ENUM ('pending', 'processing', 'shipped', 'delivered', 'cancelled'); --- Join -SELECT u.*, COUNT(p.id) as post_count +-- Composite types for reusable structures +CREATE TYPE address AS ( + street TEXT, + city TEXT, + state TEXT, + zip VARCHAR(10) +); + +-- Users table with constraints +CREATE TABLE users ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL CHECK (char_length(name) >= 1), + role user_role NOT NULL DEFAULT 'viewer', + metadata JSONB DEFAULT '{}', + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- Organizations with self-referencing hierarchy +CREATE TABLE organizations ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + name TEXT NOT NULL, + parent_id BIGINT REFERENCES organizations(id) ON DELETE SET NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- Membership join table with composite primary key +CREATE TABLE org_memberships ( + user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + org_id BIGINT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE, + role user_role NOT NULL DEFAULT 'viewer', + joined_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (user_id, org_id) +); + +-- Orders with foreign keys, check constraints, and enum status +CREATE TABLE orders ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE RESTRICT, + status order_status NOT NULL DEFAULT 'pending', + total_cents BIGINT NOT NULL CHECK (total_cents >= 0), + shipping address, + items JSONB NOT NULL DEFAULT '[]', + placed_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- Auto-update updated_at with a trigger +CREATE OR REPLACE FUNCTION set_updated_at() +RETURNS TRIGGER AS $$ +BEGIN + NEW.updated_at = now(); + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +CREATE TRIGGER trg_users_updated_at + BEFORE UPDATE ON users + FOR EACH ROW EXECUTE FUNCTION set_updated_at(); +``` + +**Key principles:** +- Use `BIGINT GENERATED ALWAYS AS IDENTITY` over `SERIAL` for new projects +- Use `TIMESTAMPTZ` (not `TIMESTAMP`) to store times with timezone awareness +- Prefer `TEXT` over `VARCHAR(n)` unless a hard length limit is business-critical +- Add `ON DELETE` actions on every foreign key (CASCADE, RESTRICT, or SET NULL) +- Use `CHECK` constraints for business rules that live at the data level + +--- + +### 2. Index Strategy + +Choose the right index type based on your query patterns. + +**Decision guide:** + +| Query Pattern | Index Type | Example | +|---------------|-----------|---------| +| Equality (`=`) and range (`<`, `>`, `BETWEEN`) | B-tree (default) | `WHERE created_at > '2025-01-01'` | +| Array containment (`@>`), JSONB queries | GIN | `WHERE tags @> '{postgres}'` | +| Full-text search (`@@`) | GIN | `WHERE to_tsvector(body) @@ query` | +| Geometry, range overlap | GiST | `WHERE location <-> point '(40.7,-74.0)' < 0.01` | +| Filtered subset of rows | Partial | `WHERE active = true` | +| Index-only scans (no heap lookup) | Covering (INCLUDE) | Frequently selected columns | + +```sql +-- B-tree: default, good for equality and range +CREATE INDEX idx_orders_placed_at ON orders(placed_at DESC); +CREATE INDEX idx_orders_user_status ON orders(user_id, status); + +-- GIN: arrays and JSONB containment +CREATE INDEX idx_users_metadata ON users USING GIN (metadata); +CREATE INDEX idx_orders_items ON orders USING GIN (items jsonb_path_ops); + +-- GIN: full-text search +ALTER TABLE articles ADD COLUMN search_vector tsvector + GENERATED ALWAYS AS ( + setweight(to_tsvector('english', coalesce(title, '')), 'A') || + setweight(to_tsvector('english', coalesce(body, '')), 'B') + ) STORED; + +CREATE INDEX idx_articles_search ON articles USING GIN (search_vector); + +-- Full-text search query +SELECT id, title, ts_rank(search_vector, query) AS rank +FROM articles, plainto_tsquery('english', 'database optimization') AS query +WHERE search_vector @@ query +ORDER BY rank DESC +LIMIT 20; + +-- GiST: geometry and range types +CREATE INDEX idx_events_duration ON events USING GiST ( + tstzrange(starts_at, ends_at) +); + +-- Find overlapping events +SELECT * FROM events +WHERE tstzrange(starts_at, ends_at) && tstzrange('2025-06-01', '2025-06-02'); + +-- Partial index: only index rows you actually query +CREATE INDEX idx_orders_pending ON orders(placed_at) + WHERE status = 'pending'; + +-- Covering index: avoids heap lookup for common queries +CREATE INDEX idx_users_email_covering ON users(email) + INCLUDE (name, role); + +-- This query can now be answered entirely from the index +SELECT name, role FROM users WHERE email = 'user@example.com'; +``` + +**When to add an index:** Run `EXPLAIN ANALYZE` first. Add an index when you see sequential scans on large tables with selective WHERE clauses. Do not index columns with very low cardinality (e.g., a boolean on a small table) unless combined with other columns. + +--- + +### 3. Query Optimization + +#### Reading EXPLAIN ANALYZE + +```sql +EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) +SELECT u.name, COUNT(o.id) AS order_count FROM users u -LEFT JOIN posts p ON p.user_id = u.id -GROUP BY u.id; +JOIN orders o ON o.user_id = u.id +WHERE o.placed_at > now() - INTERVAL '30 days' +GROUP BY u.id, u.name +ORDER BY order_count DESC +LIMIT 10; ``` -### Indexes +**What to look for in the output:** +- **Seq Scan on large tables** -- add an index or rewrite the WHERE clause +- **Nested Loop with high row counts** -- consider a Hash Join (may need more `work_mem`) +- **actual rows far exceeding estimated rows** -- run `ANALYZE tablename` to update statistics +- **Buffers: shared read** large numbers -- data not cached, check `shared_buffers` sizing +- **Sort Method: external merge** -- increase `work_mem` for this query + +#### Common Query Rewrites ```sql --- Single column index -CREATE INDEX idx_users_email ON users(email); +-- BAD: correlated subquery runs once per row +SELECT u.name, + (SELECT COUNT(*) FROM orders o WHERE o.user_id = u.id) AS order_count +FROM users u; --- Composite index -CREATE INDEX idx_posts_user_date ON posts(user_id, created_at DESC); +-- GOOD: single pass with JOIN + GROUP BY +SELECT u.name, COUNT(o.id) AS order_count +FROM users u +LEFT JOIN orders o ON o.user_id = u.id +GROUP BY u.id, u.name; --- Partial index -CREATE INDEX idx_active_users ON users(email) WHERE active = true; +-- BAD: OR on different columns defeats index usage +SELECT * FROM orders WHERE user_id = 5 OR status = 'pending'; + +-- GOOD: UNION ALL lets each branch use its own index +SELECT * FROM orders WHERE user_id = 5 +UNION ALL +SELECT * FROM orders WHERE status = 'pending' AND user_id != 5; + +-- BAD: function call on indexed column prevents index use +SELECT * FROM users WHERE LOWER(email) = 'user@example.com'; + +-- GOOD: expression index or use citext +CREATE INDEX idx_users_email_lower ON users(LOWER(email)); +-- or better: define email as CITEXT type + +-- Avoiding N+1: fetch users and their latest order in one query +SELECT DISTINCT ON (u.id) + u.id, u.name, o.id AS latest_order_id, o.total_cents, o.placed_at +FROM users u +LEFT JOIN orders o ON o.user_id = u.id +ORDER BY u.id, o.placed_at DESC; ``` -### Migrations +--- + +### 4. Migrations + +Follow the up/down pattern and plan for zero-downtime deployments. ```sql --- Add column with default -ALTER TABLE users ADD COLUMN role VARCHAR(50) DEFAULT 'user'; +-- ============================================ +-- Migration: 20250601_001_add_user_preferences +-- ============================================ --- Add constraint -ALTER TABLE users ADD CONSTRAINT unique_email UNIQUE (email); +-- UP +ALTER TABLE users ADD COLUMN preferences JSONB DEFAULT '{}'; + +-- Create index CONCURRENTLY to avoid locking the table +CREATE INDEX CONCURRENTLY idx_users_preferences + ON users USING GIN (preferences); + +-- DOWN +DROP INDEX IF EXISTS idx_users_preferences; +ALTER TABLE users DROP COLUMN IF EXISTS preferences; ``` +**Safe vs unsafe operations:** + +| Operation | Safe? | Notes | +|-----------|-------|-------| +| ADD COLUMN (nullable or with volatile default) | Yes | Instant in PG 11+ with non-volatile default too | +| ADD COLUMN NOT NULL without default | No | Fails if rows exist; add nullable first, backfill, then set NOT NULL | +| DROP COLUMN | Mostly | Quick, but ORM queries may break if they SELECT * | +| RENAME COLUMN | Dangerous | Breaks all queries referencing old name; use a transition period | +| ADD INDEX | Safe with CONCURRENTLY | Without CONCURRENTLY, locks writes for duration | +| ADD CONSTRAINT (CHECK/FK) | Careful | Use NOT VALID then VALIDATE CONSTRAINT in two steps | +| Change column type | Dangerous | Rewrites entire table; use a new column + migration instead | + +```sql +-- Zero-downtime: add NOT NULL constraint safely +-- Step 1: add column as nullable +ALTER TABLE users ADD COLUMN phone TEXT; + +-- Step 2: backfill in batches +UPDATE users SET phone = '' WHERE phone IS NULL AND id BETWEEN 1 AND 10000; +UPDATE users SET phone = '' WHERE phone IS NULL AND id BETWEEN 10001 AND 20000; +-- ... continue in batches + +-- Step 3: add constraint without full table lock +ALTER TABLE users ADD CONSTRAINT users_phone_not_null + CHECK (phone IS NOT NULL) NOT VALID; + +-- Step 4: validate (scans table but allows concurrent writes) +ALTER TABLE users VALIDATE CONSTRAINT users_phone_not_null; + +-- Step 5: optionally convert to proper NOT NULL +ALTER TABLE users ALTER COLUMN phone SET NOT NULL; +ALTER TABLE users DROP CONSTRAINT users_phone_not_null; +``` + +--- + +### 5. JSON/JSONB + +Use JSONB for semi-structured data that lives alongside relational columns. + +**When to use JSONB:** +- User preferences, settings, or metadata with varying keys +- API response caching or event payloads +- Flexible attributes that differ per row + +**When NOT to use JSONB:** +- Data you regularly JOIN on or use in WHERE clauses across tables -- normalize it +- Data that has a fixed, well-known schema -- use proper columns + +```sql +-- Querying JSONB: operators +-- -> returns JSONB element (keeps type) +-- ->> returns TEXT value +-- @> containment (left contains right) +-- ? key exists + +-- Get a nested value +SELECT + metadata->>'department' AS department, + metadata->'settings'->>'theme' AS theme +FROM users +WHERE metadata @> '{"role": "admin"}'; + +-- Check if a key exists +SELECT * FROM users WHERE metadata ? 'avatar_url'; + +-- Query inside JSONB arrays +SELECT * FROM orders +WHERE items @> '[{"sku": "WIDGET-001"}]'; + +-- Update a nested JSONB field +UPDATE users +SET metadata = jsonb_set(metadata, '{settings,notifications}', '"email"') +WHERE id = 42; + +-- Remove a key +UPDATE users +SET metadata = metadata - 'deprecated_field' +WHERE metadata ? 'deprecated_field'; + +-- Aggregate JSONB: expand array elements into rows +SELECT o.id, item->>'sku' AS sku, (item->>'qty')::int AS qty +FROM orders o, jsonb_array_elements(o.items) AS item +WHERE o.status = 'pending'; + +-- Index strategies for JSONB +-- General containment queries: GIN with jsonb_ops (default) +CREATE INDEX idx_users_metadata_gin ON users USING GIN (metadata); + +-- Containment-only queries (smaller, faster index): jsonb_path_ops +CREATE INDEX idx_orders_items_path ON orders USING GIN (items jsonb_path_ops); + +-- Specific key lookups: expression index on extracted value +CREATE INDEX idx_users_department ON users ((metadata->>'department')); +``` + +--- + +### 6. CTEs and Window Functions + +#### Common Table Expressions (CTEs) + +```sql +-- Readable multi-step query with CTEs +WITH monthly_revenue AS ( + SELECT + date_trunc('month', placed_at) AS month, + SUM(total_cents) AS revenue_cents + FROM orders + WHERE status = 'delivered' + GROUP BY 1 +), +revenue_with_growth AS ( + SELECT + month, + revenue_cents, + LAG(revenue_cents) OVER (ORDER BY month) AS prev_month, + ROUND( + 100.0 * (revenue_cents - LAG(revenue_cents) OVER (ORDER BY month)) + / NULLIF(LAG(revenue_cents) OVER (ORDER BY month), 0), + 1 + ) AS growth_pct + FROM monthly_revenue +) +SELECT * FROM revenue_with_growth ORDER BY month DESC; + +-- Recursive CTE: org hierarchy tree +WITH RECURSIVE org_tree AS ( + -- Base case: top-level orgs + SELECT id, name, parent_id, 0 AS depth, name::TEXT AS path + FROM organizations + WHERE parent_id IS NULL + + UNION ALL + + -- Recursive step + SELECT o.id, o.name, o.parent_id, t.depth + 1, t.path || ' > ' || o.name + FROM organizations o + JOIN org_tree t ON o.parent_id = t.id +) +SELECT * FROM org_tree ORDER BY path; +``` + +#### Window Functions + +```sql +-- ROW_NUMBER: assign rank within a partition +SELECT + user_id, + id AS order_id, + total_cents, + ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY placed_at DESC) AS rn +FROM orders; + +-- Get each user's most recent order +SELECT * FROM ( + SELECT + o.*, + ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY placed_at DESC) AS rn + FROM orders o +) sub WHERE rn = 1; + +-- LAG/LEAD: compare with previous/next row +SELECT + placed_at::date AS order_date, + total_cents, + LAG(total_cents) OVER (ORDER BY placed_at) AS prev_order_total, + total_cents - LAG(total_cents) OVER (ORDER BY placed_at) AS diff +FROM orders +WHERE user_id = 42; + +-- Running total +SELECT + placed_at::date AS order_date, + total_cents, + SUM(total_cents) OVER ( + ORDER BY placed_at + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW + ) AS running_total +FROM orders +WHERE user_id = 42; + +-- NTILE: divide rows into equal buckets (e.g., quartiles) +SELECT + user_id, + SUM(total_cents) AS lifetime_spend, + NTILE(4) OVER (ORDER BY SUM(total_cents) DESC) AS spend_quartile +FROM orders +GROUP BY user_id; +``` + +--- + +### 7. Transaction Isolation + +PostgreSQL supports four isolation levels. The two most commonly used: + +| Level | Dirty Read | Non-Repeatable Read | Phantom Read | Use Case | +|-------|-----------|-------------------|-------------|----------| +| READ COMMITTED (default) | No | Possible | Possible | Most OLTP workloads | +| REPEATABLE READ | No | No | No (in PG) | Reports, consistent snapshots | +| SERIALIZABLE | No | No | No | Financial transactions, inventory | + +```sql +-- Default: READ COMMITTED +-- Each statement sees the latest committed data +BEGIN; + UPDATE accounts SET balance = balance - 100 WHERE id = 1; + UPDATE accounts SET balance = balance + 100 WHERE id = 2; +COMMIT; + +-- SERIALIZABLE: full isolation, detects write conflicts +BEGIN ISOLATION LEVEL SERIALIZABLE; + -- Read current inventory + SELECT quantity FROM inventory WHERE sku = 'WIDGET-001'; + -- Decrement if sufficient (PG will abort if concurrent tx conflicts) + UPDATE inventory SET quantity = quantity - 1 WHERE sku = 'WIDGET-001'; +COMMIT; +-- If another SERIALIZABLE tx modified the same row, one will get: +-- ERROR: could not serialize access due to concurrent update +-- Your application must retry on serialization failure (SQLSTATE 40001) + +-- Advisory locks for application-level coordination +SELECT pg_advisory_xact_lock(hashtext('process-user-' || '42')); +-- Lock is held until transaction ends; no table-level contention +``` + +**Guidelines:** +- Use READ COMMITTED for general CRUD operations +- Use SERIALIZABLE when correctness requires that concurrent transactions behave as if run sequentially (e.g., balance transfers, seat reservations) +- Always implement retry logic for serialization failures +- Keep transactions as short as possible to reduce contention + +--- + +### 8. Connection Pooling + +Direct PostgreSQL connections are expensive (~1-10 MB RAM each). Use a pooler. + +**PgBouncer configuration (pgbouncer.ini):** + +```ini +[databases] +myapp = host=127.0.0.1 port=5432 dbname=myapp + +[pgbouncer] +listen_addr = 127.0.0.1 +listen_port = 6432 +auth_type = scram-sha-256 +auth_file = /etc/pgbouncer/userlist.txt + +; Pool mode: transaction is best for most web apps +pool_mode = transaction + +; Sizing: start conservative, tune with monitoring +default_pool_size = 20 +max_client_conn = 200 +min_pool_size = 5 +reserve_pool_size = 5 +reserve_pool_timeout = 3 + +; Timeouts +server_idle_timeout = 300 +client_idle_timeout = 60 +query_timeout = 30 +``` + +**Pool sizing formula:** + +``` +optimal_pool_size = ((2 * cpu_cores) + effective_disk_spindles) +``` + +For a 4-core SSD server: `(2 * 4) + 1 = 9` connections is a good starting point. More connections does not mean more throughput -- too many causes contention. + +**Pool modes:** + +| Mode | Description | Caveats | +|------|-------------|---------| +| `transaction` | Connection returned after each transaction | Cannot use session-level features (LISTEN/NOTIFY, prepared statements, temp tables) | +| `session` | Connection held for entire client session | Fewer pooling benefits; use only when session features needed | +| `statement` | Connection returned after each statement | No multi-statement transactions; rarely used | + +**Application-level pooling (Python example with asyncpg):** + +```python +import asyncpg + +pool = await asyncpg.create_pool( + dsn="postgresql://user:pass@localhost:6432/myapp", + min_size=5, + max_size=20, + max_inactive_connection_lifetime=300, + command_timeout=30, +) + +async with pool.acquire() as conn: + rows = await conn.fetch("SELECT * FROM users WHERE active = true") +``` + +--- + ## Best Practices -1. Use indexes for filtered/sorted columns -2. Use EXPLAIN ANALYZE for slow queries -3. Avoid SELECT * in production -4. Use transactions for multiple operations -5. Use connection pooling +1. **Use parameterized queries everywhere.** Never concatenate user input into SQL strings. ORMs and query builders handle this, but verify in raw SQL contexts. + +2. **Run ANALYZE after bulk data changes.** The query planner relies on statistics. After large imports or deletes, run `ANALYZE tablename` to update them. + +3. **Prefer BIGINT for primary keys.** INTEGER (max ~2.1 billion) can be exhausted sooner than expected in high-write systems. BIGINT costs 4 extra bytes per row but avoids a painful migration later. + +4. **Store money as integers (cents).** Floating-point arithmetic causes rounding errors. Use `BIGINT` for cents or `NUMERIC(19,4)` if sub-cent precision is needed. + +5. **Add indexes for foreign keys.** PostgreSQL does not automatically index the child side of a foreign key. Without it, DELETE on the parent table triggers a sequential scan on the child. + +6. **Use TIMESTAMPTZ, not TIMESTAMP.** `TIMESTAMP WITHOUT TIME ZONE` silently drops timezone info. Always use `TIMESTAMPTZ` and let the application control display timezone. + +7. **Set statement_timeout for web requests.** Prevent runaway queries from holding connections: `SET statement_timeout = '5s';` at session start, or configure per-role in PostgreSQL. + +8. **Monitor with pg_stat_statements.** Enable this extension to track query performance over time. The top queries by `total_exec_time` are your optimization targets. + +```sql +-- Find slowest queries +SELECT + calls, + round(total_exec_time::numeric, 1) AS total_ms, + round(mean_exec_time::numeric, 1) AS mean_ms, + query +FROM pg_stat_statements +ORDER BY total_exec_time DESC +LIMIT 10; +``` ## Common Pitfalls -- **N+1 queries**: Use JOINs or batch loading -- **Missing indexes**: Add indexes for WHERE/ORDER BY -- **Large transactions**: Keep transactions short +1. **N+1 queries from ORM lazy loading.** Loading a list of users and then accessing `user.orders` in a loop generates one query per user. Use eager loading (`joinedload` in SQLAlchemy, `select_related` in Django) or batch the query with a JOIN. + +2. **Locking the table during migrations.** `ALTER TABLE ... ADD COLUMN NOT NULL DEFAULT 'x'` is safe in PG 11+, but `CREATE INDEX` without `CONCURRENTLY` locks writes. Always use `CREATE INDEX CONCURRENTLY` in production migrations. + +3. **Bloated tables from UPDATE-heavy workloads.** PostgreSQL MVCC creates dead tuples on every UPDATE. If autovacuum cannot keep up, table size and query times grow. Monitor `pg_stat_user_tables.n_dead_tup` and tune autovacuum settings for hot tables. + +4. **Using OFFSET for pagination on large datasets.** `OFFSET 100000` forces PG to scan and discard 100,000 rows. Use keyset pagination instead: + +```sql +-- BAD: slow for deep pages +SELECT * FROM orders ORDER BY id LIMIT 20 OFFSET 100000; + +-- GOOD: keyset pagination +SELECT * FROM orders WHERE id > 100000 ORDER BY id LIMIT 20; +``` + +5. **Ignoring connection limits.** Each PostgreSQL connection consumes RAM. Opening hundreds of direct connections (e.g., one per serverless function invocation) will exhaust `max_connections` and crash the server. Always use PgBouncer or an application-level pool. + +6. **Storing large blobs in the database.** Files over a few KB should go in object storage (S3, R2). Store the URL/key in PostgreSQL. Large `bytea` or `TEXT` columns bloat the table, slow backups, and waste shared_buffers cache. + +## Related Skills + +- `databases/mongodb` - Document-based database patterns for non-relational data +- `patterns/caching` - Caching strategies to reduce database load +- `patterns/logging` - Logging patterns for query debugging and monitoring diff --git a/.claude/skills/databases/postgresql/references/index-decision-tree.md b/.claude/skills/databases/postgresql/references/index-decision-tree.md new file mode 100644 index 0000000..4f92dc4 --- /dev/null +++ b/.claude/skills/databases/postgresql/references/index-decision-tree.md @@ -0,0 +1,173 @@ +# PostgreSQL Index Decision Tree + +Quick reference for choosing the right index type. + +## Decision Tree + +``` +What are you querying? +| ++-- Equality (=) or Range (<, >, BETWEEN, ORDER BY)? +| | +| +-- On a single scalar column? +| | --> B-tree (default) +| | +| +-- On a timestamp/date column with append-only inserts? +| | --> BRIN (much smaller than B-tree) +| | +| +-- Need the index to also return columns without table lookup? +| --> Covering Index (B-tree with INCLUDE) +| ++-- Array containment (@>, &&) or JSONB queries? +| --> GIN +| ++-- Full-text search (tsvector, @@)? +| --> GIN +| ++-- Geometric/spatial data (points, polygons, PostGIS)? +| --> GiST +| ++-- Range types (int4range, tsrange, overlaps)? +| --> GiST +| ++-- Nearest-neighbor / distance queries (KNN)? +| --> GiST (or SP-GiST for partitioned space) +| ++-- Only a subset of rows match your WHERE clause? +| --> Partial Index (any type + WHERE filter) +| ++-- Trigram similarity (LIKE '%pattern%', pg_trgm)? +| --> GIN with pg_trgm (or GiST for smaller, slower) +| ++-- Hash equality only (= but never range)? + --> Hash index (rarely better than B-tree in practice) +``` + +## Index Type Comparison + +| Type | Best For | Operators | Size | Write Cost | Notes | +|------|----------|-----------|------|------------|-------| +| **B-tree** | Equality, range, sorting | `= < > <= >= BETWEEN IN IS NULL` | Medium | Low | Default. Covers 90% of cases. | +| **GIN** | Multi-valued data | `@> && @@ ? ?& ?|` | Large | High (slow updates) | Best for arrays, JSONB, full-text. Use `fastupdate=on`. | +| **GiST** | Spatial, ranges, nearest-neighbor | `<< >> && @> <@ <->` | Medium | Medium | Lossy for some types. Supports KNN. | +| **SP-GiST** | Partitioned search spaces | Same as GiST | Medium | Medium | Good for phone numbers, IP addresses, non-balanced trees. | +| **BRIN** | Large sequential/append-only tables | `= < > <= >=` | Tiny | Very Low | 1000x smaller than B-tree. Only effective when physical order correlates with column values. | +| **Hash** | Equality only | `=` | Medium | Low | WAL-logged since PG10. Rarely outperforms B-tree. | + +## Common Patterns + +### Covering Index (Index-Only Scans) + +Avoid heap lookups by including extra columns: + +```sql +-- Query: SELECT email, name FROM users WHERE email = ? +CREATE INDEX idx_users_email_covering + ON users (email) INCLUDE (name); +``` + +### Partial Index (Filtered) + +Index only the rows you actually query: + +```sql +-- Only index active orders (skip 95% of rows) +CREATE INDEX idx_orders_active + ON orders (created_at) + WHERE status = 'active'; +``` + +### Composite Index (Multi-Column) + +Column order matters -- put equality columns first, range columns last: + +```sql +-- Query: WHERE tenant_id = ? AND created_at > ? +CREATE INDEX idx_events_tenant_date + ON events (tenant_id, created_at); +``` + +### Expression Index + +Index a computed value: + +```sql +CREATE INDEX idx_users_lower_email + ON users (lower(email)); +``` + +### GIN for JSONB + +```sql +-- Index all keys and values in a JSONB column +CREATE INDEX idx_metadata_gin + ON products USING gin (metadata jsonb_path_ops); + +-- Supports: metadata @> '{"color": "red"}' +``` + +### GiST for Range Overlap + +```sql +CREATE INDEX idx_reservations_during + ON reservations USING gist (during); + +-- Supports: WHERE during && '[2025-01-01, 2025-01-31]'::daterange +``` + +### BRIN for Time-Series + +```sql +-- Table has millions of rows inserted in timestamp order +CREATE INDEX idx_logs_ts_brin + ON logs USING brin (created_at) + WITH (pages_per_range = 32); +``` + +## Sizing Rules of Thumb + +| Table Rows | B-tree Size | BRIN Size | GIN Size | +|------------|-------------|-----------|----------| +| 1M | ~20 MB | ~50 KB | ~30 MB | +| 10M | ~200 MB | ~500 KB | ~300 MB | +| 100M | ~2 GB | ~5 MB | ~3 GB | + +## Diagnostic Queries + +```sql +-- Check if an index is being used +EXPLAIN (ANALYZE, BUFFERS) SELECT ...; + +-- Find unused indexes +SELECT indexrelname, idx_scan +FROM pg_stat_user_indexes +WHERE idx_scan = 0 +ORDER BY pg_relation_size(indexrelid) DESC; + +-- Check index size +SELECT pg_size_pretty(pg_relation_size('idx_name')); + +-- Index bloat estimate +SELECT * FROM pgstatindex('idx_name'); +``` + +## Anti-Patterns + +| Mistake | Why It Hurts | +|---------|-------------| +| Indexing every column | Slows writes, wastes disk, confuses planner | +| Wrong column order in composite | Index cannot be used for the query | +| GIN on tiny tables | Overhead exceeds benefit | +| B-tree on low-cardinality columns | Planner prefers seq scan anyway | +| Missing `CONCURRENTLY` on production | Locks the table during index build | +| Forgetting `ANALYZE` after bulk load | Planner uses stale statistics | + +## Safe Index Creation + +```sql +-- Non-blocking index creation (no table lock) +CREATE INDEX CONCURRENTLY idx_name ON table (column); + +-- Always run ANALYZE after bulk operations +ANALYZE table; +``` diff --git a/.claude/skills/databases/postgresql/templates/migration-template.sql b/.claude/skills/databases/postgresql/templates/migration-template.sql new file mode 100644 index 0000000..8e9a875 --- /dev/null +++ b/.claude/skills/databases/postgresql/templates/migration-template.sql @@ -0,0 +1,143 @@ +-- ============================================================================= +-- Migration: [DESCRIPTION] +-- Created: [DATE] +-- Author: [AUTHOR] +-- Ticket: [TICKET-ID] +-- ============================================================================= +-- +-- SAFETY CHECKLIST (review before running): +-- [ ] Tested on staging with production-size data +-- [ ] Backward compatible with current application code +-- [ ] No exclusive locks on large tables during peak hours +-- [ ] Rollback (DOWN) section tested independently +-- [ ] Estimated run time: ___ +-- [ ] Estimated lock duration: ___ +-- + +-- ============================================================ +-- UP MIGRATION +-- ============================================================ + +BEGIN; + +-- Set a statement timeout to prevent long-running locks. +-- Adjust as needed; remove for data-only migrations. +SET LOCAL lock_timeout = '5s'; +SET LOCAL statement_timeout = '30s'; + +-- ------------------------------------ +-- 1. Schema changes +-- ------------------------------------ + +-- Add new table +-- CREATE TABLE IF NOT EXISTS example ( +-- id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, +-- name text NOT NULL, +-- created_at timestamptz NOT NULL DEFAULT now(), +-- updated_at timestamptz NOT NULL DEFAULT now() +-- ); + +-- Add column (safe: does not rewrite table) +-- ALTER TABLE example ADD COLUMN IF NOT EXISTS description text; + +-- Add column with default (PG 11+: does not rewrite table) +-- ALTER TABLE example ADD COLUMN IF NOT EXISTS is_active boolean NOT NULL DEFAULT true; + +-- Rename column (safe: metadata-only change) +-- ALTER TABLE example RENAME COLUMN old_name TO new_name; + +-- ------------------------------------ +-- 2. Constraints +-- ------------------------------------ + +-- Add NOT NULL (requires all existing rows to satisfy it) +-- ALTER TABLE example ALTER COLUMN name SET NOT NULL; + +-- Add check constraint (NOT VALID avoids full table scan, then VALIDATE separately) +-- ALTER TABLE example ADD CONSTRAINT chk_example_name CHECK (name <> '') NOT VALID; +-- ALTER TABLE example VALIDATE CONSTRAINT chk_example_name; + +-- Add foreign key (NOT VALID + VALIDATE pattern to avoid long locks) +-- ALTER TABLE example ADD CONSTRAINT fk_example_parent +-- FOREIGN KEY (parent_id) REFERENCES parent(id) NOT VALID; +-- ALTER TABLE example VALIDATE CONSTRAINT fk_example_parent; + +-- ------------------------------------ +-- 3. Indexes (use CONCURRENTLY outside transaction) +-- ------------------------------------ + +-- NOTE: CREATE INDEX CONCURRENTLY cannot run inside a transaction. +-- Run these statements separately after committing the transaction above. +-- +-- CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_example_name +-- ON example (name); +-- +-- CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_example_created_at +-- ON example USING brin (created_at); + +-- ------------------------------------ +-- 4. Data migration +-- ------------------------------------ + +-- Backfill in batches to avoid long transactions: +-- UPDATE example SET description = 'default' WHERE description IS NULL; +-- +-- For large tables, batch with: +-- DO $$ +-- DECLARE +-- batch_size int := 10000; +-- rows_updated int; +-- BEGIN +-- LOOP +-- UPDATE example +-- SET description = 'default' +-- WHERE id IN ( +-- SELECT id FROM example +-- WHERE description IS NULL +-- LIMIT batch_size +-- FOR UPDATE SKIP LOCKED +-- ); +-- GET DIAGNOSTICS rows_updated = ROW_COUNT; +-- EXIT WHEN rows_updated = 0; +-- RAISE NOTICE 'Updated % rows', rows_updated; +-- COMMIT; +-- END LOOP; +-- END $$; + +-- ------------------------------------ +-- 5. Permissions +-- ------------------------------------ + +-- GRANT SELECT, INSERT, UPDATE ON example TO app_role; +-- GRANT USAGE ON SEQUENCE example_id_seq TO app_role; + +COMMIT; + + +-- ============================================================ +-- DOWN MIGRATION (rollback) +-- ============================================================ +-- Run this section to undo the UP migration. +-- Test this independently before deploying the UP migration. + +-- BEGIN; +-- +-- -- Reverse data migration +-- -- UPDATE example SET description = NULL; +-- +-- -- Drop constraints +-- -- ALTER TABLE example DROP CONSTRAINT IF EXISTS chk_example_name; +-- -- ALTER TABLE example DROP CONSTRAINT IF EXISTS fk_example_parent; +-- +-- -- Drop columns +-- -- ALTER TABLE example DROP COLUMN IF EXISTS description; +-- -- ALTER TABLE example DROP COLUMN IF EXISTS is_active; +-- +-- -- Drop tables +-- -- DROP TABLE IF EXISTS example; +-- +-- COMMIT; +-- +-- -- Drop indexes (outside transaction) +-- -- DROP INDEX CONCURRENTLY IF EXISTS idx_example_name; +-- -- DROP INDEX CONCURRENTLY IF EXISTS idx_example_created_at; diff --git a/.claude/skills/devops/docker/SKILL.md b/.claude/skills/devops/docker/SKILL.md index f71d636..e56d2b4 100644 --- a/.claude/skills/devops/docker/SKILL.md +++ b/.claude/skills/devops/docker/SKILL.md @@ -1,70 +1,216 @@ +--- +name: docker +description: > + Use this skill whenever containerizing applications, writing Dockerfiles, configuring Docker Compose, or optimizing container images. Trigger on keywords like Docker, Dockerfile, container, docker-compose, multi-stage build, image, or container registry. Also applies when setting up local development environments with containers, debugging container networking, or preparing applications for container-based deployment in CI/CD pipelines. +--- + # Docker -## Description - -Docker containerization including Dockerfiles, compose, and best practices. - ## When to Use - Containerizing applications - Local development environments - CI/CD pipelines +## When NOT to Use + +- Serverless-only deployments where containers are not part of the architecture (e.g., pure AWS Lambda, Cloudflare Workers) +- Local development without containers where native tooling is preferred +- Simple scripts or utilities that do not need isolation or reproducible environments + --- ## Core Patterns -### Multi-stage Dockerfile (Node.js) +### 1. Multi-Stage Builds + +Multi-stage builds separate build-time dependencies from the runtime image, producing +smaller, more secure containers. + +#### Python (builder + slim runtime) ```dockerfile -# Build stage -FROM node:20-alpine AS builder -WORKDIR /app -COPY package*.json ./ -RUN npm ci -COPY . . -RUN npm run build +# ---- Build stage ---- +FROM python:3.12-slim AS builder -# Production stage -FROM node:20-alpine -WORKDIR /app -COPY --from=builder /app/dist ./dist -COPY --from=builder /app/node_modules ./node_modules -EXPOSE 3000 -CMD ["node", "dist/index.js"] -``` +WORKDIR /build -### Python Dockerfile +# Install build-only dependencies (gcc, etc.) needed by some wheels +RUN apt-get update && \ + apt-get install -y --no-install-recommends gcc libpq-dev && \ + rm -rf /var/lib/apt/lists/* -```dockerfile +COPY requirements.txt . +RUN pip install --no-cache-dir --prefix=/install -r requirements.txt + +# ---- Runtime stage ---- FROM python:3.12-slim WORKDIR /app -# Install dependencies first (caching) -COPY requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt +# Copy only the installed packages from the builder +COPY --from=builder /install /usr/local -COPY . . +# Copy application code +COPY src/ ./src/ +COPY main.py . + +# Run as non-root +RUN addgroup --system app && adduser --system --ingroup app app +USER app EXPOSE 8000 -CMD ["uvicorn", "main:app", "--host", "0.0.0.0"] + +HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ + CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" + +CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] ``` -### Docker Compose +#### Node.js (build + nginx/alpine) + +```dockerfile +# ---- Build stage ---- +FROM node:20-alpine AS builder + +WORKDIR /app + +# Install dependencies first for layer caching +COPY package.json pnpm-lock.yaml ./ +RUN corepack enable && pnpm install --frozen-lockfile + +# Copy source and build +COPY tsconfig.json ./ +COPY src/ ./src/ +COPY public/ ./public/ +RUN pnpm build + +# ---- Runtime stage (static site served by nginx) ---- +FROM nginx:1.27-alpine + +# Copy custom nginx config +COPY nginx.conf /etc/nginx/conf.d/default.conf + +# Copy built assets from builder +COPY --from=builder /app/dist /usr/share/nginx/html + +# Run as non-root +RUN chown -R nginx:nginx /usr/share/nginx/html && \ + chown -R nginx:nginx /var/cache/nginx && \ + chown -R nginx:nginx /var/log/nginx && \ + touch /var/run/nginx.pid && \ + chown -R nginx:nginx /var/run/nginx.pid +USER nginx + +EXPOSE 8080 + +HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ + CMD wget --no-verbose --tries=1 --spider http://localhost:8080/ || exit 1 + +CMD ["nginx", "-g", "daemon off;"] +``` + +#### Node.js (API server with alpine runtime) + +```dockerfile +# ---- Build stage ---- +FROM node:20-alpine AS builder + +WORKDIR /app + +COPY package.json pnpm-lock.yaml ./ +RUN corepack enable && pnpm install --frozen-lockfile + +COPY tsconfig.json ./ +COPY src/ ./src/ +RUN pnpm build + +# Prune dev dependencies for a lighter production node_modules +RUN pnpm prune --prod + +# ---- Runtime stage ---- +FROM node:20-alpine + +WORKDIR /app + +COPY --from=builder /app/dist ./dist +COPY --from=builder /app/node_modules ./node_modules +COPY --from=builder /app/package.json ./ + +RUN addgroup -S app && adduser -S app -G app +USER app + +EXPOSE 3000 + +HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ + CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1 + +CMD ["node", "dist/index.js"] +``` + +#### Go (build + scratch) + +```dockerfile +# ---- Build stage ---- +FROM golang:1.22-alpine AS builder + +WORKDIR /build + +# Download dependencies first for caching +COPY go.mod go.sum ./ +RUN go mod download + +# Copy source and build a static binary +COPY . . +RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /app/server ./cmd/server + +# ---- Runtime stage (scratch = empty image) ---- +FROM scratch + +# Copy CA certificates for HTTPS calls +COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ + +# Copy the static binary +COPY --from=builder /app/server /server + +EXPOSE 8080 + +ENTRYPOINT ["/server"] +``` + +--- + +### 2. Docker Compose for Development + +A full-featured Compose file with services, volumes, networks, healthchecks, and +environment variable management. ```yaml -version: '3.8' - services: app: - build: . + build: + context: . + dockerfile: Dockerfile + target: builder # Use builder stage for dev with hot-reload ports: - "3000:3000" environment: - - DATABASE_URL=postgresql://user:pass@db:5432/app + NODE_ENV: development + DATABASE_URL: postgresql://user:pass@db:5432/app + REDIS_URL: redis://redis:6379 + env_file: + - .env.local # Local overrides (gitignored) + volumes: + - .:/app # Bind-mount source for hot-reload + - /app/node_modules # Anonymous volume to preserve node_modules depends_on: - - db + db: + condition: service_healthy + redis: + condition: service_started + networks: + - backend + restart: unless-stopped db: image: postgres:16-alpine @@ -72,23 +218,442 @@ services: POSTGRES_USER: user POSTGRES_PASSWORD: pass POSTGRES_DB: app + ports: + - "5432:5432" volumes: - postgres_data:/var/lib/postgresql/data + - ./scripts/init.sql:/docker-entrypoint-initdb.d/init.sql + healthcheck: + test: ["CMD-SHELL", "pg_isready -U user -d app"] + interval: 10s + timeout: 5s + retries: 5 + start_period: 30s + networks: + - backend + + redis: + image: redis:7-alpine + ports: + - "6379:6379" + volumes: + - redis_data:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 5s + retries: 3 + networks: + - backend + + worker: + build: + context: . + dockerfile: Dockerfile.worker + environment: + DATABASE_URL: postgresql://user:pass@db:5432/app + REDIS_URL: redis://redis:6379 + depends_on: + db: + condition: service_healthy + redis: + condition: service_started + networks: + - backend + restart: unless-stopped volumes: postgres_data: + redis_data: + +networks: + backend: + driver: bridge ``` +--- + +### 3. Layer Caching + +Docker caches each layer. If a layer has not changed, every layer after it is also +cached. Order instructions from least-frequently-changed to most-frequently-changed. + +#### Optimal instruction order + +```dockerfile +FROM python:3.12-slim + +WORKDIR /app + +# 1. System dependencies (rarely change) +RUN apt-get update && apt-get install -y --no-install-recommends curl && \ + rm -rf /var/lib/apt/lists/* + +# 2. Dependency manifests (change when adding packages) +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +# 3. Application code (changes most often) +COPY . . + +CMD ["uvicorn", "main:app", "--host", "0.0.0.0"] +``` + +#### .dockerignore patterns + +Always include a `.dockerignore` to keep the build context small and avoid leaking +secrets into layers. + +``` +# Version control +.git +.gitignore + +# Dependencies (rebuilt inside container) +node_modules +__pycache__ +*.pyc +.venv +venv + +# Build output +dist +build +*.egg-info + +# IDE and editor files +.vscode +.idea +*.swp +*.swo + +# Environment and secrets +.env +.env.* +*.pem +*.key + +# Docker files (not needed in context) +Dockerfile* +docker-compose* +.dockerignore + +# Documentation and misc +README.md +CHANGELOG.md +LICENSE +docs/ +``` + +--- + +### 4. Health Checks + +Health checks let Docker (and orchestrators like Compose/Swarm/K8s) know when a +container is actually ready to serve traffic. + +#### HTTP health check with curl + +```dockerfile +HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ + CMD curl -f http://localhost:8000/health || exit 1 +``` + +#### HTTP health check with wget (alpine images without curl) + +```dockerfile +HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ + CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1 +``` + +#### TCP port check (for non-HTTP services) + +```dockerfile +HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ + CMD nc -z localhost 5432 || exit 1 +``` + +#### Python-native check (no extra binaries needed) + +```dockerfile +HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ + CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" +``` + +**Parameter reference:** + +| Parameter | Description | Default | +|------------------|--------------------------------------------------|---------| +| `--interval` | Time between checks | 30s | +| `--timeout` | Max time for a single check | 30s | +| `--start-period` | Grace period before checks count as failures | 0s | +| `--retries` | Consecutive failures before marking unhealthy | 3 | + +--- + +### 5. Security Hardening + +#### Run as non-root user + +```dockerfile +# Debian/Ubuntu based images +RUN addgroup --system app && adduser --system --ingroup app app +USER app + +# Alpine based images +RUN addgroup -S app && adduser -S app -G app +USER app +``` + +#### Use minimal base images + +| Base Image | Size | Use Case | +|--------------------|---------|---------------------------------------| +| `alpine` | ~5 MB | General minimal base | +| `*-slim` | ~50 MB | Debian-based with fewer packages | +| `distroless` | ~20 MB | Google's no-shell, no-package-manager | +| `scratch` | 0 MB | Static binaries only (Go, Rust) | + +```dockerfile +# Distroless for Python +FROM gcr.io/distroless/python3-debian12 +COPY --from=builder /app /app +CMD ["main.py"] +``` + +#### Never put secrets in image layers + +```dockerfile +# BAD - secret is baked into image history +COPY .env /app/.env +RUN echo "API_KEY=secret123" >> /app/.env + +# GOOD - pass secrets at runtime +CMD ["python", "main.py"] +# docker run -e API_KEY=secret123 myapp +# or docker run --env-file .env myapp +``` + +#### Multi-stage to exclude build tools + +Build tools (compilers, package managers, source code) stay in the builder stage +and never reach the runtime image. This reduces attack surface and image size. + +```dockerfile +FROM node:20-alpine AS builder +WORKDIR /app +COPY package.json pnpm-lock.yaml ./ +RUN corepack enable && pnpm install --frozen-lockfile +COPY . . +RUN pnpm build && pnpm prune --prod + +FROM node:20-alpine +WORKDIR /app +# Only the built output and production deps are copied +COPY --from=builder /app/dist ./dist +COPY --from=builder /app/node_modules ./node_modules +USER node +CMD ["node", "dist/index.js"] +``` + +--- + +### 6. Environment Configuration + +#### ARG vs ENV + +| Directive | Available at | Persists in image | Use for | +|-----------|-------------|-------------------|-----------------------------| +| `ARG` | Build time | No | Build-time variables | +| `ENV` | Build + run | Yes | Runtime configuration | + +```dockerfile +# ARG - only available during build +ARG NODE_ENV=production +ARG BUILD_VERSION=unknown + +# ENV - available at build and runtime +ENV NODE_ENV=${NODE_ENV} +ENV APP_VERSION=${BUILD_VERSION} + +# Build with: docker build --build-arg BUILD_VERSION=1.2.3 . +``` + +#### .env files with Compose + +```yaml +services: + app: + build: . + # Single .env file + env_file: + - .env + + # Multiple files (later files override earlier ones) + env_file: + - .env.defaults + - .env.local + + # Inline environment variables (override env_file) + environment: + LOG_LEVEL: debug + DEBUG: "true" +``` + +#### Secrets management with Docker Compose + +```yaml +services: + app: + build: . + secrets: + - db_password + - api_key + environment: + DB_PASSWORD_FILE: /run/secrets/db_password + +secrets: + db_password: + file: ./secrets/db_password.txt + api_key: + environment: API_KEY # Read from host environment +``` + +Inside the container, secrets are mounted at `/run/secrets/` as files. + +--- + +### 7. Networking + +#### Bridge networks for service isolation + +```yaml +services: + frontend: + build: ./frontend + ports: + - "3000:3000" + networks: + - frontend-net + - backend-net # Can reach the API + + api: + build: ./api + ports: + - "8000:8000" + networks: + - backend-net # Reachable by frontend and workers + + db: + image: postgres:16-alpine + networks: + - backend-net # Only reachable by api and workers + # No ports exposed to host + + worker: + build: ./worker + networks: + - backend-net + +networks: + frontend-net: + driver: bridge + backend-net: + driver: bridge +``` + +#### Service discovery + +Within a Docker Compose network, services reach each other by **service name** +as the hostname. + +```python +# In the api service, connect to db using its service name +DATABASE_URL = "postgresql://user:pass@db:5432/app" + +# In the frontend service, call the api by service name +API_URL = "http://api:8000" +``` + +#### Exposing ports + +```yaml +services: + app: + ports: + - "3000:3000" # host:container, binds to 0.0.0.0 + - "127.0.0.1:3000:3000" # bind to localhost only (more secure) + expose: + - "3000" # expose to other containers only, not host +``` + +--- + ## Best Practices -1. Use multi-stage builds -2. Order commands for cache efficiency -3. Use .dockerignore -4. Run as non-root user -5. Use specific image tags +1. **Use multi-stage builds** -- Separate build dependencies from the runtime + image. The final image should contain only what is needed to run the + application. + +2. **Pin image tags** -- Use `node:20.11-alpine` or a digest instead of + `node:latest` or `node:20`. Floating tags lead to unpredictable builds. + +3. **Order instructions for cache efficiency** -- Copy dependency manifests and + install dependencies before copying application code. This ensures that code + changes do not invalidate the dependency layer cache. + +4. **Use .dockerignore** -- Exclude `.git`, `node_modules`, `__pycache__`, `.env` + files, and anything not needed inside the container to keep the build context + small and avoid leaking secrets. + +5. **Run as non-root** -- Add a `USER` instruction to run the process as an + unprivileged user. Never run production containers as root. + +6. **Combine RUN commands** -- Merge related `RUN` instructions with `&&` to + reduce layers and always clean up apt/apk caches in the same layer that + installs packages. + +7. **Use COPY instead of ADD** -- `COPY` is explicit and predictable. `ADD` has + implicit behaviors (tar extraction, URL fetching) that can surprise you. + +8. **Set explicit HEALTHCHECK** -- Define health checks in the Dockerfile so + orchestrators know when the container is ready. This prevents routing traffic + to containers that are still starting up. + +--- ## Common Pitfalls -- **Large images**: Use slim/alpine bases -- **Cache busting**: Order COPY commands properly -- **Root user**: Add USER instruction +1. **Bloated images** -- Using full base images like `python:3.12` instead of + `python:3.12-slim` adds hundreds of megabytes. Always prefer slim or alpine + variants. Use multi-stage builds to exclude build tools. + +2. **Cache invalidation by COPY order** -- Placing `COPY . .` before + `RUN pip install` means every code change reinstalls all dependencies. Always + copy the dependency manifest first, install, then copy the rest of the code. + +3. **Running as root** -- Forgetting the `USER` instruction means the container + process runs as root. If the application is compromised, the attacker has full + control of the container filesystem. + +4. **Secrets baked into layers** -- Using `COPY .env .` or `ARG` for secrets + embeds them in the image layer history. Anyone with access to the image can + extract them with `docker history`. Pass secrets at runtime via environment + variables or Docker secrets. + +5. **Missing .dockerignore** -- Without a `.dockerignore`, the entire directory + (including `.git`, `node_modules`, `.env` files) is sent as build context. + This slows builds, increases image size, and risks leaking credentials. + +6. **Ignoring healthchecks in Compose** -- Using `depends_on` without + `condition: service_healthy` means the dependent service starts as soon as + the database container starts, not when the database is actually ready to + accept connections. Always pair `depends_on` with healthchecks. + +--- + +## Related Skills + +- `devops/github-actions` - CI/CD workflows for building and deploying Docker containers +- `security/owasp` - Security best practices for container hardening and vulnerability scanning +- `patterns/logging` — Container logging and log aggregation diff --git a/.claude/skills/devops/docker/references/dockerfile-best-practices.md b/.claude/skills/devops/docker/references/dockerfile-best-practices.md new file mode 100644 index 0000000..96419b7 --- /dev/null +++ b/.claude/skills/devops/docker/references/dockerfile-best-practices.md @@ -0,0 +1,196 @@ +# Dockerfile Best Practices Reference + +Quick reference for writing efficient, secure, and maintainable Dockerfiles. + +## Layer Ordering for Cache Optimization + +Order instructions from least-frequently-changed to most-frequently-changed: + +```dockerfile +# 1. Base image (changes: rarely) +FROM node:22-slim + +# 2. System dependencies (changes: rarely) +RUN apt-get update && apt-get install -y --no-install-recommends \ + curl \ + && rm -rf /var/lib/apt/lists/* + +# 3. App dependency manifest (changes: sometimes) +COPY package.json pnpm-lock.yaml ./ + +# 4. Install dependencies (changes: sometimes, cached if manifests unchanged) +RUN pnpm install --frozen-lockfile + +# 5. Copy source code (changes: frequently) +COPY . . + +# 6. Build step (changes: frequently) +RUN pnpm build + +# 7. Runtime config (changes: rarely, but placed last for clarity) +EXPOSE 3000 +CMD ["node", "dist/server.js"] +``` + +**Key rule**: If a layer changes, all subsequent layers are rebuilt. Separate dependency installation from source code copying. + +## Multi-Stage Builds + +Reduce final image size by separating build and runtime stages. + +``` ++-------------------+ +-------------------+ +| Build Stage | | Runtime Stage | +| | | | +| - Full toolchain | ---> | - Minimal base | +| - Dev deps | | - Only artifacts | +| - Source code | | - No build tools | +| - Build output | | - No source code | ++-------------------+ +-------------------+ + ~800 MB ~80 MB +``` + +**Benefits**: Smaller images, faster deploys, reduced attack surface, no build tools in production. + +## Base Image Selection + +| Image | Size | Use Case | Security | Package Manager | +|-------|------|----------|----------|-----------------| +| **alpine** | ~5 MB | Small images, CLI tools | Good (small surface) | apk | +| **slim** (Debian) | ~80 MB | Most apps (Python, Node) | Good | apt | +| **distroless** | ~20 MB | Production, no shell needed | Excellent (no shell) | None | +| **scratch** | 0 MB | Static Go/Rust binaries | Excellent (nothing) | None | +| **full** (Debian) | ~300 MB | Build stages, debugging | Fair (large surface) | apt | + +### Recommendations by Language + +| Language | Build Stage | Runtime Stage | +|----------|-------------|---------------| +| **Python** | `python:3.12-slim` | `python:3.12-slim` or `distroless/python3` | +| **Node.js** | `node:22-slim` | `node:22-slim` or `distroless/nodejs22` | +| **Go** | `golang:1.23` | `scratch` or `distroless/static` | +| **Rust** | `rust:1.83` | `scratch` or `distroless/cc` | +| **Java** | `eclipse-temurin:21-jdk` | `eclipse-temurin:21-jre-alpine` | + +## Instruction Best Practices + +### RUN: Combine and Clean Up + +```dockerfile +# BAD: Multiple layers, leftover cache +RUN apt-get update +RUN apt-get install -y curl +RUN apt-get install -y git + +# GOOD: Single layer, cache cleaned +RUN apt-get update && apt-get install -y --no-install-recommends \ + curl \ + git \ + && rm -rf /var/lib/apt/lists/* +``` + +### COPY: Be Specific + +```dockerfile +# BAD: Copies everything, including .git, node_modules, etc. +COPY . . + +# GOOD: Copy only what's needed (use .dockerignore too) +COPY package.json pnpm-lock.yaml ./ +RUN pnpm install --frozen-lockfile +COPY src/ ./src/ +COPY tsconfig.json ./ +``` + +### .dockerignore Essentials + +``` +.git +node_modules +__pycache__ +.env +*.log +dist +.venv +.pytest_cache +coverage +.DS_Store +``` + +### USER: Don't Run as Root + +```dockerfile +# Create non-root user +RUN groupadd -r appuser && useradd -r -g appuser -s /bin/false appuser +USER appuser +``` + +### HEALTHCHECK + +```dockerfile +HEALTHCHECK --interval=30s --timeout=3s --retries=3 \ + CMD curl -f http://localhost:3000/health || exit 1 +``` + +### ARG vs ENV + +| Directive | Available at | Persists in image | Use for | +|-----------|-------------|-------------------|---------| +| `ARG` | Build time only | No | Build-time toggles, versions | +| `ENV` | Build + runtime | Yes | App configuration | + +```dockerfile +ARG PYTHON_VERSION=3.12 +FROM python:${PYTHON_VERSION}-slim + +ENV APP_ENV=production +ENV PORT=8000 +``` + +## Security Checklist + +| Practice | Command/Example | +|----------|----------------| +| Pin base image digests | `FROM node:22-slim@sha256:abc123...` | +| Run as non-root | `USER appuser` | +| No secrets in layers | Use `--mount=type=secret` or build args | +| Scan for vulnerabilities | `docker scout cves`, `trivy image` | +| Read-only filesystem | `docker run --read-only` | +| Drop capabilities | `docker run --cap-drop ALL` | +| Use `.dockerignore` | Exclude `.env`, `.git`, credentials | +| Minimal base image | Use slim/distroless/scratch | + +### Secrets at Build Time (BuildKit) + +```dockerfile +# Mount a secret file without baking it into a layer +RUN --mount=type=secret,id=npm_token \ + NPM_TOKEN=$(cat /run/secrets/npm_token) \ + npm install + +# Build command: +# docker build --secret id=npm_token,src=.npmrc . +``` + +## Image Size Reduction Checklist + +1. Use multi-stage builds +2. Choose slim/alpine/distroless base +3. Combine RUN commands +4. Remove package manager caches (`rm -rf /var/lib/apt/lists/*`) +5. Use `.dockerignore` +6. Don't install dev dependencies in runtime stage +7. Remove unnecessary files after build +8. Use `--no-install-recommends` with apt + +## Common Pitfalls + +| Pitfall | Impact | Fix | +|---------|--------|-----| +| `COPY . .` before `npm install` | No dependency caching | Copy lockfile first, install, then copy source | +| Using `latest` tag | Non-reproducible builds | Pin specific version tags or digests | +| Secrets in `ENV` or `COPY` | Leaked in image layers | Use BuildKit secrets mount | +| Running as root | Security vulnerability | Add `USER` directive | +| No `.dockerignore` | Bloated context, slow builds | Add and maintain `.dockerignore` | +| Installing build tools in final stage | Bloated image | Use multi-stage; build in first stage | +| Not using `--frozen-lockfile` | Non-deterministic installs | Always use lockfile flags | diff --git a/.claude/skills/devops/docker/templates/Dockerfile.node b/.claude/skills/devops/docker/templates/Dockerfile.node new file mode 100644 index 0000000..b505b47 --- /dev/null +++ b/.claude/skills/devops/docker/templates/Dockerfile.node @@ -0,0 +1,93 @@ +# ============================================================================= +# Multi-Stage Node.js Dockerfile +# Usage: +# docker build -t myapp . +# docker run -p 3000:3000 myapp +# ============================================================================= + +# --------------------------------------------------------------------------- +# Stage 1: Install dependencies +# --------------------------------------------------------------------------- +FROM node:22-slim AS deps + +# Enable corepack for pnpm support. +RUN corepack enable + +WORKDIR /app + +# Copy only package manifests first for dependency layer caching. +# Dependencies are only reinstalled when these files change. +COPY package.json pnpm-lock.yaml ./ + +# Install production and dev dependencies (dev deps needed for build step). +RUN pnpm install --frozen-lockfile + +# --------------------------------------------------------------------------- +# Stage 2: Build the application +# --------------------------------------------------------------------------- +FROM node:22-slim AS builder + +RUN corepack enable + +WORKDIR /app + +# Copy dependencies from the deps stage. +COPY --from=deps /app/node_modules ./node_modules +COPY --from=deps /app/package.json /app/pnpm-lock.yaml ./ + +# Copy source code and config files needed for the build. +COPY tsconfig.json ./ +COPY src/ ./src/ +# COPY public/ ./public/ # Uncomment for Next.js or static assets + +# Build the application. +RUN pnpm build + +# Remove dev dependencies after build to reduce size. +RUN pnpm prune --prod + +# --------------------------------------------------------------------------- +# Stage 3: Production runtime +# --------------------------------------------------------------------------- +FROM node:22-slim AS runtime + +# Run as non-root for security. +RUN groupadd -r appuser && useradd -r -g appuser -d /app -s /bin/false appuser + +WORKDIR /app + +# Set production environment. +ENV NODE_ENV=production \ + PORT=3000 + +# Copy only production artifacts from the builder stage. +COPY --from=builder /app/node_modules ./node_modules +COPY --from=builder /app/dist ./dist +COPY --from=builder /app/package.json ./ + +# For Next.js standalone output, use instead: +# COPY --from=builder /app/.next/standalone ./ +# COPY --from=builder /app/.next/static ./.next/static +# COPY --from=builder /app/public ./public + +# Switch to non-root user. +USER appuser + +# Expose the application port. +EXPOSE 3000 + +# Health check -- adjust the endpoint to match your app. +HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ + CMD node -e "fetch('http://localhost:3000/health').then(r => { if (!r.ok) process.exit(1) })" || exit 1 + +# Run the application. +CMD ["node", "dist/server.js"] + +# For Next.js standalone: +# CMD ["node", "server.js"] + +# For NestJS: +# CMD ["node", "dist/main.js"] + +# For Express with ts-node (dev only, not recommended for production): +# CMD ["npx", "ts-node", "src/server.ts"] diff --git a/.claude/skills/devops/docker/templates/Dockerfile.python b/.claude/skills/devops/docker/templates/Dockerfile.python new file mode 100644 index 0000000..036cfa0 --- /dev/null +++ b/.claude/skills/devops/docker/templates/Dockerfile.python @@ -0,0 +1,78 @@ +# ============================================================================= +# Multi-Stage Python Dockerfile +# Usage: +# docker build -t myapp . +# docker run -p 8000:8000 myapp +# ============================================================================= + +# --------------------------------------------------------------------------- +# Stage 1: Build dependencies +# --------------------------------------------------------------------------- +# Use slim for building - it has gcc and headers available via apt. +FROM python:3.12-slim AS builder + +# Prevent Python from writing .pyc files and enable unbuffered output. +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 + +# Install build-time system dependencies (if any compiled packages need them). +RUN apt-get update && apt-get install -y --no-install-recommends \ + build-essential \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /app + +# Install Python dependencies into a virtual environment. +# Copying requirements first enables Docker layer caching -- +# dependencies are only reinstalled when requirements.txt changes. +COPY requirements.txt . +RUN python -m venv /app/.venv \ + && /app/.venv/bin/pip install --no-cache-dir --upgrade pip \ + && /app/.venv/bin/pip install --no-cache-dir -r requirements.txt + +# --------------------------------------------------------------------------- +# Stage 2: Runtime +# --------------------------------------------------------------------------- +FROM python:3.12-slim AS runtime + +# Prevent .pyc files and enable unbuffered output for logging. +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PATH="/app/.venv/bin:$PATH" + +# Install only runtime system dependencies (no build tools). +# Add packages here if your app needs them at runtime (e.g., libpq for psycopg). +# RUN apt-get update && apt-get install -y --no-install-recommends \ +# libpq5 \ +# && rm -rf /var/lib/apt/lists/* + +# Create a non-root user for security. +RUN groupadd -r appuser && useradd -r -g appuser -d /app -s /bin/false appuser + +WORKDIR /app + +# Copy the virtual environment from the builder stage. +COPY --from=builder /app/.venv /app/.venv + +# Copy application source code. +COPY src/ ./src/ + +# Switch to non-root user. +USER appuser + +# Expose the application port. +EXPOSE 8000 + +# Health check -- adjust the endpoint to match your app. +HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ + CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1 + +# Run the application. +# For FastAPI/Uvicorn: +CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"] + +# For Django/Gunicorn: +# CMD ["gunicorn", "src.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "4"] + +# For a plain script: +# CMD ["python", "-m", "src.main"] diff --git a/.claude/skills/devops/docker/templates/docker-compose.dev.yaml b/.claude/skills/devops/docker/templates/docker-compose.dev.yaml new file mode 100644 index 0000000..4aa7159 --- /dev/null +++ b/.claude/skills/devops/docker/templates/docker-compose.dev.yaml @@ -0,0 +1,100 @@ +# ============================================================================= +# Development Docker Compose +# Usage: +# docker compose -f docker-compose.dev.yaml up +# docker compose -f docker-compose.dev.yaml down -v # remove volumes too +# ============================================================================= + +services: + # --------------------------------------------------------------------------- + # Application + # --------------------------------------------------------------------------- + app: + build: + context: . + dockerfile: Dockerfile + target: builder # Use the build stage for dev (includes dev deps) + ports: + - "${APP_PORT:-3000}:3000" + environment: + NODE_ENV: development + DATABASE_URL: postgresql://postgres:postgres@postgres:5432/app_dev + REDIS_URL: redis://redis:6379/0 + volumes: + # Mount source code for hot-reload. Exclude node_modules. + - .:/app + - /app/node_modules + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_healthy + # Override CMD for development (hot-reload). + command: ["pnpm", "dev"] + # For Python: + # command: ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "3000", "--reload"] + + # --------------------------------------------------------------------------- + # PostgreSQL + # --------------------------------------------------------------------------- + postgres: + image: postgres:17-alpine + ports: + - "${POSTGRES_PORT:-5432}:5432" + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_DB: app_dev + volumes: + # Persist data across restarts. + - postgres_data:/var/lib/postgresql/data + # Run init scripts on first start. + # - ./scripts/init-db.sql:/docker-entrypoint-initdb.d/init.sql + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 5s + timeout: 3s + retries: 5 + + # --------------------------------------------------------------------------- + # Redis + # --------------------------------------------------------------------------- + redis: + image: redis:7-alpine + ports: + - "${REDIS_PORT:-6379}:6379" + volumes: + - redis_data:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 5s + timeout: 3s + retries: 5 + # Persist data to disk every 60 seconds if at least 1 key changed. + command: ["redis-server", "--save", "60", "1", "--loglevel", "warning"] + + # --------------------------------------------------------------------------- + # Optional: pgAdmin (database GUI) + # --------------------------------------------------------------------------- + # pgadmin: + # image: dpage/pgadmin4:latest + # ports: + # - "5050:80" + # environment: + # PGADMIN_DEFAULT_EMAIL: admin@local.dev + # PGADMIN_DEFAULT_PASSWORD: admin + # depends_on: + # - postgres + + # --------------------------------------------------------------------------- + # Optional: Mailpit (email testing) + # --------------------------------------------------------------------------- + # mailpit: + # image: axllent/mailpit:latest + # ports: + # - "8025:8025" # Web UI + # - "1025:1025" # SMTP + +volumes: + postgres_data: + redis_data: diff --git a/.claude/skills/devops/github-actions/SKILL.md b/.claude/skills/devops/github-actions/SKILL.md index 591a425..8de58cb 100644 --- a/.claude/skills/devops/github-actions/SKILL.md +++ b/.claude/skills/devops/github-actions/SKILL.md @@ -1,20 +1,33 @@ +--- +name: github-actions +description: > + Use this skill whenever setting up or modifying GitHub Actions CI/CD workflows, automating tests, builds, or deployments on GitHub. Trigger on keywords like GitHub Actions, workflow YAML, CI/CD pipeline, actions/checkout, matrix builds, workflow_dispatch, or .github/workflows. Also applies when configuring caching in workflows, managing GitHub secrets, or troubleshooting failed workflow runs. +--- + # GitHub Actions -## Description - -GitHub Actions CI/CD workflows including testing, building, and deployment. - ## When to Use - Setting up CI/CD pipelines - Automating tests and builds - Deployment automation +## When NOT to Use + +- GitLab CI projects using `.gitlab-ci.yml` configuration +- Jenkins pipelines using Jenkinsfile or Groovy-based configuration +- CircleCI, Travis CI, or other non-GitHub CI/CD systems + --- ## Core Patterns -### Basic CI Workflow +### 1. CI Pipeline + +Complete CI workflow covering checkout, setup, install, lint, test, and build for +both Python and Node.js projects. + +#### Node.js CI Pipeline ```yaml name: CI @@ -25,64 +38,767 @@ on: pull_request: branches: [main] +permissions: + contents: read + jobs: - test: + lint: + name: Lint runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 with: - node-version: '20' - cache: 'npm' + node-version: "20" + cache: "pnpm" + + - run: corepack enable + + - run: pnpm install --frozen-lockfile + + - run: pnpm lint + + - run: pnpm typecheck + + test: + name: Test + runs-on: ubuntu-latest + needs: lint + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: "20" + cache: "pnpm" + + - run: corepack enable + + - run: pnpm install --frozen-lockfile + + - run: pnpm test -- --coverage + + - name: Upload coverage + uses: actions/upload-artifact@v4 + with: + name: coverage-report + path: coverage/ + retention-days: 7 + + build: + name: Build + runs-on: ubuntu-latest + needs: test + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: "20" + cache: "pnpm" + + - run: corepack enable + + - run: pnpm install --frozen-lockfile + + - run: pnpm build + + - name: Upload build artifact + uses: actions/upload-artifact@v4 + with: + name: build-output + path: dist/ + retention-days: 5 +``` + +#### Python CI Pipeline + +```yaml +name: CI - Python + +on: + push: + branches: [main] + pull_request: + branches: [main] + +permissions: + contents: read + +jobs: + lint: + name: Lint + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: "pip" + + - run: pip install -r requirements-dev.txt + + - run: ruff check . + + - run: ruff format --check . + + - run: mypy src/ + + test: + name: Test + runs-on: ubuntu-latest + needs: lint + services: + postgres: + image: postgres:16-alpine + env: + POSTGRES_USER: test + POSTGRES_PASSWORD: test + POSTGRES_DB: testdb + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready -U test" + --health-interval 10s + --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: "pip" + + - run: pip install -r requirements.txt -r requirements-dev.txt + + - name: Run tests + env: + DATABASE_URL: postgresql://test:test@localhost:5432/testdb + run: pytest -v --cov=src --cov-report=xml + + - name: Upload coverage + uses: actions/upload-artifact@v4 + with: + name: coverage-xml + path: coverage.xml + retention-days: 7 +``` + +--- + +### 2. Matrix Strategy + +Matrix builds run the same job across multiple combinations of OS, language +version, or other variables. + +#### OS and version matrix + +```yaml +jobs: + test: + name: Test (${{ matrix.os }}, Node ${{ matrix.node }}) + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest, macos-latest, windows-latest] + node: [18, 20, 22] + runs-on: ${{ matrix.os }} + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: ${{ matrix.node }} + cache: "npm" + - run: npm ci + - run: npm test ``` -### Matrix Builds +#### Include and exclude ```yaml jobs: test: strategy: matrix: - os: [ubuntu-latest, macos-latest] - node: [18, 20] + os: [ubuntu-latest, macos-latest, windows-latest] + python: ["3.11", "3.12"] + exclude: + # Skip Python 3.11 on Windows + - os: windows-latest + python: "3.11" + include: + # Add a specific combination with extra env + - os: ubuntu-latest + python: "3.13" + experimental: true runs-on: ${{ matrix.os }} + continue-on-error: ${{ matrix.experimental || false }} steps: - - uses: actions/setup-node@v4 + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 with: - node-version: ${{ matrix.node }} + python-version: ${{ matrix.python }} + + - run: pip install -r requirements.txt + + - run: pytest ``` -### Caching +--- + +### 3. Caching + +Caching avoids re-downloading dependencies on every run. Use `hashFiles` to +generate cache keys from lockfiles so the cache invalidates when dependencies +change. + +#### npm cache ```yaml - uses: actions/cache@v4 with: path: ~/.npm - key: npm-${{ hashFiles('**/package-lock.json') }} - restore-keys: npm- + key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }} + restore-keys: | + npm-${{ runner.os }}- ``` -### Secrets +#### pnpm cache ```yaml -- name: Deploy - env: - API_KEY: ${{ secrets.API_KEY }} - run: deploy --key "$API_KEY" +- name: Get pnpm store directory + id: pnpm-cache + shell: bash + run: echo "store=$(pnpm store path)" >> "$GITHUB_OUTPUT" + +- uses: actions/cache@v4 + with: + path: ${{ steps.pnpm-cache.outputs.store }} + key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }} + restore-keys: | + pnpm-${{ runner.os }}- ``` +#### pip cache + +```yaml +- uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: pip-${{ runner.os }}-${{ hashFiles('**/requirements*.txt') }} + restore-keys: | + pip-${{ runner.os }}- +``` + +#### Docker layer cache + +```yaml +- name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + +- name: Build and push + uses: docker/build-push-action@v6 + with: + context: . + push: true + tags: myapp:latest + cache-from: type=gha + cache-to: type=gha,mode=max +``` + +--- + +### 4. Reusable Workflows + +Reusable workflows let you define a workflow once and call it from other +workflows, reducing duplication across repositories. + +#### Defining a reusable workflow (`.github/workflows/reusable-test.yml`) + +```yaml +name: Reusable Test Workflow + +on: + workflow_call: + inputs: + node-version: + description: "Node.js version to use" + required: false + type: string + default: "20" + working-directory: + description: "Directory to run commands in" + required: false + type: string + default: "." + secrets: + NPM_TOKEN: + required: false + +jobs: + test: + runs-on: ubuntu-latest + defaults: + run: + working-directory: ${{ inputs.working-directory }} + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: ${{ inputs.node-version }} + cache: "npm" + registry-url: "https://registry.npmjs.org" + + - run: npm ci + env: + NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} + + - run: npm test +``` + +#### Calling a reusable workflow + +```yaml +name: CI + +on: + push: + branches: [main] + +jobs: + test-app: + uses: ./.github/workflows/reusable-test.yml + with: + node-version: "20" + working-directory: "packages/app" + secrets: inherit # Pass all secrets to the called workflow + + test-lib: + uses: ./.github/workflows/reusable-test.yml + with: + node-version: "20" + working-directory: "packages/lib" + secrets: inherit +``` + +--- + +### 5. Composite Actions + +Composite actions package multiple steps into a single reusable action. Unlike +reusable workflows, they run inline within the calling job. + +#### Action definition (`.github/actions/setup-project/action.yml`) + +```yaml +name: "Setup Project" +description: "Install Node.js, enable corepack, and install dependencies" + +inputs: + node-version: + description: "Node.js version" + required: false + default: "20" + install-command: + description: "Command to install dependencies" + required: false + default: "pnpm install --frozen-lockfile" + +runs: + using: "composite" + steps: + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: ${{ inputs.node-version }} + + - name: Enable corepack + shell: bash + run: corepack enable + + - name: Get pnpm store directory + id: pnpm-cache + shell: bash + run: echo "store=$(pnpm store path)" >> "$GITHUB_OUTPUT" + + - name: Cache pnpm store + uses: actions/cache@v4 + with: + path: ${{ steps.pnpm-cache.outputs.store }} + key: pnpm-${{ runner.os }}-${{ hashFiles('**/pnpm-lock.yaml') }} + restore-keys: | + pnpm-${{ runner.os }}- + + - name: Install dependencies + shell: bash + run: ${{ inputs.install-command }} +``` + +#### Using the composite action + +```yaml +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: ./.github/actions/setup-project + with: + node-version: "20" + + - run: pnpm build +``` + +--- + +### 6. Deployment + +Deployment workflows with environment protection rules, manual approval gates, +and multi-stage promotion. + +```yaml +name: Deploy + +on: + push: + branches: [main] + workflow_dispatch: + inputs: + environment: + description: "Target environment" + required: true + type: choice + options: + - staging + - production + +permissions: + contents: read + deployments: write + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: "20" + cache: "pnpm" + + - run: corepack enable && pnpm install --frozen-lockfile + + - run: pnpm build + + - uses: actions/upload-artifact@v4 + with: + name: build-output + path: dist/ + + deploy-staging: + name: Deploy to Staging + runs-on: ubuntu-latest + needs: build + environment: + name: staging + url: https://staging.example.com + steps: + - uses: actions/download-artifact@v4 + with: + name: build-output + path: dist/ + + - name: Deploy to staging + env: + DEPLOY_TOKEN: ${{ secrets.STAGING_DEPLOY_TOKEN }} + run: | + echo "Deploying to staging..." + # Replace with your actual deploy command + # e.g., aws s3 sync, rsync, wrangler publish, etc. + + deploy-production: + name: Deploy to Production + runs-on: ubuntu-latest + needs: deploy-staging + if: github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'production' + environment: + name: production + url: https://example.com + # Production environment should have required reviewers configured + # in GitHub Settings > Environments > production > Protection rules + steps: + - uses: actions/download-artifact@v4 + with: + name: build-output + path: dist/ + + - name: Deploy to production + env: + DEPLOY_TOKEN: ${{ secrets.PRODUCTION_DEPLOY_TOKEN }} + run: | + echo "Deploying to production..." +``` + +--- + +### 7. Artifacts + +Artifacts let you share data between jobs in the same workflow or persist build +outputs for later download. + +#### Upload artifact + +```yaml +- name: Upload test results + uses: actions/upload-artifact@v4 + if: always() # Upload even if tests fail + with: + name: test-results-${{ matrix.os }}-${{ matrix.node }} + path: | + test-results/ + coverage/ + retention-days: 14 + if-no-files-found: warn # warn, error, or ignore +``` + +#### Download artifact in another job + +```yaml +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: npm ci && npm run build + - uses: actions/upload-artifact@v4 + with: + name: dist + path: dist/ + + deploy: + runs-on: ubuntu-latest + needs: build + steps: + - uses: actions/download-artifact@v4 + with: + name: dist + path: dist/ + + - run: ls -la dist/ +``` + +#### Download all artifacts + +```yaml +- uses: actions/download-artifact@v4 + with: + path: all-artifacts/ + # Each artifact is placed in a subdirectory named after the artifact +``` + +--- + +### 8. Conditional Execution + +Control when jobs and steps run using `if` expressions, job dependencies, and +path filters. + +#### Path filters on triggers + +```yaml +on: + push: + branches: [main] + paths: + - "src/**" + - "package.json" + - "pnpm-lock.yaml" + paths-ignore: + - "docs/**" + - "*.md" +``` + +#### Conditional jobs + +```yaml +jobs: + changes: + runs-on: ubuntu-latest + outputs: + backend: ${{ steps.filter.outputs.backend }} + frontend: ${{ steps.filter.outputs.frontend }} + steps: + - uses: actions/checkout@v4 + - uses: dorny/paths-filter@v3 + id: filter + with: + filters: | + backend: + - 'src/api/**' + - 'requirements*.txt' + frontend: + - 'src/web/**' + - 'package.json' + + test-backend: + needs: changes + if: needs.changes.outputs.backend == 'true' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: pip install -r requirements.txt && pytest + + test-frontend: + needs: changes + if: needs.changes.outputs.frontend == 'true' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: npm ci && npm test +``` + +#### Conditional steps with if expressions + +```yaml +steps: + - name: Run only on main branch + if: github.ref == 'refs/heads/main' + run: echo "On main" + + - name: Run only on pull requests + if: github.event_name == 'pull_request' + run: echo "PR event" + + - name: Run only when previous step failed + if: failure() + run: echo "Something failed" + + - name: Always run (cleanup) + if: always() + run: echo "Cleanup" + + - name: Run only when a label is present + if: contains(github.event.pull_request.labels.*.name, 'deploy') + run: echo "Deploy label found" + + - name: Skip for dependabot + if: github.actor != 'dependabot[bot]' + run: npm test +``` + +#### Job dependencies + +```yaml +jobs: + lint: + runs-on: ubuntu-latest + steps: + - run: echo "Linting..." + + test: + runs-on: ubuntu-latest + steps: + - run: echo "Testing..." + + # Runs after both lint and test succeed + deploy: + runs-on: ubuntu-latest + needs: [lint, test] + steps: + - run: echo "Deploying..." + + # Runs even if test fails, but only after it completes + notify: + runs-on: ubuntu-latest + needs: [test] + if: always() + steps: + - run: echo "Test job status: ${{ needs.test.result }}" +``` + +--- + ## Best Practices -1. Use caching for dependencies -2. Run jobs in parallel when possible -3. Use environment secrets -4. Pin action versions -5. Add proper triggers +1. **Pin action versions with SHA** -- Use the full commit SHA instead of a + mutable tag: `actions/checkout@b4ffde65f...` (or at minimum a major version + tag like `@v4`). This prevents supply-chain attacks where a tag is moved. + +2. **Use caching aggressively** -- Cache package manager stores (`~/.npm`, + pnpm store, `~/.cache/pip`) and Docker layers. A well-cached pipeline can + cut run times by 50-80%. + +3. **Set minimal permissions** -- Add a top-level `permissions` block and grant + only what is needed. Default permissions are overly broad and pose a security + risk, especially for pull requests from forks. + +4. **Run jobs in parallel** -- Structure independent jobs (lint, test, typecheck) + to run concurrently. Use `needs` only when there is a real dependency between + jobs. + +5. **Use `fail-fast: false` in matrix builds** -- By default a failing matrix + combination cancels all others. Setting `fail-fast: false` lets all + combinations complete so you get the full picture of what is broken. + +6. **Use environment protection rules** -- Configure required reviewers and wait + timers on production environments in GitHub Settings. This adds a human gate + before production deploys. + +7. **Extract reusable workflows and composite actions** -- If the same steps + appear in multiple workflows, factor them into a reusable workflow + (`workflow_call`) or composite action to keep things DRY. + +8. **Keep secrets out of logs** -- Never `echo` a secret. GitHub masks known + secrets, but dynamically constructed values may leak. Use `::add-mask::` for + runtime values that should be hidden. + +--- ## Common Pitfalls -- **Slow pipelines**: Add caching -- **Secret exposure**: Never echo secrets -- **Unpinned versions**: Use @v4 not @main +1. **Unpinned action versions** -- Using `actions/checkout@main` means your + workflow pulls whatever is on main today. A bad push to that action + repository could break or compromise your builds. Pin to a tag (`@v4`) or + SHA. + +2. **Missing caching** -- Running `npm ci` or `pip install` from scratch on + every run wastes minutes. Always configure caching for your package manager, + or use the built-in `cache` option in setup actions (e.g., + `actions/setup-node` has a `cache` input). + +3. **Overly broad triggers** -- Triggering on every push to every branch floods + the queue. Restrict triggers to `main` and pull requests. Use `paths` or + `paths-ignore` to skip runs when only docs or unrelated files change. + +4. **Secret exposure in pull requests from forks** -- Secrets are NOT available + in workflows triggered by `pull_request` from forks (by design). If your + workflow needs secrets for fork PRs, use `pull_request_target` carefully and + never check out untrusted code in that context. + +5. **Large artifacts without retention limits** -- Uploading artifacts without + setting `retention-days` uses the repository default (90 days), consuming + storage quota. Set short retention for transient artifacts like test results + and coverage reports. + +6. **Ignoring `if: always()` for cleanup** -- Steps after a failure are skipped + by default. If you need to upload test results, send notifications, or run + cleanup regardless of prior step results, use `if: always()` or + `if: failure()`. + +--- + +## Related Skills + +- `devops/docker` - Container patterns for building and deploying Dockerized applications in workflows +- `testing/pytest` - Python test configuration for CI pipeline integration +- `testing/vitest` - TypeScript/JavaScript test configuration for CI pipeline integration diff --git a/.claude/skills/devops/github-actions/references/gha-syntax.md b/.claude/skills/devops/github-actions/references/gha-syntax.md new file mode 100644 index 0000000..5d4f356 --- /dev/null +++ b/.claude/skills/devops/github-actions/references/gha-syntax.md @@ -0,0 +1,250 @@ +# GitHub Actions Syntax Quick Reference +## Workflow File Structure + +```yaml +name: CI # Workflow name (shown in GitHub UI) + +on: # Triggers + push: + branches: [main] + pull_request: + branches: [main] + +permissions: # Workflow-level permissions + contents: read + +env: # Workflow-level environment variables + NODE_ENV: test + +jobs: + build: # Job ID + runs-on: ubuntu-latest # Runner + steps: + - uses: actions/checkout@v4 # Action step + - run: echo "Hello" # Shell step +``` + +## Triggers (on:) + +### Common Events + +```yaml +on: + push: + branches: [main, "release/**"] + paths: ["src/**", "!src/**/*.test.ts"] # Path filtering + tags: ["v*"] + pull_request: + branches: [main] + types: [opened, synchronize, reopened] + workflow_dispatch: # Manual trigger + inputs: + environment: + type: choice + options: [staging, production] + schedule: + - cron: "0 6 * * 1" # Every Monday at 6am UTC + release: + types: [published] + workflow_call: # Reusable workflow + inputs: + node-version: { type: string, default: "22" } + secrets: + NPM_TOKEN: { required: true } +``` + +## Jobs + +```yaml +jobs: + test: + name: Run Tests + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v4 + - run: npm test + deploy: + needs: [lint, test] # Runs after lint AND test succeed + runs-on: ubuntu-latest + steps: [...] +``` + +### Matrix Strategy + +```yaml +jobs: + test: + strategy: + fail-fast: false # Don't cancel other jobs on failure + matrix: + os: [ubuntu-latest, macos-latest] + node: [20, 22] + exclude: + - os: macos-latest + node: 20 + include: + - os: ubuntu-latest + node: 22 + coverage: true + runs-on: ${{ matrix.os }} + steps: + - uses: actions/setup-node@v4 + with: + node-version: ${{ matrix.node }} +``` + +## Steps + +### Action Step + +```yaml +- uses: actions/checkout@v4 + with: + fetch-depth: 0 # Full history (needed for some tools) + +- uses: actions/setup-node@v4 + with: + node-version-file: ".nvmrc" + cache: "pnpm" +``` + +### Shell Step + +```yaml +- name: Run tests + run: npm test + working-directory: ./packages/api + env: + DATABASE_URL: ${{ secrets.DATABASE_URL }} + shell: bash + continue-on-error: true # Don't fail the job + timeout-minutes: 10 +``` + +### Multi-line Commands + +```yaml +- run: | + echo "Line 1" + echo "Line 2" + npm run build +``` + +## Conditionals (if:) + +```yaml +# Run only on main branch +- if: github.ref == 'refs/heads/main' + +# Run only on pull requests +- if: github.event_name == 'pull_request' + +# Run only when previous step failed +- if: failure() + +# Always run (even if previous steps failed) +- if: always() + +# Run only when a matrix variable is set +- if: matrix.coverage == true + +# Run based on changed files (requires dorny/paths-filter or similar) +- if: steps.filter.outputs.backend == 'true' + +# Run on specific actor +- if: github.actor != 'dependabot[bot]' +``` + +## Environment and Secrets + +```yaml +jobs: + deploy: + environment: + name: production + url: https://example.com + env: + APP_VERSION: ${{ github.sha }} + steps: + - run: deploy.sh + env: + API_KEY: ${{ secrets.API_KEY }} # Repository secret + DEPLOY_TOKEN: ${{ vars.DEPLOY_TOKEN }} # Repository variable +``` + +## Caching + +### Built-in Cache (setup actions) + +```yaml +- uses: actions/setup-node@v4 + with: + node-version: 22 + cache: "pnpm" # Automatic pnpm cache +``` + +### Manual Cache + +```yaml +- uses: actions/cache@v4 + with: + path: | + ~/.cache/pip + .mypy_cache + key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }} + restore-keys: | + ${{ runner.os }}-pip- +``` + +## Artifacts + +### Upload + +```yaml +- uses: actions/upload-artifact@v4 + with: + name: coverage-report + path: coverage/ + retention-days: 7 +``` + +### Download (in another job) + +```yaml +- uses: actions/download-artifact@v4 + with: + name: coverage-report + path: ./coverage +``` + +## Services (Containers) + +Define `services:` under a job with `image`, `env`, `ports`, and `options` (for health checks). Common: postgres, redis, mysql. + +## Outputs (Passing Data Between Steps/Jobs) + +```yaml +# Step output: echo "key=value" >> "$GITHUB_OUTPUT" +# Read in later step: ${{ steps..outputs.key }} +# Job output: declare under jobs..outputs, read via needs..outputs.key +``` + +## Permissions + +Common values: `contents: read`, `pull-requests: write`, `issues: write`, `packages: write`, `id-token: write` (OIDC). + +## Reusable Workflows + +Caller uses `uses: ./.github/workflows/reusable.yaml` with `with:` and `secrets: inherit`. Callee triggers on `workflow_call:` with `inputs:` and `secrets:` definitions. + +## Useful Expressions + +| Expression | Result | +|-----------|--------| +| `${{ github.sha }}` | Full commit SHA | +| `${{ github.ref_name }}` | Branch or tag name | +| `${{ github.event.pull_request.number }}` | PR number | +| `${{ runner.os }}` | `Linux`, `macOS`, `Windows` | +| `${{ hashFiles('**/lockfile') }}` | SHA256 of files | +| `${{ contains(github.event.head_commit.message, '[skip ci]') }}` | Check commit message | diff --git a/.claude/skills/devops/github-actions/templates/ci-node.yaml b/.claude/skills/devops/github-actions/templates/ci-node.yaml new file mode 100644 index 0000000..7a64df0 --- /dev/null +++ b/.claude/skills/devops/github-actions/templates/ci-node.yaml @@ -0,0 +1,176 @@ +# ============================================================================= +# Node.js CI Pipeline +# Runs: lint (eslint), type check (tsc), test (vitest), build +# ============================================================================= + +name: Node CI + +on: + push: + branches: [main] + paths: + - "**.ts" + - "**.tsx" + - "**.js" + - "**.jsx" + - "package.json" + - "pnpm-lock.yaml" + - "tsconfig.json" + - ".github/workflows/ci-node.yaml" + pull_request: + branches: [main] + paths: + - "**.ts" + - "**.tsx" + - "**.js" + - "**.jsx" + - "package.json" + - "pnpm-lock.yaml" + - "tsconfig.json" + - ".github/workflows/ci-node.yaml" + +permissions: + contents: read + +env: + NODE_VERSION: "22" + +jobs: + # --------------------------------------------------------------------------- + # Install dependencies (shared across jobs via cache) + # --------------------------------------------------------------------------- + install: + name: Install + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + version: latest + + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: "pnpm" + + - name: Install dependencies + run: pnpm install --frozen-lockfile + + # --------------------------------------------------------------------------- + # Lint with ESLint + # --------------------------------------------------------------------------- + lint: + name: Lint + needs: install + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + version: latest + + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: "pnpm" + + - run: pnpm install --frozen-lockfile + - run: pnpm lint + + # --------------------------------------------------------------------------- + # Type check with TypeScript compiler + # --------------------------------------------------------------------------- + type-check: + name: Type Check + needs: install + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + version: latest + + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: "pnpm" + + - run: pnpm install --frozen-lockfile + - run: pnpm tsc --noEmit + + # --------------------------------------------------------------------------- + # Test with Vitest + # --------------------------------------------------------------------------- + test: + name: Test + needs: install + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + version: latest + + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: "pnpm" + + - run: pnpm install --frozen-lockfile + + - name: Run tests with coverage + run: pnpm vitest run --coverage --reporter=junit --outputFile=junit.xml + + - name: Upload coverage report + if: always() + uses: actions/upload-artifact@v4 + with: + name: coverage-report + path: coverage/ + retention-days: 7 + + - name: Upload test results + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-results + path: junit.xml + retention-days: 7 + + # --------------------------------------------------------------------------- + # Build + # --------------------------------------------------------------------------- + build: + name: Build + needs: [lint, type-check, test] + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + + - uses: pnpm/action-setup@v4 + with: + version: latest + + - uses: actions/setup-node@v4 + with: + node-version: ${{ env.NODE_VERSION }} + cache: "pnpm" + + - run: pnpm install --frozen-lockfile + - run: pnpm build + + - name: Upload build artifacts + uses: actions/upload-artifact@v4 + with: + name: build-output + path: dist/ + retention-days: 7 diff --git a/.claude/skills/devops/github-actions/templates/ci-python.yaml b/.claude/skills/devops/github-actions/templates/ci-python.yaml new file mode 100644 index 0000000..5895e3b --- /dev/null +++ b/.claude/skills/devops/github-actions/templates/ci-python.yaml @@ -0,0 +1,164 @@ +# ============================================================================= +# Python CI Pipeline +# Runs: lint (ruff), type check (mypy), test (pytest), coverage upload +# ============================================================================= + +name: Python CI + +on: + push: + branches: [main] + paths: + - "**.py" + - "requirements*.txt" + - "pyproject.toml" + - ".github/workflows/ci-python.yaml" + pull_request: + branches: [main] + paths: + - "**.py" + - "requirements*.txt" + - "pyproject.toml" + - ".github/workflows/ci-python.yaml" + +permissions: + contents: read + +env: + PYTHON_VERSION: "3.12" + +jobs: + # --------------------------------------------------------------------------- + # Lint with Ruff + # --------------------------------------------------------------------------- + lint: + name: Lint + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: ${{ env.PYTHON_VERSION }} + + - name: Install ruff + run: pip install ruff + + - name: Ruff check (lint) + run: ruff check . + + - name: Ruff format (formatting) + run: ruff format --check . + + # --------------------------------------------------------------------------- + # Type check with mypy + # --------------------------------------------------------------------------- + type-check: + name: Type Check + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: ${{ env.PYTHON_VERSION }} + + - name: Cache pip dependencies + uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }} + restore-keys: | + ${{ runner.os }}-pip- + + - name: Install dependencies + run: | + pip install --upgrade pip + pip install -r requirements.txt + pip install mypy + + - name: Run mypy + run: mypy src/ --ignore-missing-imports + + # --------------------------------------------------------------------------- + # Test with pytest + # --------------------------------------------------------------------------- + test: + name: Test + runs-on: ubuntu-latest + timeout-minutes: 15 + + # Uncomment to add a PostgreSQL service for integration tests. + # services: + # postgres: + # image: postgres:17-alpine + # env: + # POSTGRES_USER: test + # POSTGRES_PASSWORD: test + # POSTGRES_DB: test_db + # ports: + # - 5432:5432 + # options: >- + # --health-cmd pg_isready + # --health-interval 10s + # --health-timeout 5s + # --health-retries 5 + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: ${{ env.PYTHON_VERSION }} + + - name: Cache pip dependencies + uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }} + restore-keys: | + ${{ runner.os }}-pip- + + - name: Install dependencies + run: | + pip install --upgrade pip + pip install -r requirements.txt + pip install -r requirements-dev.txt + + - name: Run tests with coverage + run: | + pytest \ + --cov=src \ + --cov-report=xml:coverage.xml \ + --cov-report=term-missing \ + --junitxml=junit.xml \ + -v + env: + PYTHONPATH: ${{ github.workspace }} + # DATABASE_URL: postgresql://test:test@localhost:5432/test_db + + - name: Upload coverage report + if: always() + uses: actions/upload-artifact@v4 + with: + name: coverage-report + path: coverage.xml + retention-days: 7 + + - name: Upload test results + if: always() + uses: actions/upload-artifact@v4 + with: + name: test-results + path: junit.xml + retention-days: 7 + + # Uncomment to upload coverage to Codecov. + # - name: Upload to Codecov + # if: github.event_name == 'push' && github.ref == 'refs/heads/main' + # uses: codecov/codecov-action@v4 + # with: + # files: coverage.xml + # token: ${{ secrets.CODECOV_TOKEN }} diff --git a/.claude/skills/frameworks/django/SKILL.md b/.claude/skills/frameworks/django/SKILL.md index 0ce2781..d7ae068 100644 --- a/.claude/skills/frameworks/django/SKILL.md +++ b/.claude/skills/frameworks/django/SKILL.md @@ -1,91 +1,715 @@ +--- +name: django +description: > + Use this skill when working with Django web applications, Django ORM models, Django REST Framework APIs, or Django admin interfaces. Trigger for any mention of Django views, serializers, migrations, URL routing, class-based views, or Django middleware. Also applies when building Python web apps with templates, managing database schemas through Django migrations, or setting up admin panels. +--- + # Django -## Description - -Django web framework with ORM, views, and REST framework patterns. - ## When to Use - Python web applications - Admin interfaces - Django REST Framework APIs +- Content-heavy sites with ORM-driven data models + +## When NOT to Use + +- FastAPI projects — use the `frameworks/fastapi` skill instead for async APIs and microservices +- JavaScript/Node.js backends (Express, NestJS) — this skill is Python-only +- Microservices architectures — consider FastAPI instead for lightweight, async services --- ## Core Patterns -### Models +### 1. Models & ORM + +#### Field types and relationships ```python from django.db import models +from django.utils import timezone + +class Organization(models.Model): + name = models.CharField(max_length=200) + slug = models.SlugField(unique=True) + created_at = models.DateTimeField(auto_now_add=True) + updated_at = models.DateTimeField(auto_now=True) + + class Meta: + ordering = ["name"] + + def __str__(self): + return self.name class User(models.Model): + class Role(models.TextChoices): + ADMIN = "admin", "Administrator" + MEMBER = "member", "Member" + VIEWER = "viewer", "Viewer" + email = models.EmailField(unique=True) name = models.CharField(max_length=100) + organization = models.ForeignKey( + Organization, + on_delete=models.CASCADE, + related_name="members", + ) + role = models.CharField(max_length=20, choices=Role.choices, default=Role.MEMBER) + is_active = models.BooleanField(default=True) created_at = models.DateTimeField(auto_now_add=True) class Meta: - ordering = ['-created_at'] + ordering = ["-created_at"] + indexes = [ + models.Index(fields=["email"]), + models.Index(fields=["organization", "role"]), + ] + constraints = [ + models.UniqueConstraint( + fields=["organization", "email"], + name="unique_org_email", + ), + ] def __str__(self): return self.email + +class Tag(models.Model): + name = models.CharField(max_length=50, unique=True) + +class Project(models.Model): + title = models.CharField(max_length=200) + description = models.TextField(blank=True) + owner = models.ForeignKey(User, on_delete=models.CASCADE, related_name="owned_projects") + organization = models.ForeignKey(Organization, on_delete=models.CASCADE) + tags = models.ManyToManyField(Tag, blank=True, related_name="projects") + # OneToOneField for 1:1 relationships + settings = models.OneToOneField( + "ProjectSettings", on_delete=models.CASCADE, null=True, blank=True + ) + +class ProjectSettings(models.Model): + is_public = models.BooleanField(default=False) + max_members = models.IntegerField(default=10) ``` -### Views (Class-based) +#### Custom managers and QuerySet methods ```python -from django.views.generic import ListView, DetailView +class ActiveManager(models.Manager): + def get_queryset(self): + return super().get_queryset().filter(is_active=True) -class UserListView(ListView): - model = User - template_name = 'users/list.html' - context_object_name = 'users' +class UserQuerySet(models.QuerySet): + def admins(self): + return self.filter(role=User.Role.ADMIN) + + def in_organization(self, org_id): + return self.filter(organization_id=org_id) + + def with_project_count(self): + return self.annotate(project_count=models.Count("owned_projects")) + +class User(models.Model): + # ... fields ... + objects = UserQuerySet.as_manager() + active = ActiveManager() +``` + +#### F objects, Q objects, and annotations + +```python +from django.db.models import F, Q, Count, Avg, Sum, Value, When, Case + +# F objects: reference model fields in queries +Project.objects.filter(updated_at__gt=F("created_at")) +User.objects.update(login_count=F("login_count") + 1) # Atomic increment + +# Q objects: complex lookups with OR, AND, NOT +User.objects.filter( + Q(role="admin") | Q(role="member"), + ~Q(is_active=False), # NOT inactive +) + +# Annotations and aggregations +orgs = Organization.objects.annotate( + member_count=Count("members"), + admin_count=Count("members", filter=Q(members__role="admin")), + avg_projects=Avg("members__owned_projects"), +).filter(member_count__gte=5) + +# Conditional expressions +users = User.objects.annotate( + tier=Case( + When(owned_projects__count__gte=10, then=Value("power")), + When(owned_projects__count__gte=3, then=Value("active")), + default=Value("starter"), + ) +) + +# Subqueries +from django.db.models import Subquery, OuterRef +latest_project = Project.objects.filter( + owner=OuterRef("pk") +).order_by("-created_at").values("title")[:1] +users = User.objects.annotate(latest_project_title=Subquery(latest_project)) +``` + +### 2. Views + +#### Function-based views + +```python +from django.shortcuts import render, get_object_or_404, redirect +from django.http import JsonResponse +from django.contrib.auth.decorators import login_required + +@login_required +def project_detail(request, project_id): + project = get_object_or_404( + Project.objects.select_related("owner", "organization"), + pk=project_id, + ) + if request.method == "POST": + form = ProjectForm(request.POST, instance=project) + if form.is_valid(): + form.save() + return redirect("project-detail", project_id=project.id) + else: + form = ProjectForm(instance=project) + + return render(request, "projects/detail.html", { + "project": project, + "form": form, + }) +``` + +#### Class-based views + +```python +from django.views.generic import ListView, DetailView, CreateView, UpdateView, DeleteView +from django.contrib.auth.mixins import LoginRequiredMixin, PermissionRequiredMixin +from django.urls import reverse_lazy + +class ProjectListView(LoginRequiredMixin, ListView): + model = Project + template_name = "projects/list.html" + context_object_name = "projects" paginate_by = 20 -class UserDetailView(DetailView): - model = User - template_name = 'users/detail.html' + def get_queryset(self): + qs = super().get_queryset().select_related("owner", "organization") + search = self.request.GET.get("q") + if search: + qs = qs.filter( + Q(title__icontains=search) | Q(description__icontains=search) + ) + return qs + +class ProjectCreateView(LoginRequiredMixin, CreateView): + model = Project + form_class = ProjectForm + template_name = "projects/form.html" + success_url = reverse_lazy("project-list") + + def form_valid(self, form): + form.instance.owner = self.request.user + form.instance.organization = self.request.user.organization + return super().form_valid(form) + +class ProjectDeleteView(PermissionRequiredMixin, DeleteView): + model = Project + permission_required = "projects.delete_project" + success_url = reverse_lazy("project-list") ``` -### Django REST Framework +#### Mixins for reuse ```python -from rest_framework import serializers, viewsets +class OrganizationFilterMixin: + """Filter queryset to the current user's organization.""" + def get_queryset(self): + return super().get_queryset().filter( + organization=self.request.user.organization + ) -class UserSerializer(serializers.ModelSerializer): +class ProjectListView(LoginRequiredMixin, OrganizationFilterMixin, ListView): + model = Project + # queryset is automatically filtered by organization +``` + +#### API views with Django REST Framework + +```python +from rest_framework.decorators import api_view, permission_classes +from rest_framework.permissions import IsAuthenticated +from rest_framework.response import Response +from rest_framework import status + +@api_view(["GET", "POST"]) +@permission_classes([IsAuthenticated]) +def project_list(request): + if request.method == "GET": + projects = Project.objects.filter(organization=request.user.organization) + serializer = ProjectSerializer(projects, many=True) + return Response(serializer.data) + + serializer = ProjectCreateSerializer(data=request.data) + serializer.is_valid(raise_exception=True) + serializer.save(owner=request.user) + return Response(serializer.data, status=status.HTTP_201_CREATED) +``` + +### 3. Migrations + +#### Creating and running migrations + +```bash +# Generate migrations after model changes +python manage.py makemigrations app_name + +# Preview SQL without applying +python manage.py sqlmigrate app_name 0001 + +# Apply migrations +python manage.py migrate + +# Show migration status +python manage.py showmigrations +``` + +#### Data migrations with RunPython + +```python +from django.db import migrations + +def populate_slugs(apps, schema_editor): + Organization = apps.get_model("myapp", "Organization") + from django.utils.text import slugify + for org in Organization.objects.filter(slug=""): + org.slug = slugify(org.name) + org.save(update_fields=["slug"]) + +def reverse_populate_slugs(apps, schema_editor): + pass # No-op reverse + +class Migration(migrations.Migration): + dependencies = [ + ("myapp", "0005_add_slug_field"), + ] + operations = [ + migrations.RunPython(populate_slugs, reverse_populate_slugs), + ] +``` + +#### Squashing migrations + +```bash +# Squash migrations 0001 through 0010 into one +python manage.py squashmigrations app_name 0001 0010 +``` + +**Tips:** +- Always provide a reverse function for `RunPython` (even if it is a no-op) +- Use `apps.get_model()` in data migrations, never import models directly +- Test migrations on a copy of production data before deploying + +### 4. Forms + +#### ModelForm with custom validation + +```python +from django import forms +from django.core.exceptions import ValidationError + +class ProjectForm(forms.ModelForm): class Meta: - model = User - fields = ['id', 'email', 'name', 'created_at'] + model = Project + fields = ["title", "description", "tags"] + widgets = { + "description": forms.Textarea(attrs={"rows": 4}), + "tags": forms.CheckboxSelectMultiple(), + } -class UserViewSet(viewsets.ModelViewSet): - queryset = User.objects.all() - serializer_class = UserSerializer + def clean_title(self): + title = self.cleaned_data["title"] + if "test" in title.lower() and not self.instance.pk: + raise ValidationError("Title cannot contain 'test' for new projects.") + return title + + def clean(self): + cleaned = super().clean() + title = cleaned.get("title", "") + description = cleaned.get("description", "") + if len(title) + len(description) < 20: + raise ValidationError("Title + description must be at least 20 characters.") + return cleaned ``` -### URLs +#### Formsets ```python -from django.urls import path, include -from rest_framework.routers import DefaultRouter +from django.forms import inlineformset_factory -router = DefaultRouter() -router.register('users', UserViewSet) +TaskFormSet = inlineformset_factory( + Project, + Task, + fields=["title", "assigned_to", "due_date"], + extra=2, # Number of empty forms + can_delete=True, + max_num=20, +) -urlpatterns = [ - path('api/', include(router.urls)), +# In a view +def project_tasks(request, project_id): + project = get_object_or_404(Project, pk=project_id) + if request.method == "POST": + formset = TaskFormSet(request.POST, instance=project) + if formset.is_valid(): + formset.save() + return redirect("project-detail", project_id=project.id) + else: + formset = TaskFormSet(instance=project) + return render(request, "projects/tasks.html", {"formset": formset}) +``` + +### 5. Signals + +```python +from django.db.models.signals import post_save, pre_save, m2m_changed +from django.dispatch import receiver + +@receiver(post_save, sender=User) +def create_user_profile(sender, instance, created, **kwargs): + if created: + UserProfile.objects.create(user=instance) + +@receiver(pre_save, sender=Project) +def set_project_slug(sender, instance, **kwargs): + if not instance.slug: + from django.utils.text import slugify + instance.slug = slugify(instance.title) + +# Custom signals +from django.dispatch import Signal + +project_published = Signal() # Accepts sender + +@receiver(project_published) +def notify_members(sender, project, **kwargs): + for member in project.organization.members.all(): + send_notification(member, f"Project '{project.title}' published") + +# Firing a custom signal +project_published.send(sender=Project, project=project) +``` + +**When to use signals vs overriding `save()`:** +- Use signals when the action is a side effect (notifications, logging, cache invalidation) +- Override `save()` when the logic is core to the model's behavior (setting computed fields) + +### 6. Middleware + +```python +import time +from django.utils.deprecation import MiddlewareMixin + +class TimingMiddleware(MiddlewareMixin): + def process_request(self, request): + request._start_time = time.perf_counter() + + def process_response(self, request, response): + if hasattr(request, "_start_time"): + duration = time.perf_counter() - request._start_time + response["X-Process-Time"] = f"{duration:.4f}" + return response + +# New-style middleware (function-based) +def organization_middleware(get_response): + def middleware(request): + if request.user.is_authenticated: + request.organization = request.user.organization + else: + request.organization = None + response = get_response(request) + return response + return middleware + +# Register in settings.py +MIDDLEWARE = [ + "django.middleware.security.SecurityMiddleware", + "django.contrib.sessions.middleware.SessionMiddleware", + "django.middleware.common.CommonMiddleware", + "django.middleware.csrf.CsrfViewMiddleware", + "django.contrib.auth.middleware.AuthenticationMiddleware", + "myapp.middleware.organization_middleware", # Custom + "myapp.middleware.TimingMiddleware", # Custom + "django.contrib.messages.middleware.MessageMiddleware", + "django.middleware.clickjacking.XFrameOptionsMiddleware", ] ``` +### 7. Django REST Framework + +#### Serializers + +```python +from rest_framework import serializers + +class UserSerializer(serializers.ModelSerializer): + project_count = serializers.IntegerField(read_only=True) + full_name = serializers.SerializerMethodField() + + class Meta: + model = User + fields = ["id", "email", "name", "role", "full_name", "project_count", "created_at"] + read_only_fields = ["id", "created_at"] + + def get_full_name(self, obj): + return f"{obj.name} ({obj.role})" + +class ProjectSerializer(serializers.ModelSerializer): + owner = UserSerializer(read_only=True) + tags = serializers.SlugRelatedField( + many=True, slug_field="name", queryset=Tag.objects.all() + ) + + class Meta: + model = Project + fields = ["id", "title", "description", "owner", "tags", "created_at"] + + def validate_title(self, value): + if len(value) < 3: + raise serializers.ValidationError("Title must be at least 3 characters.") + return value + +class ProjectCreateSerializer(serializers.ModelSerializer): + class Meta: + model = Project + fields = ["title", "description", "tags"] +``` + +#### ViewSets and routers + +```python +from rest_framework import viewsets, permissions, filters +from rest_framework.decorators import action +from rest_framework.response import Response +from django_filters.rest_framework import DjangoFilterBackend + +class ProjectViewSet(viewsets.ModelViewSet): + serializer_class = ProjectSerializer + permission_classes = [permissions.IsAuthenticated] + filter_backends = [DjangoFilterBackend, filters.SearchFilter, filters.OrderingFilter] + filterset_fields = ["owner", "tags"] + search_fields = ["title", "description"] + ordering_fields = ["created_at", "title"] + ordering = ["-created_at"] + + def get_queryset(self): + return Project.objects.filter( + organization=self.request.user.organization + ).select_related("owner").prefetch_related("tags") + + def get_serializer_class(self): + if self.action == "create": + return ProjectCreateSerializer + return ProjectSerializer + + def perform_create(self, serializer): + serializer.save( + owner=self.request.user, + organization=self.request.user.organization, + ) + + @action(detail=True, methods=["post"]) + def publish(self, request, pk=None): + project = self.get_object() + project.is_published = True + project.save(update_fields=["is_published"]) + return Response({"status": "published"}) + +# urls.py +from rest_framework.routers import DefaultRouter + +router = DefaultRouter() +router.register("projects", ProjectViewSet, basename="project") +router.register("users", UserViewSet, basename="user") + +urlpatterns = [ + path("api/", include(router.urls)), +] +``` + +#### Permissions + +```python +from rest_framework.permissions import BasePermission + +class IsOrganizationAdmin(BasePermission): + def has_permission(self, request, view): + return ( + request.user.is_authenticated + and request.user.role == User.Role.ADMIN + ) + +class IsOwnerOrReadOnly(BasePermission): + def has_object_permission(self, request, view, obj): + if request.method in permissions.SAFE_METHODS: + return True + return obj.owner == request.user +``` + +#### Pagination + +```python +from rest_framework.pagination import PageNumberPagination, CursorPagination + +class StandardPagination(PageNumberPagination): + page_size = 20 + page_size_query_param = "page_size" + max_page_size = 100 + +class TimelinePagination(CursorPagination): + page_size = 50 + ordering = "-created_at" + +# settings.py +REST_FRAMEWORK = { + "DEFAULT_PAGINATION_CLASS": "myapp.pagination.StandardPagination", + "PAGE_SIZE": 20, + "DEFAULT_PERMISSION_CLASSES": ["rest_framework.permissions.IsAuthenticated"], + "DEFAULT_AUTHENTICATION_CLASSES": [ + "rest_framework_simplejwt.authentication.JWTAuthentication", + ], +} +``` + +### 8. Admin + +```python +from django.contrib import admin +from django.utils.html import format_html + +class TaskInline(admin.TabularInline): + model = Task + extra = 0 + fields = ["title", "assigned_to", "status", "due_date"] + readonly_fields = ["created_at"] + +@admin.register(Project) +class ProjectAdmin(admin.ModelAdmin): + list_display = ["title", "owner_name", "organization", "tag_list", "created_at"] + list_filter = ["organization", "tags", "created_at"] + search_fields = ["title", "description", "owner__email"] + readonly_fields = ["created_at", "updated_at"] + autocomplete_fields = ["owner", "organization"] + prepopulated_fields = {"slug": ("title",)} + date_hierarchy = "created_at" + inlines = [TaskInline] + + fieldsets = ( + (None, { + "fields": ("title", "slug", "description"), + }), + ("Ownership", { + "fields": ("owner", "organization", "tags"), + }), + ("Metadata", { + "classes": ("collapse",), + "fields": ("created_at", "updated_at"), + }), + ) + + def owner_name(self, obj): + return obj.owner.name + owner_name.short_description = "Owner" + owner_name.admin_order_field = "owner__name" + + def tag_list(self, obj): + return ", ".join(t.name for t in obj.tags.all()) + tag_list.short_description = "Tags" + + def get_queryset(self, request): + return super().get_queryset(request).select_related( + "owner", "organization" + ).prefetch_related("tags") + + # Custom admin actions + @admin.action(description="Mark selected projects as published") + def make_published(self, request, queryset): + count = queryset.update(is_published=True) + self.message_user(request, f"{count} projects published.") + + actions = [make_published] + +@admin.register(User) +class UserAdmin(admin.ModelAdmin): + list_display = ["email", "name", "organization", "role", "is_active"] + list_filter = ["role", "is_active", "organization"] + search_fields = ["email", "name"] + list_editable = ["role", "is_active"] + list_per_page = 50 +``` + +--- + ## Best Practices -1. Use class-based views for standard CRUD -2. Define model methods for business logic -3. Use serializers for validation -4. Add proper permissions -5. Use select_related/prefetch_related for queries +1. **Use `select_related` and `prefetch_related` on every query that touches relations** — `select_related` for ForeignKey/OneToOne (SQL JOIN), `prefetch_related` for ManyToMany and reverse ForeignKey (separate query). Check queries with `django-debug-toolbar`. + +2. **Keep business logic in model methods or service functions, not in views** — views should handle HTTP, forms should handle validation, models/services should handle domain logic. This makes code testable without needing HTTP. + +3. **Use `get_queryset()` for dynamic filtering instead of hardcoding querysets** — both in views and DRF ViewSets. This enables mixin composition and per-request filtering (e.g., by organization). + +4. **Write data migrations for schema changes that require backfills** — never assume fields can be added as non-nullable without a migration to populate existing rows. Use `RunPython` with a reverse function. + +5. **Configure Django REST Framework defaults in settings** — set `DEFAULT_PAGINATION_CLASS`, `DEFAULT_PERMISSION_CLASSES`, `DEFAULT_AUTHENTICATION_CLASSES` in `REST_FRAMEWORK` dict to avoid repeating yourself on each ViewSet. + +6. **Use `TextChoices` / `IntegerChoices` for enum fields** — they integrate with admin filters, serializer validation, and migrations automatically. Avoid plain strings or integers for status/role fields. + +7. **Index frequently queried fields** — add `db_index=True` on individual fields or use `Meta.indexes` for composite indexes. Add `UniqueConstraint` for business-rule uniqueness. + +8. **Use Django's `transaction.atomic()` for multi-step writes** — wrap create/update sequences that must succeed or fail together. DRF's `perform_create` and `perform_update` are good places for this. + +```python +from django.db import transaction + +@transaction.atomic +def transfer_project(project, new_owner): + old_owner = project.owner + project.owner = new_owner + project.save(update_fields=["owner"]) + AuditLog.objects.create( + action="transfer", + project=project, + from_user=old_owner, + to_user=new_owner, + ) +``` + +--- ## Common Pitfalls -- **N+1 queries**: Use select_related/prefetch_related -- **Missing migrations**: Run makemigrations -- **No validation**: Use serializers properly +1. **N+1 queries** — accessing `project.owner.name` in a loop without `select_related("owner")` fires one query per iteration. Use `django-debug-toolbar` or `nplusone` to detect these. Always optimize queryset in `get_queryset()`. + +2. **Importing models directly in data migrations** — models change over time, but migrations are frozen. Always use `apps.get_model("app_name", "ModelName")` inside `RunPython` functions, never `from myapp.models import Model`. + +3. **Forgetting to call `full_clean()` in model saves** — Django's `save()` does NOT run validators by default. Only forms and serializers call `full_clean()`. If you save models directly, add explicit validation. + +4. **Circular imports between apps** — referencing models across apps can cause import cycles. Use string references in ForeignKey: `models.ForeignKey("other_app.ModelName", ...)` instead of importing the class. + +5. **Overusing signals for core logic** — signals make code harder to trace and debug. Use them for side effects (sending emails, cache invalidation), not for core domain logic. If logic should always run on save, override `save()` instead. + +6. **Returning entire QuerySets from service functions** — QuerySets are lazy, which is usually good, but returning them from service layers can lead to unexpected queries executing in templates. Use `.values()`, `.values_list()`, or serialize to dicts when crossing layer boundaries. + +--- + +## Related Skills + +- `languages/python` — Python language patterns and best practices +- `databases/postgresql` — Database integration and query optimization +- `testing/pytest` — Testing Django applications with pytest-django diff --git a/.claude/skills/frameworks/django/references/django-patterns.md b/.claude/skills/frameworks/django/references/django-patterns.md new file mode 100644 index 0000000..aa72a05 --- /dev/null +++ b/.claude/skills/frameworks/django/references/django-patterns.md @@ -0,0 +1,250 @@ +# Django Patterns Quick Reference + +## QuerySet Patterns + +### select_related (FK/OneToOne - single JOIN) + +```python +# BAD: N+1 queries +for order in Order.objects.all(): + print(order.customer.name) # Hits DB each iteration + +# GOOD: 1 query with JOIN +for order in Order.objects.select_related("customer"): + print(order.customer.name) + +# Chain through multiple FKs +Order.objects.select_related("customer__company") +``` + +### prefetch_related (M2M/reverse FK - separate query) + +```python +# BAD: N+1 on reverse FK +for author in Author.objects.all(): + print(author.book_set.all()) # Query per author + +# GOOD: 2 queries total +for author in Author.objects.prefetch_related("books"): + print(author.books.all()) + +# Custom prefetch with filtering +from django.db.models import Prefetch + +Author.objects.prefetch_related( + Prefetch("books", queryset=Book.objects.filter(published=True), to_attr="published_books") +) +``` + +### F Objects (reference model fields in queries) + +```python +from django.db.models import F + +Product.objects.filter(stock__lt=F("reorder_level")) # Compare fields +Product.objects.filter(id=1).update(stock=F("stock") - 1) # Atomic update +Order.objects.filter(amount__gt=F("customer__credit_limit")) # Across relations +``` + +### Q Objects (complex lookups with OR/NOT) + +```python +from django.db.models import Q + +# OR +User.objects.filter(Q(role="admin") | Q(is_superuser=True)) + +# NOT +User.objects.filter(~Q(status="banned")) + +# Complex combinations +User.objects.filter( + (Q(role="admin") | Q(role="staff")) & ~Q(status="inactive") +) + +# Dynamic query building +conditions = Q() +if name: conditions &= Q(name__icontains=name) +if email: conditions &= Q(email__icontains=email) +User.objects.filter(conditions) +``` + +### Subquery and OuterRef + +```python +from django.db.models import Subquery, OuterRef, Exists + +# Subquery: latest order date per customer +latest_order = Order.objects.filter( + customer=OuterRef("pk") +).order_by("-created_at").values("created_at")[:1] + +Customer.objects.annotate(last_order=Subquery(latest_order)) + +# Exists: customers with orders +Customer.objects.annotate( + has_orders=Exists(Order.objects.filter(customer=OuterRef("pk"))) +).filter(has_orders=True) +``` + +### Aggregation + +```python +from django.db.models import Count, Sum, Avg + +# Aggregate (returns dict) +Order.objects.aggregate(total=Sum("amount"), avg=Avg("amount")) + +# Annotate (per-row) +Customer.objects.annotate(order_count=Count("orders")) +``` + +--- + +## Model Patterns + +### Abstract Base Model + +```python +class TimestampMixin(models.Model): + created_at = models.DateTimeField(auto_now_add=True) + updated_at = models.DateTimeField(auto_now=True) + + class Meta: + abstract = True # No DB table created + +class Order(TimestampMixin): + amount = models.DecimalField(max_digits=10, decimal_places=2) + # Inherits created_at, updated_at +``` + +### Proxy Model (same table, different behavior) + +```python +class PendingOrderManager(models.Manager): + def get_queryset(self): + return super().get_queryset().filter(status="pending") + +class PendingOrder(Order): + objects = PendingOrderManager() + class Meta: + proxy = True # Same DB table as Order +``` + +### Custom Manager and QuerySet + +```python +class PublishedQuerySet(models.QuerySet): + def published(self): + return self.filter(status="published") + + def by_author(self, author): + return self.filter(author=author) + +class Article(models.Model): + objects = PublishedQuerySet.as_manager() + +# Chainable: Article.objects.published().by_author(user) +``` + +--- + +## View Patterns + +### Class-Based View Mixins + +| Mixin | Purpose | +|-------|---------| +| `LoginRequiredMixin` | Require authentication | +| `PermissionRequiredMixin` | Require specific permission | +| `UserPassesTestMixin` | Custom permission test | +| `FormView` | Handle form display + submission | +| `CreateView` / `UpdateView` | Model form CRUD | +| `ListView` / `DetailView` | Read operations | + +```python +class OrderListView(LoginRequiredMixin, PermissionRequiredMixin, ListView): + model = Order + permission_required = "orders.view_order" + paginate_by = 25 + + def get_queryset(self): + return super().get_queryset().filter(customer=self.request.user) +``` + +--- + +## Django REST Framework Patterns + +### Nested Serializers + +```python +class AddressSerializer(serializers.ModelSerializer): + class Meta: + model = Address + fields = ["street", "city", "zip_code"] + +class CustomerSerializer(serializers.ModelSerializer): + address = AddressSerializer() + + class Meta: + model = Customer + fields = ["id", "name", "address"] + + def create(self, validated_data): + address_data = validated_data.pop("address") + address = Address.objects.create(**address_data) + return Customer.objects.create(address=address, **validated_data) +``` + +### Custom Permissions + +```python +from rest_framework.permissions import BasePermission + +class IsOwner(BasePermission): + def has_object_permission(self, request, view, obj): + return obj.owner == request.user + +class IsAdminOrReadOnly(BasePermission): + def has_permission(self, request, view): + if request.method in ("GET", "HEAD", "OPTIONS"): + return True + return request.user.is_staff + +# Usage +class OrderViewSet(viewsets.ModelViewSet): + permission_classes = [IsAuthenticated, IsOwner] +``` + +### ViewSet Actions + +```python +from rest_framework.decorators import action + +class OrderViewSet(viewsets.ModelViewSet): + queryset = Order.objects.all() + serializer_class = OrderSerializer + + @action(detail=True, methods=["post"]) + def cancel(self, request, pk=None): + order = self.get_object() + order.cancel() + return Response({"status": "cancelled"}) + + @action(detail=False, methods=["get"]) + def summary(self, request): + return Response(self.get_queryset().aggregate(total=Sum("amount"), count=Count("id"))) +``` + +### Filtering (django-filter) + +```python +class OrderFilter(django_filters.FilterSet): + min_amount = django_filters.NumberFilter(field_name="amount", lookup_expr="gte") + max_amount = django_filters.NumberFilter(field_name="amount", lookup_expr="lte") + + class Meta: + model = Order + fields = ["status", "customer"] +``` diff --git a/.claude/skills/frameworks/fastapi/SKILL.md b/.claude/skills/frameworks/fastapi/SKILL.md index fc355cd..b099a60 100644 --- a/.claude/skills/frameworks/fastapi/SKILL.md +++ b/.claude/skills/frameworks/fastapi/SKILL.md @@ -1,89 +1,680 @@ +--- +name: fastapi +description: > + Use this skill when building REST APIs with Python and FastAPI, creating async web applications, or generating OpenAPI/Swagger documentation. Trigger for any mention of FastAPI, Pydantic models, async Python endpoints, dependency injection in Python APIs, or APIRouter patterns. Also applies when setting up Python microservices, adding request validation with Pydantic, or configuring ASGI applications. +--- + # FastAPI -## Description - -FastAPI web framework with async patterns, Pydantic validation, and OpenAPI documentation. - ## When to Use - Building REST APIs with Python - Async web applications - OpenAPI/Swagger documentation needed +- Python microservices +- WebSocket real-time applications + +## When NOT to Use + +- Django projects — use the `frameworks/django` skill instead +- JavaScript/Node.js backends (Express, NestJS) — this skill is Python-only +- Non-API applications such as CLI tools, desktop apps, or batch processing scripts --- ## Core Patterns -### Route Definition +### 1. Project Structure + +Recommended layout for medium-large FastAPI applications: + +``` +project/ +├── app/ +│ ├── __init__.py +│ ├── main.py # FastAPI app creation, startup/shutdown +│ ├── config.py # Settings via pydantic-settings +│ ├── dependencies.py # Shared dependencies +│ ├── exceptions.py # Custom exception handlers +│ ├── middleware.py # Custom middleware +│ ├── api/ +│ │ ├── __init__.py +│ │ ├── router.py # Root router aggregating all sub-routers +│ │ ├── v1/ +│ │ │ ├── __init__.py +│ │ │ ├── users.py # /api/v1/users endpoints +│ │ │ ├── items.py # /api/v1/items endpoints +│ │ │ └── auth.py # /api/v1/auth endpoints +│ │ └── v2/ # Future API version +│ ├── models/ +│ │ ├── __init__.py +│ │ ├── user.py # SQLAlchemy / SQLModel ORM models +│ │ └── item.py +│ ├── schemas/ +│ │ ├── __init__.py +│ │ ├── user.py # Pydantic request/response schemas +│ │ └── item.py +│ ├── services/ +│ │ ├── __init__.py +│ │ ├── user_service.py # Business logic layer +│ │ └── item_service.py +│ ├── repositories/ +│ │ ├── __init__.py +│ │ ├── user_repo.py # Data access layer +│ │ └── item_repo.py +│ ├── core/ +│ │ ├── __init__.py +│ │ ├── database.py # DB engine, session factory +│ │ └── security.py # JWT, hashing, auth utils +│ └── tests/ +│ ├── __init__.py +│ ├── conftest.py # Fixtures: test client, test DB +│ ├── test_users.py +│ └── test_items.py +├── alembic/ # Database migrations +│ ├── env.py +│ └── versions/ +├── alembic.ini +├── pyproject.toml +├── Dockerfile +└── .env +``` + +**Key conventions:** +- Separate `schemas/` (Pydantic) from `models/` (ORM) to keep concerns clean +- Use `services/` for business logic, `repositories/` for data access +- Version API routes under `api/v1/`, `api/v2/` for backward compatibility +- Keep `main.py` thin — it only wires things together ```python -from fastapi import FastAPI, HTTPException, Depends -from pydantic import BaseModel +# app/main.py +from contextlib import asynccontextmanager +from fastapi import FastAPI +from app.api.router import api_router +from app.config import settings +from app.core.database import engine +from app.middleware import add_middleware -app = FastAPI() +@asynccontextmanager +async def lifespan(app: FastAPI): + # Startup: create tables, warm caches, connect to services + async with engine.begin() as conn: + await conn.run_sync(Base.metadata.create_all) + yield + # Shutdown: close connections, flush buffers + await engine.dispose() -class UserCreate(BaseModel): - email: str - name: str +app = FastAPI( + title=settings.PROJECT_NAME, + version=settings.VERSION, + lifespan=lifespan, +) +add_middleware(app) +app.include_router(api_router, prefix="/api") +``` -class UserResponse(BaseModel): - id: int - email: str - name: str +### 2. Route Patterns -@app.post("/users", response_model=UserResponse, status_code=201) -async def create_user(user: UserCreate): - # Create user logic - return UserResponse(id=1, **user.model_dump()) +#### APIRouter with tags, prefixes, and dependencies -@app.get("/users/{user_id}", response_model=UserResponse) -async def get_user(user_id: int): - user = await get_user_by_id(user_id) +```python +from fastapi import APIRouter, Depends, Query, Path, Body, HTTPException, status +from app.schemas.user import UserCreate, UserResponse, UserUpdate, UserList +from app.dependencies import get_current_user + +router = APIRouter( + prefix="/users", + tags=["users"], + dependencies=[Depends(get_current_user)], # Applied to all routes + responses={401: {"description": "Not authenticated"}}, +) + +# Path parameters with validation +@router.get("/{user_id}", response_model=UserResponse) +async def get_user( + user_id: int = Path(..., gt=0, description="The ID of the user to retrieve"), +): + user = await user_service.get(user_id) if not user: raise HTTPException(status_code=404, detail="User not found") return user + +# Query parameters with defaults and validation +@router.get("/", response_model=UserList) +async def list_users( + skip: int = Query(0, ge=0, description="Number of records to skip"), + limit: int = Query(20, ge=1, le=100, description="Max records to return"), + search: str | None = Query(None, min_length=1, max_length=100), + sort_by: str = Query("created_at", pattern="^(created_at|name|email)$"), +): + users = await user_service.list(skip=skip, limit=limit, search=search) + return users + +# Request body with status codes +@router.post( + "/", + response_model=UserResponse, + status_code=status.HTTP_201_CREATED, + summary="Create a new user", + description="Creates a user account and sends a welcome email.", +) +async def create_user(user: UserCreate = Body(...)): + return await user_service.create(user) + +# Multiple response models for different status codes +@router.put("/{user_id}", response_model=UserResponse, responses={ + 404: {"description": "User not found"}, + 409: {"description": "Email already taken"}, +}) +async def update_user(user_id: int, user: UserUpdate): + return await user_service.update(user_id, user) + +@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT) +async def delete_user(user_id: int): + await user_service.delete(user_id) ``` -### Dependency Injection +#### Router aggregation ```python -from fastapi import Depends -from sqlalchemy.ext.asyncio import AsyncSession +# app/api/router.py +from fastapi import APIRouter +from app.api.v1 import users, items, auth +api_router = APIRouter() +api_router.include_router(auth.router, prefix="/v1") +api_router.include_router(users.router, prefix="/v1") +api_router.include_router(items.router, prefix="/v1") +``` + +### 3. Dependency Injection + +#### Basic dependency with Depends() + +```python +from fastapi import Depends, Header, HTTPException + +async def verify_api_key(x_api_key: str = Header(...)): + if x_api_key != settings.API_KEY: + raise HTTPException(status_code=403, detail="Invalid API key") + return x_api_key +``` + +#### Nested dependencies + +```python async def get_db() -> AsyncGenerator[AsyncSession, None]: async with async_session_maker() as session: - yield session + yield session # yield dependency — cleanup runs after response -@app.get("/users") -async def list_users(db: AsyncSession = Depends(get_db)): - return await db.execute(select(User)) +async def get_user_repo(db: AsyncSession = Depends(get_db)) -> UserRepository: + return UserRepository(db) + +async def get_user_service( + repo: UserRepository = Depends(get_user_repo), +) -> UserService: + return UserService(repo) + +@router.get("/users") +async def list_users(service: UserService = Depends(get_user_service)): + return await service.list_all() ``` -### Router Organization +#### Yield dependencies for cleanup ```python -from fastapi import APIRouter +async def get_redis() -> AsyncGenerator[Redis, None]: + redis = await aioredis.from_url(settings.REDIS_URL) + try: + yield redis + finally: + await redis.close() # Always runs, even on exceptions -router = APIRouter(prefix="/users", tags=["users"]) - -@router.get("/") -async def list_users(): - pass - -# In main.py -app.include_router(router) +async def get_http_client() -> AsyncGenerator[httpx.AsyncClient, None]: + async with httpx.AsyncClient(timeout=30.0) as client: + yield client ``` +#### Request-scoped dependencies with caching + +```python +# FastAPI caches dependency results per-request by default. +# The same db session is reused if multiple deps request it. + +@router.get("/dashboard") +async def dashboard( + user_service: UserService = Depends(get_user_service), + item_service: ItemService = Depends(get_item_service), + # Both services share the same db session from get_db() +): + users = await user_service.count() + items = await item_service.count() + return {"users": users, "items": items} + +# To disable caching (get fresh instance each time): +@router.get("/example") +async def example( + db1: AsyncSession = Depends(get_db), + db2: AsyncSession = Depends(get_db, use_cache=False), + # db1 and db2 are different sessions +): + pass +``` + +#### Class-based dependencies + +```python +class Pagination: + def __init__( + self, + skip: int = Query(0, ge=0), + limit: int = Query(20, ge=1, le=100), + ): + self.skip = skip + self.limit = limit + +@router.get("/items") +async def list_items(pagination: Pagination = Depends()): + # pagination.skip, pagination.limit + pass +``` + +### 4. Middleware + +#### Custom middleware + +```python +import time +from fastapi import Request +from starlette.middleware.base import BaseHTTPMiddleware + +class TimingMiddleware(BaseHTTPMiddleware): + async def dispatch(self, request: Request, call_next): + start = time.perf_counter() + response = await call_next(request) + duration = time.perf_counter() - start + response.headers["X-Process-Time"] = f"{duration:.4f}" + return response +``` + +#### Pure ASGI middleware (higher performance) + +```python +from starlette.types import ASGIApp, Receive, Scope, Send + +class RequestIDMiddleware: + def __init__(self, app: ASGIApp): + self.app = app + + async def __call__(self, scope: Scope, receive: Receive, send: Send): + if scope["type"] == "http": + request_id = uuid.uuid4().hex + scope.setdefault("state", {})["request_id"] = request_id + + async def send_with_header(message): + if message["type"] == "http.response.start": + headers = dict(message.get("headers", [])) + headers[b"x-request-id"] = request_id.encode() + message["headers"] = list(headers.items()) + await send(message) + + await self.app(scope, receive, send_with_header) + else: + await self.app(scope, receive, send) +``` + +#### Standard middleware configuration + +```python +from fastapi.middleware.cors import CORSMiddleware +from fastapi.middleware.trustedhost import TrustedHostMiddleware +from fastapi.middleware.gzip import GZipMiddleware + +def add_middleware(app: FastAPI): + # Order matters: first added = outermost (runs first on request, last on response) + + app.add_middleware( + CORSMiddleware, + allow_origins=settings.CORS_ORIGINS, # ["https://example.com"] + allow_credentials=True, + allow_methods=["GET", "POST", "PUT", "DELETE", "PATCH"], + allow_headers=["*"], + ) + + app.add_middleware( + TrustedHostMiddleware, + allowed_hosts=settings.ALLOWED_HOSTS, # ["example.com", "*.example.com"] + ) + + app.add_middleware(GZipMiddleware, minimum_size=500) # Compress responses > 500 bytes + + app.add_middleware(TimingMiddleware) +``` + +### 5. Background Tasks + +#### Simple background tasks + +```python +from fastapi import BackgroundTasks + +async def send_welcome_email(email: str, name: str): + # This runs after the response is sent + await email_service.send( + to=email, + subject="Welcome!", + body=f"Hello {name}, welcome to our platform.", + ) + +async def log_activity(user_id: int, action: str): + await activity_repo.create(user_id=user_id, action=action) + +@router.post("/users", status_code=201) +async def create_user( + user: UserCreate, + background_tasks: BackgroundTasks, +): + new_user = await user_service.create(user) + # Queue multiple background tasks + background_tasks.add_task(send_welcome_email, new_user.email, new_user.name) + background_tasks.add_task(log_activity, new_user.id, "account_created") + return new_user +``` + +#### Long-running tasks with task queues + +For tasks that take more than a few seconds, use a proper task queue: + +```python +from celery import Celery + +celery_app = Celery("worker", broker=settings.CELERY_BROKER_URL) + +@celery_app.task +def generate_report(report_id: int): + # Long-running: query data, build PDF, upload to S3 + ... + +@router.post("/reports", status_code=202) +async def request_report(params: ReportRequest): + report = await report_service.create(params) + generate_report.delay(report.id) # Dispatch to Celery worker + return {"report_id": report.id, "status": "processing"} + +@router.get("/reports/{report_id}/status") +async def report_status(report_id: int): + report = await report_service.get(report_id) + return {"status": report.status, "url": report.download_url} +``` + +### 6. WebSocket + +#### WebSocket endpoint with connection management + +```python +from fastapi import WebSocket, WebSocketDisconnect + +class ConnectionManager: + def __init__(self): + self.active_connections: dict[str, list[WebSocket]] = {} + + async def connect(self, websocket: WebSocket, room: str): + await websocket.accept() + self.active_connections.setdefault(room, []).append(websocket) + + def disconnect(self, websocket: WebSocket, room: str): + self.active_connections.get(room, []).remove(websocket) + + async def broadcast(self, message: str, room: str): + for connection in self.active_connections.get(room, []): + try: + await connection.send_text(message) + except WebSocketDisconnect: + self.disconnect(connection, room) + +manager = ConnectionManager() + +@app.websocket("/ws/{room}") +async def websocket_endpoint(websocket: WebSocket, room: str): + await manager.connect(websocket, room) + try: + while True: + data = await websocket.receive_text() + await manager.broadcast(f"Message: {data}", room) + except WebSocketDisconnect: + manager.disconnect(websocket, room) + await manager.broadcast(f"User left the room", room) +``` + +#### WebSocket with authentication + +```python +@app.websocket("/ws/private") +async def private_ws(websocket: WebSocket, token: str = Query(...)): + try: + user = verify_token(token) + except InvalidToken: + await websocket.close(code=4001, reason="Invalid token") + return + + await websocket.accept() + try: + while True: + data = await websocket.receive_json() + response = await process_message(user, data) + await websocket.send_json(response) + except WebSocketDisconnect: + pass +``` + +### 7. File Handling + +#### Upload files + +```python +from fastapi import UploadFile, File + +@router.post("/upload") +async def upload_file( + file: UploadFile = File(..., description="File to upload"), +): + # Validate file type and size + if file.content_type not in ["image/png", "image/jpeg"]: + raise HTTPException(400, "Only PNG and JPEG images are allowed") + + if file.size and file.size > 5 * 1024 * 1024: # 5 MB + raise HTTPException(400, "File too large (max 5 MB)") + + contents = await file.read() + path = f"uploads/{uuid.uuid4()}_{file.filename}" + async with aiofiles.open(path, "wb") as f: + await f.write(contents) + + return {"filename": file.filename, "path": path, "size": len(contents)} + +# Multiple file upload +@router.post("/upload-multiple") +async def upload_multiple(files: list[UploadFile] = File(...)): + results = [] + for file in files: + contents = await file.read() + results.append({"filename": file.filename, "size": len(contents)}) + return results +``` + +#### Streaming responses + +```python +from fastapi.responses import StreamingResponse +import csv +import io + +@router.get("/export/users") +async def export_users(): + async def generate_csv(): + output = io.StringIO() + writer = csv.writer(output) + writer.writerow(["id", "name", "email"]) + yield output.getvalue() + output.seek(0) + output.truncate(0) + + async for user in user_service.stream_all(): + writer.writerow([user.id, user.name, user.email]) + yield output.getvalue() + output.seek(0) + output.truncate(0) + + return StreamingResponse( + generate_csv(), + media_type="text/csv", + headers={"Content-Disposition": "attachment; filename=users.csv"}, + ) +``` + +#### Static files + +```python +from fastapi.staticfiles import StaticFiles + +app.mount("/static", StaticFiles(directory="static"), name="static") +``` + +### 8. Testing + +#### TestClient for synchronous tests + +```python +from fastapi.testclient import TestClient +from app.main import app + +client = TestClient(app) + +def test_create_user(): + response = client.post("/api/v1/users", json={ + "email": "test@example.com", + "name": "Test User", + }) + assert response.status_code == 201 + data = response.json() + assert data["email"] == "test@example.com" + +def test_get_user_not_found(): + response = client.get("/api/v1/users/99999") + assert response.status_code == 404 +``` + +#### Async testing with httpx + +```python +import pytest +from httpx import AsyncClient, ASGITransport +from app.main import app + +@pytest.fixture +async def async_client(): + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + yield client + +@pytest.mark.anyio +async def test_list_users(async_client: AsyncClient): + response = await async_client.get("/api/v1/users") + assert response.status_code == 200 + assert isinstance(response.json(), list) +``` + +#### Overriding dependencies for tests + +```python +from app.dependencies import get_db, get_current_user +from app.models.user import User + +# Mock database session +async def override_get_db(): + async with test_session_maker() as session: + yield session + +# Mock authenticated user +async def override_get_current_user(): + return User(id=1, email="test@example.com", name="Test") + +app.dependency_overrides[get_db] = override_get_db +app.dependency_overrides[get_current_user] = override_get_current_user + +# In conftest.py — clean up after tests +@pytest.fixture(autouse=True) +def clear_overrides(): + yield + app.dependency_overrides.clear() +``` + +#### Testing WebSocket endpoints + +```python +def test_websocket(): + with client.websocket_connect("/ws/test-room") as ws: + ws.send_text("hello") + data = ws.receive_text() + assert "hello" in data +``` + +--- + ## Best Practices -1. Use Pydantic models for request/response validation -2. Organize routes with APIRouter -3. Use dependency injection for services -4. Return proper HTTP status codes -5. Add OpenAPI descriptions +1. **Use Pydantic models for all request/response validation** — never pass raw dicts through your API boundary. Define separate `Create`, `Update`, and `Response` schemas for each resource. + +2. **Organize routes with APIRouter** — group related endpoints by resource and version. Apply shared dependencies at the router level, not on each individual route. + +3. **Separate business logic from routes** — route functions should only handle HTTP concerns (parsing request, returning response). Delegate logic to service classes injected via `Depends()`. + +4. **Use the lifespan context manager** — replace deprecated `on_event("startup")` and `on_event("shutdown")` with the `lifespan` async context manager for resource setup and teardown. + +5. **Return proper HTTP status codes** — 201 for creation, 204 for deletion, 202 for accepted-but-not-done, 409 for conflicts. Use `status_code` parameter on route decorators. + +6. **Add OpenAPI metadata** — provide `summary`, `description`, `tags`, and `responses` on routes. Set `title`, `version`, and `description` on the FastAPI app. This generates high-quality auto-docs. + +7. **Use async all the way down** — if your route is `async def`, every I/O call inside it must also be async. Mixing sync blocking calls (e.g., `requests.get()`) in an async route will block the event loop. + +8. **Configure settings with pydantic-settings** — load config from environment variables with validation and type coercion. Never hardcode secrets or connection strings. + +```python +from pydantic_settings import BaseSettings + +class Settings(BaseSettings): + DATABASE_URL: str + API_KEY: str + DEBUG: bool = False + + model_config = {"env_file": ".env"} + +settings = Settings() +``` + +--- ## Common Pitfalls -- **Blocking I/O in async**: Use async libraries -- **Missing response models**: Always define them -- **No error handling**: Use HTTPException properly +1. **Blocking I/O in async routes** — calling `requests.get()`, `time.sleep()`, or synchronous DB drivers inside `async def` routes starves the event loop. Use `httpx`, `asyncio.sleep()`, and async database drivers instead. If you must call sync code, use `run_in_executor`. + +2. **Missing response_model** — without `response_model`, FastAPI returns whatever you return, potentially leaking internal fields (passwords, internal IDs). Always define a Pydantic response schema. + +3. **Forgetting to await coroutines** — calling `await db.execute(query)` vs `db.execute(query)` is easy to miss. The latter returns a coroutine object instead of results. Enable linting rules that catch unawaited coroutines. + +4. **Circular imports between models and schemas** — when schemas reference ORM models and vice versa, you get import cycles. Fix by using `TYPE_CHECKING` imports or by keeping schemas and models in separate modules that do not import each other. + +5. **Not handling Pydantic validation errors** — FastAPI returns 422 by default, but the error format may confuse API consumers. Add a custom exception handler to reshape validation error responses to match your API's error format. + +6. **Sharing mutable state across requests without locks** — global mutable variables (lists, dicts) accessed from async routes can cause race conditions. Use async-safe structures or dependency-injected per-request state. + +--- + +## Related Skills + +- `languages/python` — Python language patterns and best practices +- `api/openapi` — OpenAPI specification and documentation standards +- `databases/postgresql` — Database integration with async SQLAlchemy +- `testing/pytest` — Testing FastAPI applications with pytest and httpx +- `patterns/authentication` — JWT, OAuth2, and session patterns for FastAPI endpoints +- `patterns/logging` — Structured logging for FastAPI applications diff --git a/.claude/skills/frameworks/fastapi/references/fastapi-project-structure.md b/.claude/skills/frameworks/fastapi/references/fastapi-project-structure.md new file mode 100644 index 0000000..923de12 --- /dev/null +++ b/.claude/skills/frameworks/fastapi/references/fastapi-project-structure.md @@ -0,0 +1,229 @@ +# FastAPI Project Structure Reference + +## Small Project (1-5 endpoints, single module) + +``` +project/ +├── main.py # App factory, routes, startup +├── models.py # Pydantic schemas + SQLAlchemy models +├── database.py # DB connection, session factory +├── config.py # Settings via pydantic-settings +├── requirements.txt +├── .env +└── tests/ + ├── conftest.py # Fixtures (test client, test DB) + └── test_main.py +``` + +**When to use**: Prototypes, microservices, internal tools, single-domain APIs. + +**`main.py` structure**: +```python +from fastapi import FastAPI +from contextlib import asynccontextmanager + +@asynccontextmanager +async def lifespan(app: FastAPI): + # startup + yield + # shutdown + +app = FastAPI(lifespan=lifespan) + +@app.get("/health") +async def health(): return {"status": "ok"} +``` + +--- + +## Medium Project (5-20 endpoints, feature-grouped) + +``` +project/ +├── app/ +│ ├── __init__.py +│ ├── main.py # App factory, include routers +│ ├── config.py # Settings (pydantic-settings) +│ ├── database.py # Engine, SessionLocal, Base +│ ├── dependencies.py # Shared deps (get_db, get_current_user) +│ ├── exceptions.py # Custom exception handlers +│ ├── middleware.py # CORS, logging, timing middleware +│ │ +│ ├── auth/ +│ │ ├── __init__.py +│ │ ├── router.py # POST /login, POST /register +│ │ ├── schemas.py # LoginRequest, TokenResponse +│ │ ├── models.py # User SQLAlchemy model +│ │ ├── service.py # Business logic (hash, verify, tokens) +│ │ └── dependencies.py # get_current_user, require_role +│ │ +│ ├── items/ +│ │ ├── __init__.py +│ │ ├── router.py # CRUD endpoints +│ │ ├── schemas.py # ItemCreate, ItemRead, ItemUpdate +│ │ ├── models.py # Item SQLAlchemy model +│ │ └── service.py # Business logic +│ │ +│ └── shared/ +│ ├── __init__.py +│ ├── pagination.py # Pagination params + response schema +│ └── filters.py # Common query filter patterns +│ +├── alembic/ # DB migrations +│ ├── env.py +│ └── versions/ +├── alembic.ini +├── requirements.txt +├── pyproject.toml +├── Dockerfile +├── docker-compose.yml +└── tests/ + ├── conftest.py + ├── auth/ + │ └── test_router.py + └── items/ + ├── test_router.py + └── test_service.py +``` + +**When to use**: Multi-feature APIs, team projects, typical SaaS backends. + +**Key patterns**: +- Each feature gets its own directory with router, schemas, models, service +- `router.py` uses `APIRouter(prefix="/items", tags=["items"])` +- `main.py` includes routers: `app.include_router(items.router)` +- Shared deps in root `dependencies.py`, feature-specific in feature dir + +--- + +## Large Project (20+ endpoints, domain-driven) + +``` +project/ +├── src/ +│ ├── __init__.py +│ ├── main.py # App factory only +│ ├── config.py # Layered settings +│ │ +│ ├── core/ # Framework-level concerns +│ │ ├── __init__.py +│ │ ├── database.py # Engine, session management +│ │ ├── security.py # JWT, hashing, RBAC +│ │ ├── exceptions.py # Base exceptions + handlers +│ │ ├── middleware.py # All middleware stack +│ │ ├── dependencies.py # Cross-cutting deps +│ │ ├── events.py # Domain event bus +│ │ └── pagination.py # Cursor + offset pagination +│ │ +│ ├── domain/ # Business logic (framework-agnostic) +│ │ ├── users/ +│ │ │ ├── __init__.py +│ │ │ ├── entity.py # Domain entity (plain dataclass) +│ │ │ ├── repository.py # Abstract repository interface +│ │ │ ├── service.py # Business rules +│ │ │ └── events.py # Domain events +│ │ ├── orders/ +│ │ │ └── ... +│ │ └── payments/ +│ │ └── ... +│ │ +│ ├── infrastructure/ # External system adapters +│ │ ├── database/ +│ │ │ ├── models.py # All SQLAlchemy models +│ │ │ ├── repositories/ # Concrete repo implementations +│ │ │ │ ├── user_repo.py +│ │ │ │ └── order_repo.py +│ │ │ └── migrations/ # Alembic +│ │ ├── cache/ +│ │ │ └── redis_client.py +│ │ ├── email/ +│ │ │ └── smtp_service.py +│ │ └── external/ +│ │ └── stripe_client.py +│ │ +│ └── api/ # HTTP layer only +│ ├── __init__.py +│ ├── v1/ +│ │ ├── __init__.py # v1 router aggregator +│ │ ├── users.py # Thin: parse request -> call service -> format response +│ │ ├── orders.py +│ │ └── payments.py +│ ├── v2/ +│ │ └── ... +│ ├── schemas/ # Request/response schemas +│ │ ├── user_schemas.py +│ │ ├── order_schemas.py +│ │ └── common.py +│ ├── dependencies.py # API-layer deps +│ └── websockets/ +│ └── notifications.py +│ +├── tests/ +│ ├── conftest.py +│ ├── unit/ +│ │ ├── domain/ +│ │ │ └── test_user_service.py +│ │ └── ... +│ ├── integration/ +│ │ ├── test_user_api.py +│ │ └── test_order_flow.py +│ └── e2e/ +│ └── test_checkout.py +│ +├── scripts/ # Dev/ops scripts +│ ├── seed_db.py +│ └── migrate.py +├── pyproject.toml +├── Dockerfile +├── docker-compose.yml +└── Makefile +``` + +**When to use**: Complex domains, multiple teams, long-lived products. + +**Key patterns**: +- **Domain layer** has zero framework imports (testable in isolation) +- **Infrastructure** implements domain interfaces (repository pattern) +- **API layer** is thin: validation, auth, call service, return schema +- API versioning via `/api/v1/`, `/api/v2/` +- Separate unit, integration, and e2e test directories + +--- + +## File Responsibilities + +| File | Responsibility | Dependencies | +|------|---------------|-------------| +| `router.py` | HTTP handling, request parsing, response formatting | schemas, service, dependencies | +| `schemas.py` | Pydantic models for request/response validation | None (or shared schemas) | +| `models.py` | SQLAlchemy/ODM models (DB table mapping) | database | +| `service.py` | Business logic, orchestration | repository/models, external services | +| `dependencies.py` | FastAPI `Depends()` callables | database, config, auth | +| `exceptions.py` | Custom exceptions + handlers | None | +| `config.py` | `BaseSettings` with env loading | None | + +## Router Registration Pattern + +```python +# app/main.py +from fastapi import FastAPI +from app.auth.router import router as auth_router +from app.items.router import router as items_router + +def create_app() -> FastAPI: + app = FastAPI(title="My API") + app.include_router(auth_router) + app.include_router(items_router) + return app + +app = create_app() +``` + +```python +# app/items/router.py +from fastapi import APIRouter, Depends +router = APIRouter(prefix="/items", tags=["items"]) + +@router.get("/") +async def list_items(db=Depends(get_db)): ... +``` diff --git a/.claude/skills/frameworks/nextjs/SKILL.md b/.claude/skills/frameworks/nextjs/SKILL.md index 9dabbcc..4b87930 100644 --- a/.claude/skills/frameworks/nextjs/SKILL.md +++ b/.claude/skills/frameworks/nextjs/SKILL.md @@ -1,112 +1,692 @@ +--- +name: nextjs +description: > + Use this skill when working with Next.js applications, App Router, Server Components, or Server Actions. Trigger for any mention of Next.js, next/server, next/navigation, route handlers, SSR, SSG, ISR, middleware, or the app/ directory structure. Also applies when building full-stack React applications with API routes, implementing streaming or suspense boundaries, or configuring next.config. +--- + # Next.js -## Description - -Next.js framework with App Router, Server Components, and full-stack development patterns. - ## When to Use - React applications with SSR/SSG - Full-stack applications - App Router patterns +- SEO-critical sites needing server rendering + +## When NOT to Use + +- Pure React SPAs without SSR needs — use the `frameworks/react` skill instead +- Non-React frameworks (Vue, Svelte, Angular) — this skill is React/Next.js specific +- Backend-only projects without a frontend — consider `frameworks/fastapi` or `frameworks/django` --- ## Core Patterns -### App Router Structure +### 1. App Router + +#### Directory structure ``` app/ -├── layout.tsx # Root layout -├── page.tsx # Home page -├── loading.tsx # Loading UI -├── error.tsx # Error UI +├── layout.tsx # Root layout (wraps entire app) +├── page.tsx # Home page (/) +├── loading.tsx # Root loading UI (Suspense fallback) +├── error.tsx # Root error boundary +├── not-found.tsx # Custom 404 page +├── global-error.tsx # Error boundary for root layout itself +├── favicon.ico +├── globals.css ├── api/ -│ └── users/ -│ └── route.ts # API route -└── users/ - ├── page.tsx # Users page - └── [id]/ - └── page.tsx # User detail +│ ├── users/ +│ │ └── route.ts # GET/POST /api/users +│ │ └── [id]/ +│ │ └── route.ts # GET/PUT/DELETE /api/users/:id +│ └── webhooks/ +│ └── stripe/ +│ └── route.ts # POST /api/webhooks/stripe +├── (marketing)/ # Route group (no URL segment) +│ ├── layout.tsx # Layout for marketing pages only +│ ├── page.tsx # / (same as root, can override) +│ ├── about/ +│ │ └── page.tsx # /about +│ └── pricing/ +│ └── page.tsx # /pricing +├── (app)/ # Route group for authenticated app +│ ├── layout.tsx # App shell layout (sidebar, nav) +│ ├── dashboard/ +│ │ ├── page.tsx # /dashboard +│ │ ├── loading.tsx # Loading skeleton for dashboard +│ │ └── error.tsx # Error boundary for dashboard +│ ├── projects/ +│ │ ├── page.tsx # /projects +│ │ └── [id]/ +│ │ ├── page.tsx # /projects/:id +│ │ ├── edit/ +│ │ │ └── page.tsx # /projects/:id/edit +│ │ └── layout.tsx # Shared layout for project detail +│ └── settings/ +│ └── page.tsx # /settings +└── @modal/ # Parallel route slot + └── (.)projects/ + └── [id]/ + └── page.tsx # Intercepted route modal ``` -### Server Components +#### Special files and their roles + +| File | Purpose | Renders when | +|------|---------|-------------| +| `page.tsx` | Route UI | URL matches segment | +| `layout.tsx` | Shared wrapper, preserved across navigation | Always for child routes | +| `loading.tsx` | Suspense fallback | While page/data is loading | +| `error.tsx` | Error boundary | When child throws | +| `not-found.tsx` | 404 UI | When `notFound()` is called | +| `route.ts` | API endpoint | HTTP request to segment | +| `template.tsx` | Like layout but re-mounts on navigation | Every navigation | +| `default.tsx` | Fallback for parallel routes | When slot has no match | ```tsx -// app/users/page.tsx - Server Component (default) -async function UsersPage() { - const users = await db.users.findMany(); +// app/layout.tsx — Root layout (required) +import type { Metadata } from "next"; + +export const metadata: Metadata = { + title: { default: "My App", template: "%s | My App" }, + description: "Application description", +}; + +export default function RootLayout({ children }: { children: React.ReactNode }) { + return ( + + + +
{children}
+ + + ); +} + +// app/error.tsx — Error boundary (must be client component) +"use client"; + +export default function Error({ + error, + reset, +}: { + error: Error & { digest?: string }; + reset: () => void; +}) { + return ( +
+

Something went wrong

+ +
+ ); +} + +// app/not-found.tsx +export default function NotFound() { + return ( +
+

Page not found

+

The requested resource does not exist.

+
+ ); +} +``` + +### 2. Server vs Client Components + +#### Decision guide + +| Use Server Component when | Use Client Component when | +|---------------------------|--------------------------| +| Fetching data | Using useState, useEffect, useRef | +| Accessing backend resources directly | Adding event handlers (onClick, onChange) | +| Keeping sensitive data on server | Using browser APIs (localStorage, window) | +| Reducing client bundle size | Using third-party client libraries | +| SEO-critical content | Animations, real-time updates | + +#### Composition patterns + +```tsx +// Server Component (default — no directive needed) +// app/projects/page.tsx +import { ProjectList } from "./project-list"; +import { SearchBar } from "./search-bar"; // Client component + +export default async function ProjectsPage() { + const projects = await db.project.findMany({ + orderBy: { createdAt: "desc" }, + }); + + return ( +
+

Projects

+ {/* Client component receives server data as props */} + + {/* Server component can render client children */} + +
+ ); +} + +// Client Component — must have "use client" at top +// app/projects/search-bar.tsx +"use client"; + +import { useRouter, useSearchParams } from "next/navigation"; +import { useTransition } from "react"; + +export function SearchBar() { + const router = useRouter(); + const searchParams = useSearchParams(); + const [isPending, startTransition] = useTransition(); + + function handleSearch(term: string) { + const params = new URLSearchParams(searchParams); + if (term) { + params.set("q", term); + } else { + params.delete("q"); + } + startTransition(() => { + router.replace(`/projects?${params.toString()}`); + }); + } + + return ( + handleSearch(e.target.value)} + className={isPending ? "opacity-50" : ""} + /> + ); +} +``` + +**Key rule:** The `"use client"` directive creates a boundary. Everything imported into a client component becomes part of the client bundle. Pass server data down as serializable props (no functions, no classes). + +### 3. Data Fetching + +#### Server component fetch with caching + +```tsx +// Fetch with automatic deduplication and caching +async function getProjects() { + const res = await fetch("https://api.example.com/projects", { + next: { revalidate: 60 }, // Revalidate every 60 seconds (ISR) + // next: { tags: ["projects"] }, // Tag-based revalidation + // cache: "no-store", // Always fresh (SSR) + // cache: "force-cache", // Cache indefinitely (SSG) + }); + if (!res.ok) throw new Error("Failed to fetch projects"); + return res.json(); +} + +export default async function ProjectsPage() { + const projects = await getProjects(); + return ; +} +``` + +#### generateStaticParams for static generation + +```tsx +// app/projects/[id]/page.tsx +export async function generateStaticParams() { + const projects = await db.project.findMany({ select: { id: true } }); + return projects.map((p) => ({ id: String(p.id) })); +} + +export default async function ProjectPage({ + params, +}: { + params: Promise<{ id: string }>; +}) { + const { id } = await params; + const project = await db.project.findUnique({ where: { id } }); + if (!project) notFound(); + return ; +} +``` + +#### Route handlers (API routes) + +```typescript +// app/api/projects/route.ts +import { NextRequest, NextResponse } from "next/server"; + +export async function GET(request: NextRequest) { + const searchParams = request.nextUrl.searchParams; + const page = Number(searchParams.get("page") ?? "1"); + const limit = Number(searchParams.get("limit") ?? "20"); + + const projects = await db.project.findMany({ + skip: (page - 1) * limit, + take: limit, + }); + + return NextResponse.json({ data: projects, page, limit }); +} + +export async function POST(request: NextRequest) { + const body = await request.json(); + const project = await db.project.create({ data: body }); + return NextResponse.json(project, { status: 201 }); +} + +// Dynamic route: app/api/projects/[id]/route.ts +export async function GET( + request: NextRequest, + { params }: { params: Promise<{ id: string }> }, +) { + const { id } = await params; + const project = await db.project.findUnique({ where: { id } }); + if (!project) { + return NextResponse.json({ error: "Not found" }, { status: 404 }); + } + return NextResponse.json(project); +} +``` + +### 4. Server Actions + +#### Form actions + +```tsx +// app/actions.ts +"use server"; + +import { revalidatePath } from "next/cache"; +import { redirect } from "next/navigation"; +import { z } from "zod"; + +const ProjectSchema = z.object({ + title: z.string().min(3).max(200), + description: z.string().optional(), +}); + +export async function createProject(prevState: unknown, formData: FormData) { + const parsed = ProjectSchema.safeParse({ + title: formData.get("title"), + description: formData.get("description"), + }); + + if (!parsed.success) { + return { errors: parsed.error.flatten().fieldErrors }; + } + + const project = await db.project.create({ data: parsed.data }); + revalidatePath("/projects"); + redirect(`/projects/${project.id}`); +} + +export async function deleteProject(id: string) { + await db.project.delete({ where: { id } }); + revalidatePath("/projects"); +} +``` + +#### Using actions in client components with useActionState + +```tsx +"use client"; + +import { useActionState } from "react"; +import { createProject } from "../actions"; + +export function CreateProjectForm() { + const [state, formAction, isPending] = useActionState(createProject, null); + + return ( +
+ + {state?.errors?.title && ( +

{state.errors.title[0]}

+ )} + + + +
+ + +
+
+``` + +--- + +## Modal Overlay + +```html + +
+ +
+
+

Confirm Action

+ +
+

+ Are you sure you want to proceed? This action cannot be undone. +

+
+ + +
+
+
+``` + +--- + +## Sidebar Layout + +```html +
+ + + + +
+
+

Dashboard

+
+ +
+
+
+
+``` + +--- + +## Tips + +- Use `transition` and `hover:` for interactive feedback +- Use `focus-visible:` instead of `focus:` for keyboard-only focus rings +- Use `dark:` variants when supporting dark mode +- Prefer `gap-*` over margin utilities for flex/grid spacing +- Use `max-w-7xl mx-auto px-4 sm:px-6 lg:px-8` as a standard container diff --git a/.claude/skills/languages/javascript/SKILL.md b/.claude/skills/languages/javascript/SKILL.md index 6b9545e..969e49c 100644 --- a/.claude/skills/languages/javascript/SKILL.md +++ b/.claude/skills/languages/javascript/SKILL.md @@ -1,101 +1,724 @@ +--- +name: javascript +description: > + Trigger this skill whenever working with JavaScript files (.js, .mjs, .cjs), writing Node.js applications without TypeScript, or using ES6+ patterns like destructuring, async/await, optional chaining, and modules. Activate for browser scripting, vanilla JS projects, or when the user asks about JavaScript-specific idioms, ESLint configuration, or modern syntax. Also use when dealing with package.json scripts, CommonJS vs ESM, or JavaScript class patterns. +--- + # JavaScript -## Description - -Modern JavaScript (ES6+) patterns and best practices for Node.js and browser environments. - ## When to Use - Working with JavaScript files (.js, .mjs) - Browser scripting - Node.js applications without TypeScript +## When NOT to Use + +- TypeScript projects -- use the `languages/typescript` skill instead, which covers typed JavaScript patterns +- Python-only projects with no JavaScript components + --- ## Core Patterns -### Modern Syntax +### 1. Modern Syntax + +#### Destructuring (Nested, Defaults, Rest) ```javascript -// Destructuring -const { name, email } = user; -const [first, ...rest] = items; +// Object destructuring with defaults and rename +const { name, email, role = "user", address: { city } = {} } = user; -// Spread operator -const merged = { ...defaults, ...options }; -const combined = [...array1, ...array2]; +// Nested destructuring +const { + data: { + attributes: { title, body }, + }, +} = apiResponse; -// Template literals -const message = `Hello, ${name}!`; +// Array destructuring with rest +const [first, second, ...remaining] = items; -// Optional chaining and nullish coalescing -const city = user?.address?.city ?? 'Unknown'; +// Swap variables +let a = 1, b = 2; +[a, b] = [b, a]; + +// Function parameter destructuring +function createUser({ name, email, role = "user" }) { + return { name, email, role, createdAt: new Date() }; +} ``` -### Async Patterns +#### Optional Chaining (?.) ```javascript -// Async/await -async function fetchData(url) { - const response = await fetch(url); - if (!response.ok) throw new Error('Fetch failed'); - return response.json(); +// Property access +const city = user?.address?.city; + +// Method call +const uppercased = value?.toString?.(); + +// Array element +const firstItem = data?.items?.[0]; + +// Combine with nullish coalescing for defaults +const displayName = user?.profile?.displayName ?? user?.name ?? "Anonymous"; +``` + +#### Nullish Coalescing (??) + +```javascript +// Only falls through on null/undefined (not 0, "", false) +const port = config.port ?? 3000; +const name = input ?? "default"; + +// Contrast with || which falls through on all falsy values +const count = data.count ?? 0; // preserves 0 +const count2 = data.count || 0; // replaces 0 with 0 (same here, but misleading) +const label = data.label ?? ""; // preserves "" +const label2 = data.label || "fallback"; // replaces "" with "fallback" +``` + +#### Logical Assignment (&&=, ||=, ??=) + +```javascript +// ??= assigns only if null/undefined +user.name ??= "Anonymous"; + +// ||= assigns if falsy +config.retries ||= 3; + +// &&= assigns only if truthy +user.session &&= refreshSession(user.session); + +// Practical: initialize nested objects +const cache = {}; +(cache.users ??= []).push(newUser); +``` + +--- + +### 2. Async Patterns + +#### Promises + +```javascript +function fetchJson(url) { + return fetch(url) + .then((response) => { + if (!response.ok) throw new Error(`HTTP ${response.status}`); + return response.json(); + }); } -// Promise.all for parallel -const results = await Promise.all([ - fetchData(url1), - fetchData(url2), +// Chaining +fetchJson("/api/user") + .then((user) => fetchJson(`/api/posts?userId=${user.id}`)) + .then((posts) => console.log(posts)) + .catch((error) => console.error("Failed:", error.message)); +``` + +#### async/await + +```javascript +async function loadUserDashboard(userId) { + const user = await fetchJson(`/api/users/${userId}`); + const posts = await fetchJson(`/api/users/${userId}/posts`); + return { user, posts }; +} +``` + +#### Promise.all / allSettled / race / any + +```javascript +// Promise.all -- fail fast on first rejection +const [users, posts, comments] = await Promise.all([ + fetchJson("/api/users"), + fetchJson("/api/posts"), + fetchJson("/api/comments"), ]); -// Error handling -try { - const data = await fetchData(url); -} catch (error) { - console.error('Failed:', error.message); +// Promise.allSettled -- wait for all, get status of each +const results = await Promise.allSettled([ + fetchJson("/api/fast"), + fetchJson("/api/slow"), + fetchJson("/api/flaky"), +]); +const successes = results + .filter((r) => r.status === "fulfilled") + .map((r) => r.value); +const failures = results + .filter((r) => r.status === "rejected") + .map((r) => r.reason); + +// Promise.race -- first to settle wins +const result = await Promise.race([ + fetchJson("/api/primary"), + new Promise((_, reject) => + setTimeout(() => reject(new Error("Timeout")), 5000) + ), +]); + +// Promise.any -- first to fulfill wins (ignores rejections) +const fastest = await Promise.any([ + fetchJson("/api/mirror1"), + fetchJson("/api/mirror2"), + fetchJson("/api/mirror3"), +]); +``` + +#### AbortController + +```javascript +async function fetchWithTimeout(url, timeoutMs = 5000) { + const controller = new AbortController(); + const timeoutId = setTimeout(() => controller.abort(), timeoutMs); + + try { + const response = await fetch(url, { signal: controller.signal }); + return await response.json(); + } finally { + clearTimeout(timeoutId); + } +} + +// Cancellable request pattern +function createRequest(url) { + const controller = new AbortController(); + return { + promise: fetch(url, { signal: controller.signal }), + cancel: () => controller.abort(), + }; } ``` -### Array Methods +#### Async Iterators (for await...of) ```javascript -// Map, filter, reduce -const names = users.map(u => u.name); -const active = users.filter(u => u.active); -const total = items.reduce((sum, i) => sum + i.price, 0); +async function* paginateApi(baseUrl) { + let page = 1; + while (true) { + const data = await fetchJson(`${baseUrl}?page=${page}`); + if (data.items.length === 0) break; + yield* data.items; + page++; + } +} -// Find and includes -const user = users.find(u => u.id === id); -const hasAdmin = users.some(u => u.role === 'admin'); +// Consume the async iterator +for await (const item of paginateApi("/api/records")) { + processItem(item); +} ``` -### Classes +--- + +### 3. Closures & Scope + +#### Closure Patterns ```javascript -class UserService { - #db; // Private field +// Counter with private state +function createCounter(initial = 0) { + let count = initial; + return { + increment: () => ++count, + decrement: () => --count, + getCount: () => count, + reset: () => { count = initial; }, + }; +} - constructor(database) { - this.#db = database; +const counter = createCounter(10); +counter.increment(); // 11 +counter.getCount(); // 11 +``` + +#### Module Pattern (Private State via Closures) + +```javascript +const rateLimiter = (() => { + const requests = new Map(); + + function isAllowed(clientId, maxPerMinute = 60) { + const now = Date.now(); + const windowStart = now - 60_000; + const clientRequests = (requests.get(clientId) ?? []).filter( + (t) => t > windowStart + ); + if (clientRequests.length >= maxPerMinute) return false; + clientRequests.push(now); + requests.set(clientId, clientRequests); + return true; } - async findById(id) { - return this.#db.users.find(u => u.id === id); + function reset(clientId) { + requests.delete(clientId); + } + + return { isAllowed, reset }; +})(); +``` + +#### WeakRef and FinalizationRegistry + +```javascript +// Cache that does not prevent garbage collection +const cache = new Map(); + +function getCached(key, factory) { + const ref = cache.get(key); + const cached = ref?.deref(); + if (cached !== undefined) return cached; + + const value = factory(); + cache.set(key, new WeakRef(value)); + return value; +} + +// Cleanup when objects are garbage collected +const registry = new FinalizationRegistry((key) => { + cache.delete(key); +}); +``` + +--- + +### 4. Iteration Protocols + +#### Custom Iterator + +```javascript +class Range { + constructor(start, end, step = 1) { + this.start = start; + this.end = end; + this.step = step; + } + + [Symbol.iterator]() { + let current = this.start; + const { end, step } = this; + return { + next() { + if (current < end) { + const value = current; + current += step; + return { value, done: false }; + } + return { done: true }; + }, + }; + } +} + +for (const n of new Range(0, 10, 2)) { + console.log(n); // 0, 2, 4, 6, 8 +} +``` + +#### Generators + +```javascript +function* fibonacci() { + let a = 0, b = 1; + while (true) { + yield a; + [a, b] = [b, a + b]; + } +} + +// Take first N values +function take(iterable, count) { + const result = []; + for (const value of iterable) { + result.push(value); + if (result.length >= count) break; + } + return result; +} + +take(fibonacci(), 8); // [0, 1, 1, 2, 3, 5, 8, 13] +``` + +#### Lazy Evaluation with Generators + +```javascript +function* map(iterable, fn) { + for (const item of iterable) { + yield fn(item); + } +} + +function* filter(iterable, predicate) { + for (const item of iterable) { + if (predicate(item)) yield item; + } +} + +// Compose lazily -- no intermediate arrays +const data = filter( + map(readLargeFile(), (line) => line.trim()), + (line) => line.length > 0 +); +``` + +#### Async Generators + +```javascript +async function* readChunks(reader) { + while (true) { + const { done, value } = await reader.read(); + if (done) break; + yield value; + } +} + +// Stream processing +const response = await fetch("/api/large-data"); +for await (const chunk of readChunks(response.body.getReader())) { + processChunk(chunk); +} +``` + +--- + +### 5. Proxy & Reflect + +#### Validation Proxy + +```javascript +function createValidated(target, validators) { + return new Proxy(target, { + set(obj, prop, value) { + const validate = validators[prop]; + if (validate && !validate(value)) { + throw new TypeError(`Invalid value for ${String(prop)}: ${value}`); + } + return Reflect.set(obj, prop, value); + }, + }); +} + +const user = createValidated( + { name: "", age: 0 }, + { + name: (v) => typeof v === "string" && v.length > 0, + age: (v) => typeof v === "number" && v >= 0 && v <= 150, + } +); + +user.name = "Alice"; // works +user.age = -1; // throws TypeError +``` + +#### Observable Object + +```javascript +function createObservable(target, onChange) { + return new Proxy(target, { + set(obj, prop, value) { + const oldValue = obj[prop]; + const result = Reflect.set(obj, prop, value); + if (oldValue !== value) { + onChange(prop, value, oldValue); + } + return result; + }, + deleteProperty(obj, prop) { + const oldValue = obj[prop]; + const result = Reflect.deleteProperty(obj, prop); + onChange(prop, undefined, oldValue); + return result; + }, + }); +} + +const state = createObservable({}, (prop, newVal, oldVal) => { + console.log(`${prop}: ${oldVal} -> ${newVal}`); +}); +``` + +#### Property Access Logging + +```javascript +function withLogging(target, label = "access") { + return new Proxy(target, { + get(obj, prop) { + console.log(`[${label}] get .${String(prop)}`); + return Reflect.get(obj, prop); + }, + has(obj, prop) { + console.log(`[${label}] has .${String(prop)}`); + return Reflect.has(obj, prop); + }, + }); +} +``` + +--- + +### 6. Module System + +#### ESM (import/export) + +```javascript +// Named exports +export function formatDate(date) { ... } +export const MAX_RETRIES = 3; + +// Default export +export default class ApiClient { ... } + +// Re-exports +export { formatDate } from "./utils.js"; +export { default as ApiClient } from "./api-client.js"; +``` + +#### Dynamic import() + +```javascript +// Lazy load modules +async function loadChart(type) { + const module = await import(`./charts/${type}.js`); + return new module.default(); +} + +// Conditional loading +const { marked } = await import("marked"); + +// With error handling +async function tryLoadPlugin(name) { + try { + return await import(`./plugins/${name}.js`); + } catch { + console.warn(`Plugin ${name} not available`); + return null; } } ``` +#### import.meta + +```javascript +// Current module URL +console.log(import.meta.url); + +// Resolve relative paths (Node.js) +const configPath = new URL("./config.json", import.meta.url); + +// Check if file is the entry point (Node.js) +if (import.meta.url === `file://${process.argv[1]}`) { + main(); +} + +// Vite environment variables +const apiUrl = import.meta.env.VITE_API_URL; +``` + +#### Top-level await + +```javascript +// config.js -- top-level await in ESM modules +const response = await fetch("/api/config"); +export const config = await response.json(); + +// db.js +import { createPool } from "./db-pool.js"; +export const pool = await createPool(process.env.DATABASE_URL); +``` + +--- + +### 7. Performance + +#### structuredClone + +```javascript +// Deep clone without library (replaces JSON.parse(JSON.stringify(...))) +const original = { nested: { array: [1, 2, 3], date: new Date() } }; +const clone = structuredClone(original); + +// Handles Date, Map, Set, ArrayBuffer, RegExp (but not functions) +clone.nested.array.push(4); +console.log(original.nested.array.length); // still 3 +``` + +#### requestAnimationFrame + +```javascript +// Smooth animation loop +function animate(timestamp) { + updatePosition(timestamp); + render(); + requestAnimationFrame(animate); +} +requestAnimationFrame(animate); + +// Throttle DOM updates to frame rate +let rafId = null; +function scheduleUpdate(data) { + if (rafId) cancelAnimationFrame(rafId); + rafId = requestAnimationFrame(() => { + applyDOMUpdate(data); + rafId = null; + }); +} +``` + +#### requestIdleCallback + +```javascript +// Run low-priority work when the browser is idle +function processQueue(queue) { + requestIdleCallback((deadline) => { + while (deadline.timeRemaining() > 0 && queue.length > 0) { + const task = queue.shift(); + task(); + } + if (queue.length > 0) { + processQueue(queue); // schedule remaining + } + }); +} +``` + +#### Web Workers Basics + +```javascript +// main.js +const worker = new Worker(new URL("./worker.js", import.meta.url), { + type: "module", +}); + +worker.postMessage({ data: largeDataSet, operation: "sort" }); +worker.onmessage = (event) => { + const sorted = event.data; + renderResults(sorted); +}; + +// worker.js +self.onmessage = (event) => { + const { data, operation } = event.data; + if (operation === "sort") { + self.postMessage(data.sort((a, b) => a - b)); + } +}; +``` + +#### performance.mark / measure + +```javascript +// Measure operation duration +performance.mark("fetch-start"); +const data = await fetchJson("/api/data"); +performance.mark("fetch-end"); + +performance.measure("fetch-duration", "fetch-start", "fetch-end"); +const measurement = performance.getEntriesByName("fetch-duration")[0]; +console.log(`Fetch took ${measurement.duration.toFixed(2)}ms`); + +// Clean up +performance.clearMarks(); +performance.clearMeasures(); +``` + +--- + ## Best Practices -1. Use `const` by default, `let` when needed -2. Avoid `var` - use block-scoped declarations -3. Use arrow functions for callbacks -4. Handle all promise rejections -5. Use ESLint for consistent style +1. **Use `const` by default, `let` only when reassignment is needed** -- never use `var`. Block scoping prevents entire categories of bugs from hoisting and accidental mutation. + +2. **Handle all promise rejections** -- unhandled rejections crash Node.js processes. Always use try/catch with await, or attach `.catch()` to promise chains. Add a global handler as a safety net. + ```javascript + process.on("unhandledRejection", (reason) => { + console.error("Unhandled rejection:", reason); + process.exit(1); + }); + ``` + +3. **Use arrow functions for callbacks, regular functions for methods** -- arrow functions capture `this` from the enclosing scope, which is correct for callbacks but breaks object methods that need their own `this`. + +4. **Prefer `for...of` over `for...in` for iteration** -- `for...in` iterates over all enumerable properties including inherited ones. Use `for...of` for arrays and iterables, `Object.entries()` for objects. + +5. **Use ESLint and Prettier** -- enforce consistent style automatically. Configure in the project root and run on pre-commit hooks. + +6. **Avoid mutating function arguments** -- create new objects and arrays with spread syntax instead of modifying inputs in place. This prevents action-at-a-distance bugs. + +7. **Use `structuredClone` for deep copies** -- replaces the `JSON.parse(JSON.stringify(x))` hack. Handles Dates, Maps, Sets, and circular references correctly. + +8. **Use private class fields (`#field`)** -- the `#` prefix creates truly private fields that cannot be accessed outside the class, unlike the `_` convention which is only a hint. + +--- ## Common Pitfalls -- **Implicit type coercion**: Use `===` instead of `==` -- **Callback hell**: Use async/await -- **Mutating objects**: Create new objects with spread -- **Not handling errors**: Always catch promise rejections +1. **Implicit type coercion** -- always use `===` and `!==`. The `==` operator performs type coercion with surprising rules (`"" == false`, `0 == null` is false but `0 == undefined` is also false, yet `null == undefined` is true). + +2. **Forgetting `await`** -- a missing `await` silently returns a Promise object instead of the resolved value, causing hard-to-debug issues. + ```javascript + // BAD -- data is a Promise, not the response + const data = fetchJson("/api/data"); + + // GOOD + const data = await fetchJson("/api/data"); + ``` + +3. **`this` binding in callbacks** -- regular functions in callbacks lose their `this` context. Use arrow functions or `.bind()`. + ```javascript + // BAD + class Timer { + start() { setTimeout(function() { this.tick(); }, 1000); } + } + + // GOOD + class Timer { + start() { setTimeout(() => this.tick(), 1000); } + } + ``` + +4. **Mutating objects passed by reference** -- objects and arrays are passed by reference. Modifying a parameter modifies the original. + ```javascript + // BAD + function addDefaults(config) { + config.retries = config.retries ?? 3; // mutates caller's object + return config; + } + + // GOOD + function addDefaults(config) { + return { retries: 3, ...config }; + } + ``` + +5. **`for...in` on arrays** -- iterates over indices as strings and includes inherited properties. Use `for...of` or array methods. + ```javascript + // BAD + for (const i in [10, 20, 30]) { + console.log(typeof i); // "string", not "number" + } + + // GOOD + for (const value of [10, 20, 30]) { + console.log(value); // 10, 20, 30 + } + ``` + +6. **Floating point arithmetic** -- `0.1 + 0.2 !== 0.3` in JavaScript. For financial calculations, work in integer cents or use a decimal library. + ```javascript + // BAD + const total = 0.1 + 0.2; // 0.30000000000000004 + + // GOOD + const totalCents = 10 + 20; // 30 + const total = totalCents / 100; // 0.3 + ``` + +--- + +## Related Skills + +- `languages/typescript` -- TypeScript for typed JavaScript development +- `frameworks/react` -- React component patterns +- `frameworks/nextjs` -- Next.js full-stack framework +- `testing/vitest` -- JavaScript/TypeScript testing with Vitest diff --git a/.claude/skills/languages/javascript/references/modern-js-patterns.md b/.claude/skills/languages/javascript/references/modern-js-patterns.md new file mode 100644 index 0000000..f4599bb --- /dev/null +++ b/.claude/skills/languages/javascript/references/modern-js-patterns.md @@ -0,0 +1,247 @@ +# Modern JavaScript Patterns Quick Reference + +> ES2020+ patterns. All examples work in current Node.js (18+) and modern browsers. + +## Destructuring Tricks + +```javascript +// Nested destructuring with rename and default +const { data: { users: members = [] } = {} } = response; + +// Array destructuring: skip elements +const [first, , third] = [1, 2, 3]; + +// Swap variables +[a, b] = [b, a]; + +// Rest in both arrays and objects +const { id, ...rest } = user; +const [head, ...tail] = items; + +// Destructure function parameters +function draw({ x = 0, y = 0, color = "black" } = {}) { /* ... */ } + +// Dynamic property destructuring +const key = "name"; +const { [key]: value } = { name: "Alice" }; // value = "Alice" + +// Destructure from iterables +const [a, b] = new Map([["a", 1], ["b", 2]]); +``` + +## Optional Chaining (?.) + +```javascript +// Property access +const city = user?.address?.city; + +// Method call (only calls if method exists) +const result = api?.getData?.(); + +// Bracket notation +const val = obj?.["dynamic-key"]; + +// Array index +const first = arr?.[0]; + +// Combine with nullish coalescing +const name = user?.profile?.name ?? "Anonymous"; + +// Short-circuit: stops evaluating after first nullish +const len = response?.data?.items?.length; // undefined if any is nullish +``` + +## Nullish Coalescing (??) vs OR (||) + +```javascript +// ?? only triggers on null/undefined (NOT 0, "", false) +0 ?? "fallback" // 0 +"" ?? "fallback" // "" +null ?? "fallback" // "fallback" + +// || triggers on any falsy value +0 || "fallback" // "fallback" +"" || "fallback" // "fallback" +null || "fallback" // "fallback" + +// Use ?? for values where 0/empty string are valid +const port = config.port ?? 3000; +const title = config.title ?? "Untitled"; +``` + +## Logical Assignment Operators + +```javascript +// ??= assigns only if current value is null/undefined +user.name ??= "Anonymous"; +// Equivalent: user.name = user.name ?? "Anonymous" + +// ||= assigns if current value is falsy +opts.verbose ||= false; +// Equivalent: opts.verbose = opts.verbose || false + +// &&= assigns only if current value is truthy +user.token &&= encrypt(user.token); +// Equivalent: user.token = user.token && encrypt(user.token) +``` + +## structuredClone (Deep Copy) + +```javascript +// Deep clone objects, arrays, Maps, Sets, Dates, RegExp, etc. +const original = { date: new Date(), nested: { arr: [1, 2] } }; +const clone = structuredClone(original); +clone.nested.arr.push(3); // original not affected + +// Works with circular references +const obj = { self: null }; +obj.self = obj; +const copy = structuredClone(obj); // OK + +// Does NOT clone: functions, DOM nodes, symbols, prototype chain +// Throws on: functions, Error objects (in some engines) +``` + +## Proxy + +```javascript +// Validation proxy +const validated = new Proxy({}, { + set(target, prop, value) { + if (prop === "age" && (typeof value !== "number" || value < 0)) { + throw new TypeError("Age must be a non-negative number"); + } + target[prop] = value; + return true; + } +}); + +// Read-only proxy +function readonly(target) { + return new Proxy(target, { + set() { throw new Error("Read-only object"); }, + deleteProperty() { throw new Error("Read-only object"); } + }); +} + +// Default values proxy +function withDefaults(target, defaults) { + return new Proxy(target, { + get(obj, prop) { + return prop in obj ? obj[prop] : defaults[prop]; + } + }); +} +const config = withDefaults({}, { theme: "dark", lang: "en" }); +config.theme; // "dark" + +// Logging / observation proxy +function observable(target, onChange) { + return new Proxy(target, { + set(obj, prop, value) { + const old = obj[prop]; + obj[prop] = value; + onChange(prop, old, value); + return true; + } + }); +} +``` + +## Generators + +```javascript +// Basic generator +function* range(start, end, step = 1) { + for (let i = start; i < end; i += step) { + yield i; + } +} +for (const n of range(0, 10, 2)) { /* 0, 2, 4, 6, 8 */ } + +// Infinite sequence +function* ids() { + let id = 0; + while (true) yield id++; +} +const gen = ids(); +gen.next().value; // 0 +gen.next().value; // 1 + +// Delegate to another generator +function* concat(...iterables) { + for (const it of iterables) { + yield* it; + } +} + +// Two-way communication +function* stateMachine() { + let input; + while (true) { + input = yield `received: ${input}`; + } +} +const sm = stateMachine(); +sm.next(); // { value: "received: undefined" } +sm.next("hello"); // { value: "received: hello" } +``` + +## Async Iterators + +```javascript +// for-await-of +async function processStream(stream) { + for await (const chunk of stream) { + console.log(chunk); + } +} + +// Async generator +async function* fetchPages(url) { + let page = 1; + while (true) { + const res = await fetch(`${url}?page=${page}`); + const data = await res.json(); + if (data.items.length === 0) return; + yield data.items; + page++; + } +} + +for await (const items of fetchPages("/api/users")) { + console.log(items); +} + +``` + +## Other Modern Patterns + +```javascript +// Object.groupBy (ES2024) +const grouped = Object.groupBy(users, u => u.role); + +// at() - negative indexing +[1, 2, 3].at(-1); // 3 + +// Object.hasOwn (replaces hasOwnProperty) +Object.hasOwn(obj, "key"); // true/false + +// Error cause chaining +throw new Error("DB failed", { cause: originalError }); + +// AbortSignal.timeout (built-in timeout) +fetch(url, { signal: AbortSignal.timeout(5000) }); + +// using keyword (explicit resource management, ES2024+) +{ using handle = openFile("data.txt"); } // auto-disposed at block exit +``` + +## Promise Combinators + +| Method | Settles when | Returns | +|--------|-------------|---------| +| `Promise.all(ps)` | All fulfill or one rejects | Array of values | +| `Promise.allSettled(ps)` | All settle | Array of `{status, value/reason}` | +| `Promise.race(ps)` | First settles | First value or rejection | +| `Promise.any(ps)` | First fulfills | First value (AggregateError if all reject) | diff --git a/.claude/skills/languages/python/SKILL.md b/.claude/skills/languages/python/SKILL.md index 596724d..0b196c8 100644 --- a/.claude/skills/languages/python/SKILL.md +++ b/.claude/skills/languages/python/SKILL.md @@ -1,9 +1,11 @@ +--- +name: python +description: > + Trigger this skill whenever working with Python files (.py), writing Python scripts or applications, or using Python frameworks like Django, FastAPI, or Flask. Activate for any Python-specific patterns including type hints, async/await with asyncio, dataclasses, Pydantic models, context managers, virtual environments, or PEP 8 style questions. Also use when the user references Python package management, pip, or pyproject.toml. +--- + # Python -## Description - -Python development expertise including type hints, async patterns, virtual environments, and Pythonic idioms. - ## When to Use - Working with Python files (.py) @@ -11,59 +13,148 @@ Python development expertise including type hints, async patterns, virtual envir - Using Python frameworks (Django, FastAPI, Flask) - Data processing and automation +## When NOT to Use + +- JavaScript or TypeScript-only projects with no Python components +- Non-Python environments where another language skill is more appropriate + --- ## Core Patterns -### Type Hints +### 1. Type Hints + +Use type hints on all public functions and module-level variables. Python 3.10+ syntax is preferred (use `X | Y` instead of `Union[X, Y]`). + +#### Basic Types ```python -from typing import Optional, List, Dict, Union -from collections.abc import Callable +from typing import Any -def process_items( - items: List[str], - callback: Callable[[str], None], - config: Optional[Dict[str, Any]] = None -) -> List[str]: - """Process items with optional callback.""" - return [callback(item) for item in items] +def greet(name: str) -> str: + return f"Hello, {name}" + +def process(count: int, factor: float = 1.0) -> float: + return count * factor + +def is_valid(data: bytes | None) -> bool: + return data is not None and len(data) > 0 ``` -### Async/Await +#### Optional and Union ```python -import asyncio -from typing import List +# Python 3.10+ syntax (preferred) +def find_user(user_id: int) -> User | None: + ... -async def fetch_data(url: str) -> dict: - async with aiohttp.ClientSession() as session: - async with session.get(url) as response: - return await response.json() +# Pre-3.10 fallback +from typing import Optional, Union -async def fetch_all(urls: List[str]) -> List[dict]: - return await asyncio.gather(*[fetch_data(url) for url in urls]) +def find_user(user_id: int) -> Optional[User]: + ... + +def parse_input(value: Union[str, int]) -> str: + return str(value) ``` -### Context Managers +#### Generic Collections ```python -from contextlib import contextmanager +# Python 3.9+ built-in generics (preferred) +def process_items(items: list[str]) -> dict[str, int]: + return {item: len(item) for item in items} -@contextmanager -def managed_resource(): - resource = acquire_resource() - try: - yield resource - finally: - release_resource(resource) +def merge_configs(base: dict[str, Any], overrides: dict[str, Any]) -> dict[str, Any]: + return {**base, **overrides} -# Usage -with managed_resource() as r: - r.do_something() +# Nested generics +def group_by_key(pairs: list[tuple[str, int]]) -> dict[str, list[int]]: + result: dict[str, list[int]] = {} + for key, value in pairs: + result.setdefault(key, []).append(value) + return result ``` -### Dataclasses +#### Protocol for Structural Subtyping + +```python +from typing import Protocol, runtime_checkable + +@runtime_checkable +class Renderable(Protocol): + def render(self) -> str: ... + +class HtmlWidget: + def render(self) -> str: + return "
widget
" + +def display(item: Renderable) -> None: + print(item.render()) + +# HtmlWidget satisfies Renderable without inheriting from it +display(HtmlWidget()) # works +``` + +#### TypeVar for Generic Functions + +```python +from typing import TypeVar, Sequence + +T = TypeVar("T") + +def first(items: Sequence[T]) -> T: + return items[0] + +# Bounded TypeVar +Numeric = TypeVar("Numeric", int, float) + +def clamp(value: Numeric, low: Numeric, high: Numeric) -> Numeric: + return max(low, min(high, value)) +``` + +#### @overload for Multiple Signatures + +```python +from typing import overload + +@overload +def parse(raw: str) -> dict[str, Any]: ... +@overload +def parse(raw: bytes) -> dict[str, Any]: ... +@overload +def parse(raw: str, as_list: bool) -> list[Any]: ... + +def parse(raw: str | bytes, as_list: bool = False) -> dict[str, Any] | list[Any]: + data = raw if isinstance(raw, str) else raw.decode() + parsed = json.loads(data) + return list(parsed) if as_list else parsed +``` + +#### TypeAlias and TypeGuard + +```python +from typing import TypeAlias, TypeGuard + +# TypeAlias for complex types +JsonValue: TypeAlias = str | int | float | bool | None | list["JsonValue"] | dict[str, "JsonValue"] +Headers: TypeAlias = dict[str, str] + +# TypeGuard for narrowing +def is_string_list(val: list[Any]) -> TypeGuard[list[str]]: + return all(isinstance(item, str) for item in val) + +def process(items: list[Any]) -> None: + if is_string_list(items): + # items is now list[str] inside this branch + print(", ".join(items)) +``` + +--- + +### 2. Dataclasses & Pydantic + +#### @dataclass with Options ```python from dataclasses import dataclass, field @@ -75,36 +166,535 @@ class User: email: str name: str created_at: datetime = field(default_factory=datetime.now) + tags: list[str] = field(default_factory=list) def __post_init__(self): - self.email = self.email.lower() + self.email = self.email.strip().lower() ``` -### Pydantic Models +#### Frozen and Slots ```python -from pydantic import BaseModel, EmailStr, Field +@dataclass(frozen=True, slots=True) +class Coordinate: + """Immutable, memory-efficient value object.""" + x: float + y: float + + @property + def magnitude(self) -> float: + return (self.x ** 2 + self.y ** 2) ** 0.5 +``` + +#### Pydantic BaseModel + +```python +from pydantic import BaseModel, EmailStr, Field, field_validator, computed_field, model_validator class UserCreate(BaseModel): + model_config = {"str_strip_whitespace": True, "frozen": False} + email: EmailStr name: str = Field(min_length=1, max_length=100) password: str = Field(min_length=8) + age: int = Field(ge=0, le=150) - class Config: - str_strip_whitespace = True + @field_validator("name") + @classmethod + def name_must_not_be_blank(cls, v: str) -> str: + if not v.strip(): + raise ValueError("Name must not be blank") + return v.title() + + @computed_field + @property + def display_name(self) -> str: + return f"{self.name} <{self.email}>" + + @model_validator(mode="after") + def check_consistency(self) -> "UserCreate": + if "admin" in self.name.lower() and self.age < 18: + raise ValueError("Admins must be 18+") + return self ``` +--- + +### 3. Async Patterns + +#### Basic async/await + +```python +import asyncio +import aiohttp + +async def fetch_json(url: str) -> dict: + async with aiohttp.ClientSession() as session: + async with session.get(url) as response: + response.raise_for_status() + return await response.json() +``` + +#### asyncio.gather for Parallel Work + +```python +async def fetch_all(urls: list[str]) -> list[dict]: + return await asyncio.gather(*[fetch_json(url) for url in urls]) +``` + +#### asyncio.TaskGroup (Python 3.11+) + +```python +async def fetch_all_safe(urls: list[str]) -> list[dict]: + results: list[dict] = [] + async with asyncio.TaskGroup() as tg: + for url in urls: + tg.create_task(fetch_and_append(url, results)) + return results + +async def fetch_and_append(url: str, results: list[dict]) -> None: + data = await fetch_json(url) + results.append(data) +``` + +#### Async Generators + +```python +async def paginate(url: str) -> AsyncIterator[dict]: + page = 1 + while True: + data = await fetch_json(f"{url}?page={page}") + if not data["items"]: + break + for item in data["items"]: + yield item + page += 1 + +# Usage +async for item in paginate("/api/users"): + process(item) +``` + +#### Async Context Managers + +```python +from contextlib import asynccontextmanager + +@asynccontextmanager +async def db_transaction(pool): + conn = await pool.acquire() + tx = await conn.begin() + try: + yield conn + await tx.commit() + except Exception: + await tx.rollback() + raise + finally: + await pool.release(conn) +``` + +#### Semaphores for Concurrency Limiting + +```python +async def fetch_with_limit(urls: list[str], max_concurrent: int = 10) -> list[dict]: + semaphore = asyncio.Semaphore(max_concurrent) + + async def limited_fetch(url: str) -> dict: + async with semaphore: + return await fetch_json(url) + + return await asyncio.gather(*[limited_fetch(url) for url in urls]) +``` + +--- + +### 4. Decorators + +#### Function Decorator with functools.wraps + +```python +import functools +import time + +def timing(func): + @functools.wraps(func) + def wrapper(*args, **kwargs): + start = time.perf_counter() + result = func(*args, **kwargs) + elapsed = time.perf_counter() - start + print(f"{func.__name__} took {elapsed:.4f}s") + return result + return wrapper + +@timing +def slow_operation(): + time.sleep(1) +``` + +#### Decorator with Arguments + +```python +def retry(max_attempts: int = 3, delay: float = 1.0): + def decorator(func): + @functools.wraps(func) + async def wrapper(*args, **kwargs): + last_error: Exception | None = None + for attempt in range(max_attempts): + try: + return await func(*args, **kwargs) + except Exception as e: + last_error = e + if attempt < max_attempts - 1: + await asyncio.sleep(delay * (2 ** attempt)) + raise last_error + return wrapper + return decorator + +@retry(max_attempts=5, delay=0.5) +async def unreliable_call(url: str) -> dict: + return await fetch_json(url) +``` + +#### Class Decorator + +```python +def singleton(cls): + instances: dict[type, Any] = {} + + @functools.wraps(cls) + def get_instance(*args, **kwargs): + if cls not in instances: + instances[cls] = cls(*args, **kwargs) + return instances[cls] + + return get_instance + +@singleton +class AppConfig: + def __init__(self): + self.settings = load_settings() +``` + +#### Caching Decorator + +```python +from functools import lru_cache, cache + +@lru_cache(maxsize=256) +def fibonacci(n: int) -> int: + if n < 2: + return n + return fibonacci(n - 1) + fibonacci(n - 2) + +# Python 3.9+ unbounded cache +@cache +def load_config(path: str) -> dict: + with open(path) as f: + return json.load(f) +``` + +--- + +### 5. Context Managers + +#### Basic @contextmanager + +```python +from contextlib import contextmanager + +@contextmanager +def managed_connection(dsn: str): + conn = connect(dsn) + try: + yield conn + finally: + conn.close() + +with managed_connection("postgres://...") as conn: + conn.execute("SELECT 1") +``` + +#### Temporary File Context Manager + +```python +import tempfile +import os + +@contextmanager +def temp_directory(): + dirpath = tempfile.mkdtemp() + try: + yield dirpath + finally: + shutil.rmtree(dirpath) + +with temp_directory() as tmpdir: + filepath = os.path.join(tmpdir, "data.json") + write_json(filepath, data) +``` + +#### Lock Context Manager + +```python +import threading + +@contextmanager +def timed_lock(lock: threading.Lock, timeout: float = 5.0): + acquired = lock.acquire(timeout=timeout) + if not acquired: + raise TimeoutError("Could not acquire lock") + try: + yield + finally: + lock.release() +``` + +#### Async Context Manager + +```python +from contextlib import asynccontextmanager + +@asynccontextmanager +async def http_session(): + session = aiohttp.ClientSession() + try: + yield session + finally: + await session.close() +``` + +--- + +### 6. Pattern Matching + +#### Basic match/case + +```python +def handle_command(command: str) -> str: + match command.split(): + case ["quit"]: + return "Goodbye" + case ["hello", name]: + return f"Hello, {name}" + case ["add", *items]: + return f"Adding {len(items)} items" + case _: + return "Unknown command" +``` + +#### Structural Patterns + +```python +def process_event(event: dict) -> None: + match event: + case {"type": "click", "x": int(x), "y": int(y)}: + handle_click(x, y) + case {"type": "keypress", "key": str(key)} if len(key) == 1: + handle_keypress(key) + case {"type": "resize", "width": w, "height": h}: + handle_resize(w, h) +``` + +#### Guard Clauses and OR Patterns + +```python +def classify_status(code: int) -> str: + match code: + case 200 | 201 | 204: + return "success" + case code if 300 <= code < 400: + return "redirect" + case 401 | 403: + return "auth_error" + case code if 400 <= code < 500: + return "client_error" + case code if 500 <= code < 600: + return "server_error" + case _: + return "unknown" +``` + +--- + +### 7. Error Handling + +#### Custom Exception Hierarchies + +```python +class AppError(Exception): + """Base exception for the application.""" + def __init__(self, message: str, code: str | None = None): + super().__init__(message) + self.code = code + +class NotFoundError(AppError): + """Resource was not found.""" + def __init__(self, resource: str, resource_id: str): + super().__init__(f"{resource} {resource_id} not found", code="NOT_FOUND") + self.resource = resource + self.resource_id = resource_id + +class ValidationError(AppError): + """Input validation failed.""" + def __init__(self, errors: list[str]): + super().__init__(f"Validation failed: {'; '.join(errors)}", code="VALIDATION") + self.errors = errors +``` + +#### ExceptionGroup (Python 3.11+) + +```python +async def process_batch(items: list[dict]) -> list[dict]: + results = [] + errors = [] + for item in items: + try: + results.append(await process(item)) + except Exception as e: + errors.append(e) + if errors: + raise ExceptionGroup("Batch processing errors", errors) + return results + +# Handling with except* +try: + await process_batch(items) +except* ValueError as eg: + print(f"Validation errors: {len(eg.exceptions)}") +except* ConnectionError as eg: + print(f"Connection errors: {len(eg.exceptions)}") +``` + +#### Exception Chaining + +```python +def load_config(path: str) -> dict: + try: + with open(path) as f: + return json.load(f) + except FileNotFoundError as e: + raise AppError(f"Config file missing: {path}") from e + except json.JSONDecodeError as e: + raise AppError(f"Invalid JSON in {path}") from e +``` + +#### contextlib.suppress + +```python +from contextlib import suppress + +# Instead of try/except/pass +with suppress(FileNotFoundError): + os.remove("temp_file.txt") + +# Instead of: +# try: +# os.remove("temp_file.txt") +# except FileNotFoundError: +# pass +``` + +--- + ## Best Practices -1. Use type hints for all public functions -2. Use dataclasses or Pydantic for data models -3. Prefer context managers for resource management -4. Use async for I/O-bound operations -5. Follow PEP 8 style guidelines +1. **Use type hints on all public functions** -- they serve as documentation, enable IDE autocompletion, and allow static analysis with mypy or pyright. + +2. **Prefer dataclasses or Pydantic for structured data** -- avoid passing raw dicts around. Use `@dataclass` for internal data, Pydantic `BaseModel` for external boundaries (API input/output, config files). + +3. **Use context managers for resource management** -- database connections, file handles, locks, and temporary resources should always be wrapped in `with` statements to guarantee cleanup. + +4. **Prefer `asyncio.TaskGroup` over bare `gather`** -- TaskGroup (3.11+) provides proper error handling by cancelling sibling tasks when one fails, avoiding orphaned coroutines. + +5. **Follow PEP 8 and use a formatter** -- use `ruff format` or `black` for consistent formatting, and `ruff check` for linting. Configure in `pyproject.toml`. + +6. **Write small, composable functions** -- each function should do one thing. Prefer returning values over mutating state. Limit functions to ~20 lines when practical. + +7. **Use `__all__` in public modules** -- explicitly declare the public API of a module to prevent accidental imports of internal helpers. + +8. **Use `pathlib.Path` over `os.path`** -- pathlib provides a cleaner, object-oriented API for file system operations and works cross-platform. + +--- ## Common Pitfalls -- **Mutable default arguments**: Use `None` and initialize in function -- **Not closing resources**: Use `with` statements -- **Blocking in async**: Use `asyncio.to_thread()` for CPU work -- **Catching bare exceptions**: Be specific with exception types +1. **Mutable default arguments** -- default values are shared across calls. Use `None` and initialize inside the function body. + ```python + # BAD + def add_item(item: str, items: list[str] = []) -> list[str]: ... + + # GOOD + def add_item(item: str, items: list[str] | None = None) -> list[str]: + if items is None: + items = [] + items.append(item) + return items + ``` + +2. **Blocking calls inside async functions** -- calling `time.sleep()`, `requests.get()`, or CPU-heavy code in an async function blocks the entire event loop. Use `asyncio.to_thread()` or `asyncio.sleep()`. + ```python + # BAD + async def fetch(): + return requests.get(url) # blocks event loop + + # GOOD + async def fetch(): + return await asyncio.to_thread(requests.get, url) + ``` + +3. **Catching bare `Exception`** -- always be specific about which exceptions you catch. Bare `except:` or `except Exception:` hides bugs. + ```python + # BAD + try: + result = compute() + except Exception: + pass + + # GOOD + try: + result = compute() + except (ValueError, TypeError) as e: + logger.warning("Computation failed: %s", e) + result = default_value + ``` + +4. **Using `is` for value comparison** -- `is` checks identity, not equality. Only use `is` for `None`, `True`, `False`, and sentinel objects. + ```python + # BAD + if x is 42: ... + + # GOOD + if x == 42: ... + if x is None: ... + ``` + +5. **Forgetting to close resources** -- file handles, database connections, and HTTP sessions leak if not closed. Always use context managers. + ```python + # BAD + f = open("data.txt") + data = f.read() + + # GOOD + with open("data.txt") as f: + data = f.read() + ``` + +6. **Circular imports** -- restructure code to avoid circular dependencies. Move shared types into a separate module, use `TYPE_CHECKING` for type-only imports, or use lazy imports. + ```python + from __future__ import annotations + from typing import TYPE_CHECKING + + if TYPE_CHECKING: + from myapp.models import User # only imported during type checking + ``` + +--- + +## Related Skills + +- `languages/typescript` -- TypeScript language patterns for polyglot projects +- `frameworks/fastapi` -- FastAPI web framework built on Python +- `frameworks/django` -- Django web framework for Python +- `testing/pytest` -- Python testing with pytest +- `patterns/error-handling` -- Python error handling and exception hierarchies diff --git a/.claude/skills/languages/python/references/type-hints-reference.md b/.claude/skills/languages/python/references/type-hints-reference.md new file mode 100644 index 0000000..431a160 --- /dev/null +++ b/.claude/skills/languages/python/references/type-hints-reference.md @@ -0,0 +1,216 @@ +# Python Type Hints Quick Reference + +> Python 3.10+ syntax preferred. For 3.9, use `from __future__ import annotations`. + +## Basic Types + +| Type | Example | Notes | +|------|---------|-------| +| `int` | `x: int = 1` | | +| `float` | `x: float = 1.0` | | +| `str` | `x: str = "hi"` | | +| `bool` | `x: bool = True` | | +| `bytes` | `x: bytes = b"hi"` | | +| `None` | `x: None = None` | Use as return type for side-effect functions | +| `object` | `x: object` | Accepts anything, but no attribute access | +| `Any` | `x: Any` | Escapes type checking entirely | + +## Collection Types (3.10+) + +| Type | Example | Notes | +|------|---------|-------| +| `list[int]` | `x: list[int] = [1, 2]` | Mutable sequence | +| `tuple[int, str]` | `x: tuple[int, str]` | Fixed length | +| `tuple[int, ...]` | `x: tuple[int, ...]` | Variable length | +| `dict[str, int]` | `x: dict[str, int]` | | +| `set[str]` | `x: set[str]` | | +| `frozenset[str]` | `x: frozenset[str]` | | + +## Union and Optional + +```python +# 3.10+ syntax +def f(x: int | str) -> None: ... +def g(x: int | None = None) -> None: ... + +# Pre-3.10 +from typing import Union, Optional +def f(x: Union[int, str]) -> None: ... +def g(x: Optional[int] = None) -> None: ... +``` + +## TypeAlias + +```python +from typing import TypeAlias + +# Explicit alias (3.10+) +Vector: TypeAlias = list[float] + +# 3.12+ syntax +type Vector = list[float] +type Tree[T] = T | list["Tree[T]"] # recursive +``` + +## Generics with TypeVar + +```python +from typing import TypeVar + +T = TypeVar("T") +K = TypeVar("K", bound=str) # upper bound +N = TypeVar("N", int, float) # constrained + +def first(items: list[T]) -> T: + return items[0] + +# 3.12+ syntax (no TypeVar needed) +def first[T](items: list[T]) -> T: + return items[0] +``` + +## ParamSpec and Concatenate + +```python +from typing import ParamSpec, Concatenate, Callable + +P = ParamSpec("P") +T = TypeVar("T") + +# Preserve function signatures through decorators +def logged(fn: Callable[P, T]) -> Callable[P, T]: + def wrapper(*args: P.args, **kwargs: P.kwargs) -> T: + print(f"Calling {fn.__name__}") + return fn(*args, **kwargs) + return wrapper + +# Add a parameter to a function signature +def with_user( + fn: Callable[Concatenate[User, P], T] +) -> Callable[P, T]: + def wrapper(*args: P.args, **kwargs: P.kwargs) -> T: + return fn(get_current_user(), *args, **kwargs) + return wrapper +``` + +## Protocol (Structural Typing) + +```python +from typing import Protocol, runtime_checkable + +class Renderable(Protocol): + def render(self) -> str: ... + +class Widget(Protocol): + name: str + def resize(self, width: int, height: int) -> None: ... + +# Any class with matching methods satisfies the protocol +class Button: + def render(self) -> str: + return " + + ); +} + +function App() { + return ( + { + // Send to error tracking service + reportError({ error, componentStack: info.componentStack }); + }} + onReset={() => { + // Clear any stale state before retry + queryClient.clear(); + }} + > + + + ); +} +``` + +**Granular boundaries per feature** + +```typescript +function DashboardPage() { + return ( +
+ {/* Each widget fails independently */} + + + + + + + + + +
+ ); +} + +function WidgetErrorFallback({ error, resetErrorBoundary }: FallbackProps) { + return ( +
+

This widget failed to load.

+ +
+ ); +} +``` + +**Class-based error boundary (when you need full control)** + +```typescript +import { Component, type ErrorInfo, type ReactNode } from "react"; + +interface Props { + children: ReactNode; + fallback?: ReactNode; +} + +interface State { + hasError: boolean; + error: Error | null; +} + +class ManualErrorBoundary extends Component { + state: State = { hasError: false, error: null }; + + static getDerivedStateFromError(error: Error): State { + return { hasError: true, error }; + } + + componentDidCatch(error: Error, info: ErrorInfo) { + console.error("ErrorBoundary caught:", error, info.componentStack); + } + + render() { + if (this.state.hasError) { + return this.props.fallback ??

Something went wrong.

; + } + return this.props.children; + } +} +``` + +--- + +### 3. Retry Patterns + +Retry transient failures with exponential backoff and jitter to avoid thundering herd. + +**Python - Retry decorator with exponential backoff** + +```python +import asyncio +import random +import logging +from functools import wraps +from typing import TypeVar, Callable, Awaitable + +logger = logging.getLogger(__name__) + +T = TypeVar("T") + + +def retry( + max_attempts: int = 3, + base_delay: float = 1.0, + max_delay: float = 30.0, + retryable: tuple[type[Exception], ...] = (Exception,), +) -> Callable: + """Retry decorator with exponential backoff and full jitter. + + Args: + max_attempts: Total number of attempts including the first call. + base_delay: Initial delay in seconds before the first retry. + max_delay: Upper bound on the computed delay. + retryable: Exception types eligible for retry. + """ + + def decorator(fn: Callable[..., Awaitable[T]]) -> Callable[..., Awaitable[T]]: + @wraps(fn) + async def wrapper(*args, **kwargs) -> T: + last_exc: Exception | None = None + for attempt in range(1, max_attempts + 1): + try: + return await fn(*args, **kwargs) + except retryable as exc: + last_exc = exc + if attempt == max_attempts: + break + delay = min(base_delay * (2 ** (attempt - 1)), max_delay) + jitter = random.uniform(0, delay) + logger.warning( + "Attempt %d/%d failed (%s), retrying in %.2fs", + attempt, + max_attempts, + exc, + jitter, + ) + await asyncio.sleep(jitter) + raise last_exc # type: ignore[misc] + + return wrapper + + return decorator + + +# Usage +@retry(max_attempts=3, base_delay=0.5, retryable=(ConnectionError, TimeoutError)) +async def fetch_remote_config(url: str) -> dict: + async with httpx.AsyncClient(timeout=5) as client: + resp = await client.get(url) + resp.raise_for_status() + return resp.json() +``` + +**TypeScript - Retry wrapper** + +```typescript +interface RetryOptions { + maxAttempts?: number; + baseDelayMs?: number; + maxDelayMs?: number; + isRetryable?: (error: unknown) => boolean; +} + +async function withRetry( + fn: () => Promise, + options: RetryOptions = {} +): Promise { + const { + maxAttempts = 3, + baseDelayMs = 1000, + maxDelayMs = 30_000, + isRetryable = () => true, + } = options; + + let lastError: unknown; + + for (let attempt = 1; attempt <= maxAttempts; attempt++) { + try { + return await fn(); + } catch (error) { + lastError = error; + + if (attempt === maxAttempts || !isRetryable(error)) { + throw error; + } + + const exponential = baseDelayMs * 2 ** (attempt - 1); + const capped = Math.min(exponential, maxDelayMs); + const jitter = Math.random() * capped; + + console.warn( + `Attempt ${attempt}/${maxAttempts} failed, retrying in ${jitter.toFixed(0)}ms` + ); + await new Promise((resolve) => setTimeout(resolve, jitter)); + } + } + + throw lastError; +} + +// Usage +const data = await withRetry( + () => fetch("/api/config").then((r) => r.json()), + { + maxAttempts: 3, + baseDelayMs: 500, + isRetryable: (err) => + err instanceof TypeError || (err as Response)?.status >= 500, + } +); +``` + +--- + +### 4. Graceful Degradation + +When a dependency fails, fall back to a degraded but functional state instead of crashing. + +**Circuit breaker pattern (Python)** + +```python +import time +from enum import Enum + + +class CircuitState(Enum): + CLOSED = "closed" # normal operation + OPEN = "open" # failing, reject immediately + HALF_OPEN = "half_open" # testing recovery + + +class CircuitBreaker: + """Prevents cascading failures by short-circuiting calls to an unhealthy dependency.""" + + def __init__( + self, + failure_threshold: int = 5, + recovery_timeout: float = 30.0, + ) -> None: + self.failure_threshold = failure_threshold + self.recovery_timeout = recovery_timeout + self.state = CircuitState.CLOSED + self.failure_count = 0 + self.last_failure_time = 0.0 + + def _trip(self) -> None: + self.state = CircuitState.OPEN + self.last_failure_time = time.monotonic() + + def _reset(self) -> None: + self.state = CircuitState.CLOSED + self.failure_count = 0 + + async def call(self, fn, *args, fallback=None, **kwargs): + if self.state == CircuitState.OPEN: + if time.monotonic() - self.last_failure_time > self.recovery_timeout: + self.state = CircuitState.HALF_OPEN + else: + if fallback is not None: + return fallback() if callable(fallback) else fallback + raise ExternalServiceError("circuit-breaker", RuntimeError("Circuit open")) + + try: + result = await fn(*args, **kwargs) + if self.state == CircuitState.HALF_OPEN: + self._reset() + return result + except Exception as exc: + self.failure_count += 1 + if self.failure_count >= self.failure_threshold: + self._trip() + raise + + +# Usage +recommendations_circuit = CircuitBreaker(failure_threshold=3, recovery_timeout=60) + +async def get_recommendations(user_id: str) -> list[dict]: + return await recommendations_circuit.call( + recommendation_service.fetch, + user_id, + fallback=lambda: [], # empty list when service is down + ) +``` + +**Feature-flag degraded mode (TypeScript)** + +```typescript +interface FeatureFlags { + enableRecommendations: boolean; + enableRealTimeUpdates: boolean; + enableAdvancedSearch: boolean; +} + +const defaultFlags: FeatureFlags = { + enableRecommendations: true, + enableRealTimeUpdates: true, + enableAdvancedSearch: true, +}; + +async function getFlags(): Promise { + try { + const resp = await fetch("/api/feature-flags"); + if (!resp.ok) throw new Error(`HTTP ${resp.status}`); + return await resp.json(); + } catch { + // Fall back to safe defaults when flag service is unavailable + console.warn("Feature flag service unavailable, using defaults"); + return defaultFlags; + } +} + +// Component that degrades gracefully +function SearchPage() { + const flags = useFeatureFlags(); + + return ( +
+ + {flags.enableAdvancedSearch ? ( + + ) : ( +

+ Advanced search is temporarily unavailable. +

+ )} +
+ ); +} +``` + +--- + +### 5. API Error Responses + +Return consistent, machine-readable error payloads following RFC 7807 Problem Details. + +**Python (FastAPI)** + +```python +from fastapi import FastAPI, Request +from fastapi.responses import JSONResponse +from starlette.status import ( + HTTP_400_BAD_REQUEST, + HTTP_404_NOT_FOUND, + HTTP_429_TOO_MANY_REQUESTS, + HTTP_500_INTERNAL_SERVER_ERROR, +) + +app = FastAPI() + +# Map domain error codes to HTTP status codes +STATUS_MAP: dict[ErrorCode, int] = { + ErrorCode.NOT_FOUND: HTTP_404_NOT_FOUND, + ErrorCode.VALIDATION_FAILED: HTTP_400_BAD_REQUEST, + ErrorCode.DUPLICATE_ENTRY: 409, + ErrorCode.UNAUTHORIZED: 401, + ErrorCode.RATE_LIMITED: HTTP_429_TOO_MANY_REQUESTS, + ErrorCode.EXTERNAL_SERVICE: 502, + ErrorCode.INTERNAL: HTTP_500_INTERNAL_SERVER_ERROR, +} + + +@app.exception_handler(AppError) +async def app_error_handler(request: Request, exc: AppError) -> JSONResponse: + status = STATUS_MAP.get(exc.code, 500) + return JSONResponse( + status_code=status, + content={ + "type": f"https://docs.example.com/errors/{exc.code.value.lower()}", + "title": exc.code.value.replace("_", " ").title(), + "status": status, + "detail": str(exc), + **exc.details, + }, + ) + + +@app.exception_handler(Exception) +async def unhandled_error_handler(request: Request, exc: Exception) -> JSONResponse: + # Never leak internal details to the client + logger.exception("Unhandled exception on %s %s", request.method, request.url.path) + return JSONResponse( + status_code=500, + content={ + "type": "https://docs.example.com/errors/internal", + "title": "Internal Server Error", + "status": 500, + "detail": "An unexpected error occurred.", + }, + ) +``` + +**TypeScript (Express middleware)** + +```typescript +import type { Request, Response, NextFunction } from "express"; + +const STATUS_MAP: Record = { + NOT_FOUND: 404, + VALIDATION_FAILED: 400, + DUPLICATE_ENTRY: 409, + UNAUTHORIZED: 401, + RATE_LIMITED: 429, + EXTERNAL_SERVICE: 502, + INTERNAL: 500, +}; + +function errorHandler( + err: Error, + _req: Request, + res: Response, + _next: NextFunction +) { + if (err instanceof AppError) { + const status = STATUS_MAP[err.code] ?? 500; + res.status(status).json({ + type: `https://docs.example.com/errors/${err.code.toLowerCase()}`, + title: err.code.replace(/_/g, " ").toLowerCase(), + status, + detail: err.message, + ...err.details, + }); + return; + } + + // Unhandled errors - log full details, return generic message + console.error("Unhandled error:", err); + res.status(500).json({ + type: "https://docs.example.com/errors/internal", + title: "Internal Server Error", + status: 500, + detail: "An unexpected error occurred.", + }); +} + +// Register as the last middleware +app.use(errorHandler); +``` + +--- + +### 6. Error Logging Integration + +Attach structured context to errors so they are searchable and actionable in observability tools. + +**Python - Structured error logging** + +```python +import logging +import traceback +from contextvars import ContextVar + +request_id_var: ContextVar[str] = ContextVar("request_id", default="unknown") + + +class StructuredErrorLogger: + """Wraps the standard logger to attach error context automatically.""" + + def __init__(self, name: str) -> None: + self.logger = logging.getLogger(name) + + def error( + self, + msg: str, + *, + exc: Exception | None = None, + extra: dict | None = None, + ) -> None: + context = { + "request_id": request_id_var.get(), + **(extra or {}), + } + + if exc is not None: + context["error_type"] = type(exc).__name__ + context["error_message"] = str(exc) + context["stacktrace"] = traceback.format_exception(exc) + + if isinstance(exc, AppError): + context["error_code"] = exc.code.value + context["error_details"] = exc.details + + self.logger.error(msg, extra={"structured": context}, exc_info=exc) + + +# Usage in a FastAPI middleware +from starlette.middleware.base import BaseHTTPMiddleware +import uuid + +log = StructuredErrorLogger(__name__) + + +class ErrorLoggingMiddleware(BaseHTTPMiddleware): + async def dispatch(self, request, call_next): + rid = request.headers.get("x-request-id", str(uuid.uuid4())) + request_id_var.set(rid) + try: + response = await call_next(request) + return response + except Exception as exc: + log.error( + "Request failed", + exc=exc, + extra={"method": request.method, "path": request.url.path}, + ) + raise +``` + +**TypeScript - Error context enrichment** + +```typescript +interface ErrorContext { + requestId?: string; + userId?: string; + operation?: string; + [key: string]: unknown; +} + +function logError( + message: string, + error: unknown, + context: ErrorContext = {} +): void { + const payload: Record = { + message, + timestamp: new Date().toISOString(), + ...context, + }; + + if (error instanceof AppError) { + payload.errorCode = error.code; + payload.errorMessage = error.message; + payload.errorDetails = error.details; + } else if (error instanceof Error) { + payload.errorType = error.name; + payload.errorMessage = error.message; + payload.stack = error.stack; + } else { + payload.errorRaw = String(error); + } + + // Structured JSON log for ingestion by Datadog, CloudWatch, etc. + console.error(JSON.stringify(payload)); +} + +// Usage +try { + await processOrder(orderId); +} catch (error) { + logError("Order processing failed", error, { + requestId: req.headers["x-request-id"], + operation: "processOrder", + orderId, + }); + throw error; +} +``` + +--- + +### 7. Result Pattern + +Use a Result type for operations where failure is an expected outcome. Avoids exception overhead and makes the failure path explicit in the type signature. + +**Python - Using the `result` library** + +```python +# pip install result +from result import Ok, Err, Result + + +def parse_age(value: str) -> Result[int, str]: + """Parse a string to a valid age. Returns Err for invalid input.""" + try: + age = int(value) + except ValueError: + return Err(f"'{value}' is not a number") + + if age < 0 or age > 150: + return Err(f"Age {age} is out of valid range (0-150)") + + return Ok(age) + + +def validate_registration(data: dict) -> Result[dict, list[str]]: + """Validate all fields, collecting every error instead of failing on the first.""" + errors: list[str] = [] + + match parse_age(data.get("age", "")): + case Ok(age): + data["age"] = age + case Err(msg): + errors.append(msg) + + name = data.get("name", "").strip() + if not name: + errors.append("Name is required") + if len(name) > 100: + errors.append("Name must be 100 characters or fewer") + + if errors: + return Err(errors) + return Ok(data) + + +# Caller handles both paths explicitly +match validate_registration(form_data): + case Ok(valid): + user = create_user(valid) + case Err(errs): + return {"errors": errs}, 400 +``` + +**TypeScript - Discriminated union Result** + +```typescript +type Result = + | { ok: true; value: T } + | { ok: false; error: E }; + +function ok(value: T): Result { + return { ok: true, value }; +} + +function err(error: E): Result { + return { ok: false, error }; +} + +// Usage +function parseAge(input: string): Result { + const age = Number(input); + if (Number.isNaN(age)) return err(`'${input}' is not a number`); + if (age < 0 || age > 150) return err(`Age ${age} out of range (0-150)`); + return ok(age); +} + +function validateRegistration( + data: Record +): Result<{ name: string; age: number }, string[]> { + const errors: string[] = []; + + const ageResult = parseAge(data.age ?? ""); + if (!ageResult.ok) errors.push(ageResult.error); + + const name = data.name?.trim() ?? ""; + if (!name) errors.push("Name is required"); + if (name.length > 100) errors.push("Name must be 100 characters or fewer"); + + if (errors.length > 0) return err(errors); + return ok({ name, age: (ageResult as { ok: true; value: number }).value }); +} + +// Caller +const result = validateRegistration(formData); +if (!result.ok) { + return res.status(400).json({ errors: result.error }); +} +const user = await createUser(result.value); +``` + +--- + +## Best Practices + +1. **Catch specific exceptions, not bare `except` or `catch`.** A catch-all hides bugs. Catch only the errors you know how to handle and let everything else propagate. + +2. **Translate errors at architectural boundaries.** A database `IntegrityError` should become a domain `DuplicateEntryError` at the repository layer, then an HTTP 409 at the API layer. Each layer speaks its own error language. + +3. **Preserve the original cause.** Always chain the original exception (`raise X from original` in Python, `{ cause }` in TypeScript) so the root cause is visible in logs and debuggers. + +4. **Fail fast, recover high.** Detect errors as early as possible (validate inputs at the boundary) but handle them at the highest level that has enough context to decide what to do (e.g., return an HTTP response, show a fallback UI). + +5. **Never swallow errors silently.** An empty `except: pass` or `catch {}` is almost always a bug. At minimum, log the error. If you intentionally ignore it, leave a comment explaining why. + +6. **Use the Result pattern for expected failures.** When a function can legitimately fail (parsing, validation, lookups), return a Result instead of throwing. Reserve exceptions for truly unexpected situations. + +7. **Make errors actionable.** Every error message should help the reader fix the problem. Include what happened, what was expected, and what the caller can do about it. `"User not found"` is worse than `"User with id '123' not found. Verify the id and check that the user has not been deleted."`. + +8. **Test the error paths.** Write explicit tests for every error branch. Verify the error type, message, and status code. Error paths that are never tested are error paths that will break in production. + +--- + +## Common Pitfalls + +1. **Catching too broadly.** Using `except Exception` or `catch (e: any)` silences programming errors like `TypeError` or `ReferenceError` that should crash loudly during development. + +2. **Logging and re-throwing without deduplication.** If every layer logs the same error, you get five log entries for one failure. Log at the outermost handler and let inner layers propagate. + +3. **Returning error data in the wrong shape.** Mixing `{ error: "..." }`, `{ message: "..." }`, and `{ errors: [...] }` across endpoints forces every client to handle multiple formats. Pick one shape and enforce it globally. + +4. **Leaking internal details to clients.** Stack traces, database table names, and file paths in API responses are a security risk. Sanitize errors before they leave the server. + +5. **Retrying non-idempotent operations.** Retrying a `POST /orders` that partially succeeded can create duplicate orders. Only retry operations that are safe to repeat, or use idempotency keys. + +6. **Ignoring async error boundaries.** In React, error boundaries do not catch errors inside event handlers or async callbacks. Use try/catch inside `onClick`, `useEffect` cleanup, and promise chains separately. + +--- + +## Related Skills + +- `patterns/logging` - Structured logging setup and conventions +- `patterns/api-client` - HTTP client wrappers with built-in error handling +- `security/owasp` - Preventing information leakage through error messages +- `languages/python` - Python exception syntax and idioms +- `languages/typescript` - TypeScript error types and narrowing diff --git a/.claude/skills/patterns/error-handling/references/error-taxonomy.md b/.claude/skills/patterns/error-handling/references/error-taxonomy.md new file mode 100644 index 0000000..192cb2f --- /dev/null +++ b/.claude/skills/patterns/error-handling/references/error-taxonomy.md @@ -0,0 +1,170 @@ +# Error Taxonomy Quick Reference + +## Two Fundamental Categories + +### Operational Errors (Expected, Recoverable) + +Errors that occur in correctly written programs due to external conditions. + +**Response**: Handle gracefully. Log, retry, return error to user. + +| Category | Description | HTTP Status | Example | +|----------|-------------|-------------|---------| +| Validation | Invalid input data | `400` | Missing required field, wrong format | +| Authentication | Identity not established | `401` | Missing/expired/invalid token | +| Authorization | Insufficient permissions | `403` | User lacks role for this action | +| Not Found | Resource does not exist | `404` | Item with given ID not in DB | +| Conflict | State conflict | `409` | Duplicate email, concurrent edit | +| Rate Limit | Too many requests | `429` | API quota exceeded | +| Payload Too Large | Request body too big | `413` | File upload exceeds limit | +| Unprocessable | Valid syntax, invalid semantics | `422` | Transfer amount exceeds balance | +| External Dependency | Third-party service failed | `502` / `503` | Payment gateway timeout | +| Service Unavailable | System overloaded or in maintenance | `503` | DB connection pool exhausted | + +### Programmer Errors (Bugs, Fix the Code) + +Errors caused by mistakes in the code itself. + +**Response**: Fix the code. Do NOT catch and continue. Crash, log, alert. + +| Category | Example | Fix | +|----------|---------|-----| +| TypeError | Calling method on undefined | Add null check or fix data flow | +| ReferenceError | Using undeclared variable | Fix variable name/scope | +| Assertion failure | Invariant violated | Fix logic that broke invariant | +| Wrong argument type | Passing string where number expected | Fix caller or add validation | +| Missing error handling | Unhandled promise rejection | Add try/catch or .catch() | +| Off-by-one | Array index out of bounds | Fix loop/index logic | + +--- + +## Error Handling by Category + +### Validation Errors (400) + +```python +# Python/FastAPI +from pydantic import BaseModel, validator + +class CreateUser(BaseModel): + email: str + age: int + + @validator("age") + def validate_age(cls, v): + if v < 0 or v > 150: + raise ValueError("Age must be between 0 and 150") + return v +``` + +```typescript +// TypeScript +class ValidationError extends AppError { + constructor(public fields: Record) { + super("Validation failed", 400); + } +} +``` + +### Authentication Errors (401) + +| Scenario | Response | Action | +|----------|----------|--------| +| No token provided | 401 + `WWW-Authenticate` header | Client should authenticate | +| Token expired | 401 + error code `token_expired` | Client should refresh token | +| Token invalid | 401 + error code `invalid_token` | Client should re-authenticate | + +### Not Found (404) + +| Scenario | Use 404? | +|----------|----------| +| Resource by ID doesn't exist | Yes | +| Search returns no results | **No** -- return empty list with 200 | +| Resource soft-deleted | Depends on visibility rules | +| User lacks access to resource | Consider 403, or 404 to hide existence | + +### Conflict (409) + +| Scenario | Resolution | +|----------|------------| +| Duplicate unique field | Return which field conflicts | +| Optimistic locking failure | Return current version, client retries | +| State transition invalid | Return current state and valid transitions | + +### External Dependency (502/503) + +| Strategy | When | +|----------|------| +| Retry with backoff | Transient failures (timeouts, 503) | +| Circuit breaker | Repeated failures from same service | +| Fallback / degraded mode | Non-critical dependency | +| Queue for later | Async-compatible operations | + +--- + +## Error Response Format + +### Standard Error Response (JSON) + +```json +{ + "error": { + "code": "VALIDATION_ERROR", + "message": "Validation failed", + "details": [ + { "field": "email", "message": "Invalid email format" }, + { "field": "age", "message": "Must be a positive number" } + ], + "request_id": "req_abc123" + } +} +``` + +### Error Code Convention + +| Code Pattern | Meaning | +|-------------|---------| +| `VALIDATION_ERROR` | Input validation failed | +| `AUTH_TOKEN_EXPIRED` | Token needs refresh | +| `RESOURCE_NOT_FOUND` | Entity doesn't exist | +| `RATE_LIMIT_EXCEEDED` | Throttled | +| `CONFLICT_DUPLICATE` | Uniqueness violation | +| `INTERNAL_ERROR` | Unexpected server error (hide details) | + +--- + +## Decision Guide + +``` +Error occurred + | + +--> Can the program continue safely? + | | + | +--> YES (operational error) + | | | + | | +--> Is it the client's fault? --> 4xx + | | +--> Is it our fault? --> 5xx + | | +--> Is it a dependency? --> 502/503 + | | + | +--> NO (programmer error) + | | + | +--> Log full stack trace + | +--> Return generic 500 to client + | +--> Alert on-call + | +--> Fix the code + | + +--> Should the client see details? + | + +--> 4xx: Yes, help them fix their request + +--> 5xx: No, generic message + request_id for support +``` + +## Anti-Patterns + +| Anti-Pattern | Why It's Bad | Instead | +|-------------|-------------|---------| +| Catch-all silently | Hides bugs | Catch specific errors, rethrow unknown | +| Return 200 with error body | Breaks HTTP semantics | Use proper status codes | +| Expose stack traces in prod | Security risk | Log internally, return request_id | +| String error matching | Fragile, breaks on message change | Use error codes/classes | +| Catch and log only | Request hangs or returns wrong data | Handle or propagate | diff --git a/.claude/skills/patterns/logging/SKILL.md b/.claude/skills/patterns/logging/SKILL.md new file mode 100644 index 0000000..fd33703 --- /dev/null +++ b/.claude/skills/patterns/logging/SKILL.md @@ -0,0 +1,962 @@ +--- +name: logging +description: > + Structured logging patterns for Python and Node.js applications. Use this skill when setting up loggers, choosing log levels, implementing correlation IDs for request tracing, redacting sensitive data from logs, or configuring log aggregation. Trigger whenever code uses console.log, print(), logging module, winston, pino, structlog, or any logging library. Also applies when building observability, debugging production issues, or adding telemetry. +--- + +# Logging + +## When to Use + +- Setting up structured logging in a new application or service +- Replacing `console.log` or `print()` with proper logging infrastructure +- Adding request tracing with correlation IDs across microservices +- Redacting sensitive data (passwords, tokens, PII) from log output +- Building observability pipelines with log aggregation (ELK, Datadog, CloudWatch) + +## When NOT to Use + +- Static analysis or linting tasks that do not involve runtime output +- Pure computation functions where logging would add unnecessary noise +- Test assertions — use testing frameworks' built-in assertion messages, not log output + +--- + +## Core Patterns + +### 1. Structured Logging Setup + +Structured logging outputs machine-parseable JSON instead of free-form strings. This enables searching, filtering, and alerting in log aggregation systems. + +#### Python with structlog + +```python +# logging_config.py +import logging +import structlog + +def configure_logging(log_level: str = "INFO", json_output: bool = True) -> None: + """Configure structured logging for the application. + + Call this once at application startup, before any loggers are created. + """ + # Set the stdlib logging level as the baseline filter + logging.basicConfig( + format="%(message)s", + level=getattr(logging, log_level.upper()), + ) + + # Choose renderers based on environment + if json_output: + renderer = structlog.processors.JSONRenderer() + else: + # Human-readable output for local development + renderer = structlog.dev.ConsoleRenderer(colors=True) + + structlog.configure( + processors=[ + structlog.contextvars.merge_contextvars, + structlog.stdlib.filter_by_level, + structlog.stdlib.add_logger_name, + structlog.stdlib.add_log_level, + structlog.processors.TimeStamper(fmt="iso"), + structlog.processors.StackInfoRenderer(), + structlog.processors.format_exc_info, + structlog.processors.UnicodeDecoder(), + renderer, + ], + context_class=dict, + logger_factory=structlog.stdlib.LoggerFactory(), + wrapper_class=structlog.stdlib.BoundLogger, + cache_logger_on_first_use=True, + ) +``` + +```python +# Usage anywhere in the application +import structlog + +logger = structlog.get_logger(__name__) + +async def create_user(email: str) -> User: + logger.info("creating_user", email=email) + user = await user_repo.create(email=email) + logger.info("user_created", user_id=user.id, email=email) + return user +``` + +**Output (JSON mode):** +```json +{"event": "user_created", "user_id": 42, "email": "alice@example.com", "logger": "app.services.user", "level": "info", "timestamp": "2025-06-15T10:30:00.123Z"} +``` + +#### Node.js with pino + +```typescript +// logger.ts +import pino from "pino"; + +export const logger = pino({ + level: process.env.LOG_LEVEL ?? "info", + // Use pretty printing only in development + transport: + process.env.NODE_ENV === "development" + ? { target: "pino-pretty", options: { colorize: true } } + : undefined, + // Base fields included in every log line + base: { + service: process.env.SERVICE_NAME ?? "api", + version: process.env.APP_VERSION ?? "unknown", + }, + // Customize serialization + serializers: { + err: pino.stdSerializers.err, + req: pino.stdSerializers.req, + res: pino.stdSerializers.res, + }, + // Redact sensitive fields (see Pattern 4) + redact: ["req.headers.authorization", "req.headers.cookie"], +}); + +// Create child loggers for specific modules +export function createLogger(module: string): pino.Logger { + return logger.child({ module }); +} +``` + +```typescript +// Usage in a service +import { createLogger } from "./logger"; + +const log = createLogger("user-service"); + +export async function createUser(email: string): Promise { + log.info({ email }, "creating_user"); + const user = await userRepo.create({ email }); + log.info({ userId: user.id, email }, "user_created"); + return user; +} +``` + +**Output (JSON):** +```json +{"level":30,"time":1718444400123,"service":"api","module":"user-service","userId":42,"email":"alice@example.com","msg":"user_created"} +``` + +### 2. Log Levels + +Use log levels consistently to control verbosity and enable filtering in production. + +| Level | When to Use | Example | +|-------|-------------|---------| +| `DEBUG` | Detailed diagnostic information useful only during development or debugging | Variable values, SQL queries, cache hits/misses | +| `INFO` | Confirmation that things are working as expected | Request received, user created, job completed | +| `WARNING` | Something unexpected happened but the application can continue | Deprecated API called, retry attempt, approaching rate limit | +| `ERROR` | A specific operation failed but the application continues running | Database query failed, external API returned 500, payment declined | +| `CRITICAL` | The application cannot continue or is in an unrecoverable state | Database connection pool exhausted, out of disk space, configuration missing | + +#### Python examples + +```python +import structlog + +logger = structlog.get_logger(__name__) + +# DEBUG: Detailed internals for troubleshooting +logger.debug("cache_lookup", key="user:42", hit=True, ttl_remaining=120) + +# INFO: Normal business events +logger.info("order_placed", order_id="ORD-123", total=99.99, items=3) + +# WARNING: Degraded but functional +logger.warning( + "rate_limit_approaching", + current_rate=450, + limit=500, + window_seconds=60, +) + +# ERROR: Operation failed, needs attention +logger.error( + "payment_failed", + order_id="ORD-123", + provider="stripe", + error_code="card_declined", + exc_info=True, # Include stack trace +) + +# CRITICAL: System-level failure +logger.critical( + "database_pool_exhausted", + active_connections=100, + max_connections=100, + waiting_requests=47, +) +``` + +#### TypeScript examples + +```typescript +import { createLogger } from "./logger"; + +const log = createLogger("order-service"); + +// DEBUG: Internal details +log.debug({ key: "user:42", hit: true, ttlRemaining: 120 }, "cache_lookup"); + +// INFO: Normal events +log.info({ orderId: "ORD-123", total: 99.99, items: 3 }, "order_placed"); + +// WARNING: Degraded state +log.warn( + { currentRate: 450, limit: 500, windowSeconds: 60 }, + "rate_limit_approaching" +); + +// ERROR: Operation failure +log.error( + { orderId: "ORD-123", provider: "stripe", errorCode: "card_declined" }, + "payment_failed" +); + +// FATAL: Unrecoverable +log.fatal( + { activeConnections: 100, maxConnections: 100, waitingRequests: 47 }, + "database_pool_exhausted" +); +``` + +**Level selection rule of thumb:** If you would page someone at 3 AM, it is ERROR or CRITICAL. If it is useful context for investigating an issue, it is INFO. If it is only useful when actively debugging a specific problem, it is DEBUG. + +### 3. Correlation IDs + +Correlation IDs (also called request IDs or trace IDs) tie together all log entries from a single request as it flows through your system. + +#### Python — FastAPI middleware with contextvars + +```python +# middleware/correlation.py +import uuid +from contextvars import ContextVar + +import structlog +from starlette.middleware.base import BaseHTTPMiddleware +from starlette.requests import Request + +# Context variable accessible from any async task in the same request +correlation_id_var: ContextVar[str] = ContextVar("correlation_id", default="") + +class CorrelationIDMiddleware(BaseHTTPMiddleware): + async def dispatch(self, request: Request, call_next): + # Accept an incoming correlation ID or generate a new one + correlation_id = request.headers.get("X-Correlation-ID", uuid.uuid4().hex) + correlation_id_var.set(correlation_id) + + # Bind to structlog context so all logs in this request include it + structlog.contextvars.clear_contextvars() + structlog.contextvars.bind_contextvars(correlation_id=correlation_id) + + response = await call_next(request) + response.headers["X-Correlation-ID"] = correlation_id + return response +``` + +```python +# Register the middleware +from middleware.correlation import CorrelationIDMiddleware + +app.add_middleware(CorrelationIDMiddleware) +``` + +```python +# Any logger call in any module now includes correlation_id automatically +logger = structlog.get_logger(__name__) + +async def get_user(user_id: int) -> User: + logger.info("fetching_user", user_id=user_id) + # Output: {"event": "fetching_user", "user_id": 42, "correlation_id": "a1b2c3d4...", ...} + return await user_repo.get(user_id) +``` + +#### TypeScript — Express middleware with AsyncLocalStorage + +```typescript +// middleware/correlation.ts +import { AsyncLocalStorage } from "node:async_hooks"; +import { randomUUID } from "node:crypto"; +import type { Request, Response, NextFunction } from "express"; +import { logger } from "../logger"; + +interface RequestContext { + correlationId: string; +} + +export const asyncLocalStorage = new AsyncLocalStorage(); + +export function correlationMiddleware( + req: Request, + res: Response, + next: NextFunction +): void { + const correlationId = + (req.headers["x-correlation-id"] as string) ?? randomUUID(); + + res.setHeader("X-Correlation-ID", correlationId); + + asyncLocalStorage.run({ correlationId }, () => { + next(); + }); +} +``` + +```typescript +// logger.ts — augment the logger to include correlation ID +import { asyncLocalStorage } from "./middleware/correlation"; + +export function getContextLogger(): pino.Logger { + const store = asyncLocalStorage.getStore(); + if (store) { + return logger.child({ correlationId: store.correlationId }); + } + return logger; +} +``` + +```typescript +// Usage in any module +import { getContextLogger } from "./logger"; + +export async function getUser(userId: number): Promise { + const log = getContextLogger(); + log.info({ userId }, "fetching_user"); + // Output includes correlationId automatically + return await userRepo.findById(userId); +} +``` + +#### Propagating to downstream services + +When calling other microservices, forward the correlation ID: + +```python +# Python — httpx client +import httpx +from middleware.correlation import correlation_id_var + +async def call_billing_service(user_id: int) -> dict: + correlation_id = correlation_id_var.get() + async with httpx.AsyncClient() as client: + response = await client.get( + f"http://billing-service/api/v1/invoices?user_id={user_id}", + headers={"X-Correlation-ID": correlation_id}, + ) + return response.json() +``` + +```typescript +// TypeScript — fetch with correlation ID +import { asyncLocalStorage } from "./middleware/correlation"; + +export async function callBillingService(userId: number): Promise { + const store = asyncLocalStorage.getStore(); + const response = await fetch( + `http://billing-service/api/v1/invoices?user_id=${userId}`, + { + headers: { + "X-Correlation-ID": store?.correlationId ?? "", + }, + } + ); + return response.json(); +} +``` + +### 4. Sensitive Data Redaction + +Never log passwords, API keys, tokens, credit card numbers, or personally identifiable information (PII). Build redaction into the logging pipeline so developers cannot accidentally leak secrets. + +#### Python — structlog processor + +```python +# processors/redact.py +import re +from typing import Any + +# Patterns for sensitive field names (case-insensitive matching) +SENSITIVE_KEYS = re.compile( + r"(password|passwd|secret|token|api_key|apikey|authorization|" + r"credit_card|card_number|cvv|ssn|social_security)", + re.IGNORECASE, +) + +# Pattern for credit card numbers in string values +CREDIT_CARD_PATTERN = re.compile(r"\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b") + +# Pattern for bearer tokens in string values +BEARER_PATTERN = re.compile(r"Bearer\s+[A-Za-z0-9\-._~+/]+=*", re.IGNORECASE) + + +def redact_sensitive_data( + logger: Any, method_name: str, event_dict: dict +) -> dict: + """Structlog processor that masks sensitive values.""" + return _redact_dict(event_dict) + + +def _redact_dict(data: dict) -> dict: + result = {} + for key, value in data.items(): + if SENSITIVE_KEYS.search(key): + result[key] = "***REDACTED***" + elif isinstance(value, dict): + result[key] = _redact_dict(value) + elif isinstance(value, str): + result[key] = _redact_string(value) + elif isinstance(value, list): + result[key] = [ + _redact_dict(item) if isinstance(item, dict) else item + for item in value + ] + else: + result[key] = value + return result + + +def _redact_string(value: str) -> str: + value = CREDIT_CARD_PATTERN.sub("****-****-****-****", value) + value = BEARER_PATTERN.sub("Bearer ***REDACTED***", value) + return value +``` + +```python +# Add the processor to structlog configuration +structlog.configure( + processors=[ + structlog.contextvars.merge_contextvars, + redact_sensitive_data, # Add before the renderer + structlog.processors.JSONRenderer(), + ], + # ... +) +``` + +#### TypeScript — pino redaction + +```typescript +// pino has built-in redaction support +import pino from "pino"; + +export const logger = pino({ + level: "info", + redact: { + paths: [ + "password", + "secret", + "token", + "apiKey", + "authorization", + "creditCard", + "req.headers.authorization", + "req.headers.cookie", + "body.password", + "body.creditCardNumber", + "*.password", + "*.secret", + ], + censor: "***REDACTED***", + }, +}); +``` + +For more complex redaction (regex-based), use a custom serializer: + +```typescript +// redact.ts +const CREDIT_CARD_RE = /\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b/g; +const BEARER_RE = /Bearer\s+[A-Za-z0-9\-._~+/]+=*/gi; + +export function redactValue(value: unknown): unknown { + if (typeof value === "string") { + let result = value.replace(CREDIT_CARD_RE, "****-****-****-****"); + result = result.replace(BEARER_RE, "Bearer ***REDACTED***"); + return result; + } + if (typeof value === "object" && value !== null) { + return redactObject(value as Record); + } + return value; +} + +const SENSITIVE_KEYS = + /^(password|passwd|secret|token|api_?key|authorization|credit_?card|cvv|ssn)$/i; + +function redactObject(obj: Record): Record { + const result: Record = {}; + for (const [key, value] of Object.entries(obj)) { + if (SENSITIVE_KEYS.test(key)) { + result[key] = "***REDACTED***"; + } else { + result[key] = redactValue(value); + } + } + return result; +} +``` + +### 5. Request/Response Logging + +Log every HTTP request and response with method, path, status code, duration, and body size. This is the single most valuable log line for production debugging. + +#### Python — FastAPI middleware + +```python +# middleware/request_logging.py +import time +import structlog +from starlette.middleware.base import BaseHTTPMiddleware +from starlette.requests import Request + +logger = structlog.get_logger("http") + +class RequestLoggingMiddleware(BaseHTTPMiddleware): + async def dispatch(self, request: Request, call_next): + start_time = time.perf_counter() + + # Log request + logger.info( + "http_request_started", + method=request.method, + path=request.url.path, + query=str(request.url.query) or None, + client_ip=request.client.host if request.client else None, + user_agent=request.headers.get("user-agent"), + ) + + try: + response = await call_next(request) + except Exception: + duration_ms = (time.perf_counter() - start_time) * 1000 + logger.error( + "http_request_failed", + method=request.method, + path=request.url.path, + duration_ms=round(duration_ms, 2), + exc_info=True, + ) + raise + + duration_ms = (time.perf_counter() - start_time) * 1000 + content_length = response.headers.get("content-length") + + # Choose log level based on status code + log_method = logger.info + if response.status_code >= 500: + log_method = logger.error + elif response.status_code >= 400: + log_method = logger.warning + + log_method( + "http_request_completed", + method=request.method, + path=request.url.path, + status_code=response.status_code, + duration_ms=round(duration_ms, 2), + content_length=int(content_length) if content_length else None, + ) + + return response +``` + +#### TypeScript — Express middleware + +```typescript +// middleware/request-logging.ts +import type { Request, Response, NextFunction } from "express"; +import { getContextLogger } from "../logger"; + +export function requestLogging( + req: Request, + res: Response, + next: NextFunction +): void { + const start = process.hrtime.bigint(); + const log = getContextLogger(); + + log.info( + { + method: req.method, + path: req.originalUrl, + clientIp: req.ip, + userAgent: req.get("user-agent"), + }, + "http_request_started" + ); + + // Hook into the response finish event + res.on("finish", () => { + const durationMs = + Number(process.hrtime.bigint() - start) / 1_000_000; + + const logFn = + res.statusCode >= 500 + ? log.error.bind(log) + : res.statusCode >= 400 + ? log.warn.bind(log) + : log.info.bind(log); + + logFn( + { + method: req.method, + path: req.originalUrl, + statusCode: res.statusCode, + durationMs: Math.round(durationMs * 100) / 100, + contentLength: res.get("content-length") + ? parseInt(res.get("content-length")!, 10) + : undefined, + }, + "http_request_completed" + ); + }); + + next(); +} +``` + +```typescript +// Register middleware (order matters) +app.use(correlationMiddleware); +app.use(requestLogging); +``` + +### 6. Error Logging + +When logging errors, always include the stack trace, relevant context, and enough information to reproduce the issue without accessing the production environment. + +#### Python — exception logging with context + +```python +import structlog + +logger = structlog.get_logger(__name__) + +async def process_order(order_id: str) -> Order: + logger.info("processing_order", order_id=order_id) + + try: + order = await order_repo.get(order_id) + if not order: + logger.error("order_not_found", order_id=order_id) + raise OrderNotFoundError(order_id) + + payment = await payment_service.charge( + amount=order.total, + currency=order.currency, + customer_id=order.customer_id, + ) + logger.info( + "payment_processed", + order_id=order_id, + payment_id=payment.id, + amount=order.total, + ) + + except PaymentError as exc: + # Log the error with full context for debugging + logger.error( + "payment_failed", + order_id=order_id, + customer_id=order.customer_id, + amount=order.total, + error_code=exc.code, + error_message=str(exc), + exc_info=True, # Includes full stack trace + ) + raise + + except Exception as exc: + # Catch-all for unexpected errors + logger.exception( + "order_processing_unexpected_error", + order_id=order_id, + error_type=type(exc).__name__, + ) + raise +``` + +#### TypeScript — error logging with breadcrumbs + +```typescript +import { getContextLogger } from "./logger"; + +const log = getContextLogger(); + +export async function processOrder(orderId: string): Promise { + log.info({ orderId }, "processing_order"); + + try { + const order = await orderRepo.findById(orderId); + if (!order) { + log.error({ orderId }, "order_not_found"); + throw new OrderNotFoundError(orderId); + } + + const payment = await paymentService.charge({ + amount: order.total, + currency: order.currency, + customerId: order.customerId, + }); + + log.info( + { orderId, paymentId: payment.id, amount: order.total }, + "payment_processed" + ); + return order; + } catch (err) { + if (err instanceof PaymentError) { + log.error( + { + orderId, + errorCode: err.code, + errorMessage: err.message, + err, // pino serializes Error objects with stack traces + }, + "payment_failed" + ); + } else { + log.error( + { + orderId, + err, + errorType: (err as Error).constructor.name, + }, + "order_processing_unexpected_error" + ); + } + throw err; + } +} +``` + +**Key principle:** Log at the point where you have the most context, not at every layer. A single error log with full context is more useful than five partial logs scattered across the call stack. + +### 7. Performance Logging + +Track operation durations to identify slow endpoints, queries, and external calls. + +#### Python — timing decorator + +```python +import functools +import time +from typing import Callable, TypeVar + +import structlog + +logger = structlog.get_logger("performance") + +F = TypeVar("F", bound=Callable) + +def log_duration(operation: str, slow_threshold_ms: float = 1000.0): + """Decorator that logs the duration of a function call. + + Args: + operation: A descriptive name for the operation. + slow_threshold_ms: Threshold in milliseconds above which + the log level escalates to WARNING. + """ + def decorator(func: F) -> F: + @functools.wraps(func) + async def async_wrapper(*args, **kwargs): + start = time.perf_counter() + try: + result = await func(*args, **kwargs) + return result + finally: + duration_ms = (time.perf_counter() - start) * 1000 + log_fn = ( + logger.warning + if duration_ms > slow_threshold_ms + else logger.debug + ) + log_fn( + "operation_duration", + operation=operation, + duration_ms=round(duration_ms, 2), + slow=duration_ms > slow_threshold_ms, + ) + + @functools.wraps(func) + def sync_wrapper(*args, **kwargs): + start = time.perf_counter() + try: + result = func(*args, **kwargs) + return result + finally: + duration_ms = (time.perf_counter() - start) * 1000 + log_fn = ( + logger.warning + if duration_ms > slow_threshold_ms + else logger.debug + ) + log_fn( + "operation_duration", + operation=operation, + duration_ms=round(duration_ms, 2), + slow=duration_ms > slow_threshold_ms, + ) + + import asyncio + if asyncio.iscoroutinefunction(func): + return async_wrapper # type: ignore + return sync_wrapper # type: ignore + + return decorator + + +# Usage +@log_duration("fetch_user_profile", slow_threshold_ms=200) +async def get_user_profile(user_id: int) -> UserProfile: + return await user_repo.get_with_preferences(user_id) +``` + +#### Python — context manager for ad-hoc timing + +```python +import time +from contextlib import contextmanager + +import structlog + +logger = structlog.get_logger("performance") + +@contextmanager +def log_timing(operation: str, **extra_fields): + """Context manager for timing arbitrary code blocks.""" + start = time.perf_counter() + yield + duration_ms = (time.perf_counter() - start) * 1000 + logger.info( + "operation_duration", + operation=operation, + duration_ms=round(duration_ms, 2), + **extra_fields, + ) + +# Usage +async def rebuild_search_index(): + with log_timing("rebuild_search_index", index="products"): + products = await product_repo.get_all() + await search_service.reindex(products) +``` + +#### TypeScript — timing wrapper + +```typescript +import { createLogger } from "./logger"; + +const perfLog = createLogger("performance"); + +export async function withTiming( + operation: string, + fn: () => Promise, + slowThresholdMs = 1000 +): Promise { + const start = performance.now(); + try { + const result = await fn(); + return result; + } finally { + const durationMs = performance.now() - start; + const logFn = durationMs > slowThresholdMs ? perfLog.warn : perfLog.debug; + logFn( + { + operation, + durationMs: Math.round(durationMs * 100) / 100, + slow: durationMs > slowThresholdMs, + }, + "operation_duration" + ); + } +} + +// Usage +const profile = await withTiming("fetch_user_profile", () => + userRepo.getWithPreferences(userId) +); +``` + +#### Slow query logging + +```python +# Python — SQLAlchemy event listener for slow queries +from sqlalchemy import event +from sqlalchemy.engine import Engine + +import structlog + +logger = structlog.get_logger("database") + +SLOW_QUERY_THRESHOLD_MS = 500 + +@event.listens_for(Engine, "before_cursor_execute") +def before_cursor_execute(conn, cursor, statement, parameters, context, executemany): + conn.info.setdefault("query_start_time", []).append(time.perf_counter()) + +@event.listens_for(Engine, "after_cursor_execute") +def after_cursor_execute(conn, cursor, statement, parameters, context, executemany): + total_ms = (time.perf_counter() - conn.info["query_start_time"].pop()) * 1000 + if total_ms > SLOW_QUERY_THRESHOLD_MS: + logger.warning( + "slow_query", + duration_ms=round(total_ms, 2), + statement=statement[:500], # Truncate long queries + parameters=str(parameters)[:200], + ) +``` + +--- + +## Best Practices + +1. **Use structured logging from day one** — start with JSON output and key-value pairs instead of formatted strings. Switching from `f"User {user_id} created"` to `logger.info("user_created", user_id=user_id)` costs nothing upfront but saves hours when debugging in production. + +2. **Log events, not sentences** — use snake_case event names (`order_placed`, `payment_failed`) rather than prose messages (`"An order was placed by the user"`). Event names are searchable, filterable, and easy to aggregate. + +3. **Include the right context at the right level** — every log line should include enough context to be useful in isolation: relevant IDs (user, order, request), operation name, and outcome. Avoid logging the same error at every layer of the call stack. + +4. **Set log levels per environment** — use DEBUG in development, INFO in staging, and INFO or WARNING in production. Never leave DEBUG enabled in production — it generates excessive volume and may expose sensitive internals. + +5. **Centralize logging configuration** — configure loggers once at application startup, not in individual modules. Every module should call `get_logger(__name__)` and inherit the shared configuration. + +6. **Always redact sensitive data** — build redaction into the logging pipeline as a processor or serializer. Do not rely on developers remembering to exclude passwords or tokens from log calls. + +7. **Use correlation IDs for every request** — generate a unique ID at the entry point and propagate it through all downstream calls. This is the single most important pattern for debugging distributed systems. + +8. **Set up log rotation and retention policies** — configure maximum file sizes, rotation intervals, and retention periods. Production logs without rotation will fill disks. Use log aggregation services (ELK, Datadog, CloudWatch) rather than relying on local files. + +--- + +## Common Pitfalls + +1. **Logging sensitive data** — passwords, API keys, JWTs, credit card numbers, and PII end up in logs more often than expected. Once written, they persist in log storage and backups. Build redaction into the pipeline rather than relying on code review to catch every instance. + +2. **Using print() or console.log in production** — `print()` in Python and `console.log` in Node.js write to stdout without timestamps, levels, or structure. They cannot be filtered, aggregated, or searched. Replace them with a proper logger before deploying. + +3. **Logging too much at high levels** — setting every log call to INFO or ERROR creates alert fatigue and obscures real problems. Use DEBUG for diagnostic details and reserve ERROR for situations that require action. + +4. **Missing stack traces on errors** — logging `str(exception)` loses the stack trace. In Python, use `exc_info=True` or `logger.exception()`. In pino, pass the error as `{ err }` to get the full stack serialized. + +5. **Not testing log output** — logging code is code. If your redaction processor has a bug, secrets leak. Write unit tests that capture log output and assert on structure, redacted fields, and expected context. + +6. **Synchronous logging in async applications** — writing to files or network sinks synchronously from an async event loop blocks request processing. Use async-compatible transports (pino's worker thread, structlog with stdlib async handlers) or write to stdout and let the infrastructure handle routing. + +--- + +## Related Skills + +- `patterns/error-handling` — Exception handling patterns that complement error logging +- `patterns/api-client` — HTTP client patterns including logging outbound requests +- `frameworks/fastapi` — FastAPI middleware setup for request logging and correlation IDs +- `devops/docker` — Container logging drivers and log aggregation in Docker environments +- `databases/postgresql` — Logging database queries and slow query detection +- `databases/mongodb` — Logging database operations and aggregation pipelines diff --git a/.claude/skills/patterns/logging/references/log-levels.md b/.claude/skills/patterns/logging/references/log-levels.md new file mode 100644 index 0000000..bddd019 --- /dev/null +++ b/.claude/skills/patterns/logging/references/log-levels.md @@ -0,0 +1,157 @@ +# Log Levels Quick Reference + +## Level Summary + +| Level | When to Use | Audience | Production Default | +|-------|------------|----------|-------------------| +| **DEBUG** | Detailed diagnostic info | Developers debugging | Off | +| **INFO** | Routine operational events | Ops team monitoring | On | +| **WARNING** | Something unexpected but handled | Ops + Devs | On | +| **ERROR** | Operation failed, needs attention | On-call engineers | On | +| **CRITICAL** | System is unusable or data at risk | On-call + management | On + alert | + +--- + +## DEBUG + +**Purpose**: Fine-grained information useful only when diagnosing problems. + +**Turn on**: During local development or when investigating a specific issue. + +| Good | Bad | +|------|-----| +| `Parsing config file: /etc/app/config.yaml` | `Entering function parse_config` | +| `Cache miss for key user:123, fetching from DB` | `x = 5` | +| `SQL: SELECT * FROM users WHERE id=$1 [params: 123]` | `Here we go!` | +| `Retry attempt 2/3 for payment gateway` | `Debug debug debug` | +| `JWT token expires at 2025-01-29T10:00:00Z` | `token = eyJhbG...` (secret!) | + +**Rule**: Never log secrets, tokens, passwords, or PII at any level. + +--- + +## INFO + +**Purpose**: Confirm the system is working as expected. Key business events. + +| Good | Bad | +|------|-----| +| `Server started on port 8080` | `Server is running` (which port? which version?) | +| `User user:456 created account via OAuth (Google)` | `New user` | +| `Order ord:789 placed, total=$45.00, items=3` | `Order created` | +| `Migration v42 applied successfully (12 tables)` | `Migration done` | +| `Scheduled job "daily-report" completed in 4.2s` | `Job finished` | +| `Payment processed: txn:abc, amount=$99, method=card` | `Payment OK` | + +**Rule**: Include enough context to answer "what happened, to what, and relevant numbers." + +--- + +## WARNING + +**Purpose**: Something unexpected happened, but the system handled it. May indicate a future problem. + +| Good | Bad | +|------|-----| +| `Connection pool at 85% capacity (17/20)` | `Pool getting full` | +| `Deprecated API v1 called by client app:legacy (use v2)` | `Old API used` | +| `Disk space below 10% on /data (2.1 GB remaining)` | `Low disk` | +| `Request took 4.8s (threshold: 5s) for GET /api/search` | `Slow request` | +| `Config REDIS_URL missing, falling back to in-memory cache` | `No Redis` | +| `Rate limit approaching for IP 10.0.0.5: 90/100 requests` | `Almost rate limited` | + +**Rule**: Warnings should be actionable. If nobody would investigate, it's DEBUG or INFO. + +--- + +## ERROR + +**Purpose**: An operation failed. The system can continue, but something broke. + +| Good | Bad | +|------|-----| +| `Failed to send email to user:123: SMTP timeout after 30s` | `Email error` | +| `Payment declined for order:789: card_expired (Stripe)` | `Payment failed` | +| `Database query timeout after 10s: SELECT FROM orders WHERE...` | `DB error` | +| `File upload failed: S3 returned 503, bucket=media, key=img/456.jpg` | `Upload error` | +| `Unhandled exception in POST /api/orders: ValueError("...")` | (stack trace only, no context) | + +**Rule**: Include the operation, the target/ID, the error detail, and what was attempted. + +--- + +## CRITICAL + +**Purpose**: System is unusable or data integrity is at risk. Requires immediate human intervention. + +| Good | Bad | +|------|-----| +| `Database connection lost, all pools exhausted, 0/20 available` | `DB down` | +| `Disk full on /data, writes failing, data loss possible` | `No disk space` | +| `Security: 500 failed login attempts from IP 10.0.0.5 in 60s` | `Too many logins` | +| `Data corruption detected: order:789 total=-$50.00` | `Bad data` | +| `TLS certificate expires in 24h, auto-renewal failed` | `Cert expiring` | + +**Rule**: Every CRITICAL log should trigger an alert (PagerDuty, Slack, etc.). + +--- + +## Structured Logging Format + +### Python (structlog) + +```python +import structlog + +log = structlog.get_logger() + +log.info("order.placed", order_id="ord:789", total=45.00, items=3) +log.error("email.send_failed", user_id="user:123", error="SMTP timeout", retry=2) +``` + +### TypeScript (pino) + +```typescript +import pino from "pino"; + +const log = pino({ level: "info" }); + +log.info({ orderId: "ord:789", total: 45.0, items: 3 }, "order.placed"); +log.error({ userId: "user:123", err, retry: 2 }, "email.send_failed"); +``` + +### Key-Value Best Practices + +| Field | Purpose | Example | +|-------|---------|---------| +| `event` / message | What happened | `"order.placed"` | +| `request_id` | Trace across services | `"req_abc123"` | +| `user_id` | Who triggered it | `"user:456"` | +| `duration_ms` | How long it took | `142` | +| `error` | Error message (not stack in prod) | `"connection refused"` | +| `component` | Which module/service | `"payment-gateway"` | + +--- + +## Configuration by Environment + +| Environment | Minimum Level | Structured? | Destination | +|-------------|--------------|-------------|-------------| +| Local dev | DEBUG | No (human-readable) | stdout | +| CI/Test | WARNING | No | stdout | +| Staging | DEBUG | Yes (JSON) | Log aggregator | +| Production | INFO | Yes (JSON) | Log aggregator | + +--- + +## Anti-Patterns + +| Anti-Pattern | Problem | Fix | +|-------------|---------|-----| +| Logging PII/secrets | Security/compliance violation | Redact or mask sensitive fields | +| `log.error()` in a loop | Log flooding, storage cost | Log once with count | +| `log.error("Error: " + err)` | Missing context, hard to search | Use structured fields | +| Logging at wrong level | Alert fatigue or missed issues | Follow the guide above | +| Catch-log-rethrow | Duplicate log entries | Log at the handling site only | +| No request_id | Cannot correlate logs | Add correlation ID middleware | +| Logging full request bodies | Performance, storage, PII risk | Log summary fields only | diff --git a/.claude/skills/patterns/state-management/SKILL.md b/.claude/skills/patterns/state-management/SKILL.md new file mode 100644 index 0000000..29c51d1 --- /dev/null +++ b/.claude/skills/patterns/state-management/SKILL.md @@ -0,0 +1,782 @@ +--- +name: state-management +description: > + State management patterns for React and Python applications. Use this skill when choosing between useState, useReducer, context, Zustand, Jotai, or TanStack Query. Also applies to server state, form state, URL state, and Python application state with dataclasses and Pydantic. Trigger whenever someone asks about state architecture, global state, caching API responses, or managing complex form state. +--- + +# State Management Patterns + +## When to Use + +- Choosing between local, shared, or global state in a React application +- Setting up server state caching with TanStack Query or SWR +- Building forms with validation, arrays, and nested fields +- Syncing application state with URL search parameters +- Designing Python domain models with dataclasses or Pydantic +- Refactoring prop-drilling into a shared store +- Deciding whether to add a state management library or keep things simple + +## When NOT to Use + +- Static sites with no interactive state (pure content pages, docs) +- Server-only rendering with no client-side interactivity +- Simple CRUD backends where database is the only source of truth and there is no in-process state to manage + +--- + +## Core Patterns + +### 1. Local vs Global State Decision Tree + +Before reaching for a library, walk through this decision tree. + +``` +Is the state used by a single component? +├── YES --> useState or useReducer +└── NO + Is it shared by a parent and 1-2 direct children? + ├── YES --> Lift state up to the common parent, pass via props + └── NO + Is it server data (fetched from an API)? + ├── YES --> TanStack Query (useQuery / useMutation) + └── NO + Is it URL-representable (filters, pagination, tabs)? + ├── YES --> URL state (useSearchParams / nuqs) + └── NO + Is it form data with validation? + ├── YES --> react-hook-form + zod + └── NO + Zustand store (or Jotai for atomic state) +``` + +**Rules of thumb:** + +- Start with the simplest option. Only add a library when props become painful. +- Server state and client state are different concerns. Never put fetched API data in Zustand; use TanStack Query instead. +- URL state is free persistence. If the user should be able to bookmark or share the current view, put it in the URL. +- Form state belongs to the form library. Do not mirror react-hook-form values in a Zustand store. + +--- + +### 2. React State Patterns + +**useState for simple values** + +```typescript +function Counter() { + const [count, setCount] = useState(0); + + return ( + + ); +} +``` + +**useReducer for complex state with multiple transitions** + +```typescript +interface TimerState { + status: "idle" | "running" | "paused"; + elapsed: number; +} + +type TimerAction = + | { type: "start" } + | { type: "pause" } + | { type: "reset" } + | { type: "tick" }; + +function timerReducer(state: TimerState, action: TimerAction): TimerState { + switch (action.type) { + case "start": + return { ...state, status: "running" }; + case "pause": + return { ...state, status: "paused" }; + case "reset": + return { status: "idle", elapsed: 0 }; + case "tick": + return state.status === "running" + ? { ...state, elapsed: state.elapsed + 1 } + : state; + } +} + +function Timer() { + const [state, dispatch] = useReducer(timerReducer, { + status: "idle", + elapsed: 0, + }); + + useEffect(() => { + if (state.status !== "running") return; + const id = setInterval(() => dispatch({ type: "tick" }), 1000); + return () => clearInterval(id); + }, [state.status]); + + return ( +
+

{state.elapsed}s

+ {state.status !== "running" && ( + + )} + {state.status === "running" && ( + + )} + +
+ ); +} +``` + +**When to pick which:** + +| Criteria | useState | useReducer | +|----------|----------|------------| +| Single primitive value | Yes | Overkill | +| Multiple related fields | Possible | Preferred | +| Complex transitions | Messy | Clean | +| Needs testing in isolation | Hard | Easy (test the reducer) | + +--- + +### 3. Global State (Zustand) + +Zustand is lightweight, TypeScript-friendly, and avoids the boilerplate of Redux. + +**Basic store** + +```typescript +import { create } from "zustand"; + +interface AuthStore { + user: User | null; + token: string | null; + login: (user: User, token: string) => void; + logout: () => void; +} + +const useAuthStore = create((set) => ({ + user: null, + token: null, + login: (user, token) => set({ user, token }), + logout: () => set({ user: null, token: null }), +})); + +// In components - use selectors to avoid unnecessary re-renders +function UserMenu() { + const user = useAuthStore((s) => s.user); + const logout = useAuthStore((s) => s.logout); + + if (!user) return ; + return ( +
+ {user.name} + +
+ ); +} +``` + +**Slices pattern for large stores** + +```typescript +import { create, type StateCreator } from "zustand"; +import { devtools, persist } from "zustand/middleware"; + +// Each slice is its own interface + creator +interface CartSlice { + items: CartItem[]; + addItem: (item: CartItem) => void; + removeItem: (id: string) => void; + clearCart: () => void; +} + +interface UISlice { + sidebarOpen: boolean; + toggleSidebar: () => void; +} + +const createCartSlice: StateCreator< + CartSlice & UISlice, + [], + [], + CartSlice +> = (set) => ({ + items: [], + addItem: (item) => + set((state) => ({ items: [...state.items, item] })), + removeItem: (id) => + set((state) => ({ items: state.items.filter((i) => i.id !== id) })), + clearCart: () => set({ items: [] }), +}); + +const createUISlice: StateCreator< + CartSlice & UISlice, + [], + [], + UISlice +> = (set) => ({ + sidebarOpen: false, + toggleSidebar: () => set((state) => ({ sidebarOpen: !state.sidebarOpen })), +}); + +// Combine slices with middleware +const useAppStore = create()( + devtools( + persist( + (...args) => ({ + ...createCartSlice(...args), + ...createUISlice(...args), + }), + { + name: "app-store", + partialize: (state) => ({ items: state.items }), // only persist cart + } + ) + ) +); +``` + +**Zustand best practices:** + +- Always use selectors (`useStore((s) => s.field)`) instead of the whole store. +- Keep stores small and focused. One store per domain, not one mega-store. +- Use `persist` middleware for state that should survive page reloads. +- Use `devtools` middleware in development for Redux DevTools integration. + +--- + +### 4. Server State (TanStack Query) + +Server state (data from APIs) has different needs than client state: caching, background refetching, deduplication, pagination. + +**Basic query** + +```typescript +import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query"; + +// Query key conventions: [entity, ...params] +const userKeys = { + all: ["users"] as const, + list: (filters: UserFilters) => ["users", "list", filters] as const, + detail: (id: string) => ["users", "detail", id] as const, +}; + +function useUser(id: string) { + return useQuery({ + queryKey: userKeys.detail(id), + queryFn: () => api.users.get(id), + staleTime: 5 * 60 * 1000, // 5 minutes + }); +} + +function UserProfile({ id }: { id: string }) { + const { data: user, isLoading, error } = useUser(id); + + if (isLoading) return ; + if (error) return ; + + return
{user.name}
; +} +``` + +**Mutations with optimistic updates** + +```typescript +function useUpdateUser() { + const queryClient = useQueryClient(); + + return useMutation({ + mutationFn: (data: UpdateUserInput) => api.users.update(data.id, data), + onMutate: async (newData) => { + // Cancel outgoing refetches + await queryClient.cancelQueries({ + queryKey: userKeys.detail(newData.id), + }); + + // Snapshot current value for rollback + const previous = queryClient.getQueryData(userKeys.detail(newData.id)); + + // Optimistically update + queryClient.setQueryData(userKeys.detail(newData.id), (old: User) => ({ + ...old, + ...newData, + })); + + return { previous }; + }, + onError: (_err, newData, context) => { + // Rollback on failure + if (context?.previous) { + queryClient.setQueryData( + userKeys.detail(newData.id), + context.previous + ); + } + }, + onSettled: (_data, _err, variables) => { + // Always refetch after mutation to ensure consistency + queryClient.invalidateQueries({ + queryKey: userKeys.detail(variables.id), + }); + }, + }); +} +``` + +**Prefetching for instant navigation** + +```typescript +function UserListItem({ user }: { user: UserSummary }) { + const queryClient = useQueryClient(); + + const prefetch = () => { + queryClient.prefetchQuery({ + queryKey: userKeys.detail(user.id), + queryFn: () => api.users.get(user.id), + staleTime: 60_000, + }); + }; + + return ( + + {user.name} + + ); +} +``` + +--- + +### 5. Form State + +Use react-hook-form for performance (uncontrolled inputs) and zod for schema validation. + +**Basic form with validation** + +```typescript +import { useForm } from "react-hook-form"; +import { zodResolver } from "@hookform/resolvers/zod"; +import { z } from "zod"; + +const createUserSchema = z.object({ + name: z.string().min(1, "Name is required").max(100), + email: z.string().email("Invalid email address"), + role: z.enum(["admin", "editor", "viewer"]), + notifications: z.boolean().default(true), +}); + +type CreateUserForm = z.infer; + +function CreateUserForm() { + const { + register, + handleSubmit, + formState: { errors, isSubmitting }, + } = useForm({ + resolver: zodResolver(createUserSchema), + defaultValues: { + role: "viewer", + notifications: true, + }, + }); + + const onSubmit = async (data: CreateUserForm) => { + await api.users.create(data); + }; + + return ( +
+
+ + + {errors.name &&

{errors.name.message}

} +
+ +
+ + + {errors.email &&

{errors.email.message}

} +
+ +
+ + +
+ + +
+ ); +} +``` + +**Dynamic field arrays** + +```typescript +import { useFieldArray } from "react-hook-form"; + +const orderSchema = z.object({ + customer: z.string().min(1), + items: z + .array( + z.object({ + productId: z.string().min(1), + quantity: z.number().int().min(1).max(999), + }) + ) + .min(1, "At least one item is required"), +}); + +type OrderForm = z.infer; + +function OrderForm() { + const { register, control, handleSubmit } = useForm({ + resolver: zodResolver(orderSchema), + defaultValues: { items: [{ productId: "", quantity: 1 }] }, + }); + + const { fields, append, remove } = useFieldArray({ + control, + name: "items", + }); + + return ( +
+ {fields.map((field, index) => ( +
+ + + +
+ ))} + + +
+ ); +} +``` + +--- + +### 6. URL State + +Encode filters, pagination, and view settings in the URL so users can bookmark and share. + +**Using nuqs (type-safe URL search params)** + +```typescript +import { useQueryState, parseAsInteger, parseAsStringEnum } from "nuqs"; + +const sortOptions = ["name", "date", "price"] as const; + +function ProductList() { + const [search, setSearch] = useQueryState("q", { defaultValue: "" }); + const [page, setPage] = useQueryState("page", parseAsInteger.withDefault(1)); + const [sort, setSort] = useQueryState( + "sort", + parseAsStringEnum(sortOptions).withDefault("name") + ); + + // URL looks like: /products?q=shoes&page=2&sort=price + const { data } = useQuery({ + queryKey: ["products", { search, page, sort }], + queryFn: () => api.products.list({ search, page, sort }), + }); + + return ( +
+ setSearch(e.target.value || null)} + placeholder="Search products..." + /> + + + + + +
+ ); +} +``` + +**Using React Router useSearchParams** + +```typescript +import { useSearchParams } from "react-router-dom"; + +function FilteredList() { + const [searchParams, setSearchParams] = useSearchParams(); + + const status = searchParams.get("status") ?? "all"; + const page = Number(searchParams.get("page") ?? "1"); + + const updateFilter = (key: string, value: string | null) => { + setSearchParams((prev) => { + const next = new URLSearchParams(prev); + if (value === null) { + next.delete(key); + } else { + next.set(key, value); + } + // Reset page when filter changes + if (key !== "page") next.set("page", "1"); + return next; + }); + }; + + return ( +
+ +
+ ); +} +``` + +--- + +### 7. Python State + +Use dataclasses for lightweight domain objects and Pydantic for validated external data. Combine with the repository pattern for persistence. + +**Dataclasses for domain objects** + +```python +from dataclasses import dataclass, field +from datetime import datetime +from enum import Enum +from uuid import UUID, uuid4 + + +class OrderStatus(str, Enum): + DRAFT = "draft" + CONFIRMED = "confirmed" + SHIPPED = "shipped" + DELIVERED = "delivered" + CANCELLED = "cancelled" + + +@dataclass +class OrderItem: + product_id: str + quantity: int + unit_price: float + + @property + def total(self) -> float: + return self.quantity * self.unit_price + + +@dataclass +class Order: + customer_id: str + items: list[OrderItem] = field(default_factory=list) + id: UUID = field(default_factory=uuid4) + status: OrderStatus = OrderStatus.DRAFT + created_at: datetime = field(default_factory=datetime.utcnow) + + @property + def subtotal(self) -> float: + return sum(item.total for item in self.items) + + def confirm(self) -> None: + if self.status != OrderStatus.DRAFT: + raise ValueError(f"Cannot confirm order in '{self.status}' state") + if not self.items: + raise ValueError("Cannot confirm an empty order") + self.status = OrderStatus.CONFIRMED + + def cancel(self) -> None: + if self.status in (OrderStatus.DELIVERED, OrderStatus.CANCELLED): + raise ValueError(f"Cannot cancel order in '{self.status}' state") + self.status = OrderStatus.CANCELLED +``` + +**Pydantic for validated external input** + +```python +from pydantic import BaseModel, Field, field_validator + + +class CreateOrderRequest(BaseModel): + customer_id: str = Field(min_length=1, max_length=50) + items: list["OrderItemInput"] = Field(min_length=1) + + @field_validator("items") + @classmethod + def no_duplicate_products(cls, items: list["OrderItemInput"]) -> list["OrderItemInput"]: + product_ids = [item.product_id for item in items] + if len(product_ids) != len(set(product_ids)): + raise ValueError("Duplicate product IDs are not allowed") + return items + + +class OrderItemInput(BaseModel): + product_id: str = Field(min_length=1) + quantity: int = Field(ge=1, le=999) +``` + +**Repository pattern for persistence** + +```python +from abc import ABC, abstractmethod + + +class OrderRepository(ABC): + @abstractmethod + async def save(self, order: Order) -> None: ... + + @abstractmethod + async def get(self, order_id: UUID) -> Order | None: ... + + @abstractmethod + async def list_by_customer(self, customer_id: str) -> list[Order]: ... + + +class PostgresOrderRepository(OrderRepository): + def __init__(self, pool) -> None: + self.pool = pool + + async def save(self, order: Order) -> None: + async with self.pool.acquire() as conn: + await conn.execute( + """ + INSERT INTO orders (id, customer_id, status, created_at) + VALUES ($1, $2, $3, $4) + ON CONFLICT (id) DO UPDATE SET status = $3 + """, + order.id, + order.customer_id, + order.status.value, + order.created_at, + ) + # Upsert items... + + async def get(self, order_id: UUID) -> Order | None: + async with self.pool.acquire() as conn: + row = await conn.fetchrow( + "SELECT * FROM orders WHERE id = $1", order_id + ) + if row is None: + return None + items = await conn.fetch( + "SELECT * FROM order_items WHERE order_id = $1", order_id + ) + return self._row_to_order(row, items) + + async def list_by_customer(self, customer_id: str) -> list[Order]: + async with self.pool.acquire() as conn: + rows = await conn.fetch( + "SELECT * FROM orders WHERE customer_id = $1 ORDER BY created_at DESC", + customer_id, + ) + # Fetch items for each order... + return [self._row_to_order(row, items) for row, items in results] + + def _row_to_order(self, row, item_rows) -> Order: + return Order( + id=row["id"], + customer_id=row["customer_id"], + status=OrderStatus(row["status"]), + created_at=row["created_at"], + items=[ + OrderItem( + product_id=r["product_id"], + quantity=r["quantity"], + unit_price=float(r["unit_price"]), + ) + for r in item_rows + ], + ) +``` + +--- + +## Best Practices + +1. **Start local, promote when needed.** Begin with `useState`. Only move state up or into a store when two or more unrelated components need the same data. Premature globalization makes refactoring painful. + +2. **Separate server state from client state.** Use TanStack Query (or SWR) for anything fetched from an API. These libraries handle caching, deduplication, background refetching, and stale-while-revalidate. Do not duplicate fetched data in Zustand. + +3. **Use selectors to prevent re-renders.** In Zustand, always select the specific field you need: `useStore((s) => s.count)`, not `useStore()`. The latter re-renders on every store change. + +4. **Co-locate state with the component that owns it.** If only `` uses `isOpen`, keep that state inside ``. Moving it to a global store just because "it might be needed later" creates unnecessary coupling. + +5. **Derive, do not duplicate.** If `fullName` can be computed from `firstName` and `lastName`, compute it on the fly or with `useMemo`. Storing derived values introduces synchronization bugs. + +6. **Validate at the boundary, trust internally.** Use Pydantic or Zod to validate data when it enters the system (API requests, form submissions, external events). Once validated, pass typed objects without re-checking. + +7. **Keep URL state minimal.** Only encode values the user would want to bookmark or share: active tab, search query, page number, sort column. Do not put ephemeral UI state (hover, open dropdown) in the URL. + +8. **Treat form state as its own domain.** Let react-hook-form manage form values, dirty tracking, and validation. Submit the validated result to your mutation or API call. Do not synchronize form fields with external stores. + +--- + +## Common Pitfalls + +1. **Putting everything in global state.** Not all state needs to be global. A modal's open/closed state, an input's current text, or a component's loading spinner should stay local. Global stores should hold state that genuinely needs to be shared across distant parts of the tree. + +2. **Storing server data in Zustand.** Zustand has no built-in cache invalidation, stale detection, or background refetch. Using it for API data means you are rebuilding TanStack Query poorly. Use the right tool for the job. + +3. **Forgetting to invalidate queries after mutations.** After a `useMutation` succeeds, call `queryClient.invalidateQueries` with the affected keys. Without this, the UI shows stale data until the next refetch interval. + +4. **Over-using React Context for frequently changing state.** Every Context value change re-renders every consumer. Context is good for low-frequency values (theme, locale, auth). For high-frequency updates (cursor position, scroll offset), use Zustand or a ref. + +5. **Duplicating form state.** Calling `useForm()` and then also storing the same values in `useState` or Zustand means two sources of truth that can drift apart. Let the form library be the single owner. + +6. **Ignoring URL state for filterable lists.** If a user applies filters and then hits the back button or refreshes, losing the filters is a bad experience. Encode filters in the URL so they survive navigation. + +--- + +## Related Skills + +- `frameworks/react` - React component patterns and hooks +- `frameworks/nextjs` - Next.js server components and data fetching +- `languages/typescript` - TypeScript types and generics +- `patterns/caching` - Cache strategies and invalidation patterns diff --git a/.claude/skills/patterns/state-management/references/state-decision-tree.md b/.claude/skills/patterns/state-management/references/state-decision-tree.md new file mode 100644 index 0000000..674bc4c --- /dev/null +++ b/.claude/skills/patterns/state-management/references/state-decision-tree.md @@ -0,0 +1,143 @@ +# State Management Decision Tree + +## Primary Decision Tree + +``` +What kind of state is it? +│ +├─ SERVER DATA (fetched from API/DB) +│ │ +│ ├─ React/Next.js project? +│ │ ├─ YES ──> TanStack Query (React Query) +│ │ │ - Auto caching, dedup, background refresh +│ │ │ - Stale-while-revalidate out of the box +│ │ │ - DevTools for debugging +│ │ │ +│ │ └─ Next.js App Router with Server Components? +│ │ └─ Consider: fetch() in Server Components + revalidation +│ │ - No client-side state library needed +│ │ - Use TanStack Query only for client-interactive data +│ │ +│ └─ Need real-time sync? +│ ├─ WebSocket data ──> TanStack Query + custom subscription +│ └─ Collaborative ──> Liveblocks, Yjs, or Partykit +│ +├─ URL STATE (filters, pagination, search, tabs) +│ │ +│ ├─ Next.js App Router ──> useSearchParams() + useRouter() +│ ├─ React Router ──> useSearchParams() +│ └─ Plain React ──> nuqs or custom URL sync hook +│ +│ Why URL state? Shareable links, back/forward navigation, +│ bookmarkable, SSR-friendly. +│ +├─ FORM STATE (input values, validation, dirty/touched) +│ │ +│ ├─ Complex forms (multi-step, dynamic fields, arrays) +│ │ └─ react-hook-form + zod +│ │ - Uncontrolled by default (performant) +│ │ - Schema validation with zod resolver +│ │ +│ ├─ Simple forms (login, search, contact) +│ │ └─ Server Actions (Next.js) or native form + useState +│ │ +│ └─ Form + server state (edit existing record) +│ └─ TanStack Query (fetch) + react-hook-form (edit) +│ - Populate form with query data +│ - Submit with mutation +│ +├─ GLOBAL CLIENT STATE (shared across many components) +│ │ +│ ├─ Simple (theme, sidebar open, user preferences) +│ │ │ +│ │ ├─ Changes rarely ──> React Context +│ │ │ (Wrap app, useContext to consume) +│ │ │ +│ │ └─ Changes often or many consumers ──> Zustand +│ │ - Avoids Context re-render problem +│ │ - Selector-based subscriptions +│ │ - Tiny bundle, minimal boilerplate +│ │ +│ ├─ Complex (shopping cart, multi-step wizard, editor) +│ │ └─ Zustand (or Jotai for atomic state) +│ │ +│ └─ Need devtools and time-travel debugging? +│ └─ Zustand with devtools middleware +│ +├─ LOCAL COMPONENT STATE (only used in one component) +│ │ +│ ├─ Single value ──> useState +│ │ const [count, setCount] = useState(0); +│ │ +│ ├─ Related values or complex transitions ──> useReducer +│ │ const [state, dispatch] = useReducer(reducer, initial); +│ │ +│ └─ Derived value (computed from other state) ──> useMemo +│ const total = useMemo(() => items.reduce(...), [items]); +│ +└─ TRANSIENT UI STATE (animations, hover, drag position) + │ + ├─ CSS can handle it? ──> Use CSS (transitions, :hover) + ├─ Ref-based (no re-render needed) ──> useRef + └─ Needs re-render ──> useState (local) +``` + +## Quick Lookup Table + +| State Type | Recommended Tool | When NOT to Use | +|-----------|-----------------|-----------------| +| Server data | TanStack Query | Data never changes, or SSR-only | +| URL params | useSearchParams | Ephemeral UI state (hover, etc.) | +| Form inputs | react-hook-form | Single `` | +| Global UI | Zustand | Only 1-2 consumers (use Context) | +| Global UI (simple) | React Context | Frequent updates with many consumers | +| Local state | useState | Complex state transitions | +| Complex local | useReducer | Single boolean toggle | +| Derived data | useMemo | Cheap computations | +| No re-render needed | useRef | Value that should trigger re-render | + +## Library Comparison + +| Library | Bundle Size | Boilerplate | Learning Curve | Best For | +|---------|------------|-------------|----------------|----------| +| useState/useReducer | 0 KB | Minimal | Low | Local state | +| React Context | 0 KB | Low | Low | Rarely-changing global state | +| Zustand | ~1 KB | Minimal | Low | Global client state | +| Jotai | ~3 KB | Minimal | Medium | Atomic/derived state | +| TanStack Query | ~12 KB | Medium | Medium | Server state | +| Redux Toolkit | ~30 KB | High | High | Large teams needing strict patterns | + +## Common Mistakes + +| Mistake | Problem | Fix | +|---------|---------|-----| +| Storing server data in Zustand/Redux | Manual cache invalidation, stale data | Use TanStack Query | +| Storing URL state in useState | Not shareable, lost on refresh | Use URL search params | +| Putting everything in global state | Unnecessary re-renders, complexity | Colocate state where used | +| Context for frequently changing data | Re-renders all consumers | Use Zustand with selectors | +| Duplicating derived state | Out-of-sync bugs | Compute with useMemo | +| useState for complex transitions | Inconsistent intermediate states | Use useReducer | + +## Zustand Quick Setup + +```typescript +import { create } from "zustand"; + +interface CartStore { + items: CartItem[]; + addItem: (item: CartItem) => void; + removeItem: (id: string) => void; + total: () => number; +} + +const useCartStore = create((set, get) => ({ + items: [], + addItem: (item) => set((s) => ({ items: [...s.items, item] })), + removeItem: (id) => set((s) => ({ items: s.items.filter(i => i.id !== id) })), + total: () => get().items.reduce((sum, i) => sum + i.price, 0), +})); + +// Usage with selector (only re-renders when items change) +const items = useCartStore((s) => s.items); +const addItem = useCartStore((s) => s.addItem); +``` diff --git a/.claude/skills/security/owasp/SKILL.md b/.claude/skills/security/owasp/SKILL.md index afad416..f5f2824 100644 --- a/.claude/skills/security/owasp/SKILL.md +++ b/.claude/skills/security/owasp/SKILL.md @@ -1,74 +1,556 @@ -# OWASP Security +--- +name: owasp +description: > + Use this skill when reviewing code for security vulnerabilities, implementing authentication or authorization flows, handling user input validation, or building web endpoints exposed to untrusted data. Trigger on keywords like XSS, SQL injection, CSRF, input sanitization, password hashing, and security headers. Also apply when auditing existing code for OWASP Top 10 compliance or conducting security-focused code reviews. +--- -## Description - -OWASP Top 10 security practices and secure coding patterns. +# OWASP Web Application Security ## When to Use - Security code reviews -- Implementing authentication -- Handling user input +- Implementing authentication or authorization +- Handling user input from untrusted sources +- Building or auditing web API endpoints +- Configuring CORS, CSP, or other security headers +- Managing secrets, tokens, or credentials in code +- Setting up rate limiting or brute force protection + +## When NOT to Use + +- General code style or formatting reviews with no security implications +- Non-web applications such as CLI tools, batch scripts, or desktop utilities +- Performance optimization tasks where security is not the concern +- Infrastructure-level security (firewall rules, network segmentation) --- ## Core Patterns -### Input Validation +### 1. Input Validation & Sanitization + +Always validate input at the boundary. Use allowlists over denylists. + +**Python (Pydantic)** ```python -# Always validate and sanitize -from pydantic import BaseModel, EmailStr +# BAD - no validation, accepts anything +@app.post("/users") +async def create_user(request: Request): + data = await request.json() + name = data["name"] # no length check, no type check + email = data["email"] # no format validation + role = data["role"] # user controls their own role + db.execute(f"INSERT INTO users VALUES ('{name}', '{email}', '{role}')") -class UserInput(BaseModel): +# GOOD - strict schema validation with Pydantic +from pydantic import BaseModel, EmailStr, Field +from enum import Enum + +class UserRole(str, Enum): + viewer = "viewer" + editor = "editor" + +class CreateUserRequest(BaseModel): + name: str = Field(min_length=1, max_length=100, pattern=r"^[a-zA-Z\s\-]+$") email: EmailStr - name: str = Field(min_length=1, max_length=100) + role: UserRole = UserRole.viewer # default to least privilege + +@app.post("/users") +async def create_user(payload: CreateUserRequest): + # Pydantic rejects invalid data before this code runs + db.add(User(name=payload.name, email=payload.email, role=payload.role)) ``` -### SQL Injection Prevention - -```python -# Never concatenate user input -# Bad -query = f"SELECT * FROM users WHERE id = {user_id}" - -# Good - parameterized -cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,)) -``` - -### XSS Prevention +**TypeScript (Zod)** ```typescript -// Never use innerHTML with user data -// Bad -element.innerHTML = userInput; +// BAD - trusting req.body directly +app.post("/users", (req, res) => { + const { name, email, role } = req.body; // no validation + db.query(`INSERT INTO users VALUES ('${name}', '${email}', '${role}')`); +}); -// Good -element.textContent = userInput; +// GOOD - validate with Zod at the boundary +import { z } from "zod"; + +const CreateUserSchema = z.object({ + name: z.string().min(1).max(100).regex(/^[a-zA-Z\s\-]+$/), + email: z.string().email(), + role: z.enum(["viewer", "editor"]).default("viewer"), +}); + +app.post("/users", (req, res) => { + const result = CreateUserSchema.safeParse(req.body); + if (!result.success) { + return res.status(400).json({ errors: result.error.flatten() }); + } + // result.data is typed and validated + await prisma.user.create({ data: result.data }); +}); ``` -### Authentication +**File Upload Validation** ```python -# Hash passwords properly +# GOOD - validate MIME type (not just extension), size, and sanitize filename +import magic + +ALLOWED_TYPES = {"image/jpeg", "image/png", "application/pdf"} +MAX_SIZE = 5 * 1024 * 1024 # 5 MB + +def validate_upload(file_bytes: bytes, filename: str) -> bool: + if len(file_bytes) > MAX_SIZE: + raise ValueError("File too large") + if magic.from_buffer(file_bytes, mime=True) not in ALLOWED_TYPES: + raise ValueError("Disallowed file type") + if ".." in filename or filename.startswith("."): + raise ValueError("Invalid filename") + return True +``` + +### 2. SQL Injection Prevention + +Never concatenate user input into SQL strings. Always use parameterized queries or ORM methods. + +**Raw SQL (Python)** + +```python +# BAD - string interpolation creates injection vector +def get_user(user_id: str): + query = f"SELECT * FROM users WHERE id = '{user_id}'" + # Input: "'; DROP TABLE users; --" destroys the table + cursor.execute(query) + +# GOOD - parameterized query +def get_user(user_id: str): + cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,)) + return cursor.fetchone() +``` + +**SQLAlchemy (Python)** + +```python +# BAD - text() with f-string +from sqlalchemy import text +result = session.execute(text(f"SELECT * FROM users WHERE name = '{name}'")) + +# GOOD - bound parameters with text() +result = session.execute(text("SELECT * FROM users WHERE name = :name"), {"name": name}) + +# GOOD - ORM query (automatically parameterized) +user = session.query(User).filter(User.name == name).first() +``` + +**Prisma (TypeScript)** + +```typescript +// BAD - raw query with interpolation +const user = await prisma.$queryRawUnsafe(`SELECT * FROM users WHERE id = '${id}'`); + +// GOOD - tagged template (auto-parameterized) +const user = await prisma.$queryRaw`SELECT * FROM users WHERE id = ${id}`; + +// GOOD - Prisma client methods (always safe) +const user = await prisma.user.findUnique({ where: { id } }); +``` + +### 3. XSS Prevention + +Prevent cross-site scripting by encoding output, setting CSP headers, and sanitizing HTML. + +**Output Encoding** + +```typescript +// BAD - renders raw user content as HTML +element.innerHTML = userComment; + +// GOOD - use textContent for plain text +element.textContent = userComment; + +// GOOD - React auto-escapes by default (don't bypass it) +return
{userComment}
; + +// BAD - dangerouslySetInnerHTML defeats React's protection +return
; +``` + +**Sanitizing HTML When You Must Render It** + +```typescript +// GOOD - sanitize with DOMPurify when HTML rendering is required +import DOMPurify from "dompurify"; + +const cleanHtml = DOMPurify.sanitize(userHtml, { + ALLOWED_TAGS: ["b", "i", "em", "strong", "a", "p", "br"], + ALLOWED_ATTR: ["href", "title"], +}); +return
; +``` + +### 4. Authentication Patterns + +**Password Hashing** + +```python +# BAD - plain text or weak hashing +hashed = hashlib.md5(password.encode()).hexdigest() # trivially crackable + +# GOOD - use argon2 (preferred) or bcrypt with proper cost from passlib.hash import argon2 hashed = argon2.hash(password) -verified = argon2.verify(password, hashed) +is_valid = argon2.verify(password, hashed) ``` -## Security Checklist +```typescript +// GOOD - bcrypt in Node.js +import bcrypt from "bcrypt"; -- [ ] Input validation on all user data -- [ ] Parameterized queries -- [ ] Output encoding -- [ ] Strong password hashing -- [ ] Secure session management -- [ ] HTTPS everywhere -- [ ] Security headers configured +const SALT_ROUNDS = 12; +const hashed = await bcrypt.hash(password, SALT_ROUNDS); +const isValid = await bcrypt.compare(password, hashed); +``` + +**JWT Best Practices** + +```python +# BAD - long-lived token, weak secret +token = jwt.encode({"user_id": 1, "exp": datetime.utcnow() + timedelta(days=365)}, + "secret123", algorithm="HS256") + +# GOOD - short expiry, strong secret, httpOnly cookie delivery +ACCESS_TOKEN_EXPIRY = timedelta(minutes=15) + +def create_access_token(user_id: int) -> str: + return jwt.encode( + {"sub": user_id, "exp": datetime.now(timezone.utc) + ACCESS_TOKEN_EXPIRY}, + os.environ["JWT_SECRET_KEY"], algorithm="HS256", + ) + +def set_token_cookie(response: Response, token: str): + response.set_cookie( + key="access_token", value=token, + httponly=True, secure=True, samesite="lax", # not accessible via JS, HTTPS only + max_age=int(ACCESS_TOKEN_EXPIRY.total_seconds()), + ) +``` + +**Session Management Rules** + +- Set session timeouts (30 minutes idle, 8 hours absolute) +- Regenerate session ID after login to prevent session fixation +- Store sessions server-side (Redis, database), not in cookies +- Clear sessions on logout (`request.session.clear()`) +- Use `httponly`, `secure`, and `samesite=lax` on session cookies + +### 5. Authorization & Access Control + +**RBAC Pattern** + +```python +# GOOD - role-based access control with decorator +from enum import Enum + +class Role(str, Enum): + admin = "admin" + editor = "editor" + viewer = "viewer" + +ROLE_HIERARCHY = {Role.admin: 3, Role.editor: 2, Role.viewer: 1} + +def require_role(minimum_role: Role): + def decorator(func): + async def wrapper(request: Request, *args, **kwargs): + user = request.state.user + if ROLE_HIERARCHY.get(user.role, 0) < ROLE_HIERARCHY[minimum_role]: + raise HTTPException(status_code=403) + return await func(request, *args, **kwargs) + return wrapper + return decorator + +@app.delete("/posts/{post_id}") +@require_role(Role.editor) +async def delete_post(request: Request, post_id: int): ... +``` + +**Middleware-Based Authorization (Express)** + +```typescript +// GOOD - authorization middleware +function requireRole(...allowedRoles: string[]) { + return (req: Request, res: Response, next: NextFunction) => { + if (!req.user || !allowedRoles.includes(req.user.role)) { + return res.status(403).json({ error: "Forbidden" }); + } + next(); + }; +} + +app.delete("/posts/:id", requireRole("admin", "editor"), deletePostHandler); +``` + +**Object-Level Permissions** + +```python +# BAD - checks auth but not ownership (any user can edit any document) +@app.put("/documents/{doc_id}") +async def update_document(doc_id: int, payload: UpdateDoc, user=Depends(get_current_user)): + doc = await db.get(Document, doc_id) + doc.content = payload.content + +# GOOD - verify ownership or admin role on every mutation +@app.put("/documents/{doc_id}") +async def update_document(doc_id: int, payload: UpdateDoc, user=Depends(get_current_user)): + doc = await db.get(Document, doc_id) + if not doc: + raise HTTPException(status_code=404) + if doc.owner_id != user.id and user.role != Role.admin: + raise HTTPException(status_code=403) + doc.content = payload.content +``` + +### 6. CORS Configuration + +**FastAPI** + +```python +# BAD - allows everything +from fastapi.middleware.cors import CORSMiddleware +app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_credentials=True, + allow_methods=["*"], allow_headers=["*"]) + +# GOOD - restrictive CORS +app.add_middleware( + CORSMiddleware, + allow_origins=["https://app.example.com", "https://staging.example.com"], + allow_credentials=True, + allow_methods=["GET", "POST", "PUT", "DELETE"], + allow_headers=["Authorization", "Content-Type"], +) +``` + +**Express** + +```typescript +// BAD +app.use(cors({ origin: true, credentials: true })); + +// GOOD - explicit allowlist with callback +const ALLOWED_ORIGINS = ["https://app.example.com"]; +app.use(cors({ + origin: (origin, cb) => { + if (!origin || ALLOWED_ORIGINS.includes(origin)) cb(null, true); + else cb(new Error("Not allowed by CORS")); + }, + credentials: true, + methods: ["GET", "POST", "PUT", "DELETE"], +})); +``` + +### 7. Security Headers + +**Express with Helmet** + +```typescript +// GOOD - Helmet sets secure defaults for all critical headers +import helmet from "helmet"; + +app.use(helmet({ + contentSecurityPolicy: { + directives: { + defaultSrc: ["'self'"], + scriptSrc: ["'self'"], + styleSrc: ["'self'", "'unsafe-inline'"], + imgSrc: ["'self'", "data:"], + frameAncestors: ["'none'"], + }, + }, + hsts: { maxAge: 31536000, includeSubDomains: true, preload: true }, +})); +``` + +**FastAPI** + +```python +# GOOD - security headers middleware +@app.middleware("http") +async def security_headers(request, call_next): + response = await call_next(request) + response.headers["Strict-Transport-Security"] = "max-age=31536000; includeSubDomains; preload" + response.headers["X-Content-Type-Options"] = "nosniff" + response.headers["X-Frame-Options"] = "DENY" + response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin" + response.headers["Permissions-Policy"] = "camera=(), microphone=(), geolocation=()" + response.headers["Content-Security-Policy"] = "default-src 'self'; frame-ancestors 'none';" + return response +``` + +### 8. Secret Management + +```python +# BAD - hardcoded secrets +DATABASE_URL = "postgresql://admin:p@ssw0rd@localhost/mydb" +API_KEY = "sk-1234567890abcdef" +JWT_SECRET = "mysecret" + +# GOOD - environment variables with validation +import os + +def get_required_env(key: str) -> str: + value = os.environ.get(key) + if not value: + raise RuntimeError(f"Required environment variable {key} is not set") + return value + +DATABASE_URL = get_required_env("DATABASE_URL") +API_KEY = get_required_env("API_KEY") +JWT_SECRET = get_required_env("JWT_SECRET") +``` + +**.env and .gitignore** + +```bash +# .env (NEVER commit this file) +DATABASE_URL=postgresql://admin:securepass@localhost/mydb +JWT_SECRET=a-very-long-random-string-from-openssl-rand +API_KEY=sk-prod-xxxxxxxxxxxx +``` + +```gitignore +# .gitignore - always include these +.env +.env.* +!.env.example +*.pem +*.key +credentials.json +``` + +Commit a `.env.example` with empty values to document required variables without exposing secrets. + +### 9. Rate Limiting + +**Python (FastAPI with slowapi)** + +```python +# GOOD - rate limiting on sensitive endpoints +from slowapi import Limiter +from slowapi.util import get_remote_address + +limiter = Limiter(key_func=get_remote_address) +app.state.limiter = limiter + +@app.post("/login") +@limiter.limit("5/minute") # brute force protection +async def login(request: Request, credentials: LoginRequest): + ... + +@app.post("/api/data") +@limiter.limit("100/minute") # general API rate limit +async def get_data(request: Request): + ... +``` + +**Express (express-rate-limit)** + +```typescript +// GOOD - tiered rate limiting +import rateLimit from "express-rate-limit"; + +const generalLimiter = rateLimit({ windowMs: 15 * 60 * 1000, max: 100 }); +const authLimiter = rateLimit({ windowMs: 15 * 60 * 1000, max: 5 }); + +app.use("/api/", generalLimiter); +app.use("/auth/login", authLimiter); +app.use("/auth/register", authLimiter); +``` + +### 10. Dependency Security + +```bash +# Python - audit dependencies +pip install pip-audit +pip-audit # scan for known vulnerabilities +pip-audit --fix # auto-fix where possible + +# Node.js - audit dependencies +npm audit # list vulnerabilities +npm audit fix # auto-fix compatible updates +pnpm audit # pnpm equivalent + +# Always commit lock files to ensure reproducible builds +# Python: requirements.txt or poetry.lock +# Node.js: package-lock.json, pnpm-lock.yaml, or yarn.lock +``` + +Run `npm audit --audit-level=high` and `pip-audit --strict` in CI (e.g., GitHub Actions on every PR and weekly schedule). Treat high-severity findings as build failures. + +--- + +## Best Practices + +1. **Validate at the boundary, trust nothing inside.** Every piece of user input -- query params, headers, request bodies, file uploads -- must be validated before processing. Use Pydantic or Zod schemas, not manual checks. + +2. **Apply the principle of least privilege everywhere.** Default to the most restrictive access. Grant permissions explicitly. Use role-based access control and verify object-level ownership on every mutation. + +3. **Never store or log secrets in plain text.** Use environment variables, a secret manager, or encrypted storage. Ensure secrets never appear in logs, error messages, or version control. + +4. **Use strong, adaptive password hashing.** Always use argon2 or bcrypt with a sufficient work factor. Never use MD5, SHA-1, or SHA-256 alone for password storage. + +5. **Set security headers on every response.** Enable HSTS, CSP, X-Content-Type-Options, X-Frame-Options, and Referrer-Policy. Use Helmet for Express and middleware for FastAPI. + +6. **Fail closed, not open.** When authentication or authorization checks encounter errors, deny access by default. Never fall through to an unprotected code path on exception. + +7. **Keep dependencies updated and audited.** Run `npm audit` and `pip-audit` in CI pipelines. Pin dependency versions with lock files. Review changelogs before major upgrades. + +8. **Enforce rate limiting on all public-facing endpoints.** Apply stricter limits on authentication and password reset endpoints. Use IP-based and account-based limiting together for defense in depth. + +--- ## Common Pitfalls -- **Trusting user input**: Always validate -- **SQL concatenation**: Use parameters -- **Storing plain passwords**: Use argon2/bcrypt +1. **Trusting client-side validation alone.** Attackers bypass browser validation trivially. Always re-validate on the server. + +2. **Using wildcard CORS with credentials.** `allow_origins=["*"]` with credentials is insecure and browsers reject it. Specify exact origins. + +3. **Storing JWTs in localStorage.** Any XSS can steal them. Use httpOnly, secure, sameSite cookies instead. + +4. **Returning detailed error messages in production.** Stack traces help attackers. Return generic messages to clients, log details server-side. + +5. **Using ORM raw query methods unsafely.** `$queryRawUnsafe` and `text()` with f-strings bypass ORM protections. Audit every raw SQL call. + +6. **Checking authentication but not authorization.** "Logged in" does not mean "authorized." Check object-level permissions on every write. + +7. **Disabling security in dev and shipping it.** CSP, CORS, HTTPS disabled for convenience can reach production. Use environment-aware config. + +8. **Ignoring dependency vulnerabilities.** Known CVEs in transitive deps are a top attack vector. Automate auditing in CI. + +--- + +## Security Review Checklist + +- [ ] All user input validated with schema (Pydantic / Zod) before processing +- [ ] No string concatenation or interpolation in SQL queries +- [ ] Passwords hashed with argon2 or bcrypt (never MD5/SHA) +- [ ] JWTs have short expiry, use httpOnly cookies, strong secret from env +- [ ] Authorization checked at object level, not just authentication +- [ ] CORS configured with explicit origin allowlist (no wildcards with credentials) +- [ ] Security headers set: CSP, HSTS, X-Content-Type-Options, X-Frame-Options +- [ ] No secrets hardcoded in source -- all from environment variables +- [ ] .env files listed in .gitignore, .env.example committed +- [ ] Rate limiting applied to login, registration, and password reset endpoints +- [ ] File uploads validated by MIME type, size, and sanitized filename +- [ ] Error responses do not leak stack traces or internal details +- [ ] Dependencies audited with npm audit / pip-audit (no high-severity CVEs) +- [ ] HTTPS enforced in production with HSTS preload +- [ ] No use of eval(), dangerouslySetInnerHTML (without DOMPurify), or innerHTML + +--- + +## Related Skills + +- `patterns/authentication` - Authentication and authorization implementation patterns +- `patterns/error-handling` - Secure error handling that avoids leaking sensitive information +- `devops/docker` — Container security hardening +- `methodology/defense-in-depth` — Multi-layer security validation diff --git a/.claude/skills/security/owasp/references/owasp-top10-cheatsheet.md b/.claude/skills/security/owasp/references/owasp-top10-cheatsheet.md new file mode 100644 index 0000000..97b9a73 --- /dev/null +++ b/.claude/skills/security/owasp/references/owasp-top10-cheatsheet.md @@ -0,0 +1,193 @@ +# OWASP Top 10 (2021) Cheat Sheet + +Quick reference for the OWASP Top 10 web application security risks. + +--- + +## A01: Broken Access Control + +**Risk**: Users act outside intended permissions (view other users' data, modify access). + +**Prevention**: Deny by default. Enforce ownership. Disable directory listing. Log failures. + +```python +# Enforce ownership check +def get_order(order_id, current_user): + order = db.query(Order).get(order_id) + if order.user_id != current_user.id: + raise PermissionError("Access denied") + return order +``` + +## A02: Cryptographic Failures + +**Risk**: Exposure of sensitive data due to weak or missing encryption. + +**Prevention**: Encrypt data at rest and in transit. Use strong algorithms (AES-256, bcrypt). Never store plaintext passwords. + +```python +from passlib.hash import bcrypt +hashed = bcrypt.hash(password) +assert bcrypt.verify(password, hashed) +``` + +## A03: Injection + +**Risk**: Untrusted data sent to an interpreter as part of a command or query. + +**Prevention**: Use parameterized queries. Validate and sanitize all input. Use ORMs. + +```python +# WRONG: cursor.execute(f"SELECT * FROM users WHERE id = {user_id}") +# RIGHT: +cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,)) +``` + +```typescript +// WRONG: db.query(`SELECT * FROM users WHERE id = ${id}`) +// RIGHT: +db.query("SELECT * FROM users WHERE id = $1", [id]); +``` + +## A04: Insecure Design + +**Risk**: Missing or ineffective security controls due to flawed architecture. + +**Prevention**: Use threat modeling. Apply secure design patterns. Establish reference architectures. Write abuse-case tests. + +```python +# Rate-limit sensitive operations +from functools import lru_cache +from datetime import datetime, timedelta + +LOGIN_ATTEMPTS = {} # Use Redis in production + +def check_rate_limit(ip: str, max_attempts=5, window=300): + now = datetime.now().timestamp() + attempts = [t for t in LOGIN_ATTEMPTS.get(ip, []) if now - t < window] + if len(attempts) >= max_attempts: + raise RateLimitExceeded() + attempts.append(now) + LOGIN_ATTEMPTS[ip] = attempts +``` + +## A05: Security Misconfiguration + +**Risk**: Default configs, incomplete setups, open cloud storage, verbose errors. + +**Prevention**: Repeatable hardening process. Minimal platform. Remove unused features. Review cloud permissions. + +```yaml +# Docker: don't run as root +FROM python:3.12-slim +RUN useradd -m appuser +USER appuser +``` + +## A06: Vulnerable and Outdated Components + +**Risk**: Using components with known vulnerabilities. + +**Prevention**: Remove unused dependencies. Monitor CVEs. Use `pip audit`, `npm audit`. Pin versions. + +```bash +pip audit # Python +npm audit # Node.js +npx depcheck # Find unused deps +``` + +## A07: Identification and Authentication Failures + +**Risk**: Weak authentication, credential stuffing, session fixation. + +**Prevention**: MFA. Strong password policies. Secure session management. Throttle failed logins. + +```python +# Secure session config (Flask) +app.config.update( + SESSION_COOKIE_SECURE=True, + SESSION_COOKIE_HTTPONLY=True, + SESSION_COOKIE_SAMESITE="Lax", + PERMANENT_SESSION_LIFETIME=timedelta(hours=1), +) +``` + +## A08: Software and Data Integrity Failures + +**Risk**: Code and infrastructure that does not protect against integrity violations (CI/CD, unsigned updates). + +**Prevention**: Verify signatures. Use lock files. Review CI/CD pipelines. Use Subresource Integrity. + +```html + + +``` + +## A09: Security Logging and Monitoring Failures + +**Risk**: Insufficient logging makes breaches undetectable. + +**Prevention**: Log auth events, access control failures, input validation failures. Set up alerts. + +```python +import logging + +logger = logging.getLogger("security") + +def login(username, password): + user = authenticate(username, password) + if not user: + logger.warning("Failed login attempt", extra={ + "username": username, + "ip": request.remote_addr, + "timestamp": datetime.utcnow().isoformat(), + }) + raise AuthenticationError() + logger.info("Successful login", extra={"user_id": user.id}) +``` + +## A10: Server-Side Request Forgery (SSRF) + +**Risk**: Application fetches remote resources without validating user-supplied URLs. + +**Prevention**: Allowlist URLs/domains. Block private IP ranges. Disable redirects. + +```python +from urllib.parse import urlparse +import ipaddress + +ALLOWED_HOSTS = {"api.example.com", "cdn.example.com"} + +def validate_url(url: str) -> bool: + parsed = urlparse(url) + if parsed.hostname not in ALLOWED_HOSTS: + return False + try: + ip = ipaddress.ip_address(parsed.hostname) + if ip.is_private or ip.is_loopback: + return False + except ValueError: + pass # hostname, not IP — already checked against allowlist + return True +``` + +--- + +## Quick Reference Table + +| ID | Name | Key Control | +|-----|-------------------------------|--------------------------------| +| A01 | Broken Access Control | Deny by default, enforce ownership | +| A02 | Cryptographic Failures | Encrypt in transit + at rest | +| A03 | Injection | Parameterized queries | +| A04 | Insecure Design | Threat modeling, abuse cases | +| A05 | Security Misconfiguration | Hardened defaults, minimal surface | +| A06 | Vulnerable Components | Audit deps, pin versions | +| A07 | Auth Failures | MFA, session security | +| A08 | Integrity Failures | Verify signatures, lock files | +| A09 | Logging Failures | Log security events, alert | +| A10 | SSRF | Allowlist URLs, block private IPs | + +*Source: [OWASP Top 10 (2021)](https://owasp.org/Top10/)* diff --git a/.claude/skills/security/owasp/references/security-headers.md b/.claude/skills/security/owasp/references/security-headers.md new file mode 100644 index 0000000..47f5cee --- /dev/null +++ b/.claude/skills/security/owasp/references/security-headers.md @@ -0,0 +1,217 @@ +# Security Headers Reference + +Comprehensive reference for HTTP security headers with recommended values and implementation examples. + +--- + +## Header Reference Table + +| Header | Purpose | Recommended Value | +|--------|---------|-------------------| +| `Content-Security-Policy` | Prevent XSS, data injection | See detailed section below | +| `Strict-Transport-Security` | Force HTTPS | `max-age=63072000; includeSubDomains; preload` | +| `X-Frame-Options` | Prevent clickjacking | `DENY` or `SAMEORIGIN` | +| `X-Content-Type-Options` | Prevent MIME sniffing | `nosniff` | +| `Referrer-Policy` | Control referer leakage | `strict-origin-when-cross-origin` | +| `Permissions-Policy` | Restrict browser features | See detailed section below | + +--- + +## Content-Security-Policy (CSP) + +Controls which resources the browser is allowed to load. + +**Starter policy (strict):** +``` +Content-Security-Policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self'; connect-src 'self'; frame-ancestors 'none'; base-uri 'self'; form-action 'self' +``` + +**Key directives:** + +| Directive | Controls | Example | +|-----------|----------|---------| +| `default-src` | Fallback for all resource types | `'self'` | +| `script-src` | JavaScript sources | `'self' https://cdn.example.com` | +| `style-src` | CSS sources | `'self' 'unsafe-inline'` | +| `img-src` | Image sources | `'self' data: https:` | +| `connect-src` | Fetch, XHR, WebSocket targets | `'self' https://api.example.com` | +| `frame-ancestors` | Who can embed this page | `'none'` | +| `form-action` | Form submission targets | `'self'` | + +## Strict-Transport-Security (HSTS) + +Forces browsers to use HTTPS for all future requests to this domain. + +``` +Strict-Transport-Security: max-age=63072000; includeSubDomains; preload +``` + +- `max-age=63072000` — 2 years (minimum for preload list) +- `includeSubDomains` — apply to all subdomains +- `preload` — opt into browser preload lists + +## X-Frame-Options + +Prevents the page from being embedded in iframes (clickjacking protection). + +``` +X-Frame-Options: DENY +``` + +| Value | Behavior | +|-------|----------| +| `DENY` | Never allow framing | +| `SAMEORIGIN` | Allow framing by same origin only | + +Note: `frame-ancestors` in CSP is the modern replacement but set both for backward compatibility. + +## X-Content-Type-Options + +Prevents browsers from MIME-sniffing the response content type. + +``` +X-Content-Type-Options: nosniff +``` + +Always pair with correct `Content-Type` headers on responses. + +## Referrer-Policy + +Controls how much referrer information is sent with requests. + +``` +Referrer-Policy: strict-origin-when-cross-origin +``` + +| Value | Cross-Origin Sends | Same-Origin Sends | +|-------|-------------------|-------------------| +| `no-referrer` | Nothing | Nothing | +| `origin` | Origin only | Origin only | +| `strict-origin-when-cross-origin` | Origin (HTTPS only) | Full URL | +| `same-origin` | Nothing | Full URL | + +## Permissions-Policy + +Restricts which browser features the page can use. + +``` +Permissions-Policy: camera=(), microphone=(), geolocation=(), payment=() +``` + +| Feature | Recommended | Description | +|---------|-------------|-------------| +| `camera` | `()` | Disable camera access | +| `microphone` | `()` | Disable microphone | +| `geolocation` | `()` | Disable location | +| `payment` | `()` | Disable Payment API | +| `usb` | `()` | Disable USB access | +| `fullscreen` | `(self)` | Allow fullscreen for same origin | + +--- + +## Implementation: Python (FastAPI) + +```python +from fastapi import FastAPI +from starlette.middleware.base import BaseHTTPMiddleware +from starlette.requests import Request +from starlette.responses import Response + +app = FastAPI() + +class SecurityHeadersMiddleware(BaseHTTPMiddleware): + async def dispatch(self, request: Request, call_next) -> Response: + response = await call_next(request) + response.headers["Content-Security-Policy"] = ( + "default-src 'self'; script-src 'self'; " + "style-src 'self' 'unsafe-inline'; " + "img-src 'self' data: https:; " + "frame-ancestors 'none'; base-uri 'self'; form-action 'self'" + ) + response.headers["Strict-Transport-Security"] = ( + "max-age=63072000; includeSubDomains; preload" + ) + response.headers["X-Frame-Options"] = "DENY" + response.headers["X-Content-Type-Options"] = "nosniff" + response.headers["Referrer-Policy"] = "strict-origin-when-cross-origin" + response.headers["Permissions-Policy"] = ( + "camera=(), microphone=(), geolocation=(), payment=()" + ) + return response + +app.add_middleware(SecurityHeadersMiddleware) +``` + +## Implementation: Node.js (Express) + +```typescript +import helmet from "helmet"; +import express from "express"; + +const app = express(); + +app.use( + helmet({ + contentSecurityPolicy: { + directives: { + defaultSrc: ["'self'"], + scriptSrc: ["'self'"], + styleSrc: ["'self'", "'unsafe-inline'"], + imgSrc: ["'self'", "data:", "https:"], + frameAncestors: ["'none'"], + baseUri: ["'self'"], + formAction: ["'self'"], + }, + }, + strictTransportSecurity: { + maxAge: 63072000, + includeSubDomains: true, + preload: true, + }, + frameguard: { action: "deny" }, + referrerPolicy: { policy: "strict-origin-when-cross-origin" }, + permissionsPolicy: { + features: { + camera: [], + microphone: [], + geolocation: [], + payment: [], + }, + }, + }) +); +``` + +## Implementation: Next.js + +```typescript +// next.config.ts +const securityHeaders = [ + { key: "Content-Security-Policy", value: "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; frame-ancestors 'none'" }, + { key: "Strict-Transport-Security", value: "max-age=63072000; includeSubDomains; preload" }, + { key: "X-Frame-Options", value: "DENY" }, + { key: "X-Content-Type-Options", value: "nosniff" }, + { key: "Referrer-Policy", value: "strict-origin-when-cross-origin" }, + { key: "Permissions-Policy", value: "camera=(), microphone=(), geolocation=(), payment=()" }, +]; + +export default { + async headers() { + return [{ source: "/(.*)", headers: securityHeaders }]; + }, +}; +``` + +--- + +## Verification + +```bash +# Check headers on a live site +curl -I https://example.com + +# Use securityheaders.com for a grade +# https://securityheaders.com/?q=https://example.com +``` + +*Source: [MDN HTTP Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers), [OWASP Secure Headers](https://owasp.org/www-project-secure-headers/)* diff --git a/.claude/skills/security/owasp/scripts/security-audit.py b/.claude/skills/security/owasp/scripts/security-audit.py new file mode 100644 index 0000000..2abad1a --- /dev/null +++ b/.claude/skills/security/owasp/scripts/security-audit.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""Security audit scanner for common vulnerabilities. + +Scans source files for hardcoded secrets, eval() usage, SQL string +concatenation, and sensitive data in console output. Outputs JSON. + +Usage: + python security-audit.py ./src + python security-audit.py ./src --severity high --format pretty +""" + +import argparse +import json +import os +import re +import sys +from dataclasses import asdict, dataclass, field +from pathlib import Path + +SCAN_EXTENSIONS = { + ".py", ".js", ".ts", ".jsx", ".tsx", ".java", ".go", + ".rb", ".php", ".env", ".yaml", ".yml", ".toml", ".json", +} + +SKIP_DIRS = { + "node_modules", ".git", "__pycache__", ".venv", "venv", + "dist", "build", ".next", ".nuxt", "vendor", +} + + +@dataclass +class Finding: + file: str + line: int + rule: str + severity: str + message: str + snippet: str + + +@dataclass +class AuditReport: + scanned_files: int = 0 + findings: list = field(default_factory=list) + summary: dict = field(default_factory=dict) + + +# --- Detection Rules --- + +SECRET_PATTERNS = [ + (r'(?i)(api[_-]?key|apikey)\s*[=:]\s*["\'][A-Za-z0-9_\-]{16,}["\']', "Possible API key"), + (r'(?i)(secret|password|passwd|pwd)\s*[=:]\s*["\'][^"\']{8,}["\']', "Possible hardcoded secret"), + (r'(?i)(aws_access_key_id|aws_secret_access_key)\s*[=:]\s*["\'][^"\']+["\']', "AWS credential"), + (r'(?i)bearer\s+[A-Za-z0-9_\-\.]{20,}', "Possible bearer token"), + (r'(?i)(ghp_|gho_|github_pat_)[A-Za-z0-9_]{20,}', "GitHub token"), + (r'(?i)(sk-|pk_live_|pk_test_|sk_live_|sk_test_)[A-Za-z0-9]{20,}', "API secret key"), + (r'-----BEGIN\s+(RSA\s+)?PRIVATE\s+KEY-----', "Private key in source"), +] + +EVAL_PATTERNS = [ + (r'\beval\s*\(', "eval() usage detected"), + (r'\bexec\s*\(', "exec() usage detected (Python)"), + (r'new\s+Function\s*\(', "new Function() usage (dynamic code)"), + (r'\bchild_process\.exec\s*\(', "child_process.exec (command injection risk)"), + (r'subprocess\.call\s*\([^)]*shell\s*=\s*True', "subprocess with shell=True"), + (r'os\.system\s*\(', "os.system() usage (command injection risk)"), +] + +SQL_PATTERNS = [ + (r'(?i)(SELECT|INSERT|UPDATE|DELETE|DROP)\s+.*([\+]|\.format\(|f["\']|%\s)', "SQL string concatenation"), + (r'(?i)execute\s*\(\s*f["\']', "SQL f-string in execute()"), + (r'(?i)\.query\s*\(\s*`[^`]*\$\{', "SQL template literal injection"), + (r'(?i)\.raw\s*\(\s*f["\']', "Raw SQL with f-string"), +] + +SENSITIVE_LOG_PATTERNS = [ + (r'console\.log\s*\(.*(?i)(password|secret|token|key|credential)', "Sensitive data in console.log"), + (r'print\s*\(.*(?i)(password|secret|token|key|credential)', "Sensitive data in print()"), + (r'logger?\.(info|debug|warn)\s*\(.*(?i)(password|secret|token)', "Sensitive data in logger"), +] + +RULES = [ + ("hardcoded-secret", "high", SECRET_PATTERNS), + ("dangerous-eval", "high", EVAL_PATTERNS), + ("sql-injection", "high", SQL_PATTERNS), + ("sensitive-logging", "medium", SENSITIVE_LOG_PATTERNS), +] + + +def should_scan(path: Path) -> bool: + if path.suffix not in SCAN_EXTENSIONS: + return False + for part in path.parts: + if part in SKIP_DIRS: + return False + return True + + +def scan_file(filepath: Path) -> list[Finding]: + findings = [] + try: + content = filepath.read_text(encoding="utf-8", errors="ignore") + except (OSError, PermissionError): + return findings + + lines = content.splitlines() + for line_num, line in enumerate(lines, start=1): + stripped = line.strip() + if stripped.startswith(("#", "//", "*", "/*")): + continue + for rule_name, severity, patterns in RULES: + for pattern, message in patterns: + if re.search(pattern, line): + findings.append(Finding( + file=str(filepath), + line=line_num, + rule=rule_name, + severity=severity, + message=message, + snippet=line.strip()[:120], + )) + return findings + + +def scan_directory(target: Path, severity_filter: str | None = None) -> AuditReport: + report = AuditReport() + severity_order = {"high": 3, "medium": 2, "low": 1} + min_severity = severity_order.get(severity_filter, 0) if severity_filter else 0 + + for root, dirs, files in os.walk(target): + dirs[:] = [d for d in dirs if d not in SKIP_DIRS] + for fname in files: + fpath = Path(root) / fname + if not should_scan(fpath): + continue + report.scanned_files += 1 + for finding in scan_file(fpath): + if severity_order.get(finding.severity, 0) >= min_severity: + report.findings.append(finding) + + report.summary = { + "total": len(report.findings), + "high": sum(1 for f in report.findings if f.severity == "high"), + "medium": sum(1 for f in report.findings if f.severity == "medium"), + "low": sum(1 for f in report.findings if f.severity == "low"), + "by_rule": {}, + } + for f in report.findings: + report.summary["by_rule"][f.rule] = report.summary["by_rule"].get(f.rule, 0) + 1 + + return report + + +def main(): + parser = argparse.ArgumentParser( + description="Scan source files for common security issues.", + epilog="Example: python security-audit.py ./src --severity high", + ) + parser.add_argument("target", help="Directory or file to scan") + parser.add_argument( + "--severity", choices=["low", "medium", "high"], + help="Minimum severity to report (default: all)", + ) + parser.add_argument( + "--format", choices=["json", "pretty"], default="json", + help="Output format (default: json)", + ) + args = parser.parse_args() + + target = Path(args.target) + if not target.exists(): + print(f"Error: {target} does not exist", file=sys.stderr) + sys.exit(1) + + report = scan_directory(target, args.severity) + output = { + "scanned_files": report.scanned_files, + "summary": report.summary, + "findings": [asdict(f) for f in report.findings], + } + + if args.format == "pretty": + print(f"\nScanned {report.scanned_files} files\n") + print(f"Findings: {report.summary['total']} total " + f"({report.summary['high']} high, {report.summary['medium']} medium)") + print("-" * 60) + for f in report.findings: + print(f"[{f.severity.upper()}] {f.file}:{f.line}") + print(f" Rule: {f.rule}") + print(f" {f.message}") + print(f" > {f.snippet}") + print() + else: + print(json.dumps(output, indent=2)) + + sys.exit(1 if report.summary.get("high", 0) > 0 else 0) + + +if __name__ == "__main__": + main() diff --git a/.claude/skills/security/owasp/templates/security-checklist.md b/.claude/skills/security/owasp/templates/security-checklist.md new file mode 100644 index 0000000..66dd5c3 --- /dev/null +++ b/.claude/skills/security/owasp/templates/security-checklist.md @@ -0,0 +1,120 @@ +# Security Code Review Checklist + +**Project**: _______________ +**Reviewer**: _______________ +**Date**: _______________ +**Scope**: _______________ + +--- + +## Authentication and Session Management + +- [ ] Passwords hashed with bcrypt/argon2 (not MD5/SHA1) +- [ ] Session tokens are cryptographically random +- [ ] Session cookies use `Secure`, `HttpOnly`, `SameSite` flags +- [ ] Session timeout is enforced (idle and absolute) +- [ ] Failed login attempts are rate-limited +- [ ] MFA is available for sensitive accounts +- [ ] Password reset tokens expire and are single-use + +## Authorization and Access Control + +- [ ] Access denied by default (allowlist approach) +- [ ] Server-side authorization on every request +- [ ] Resource ownership verified before access +- [ ] Role/permission checks cannot be bypassed via direct URL +- [ ] Admin endpoints have separate authentication +- [ ] CORS policy restricts allowed origins + +## Input Validation + +- [ ] All user input validated server-side +- [ ] Parameterized queries used for all database access +- [ ] No string concatenation in SQL/commands +- [ ] File uploads validated (type, size, content) +- [ ] Path traversal prevented on file operations +- [ ] JSON/XML parsers configured against XXE + +## Output Encoding + +- [ ] HTML output properly escaped (XSS prevention) +- [ ] Content-Type headers set correctly on all responses +- [ ] API responses do not leak stack traces in production +- [ ] Error messages do not reveal system internals +- [ ] Sensitive data excluded from logs + +## Cryptography + +- [ ] TLS 1.2+ enforced for all connections +- [ ] Sensitive data encrypted at rest +- [ ] No hardcoded secrets, keys, or passwords in source +- [ ] Secrets loaded from environment variables or vault +- [ ] Strong algorithms used (AES-256, RSA-2048+, SHA-256+) +- [ ] No custom cryptographic implementations + +## Security Headers + +- [ ] Content-Security-Policy configured +- [ ] Strict-Transport-Security enabled +- [ ] X-Frame-Options set to DENY +- [ ] X-Content-Type-Options set to nosniff +- [ ] Referrer-Policy configured +- [ ] Permissions-Policy restricts unused features + +## Dependencies + +- [ ] No known vulnerabilities (`npm audit` / `pip audit` clean) +- [ ] Unused dependencies removed +- [ ] Dependencies pinned to specific versions +- [ ] Lock file committed and up to date + +## Logging and Monitoring + +- [ ] Authentication events logged (success and failure) +- [ ] Authorization failures logged +- [ ] Sensitive data not written to logs +- [ ] Log injection prevented (user input sanitized in logs) +- [ ] Alerts configured for suspicious patterns + +## API Security + +- [ ] Rate limiting on all public endpoints +- [ ] Request size limits configured +- [ ] API keys/tokens not exposed in URLs +- [ ] Pagination enforced on list endpoints +- [ ] HTTPS required (HTTP redirects or blocks) + +## Infrastructure + +- [ ] Debug mode disabled in production +- [ ] Default credentials changed +- [ ] Unnecessary ports/services disabled +- [ ] Container runs as non-root user +- [ ] Environment variables not logged at startup + +--- + +## Summary + +| Category | Pass | Fail | N/A | +|----------|------|------|-----| +| Authentication | | | | +| Authorization | | | | +| Input Validation | | | | +| Output Encoding | | | | +| Cryptography | | | | +| Security Headers | | | | +| Dependencies | | | | +| Logging | | | | +| API Security | | | | +| Infrastructure | | | | + +**Overall Assessment**: [ ] Pass / [ ] Conditional Pass / [ ] Fail + +**Notes**: + + + +**Follow-up Actions**: + + diff --git a/.claude/skills/testing/pytest/SKILL.md b/.claude/skills/testing/pytest/SKILL.md index cb9d194..8b61c88 100644 --- a/.claude/skills/testing/pytest/SKILL.md +++ b/.claude/skills/testing/pytest/SKILL.md @@ -1,90 +1,689 @@ +--- +name: pytest +description: > + Trigger this skill whenever writing, debugging, or refactoring Python tests, or when pytest fixtures, parametrization, mocking, or coverage are mentioned. Activate for any .py test file, test_* function, conftest.py, pytest.ini, or pyproject.toml [tool.pytest] reference. Also use when the user asks about Python test patterns, test organization, or test-driven development in a Python context. +--- + # pytest -## Description - -Python testing with pytest including fixtures, parametrization, and mocking. - ## When to Use - Writing Python tests - Test fixtures and setup - Mocking dependencies +## When NOT to Use + +- JavaScript or TypeScript testing -- use the `testing/vitest` skill instead +- Projects that explicitly mandate unittest-only by convention with no pytest dependency +- Non-Python test files or environments + --- ## Core Patterns -### Basic Tests +### 1. Fixtures + +Fixtures provide reusable setup and teardown logic. They are requested by name as test function parameters. + +#### Function-Scoped Fixtures (default) + +A new instance is created for every test that requests it. ```python import pytest +from myapp.models import User +from myapp.db import Session -def test_addition(): - assert 1 + 1 == 2 -def test_exception(): - with pytest.raises(ValueError, match="Invalid"): - raise ValueError("Invalid input") -``` - -### Fixtures - -```python @pytest.fixture def user(): - return User(id=1, name="Test") + """Fresh user instance per test.""" + return User(id=1, name="Alice", email="alice@example.com") -@pytest.fixture -def db_session(): - session = create_session() - yield session - session.close() -def test_with_fixtures(user, db_session): - db_session.add(user) - assert user.id is not None +def test_user_display_name(user): + assert user.display_name() == "Alice" + + +def test_user_email_domain(user): + assert user.email_domain() == "example.com" ``` -### Parametrization +#### Class and Module Scope + +Use broader scopes for expensive resources that are safe to share. ```python -@pytest.mark.parametrize("input,expected", [ +@pytest.fixture(scope="class") +def api_client(): + """Shared across all tests in a test class.""" + client = APIClient(base_url="http://testserver") + client.authenticate(token="test-token") + return client + + +@pytest.fixture(scope="module") +def database_schema(): + """Created once per test module, shared across all tests in the file.""" + engine = create_engine("sqlite:///:memory:") + Base.metadata.create_all(engine) + yield engine + engine.dispose() + + +@pytest.fixture(scope="session") +def redis_connection(): + """Created once for the entire test session.""" + conn = Redis(host="localhost", port=6379, db=15) + conn.flushdb() + yield conn + conn.flushdb() + conn.close() +``` + +#### Yield Fixtures for Teardown + +`yield` separates setup from teardown. Code after `yield` runs after the test completes, even if the test fails. + +```python +@pytest.fixture +def db_session(): + session = Session() + session.begin() + yield session + session.rollback() + session.close() + + +@pytest.fixture +def temp_config(tmp_path): + config_file = tmp_path / "config.yaml" + config_file.write_text("debug: true\nlog_level: INFO\n") + yield config_file + # tmp_path is automatically cleaned up by pytest +``` + +#### Autouse Fixtures + +Apply a fixture to every test automatically without requesting it by name. + +```python +@pytest.fixture(autouse=True) +def reset_environment(monkeypatch): + """Ensure each test starts with clean environment variables.""" + monkeypatch.delenv("API_KEY", raising=False) + monkeypatch.delenv("DATABASE_URL", raising=False) + + +@pytest.fixture(autouse=True) +def freeze_time(): + """Pin time for deterministic tests.""" + with freeze_time("2025-06-15T12:00:00Z"): + yield +``` + +#### Factory Fixtures + +Return a factory function when tests need multiple instances with varying parameters. + +```python +@pytest.fixture +def make_user(): + """Factory that creates users with sensible defaults.""" + created = [] + + def _make_user(name="Test User", role="viewer", active=True): + user = User(name=name, role=role, active=active) + created.append(user) + return user + + yield _make_user + + # Teardown: clean up all created users + for u in created: + u.delete() + + +def test_admin_permissions(make_user): + admin = make_user(name="Admin", role="admin") + viewer = make_user(name="Viewer", role="viewer") + assert admin.can_delete_users() is True + assert viewer.can_delete_users() is False +``` + +#### Parametrized Fixtures with request.param + +Run the same test against multiple fixture variants. + +```python +@pytest.fixture(params=["sqlite", "postgresql"]) +def db_engine(request): + """Test against multiple database backends.""" + if request.param == "sqlite": + engine = create_engine("sqlite:///:memory:") + elif request.param == "postgresql": + engine = create_engine("postgresql://test:test@localhost/testdb") + yield engine + engine.dispose() + + +def test_insert_and_query(db_engine): + # This test runs twice: once with sqlite, once with postgresql + with db_engine.connect() as conn: + conn.execute(text("CREATE TABLE t (id INT)")) + conn.execute(text("INSERT INTO t VALUES (1)")) + result = conn.execute(text("SELECT * FROM t")).fetchall() + assert len(result) == 1 +``` + +--- + +### 2. Parametrize + +#### Single Parameter + +```python +@pytest.mark.parametrize("email", [ + "user@example.com", + "admin@test.org", + "name+tag@domain.co.uk", +]) +def test_valid_email_accepted(email): + assert is_valid_email(email) is True +``` + +#### Multiple Parameters + +```python +@pytest.mark.parametrize("input_text, expected", [ ("hello", "HELLO"), ("world", "WORLD"), ("", ""), + ("already UPPER", "ALREADY UPPER"), ]) -def test_uppercase(input, expected): - assert input.upper() == expected +def test_uppercase(input_text, expected): + assert input_text.upper() == expected ``` -### Mocking +#### Custom IDs for Readable Output ```python -from unittest.mock import Mock, patch - -def test_with_mock(): - service = Mock() - service.get_user.return_value = {"id": 1} - - result = service.get_user(1) - assert result["id"] == 1 - -@patch('module.external_api') -def test_with_patch(mock_api): - mock_api.fetch.return_value = {"data": []} - # Test code that uses external_api +@pytest.mark.parametrize("status_code, should_retry", [ + pytest.param(200, False, id="success-no-retry"), + pytest.param(429, True, id="rate-limited-retry"), + pytest.param(500, True, id="server-error-retry"), + pytest.param(404, False, id="not-found-no-retry"), +]) +def test_retry_logic(status_code, should_retry): + response = MockResponse(status_code=status_code) + assert should_retry_request(response) is should_retry ``` +#### Indirect Parametrize + +Pass parameters through a fixture rather than directly to the test. + +```python +@pytest.fixture +def user_role(request): + """Create a user with the given role.""" + return User(name="Test", role=request.param) + + +@pytest.mark.parametrize("user_role", ["admin", "editor", "viewer"], indirect=True) +def test_dashboard_access(user_role): + if user_role.role == "admin": + assert user_role.can_access("/admin/dashboard") is True + else: + assert user_role.can_access("/admin/dashboard") is False +``` + +#### Stacking Parametrize Decorators + +Creates the cartesian product of all parameter sets. + +```python +@pytest.mark.parametrize("method", ["GET", "POST", "PUT", "DELETE"]) +@pytest.mark.parametrize("auth", ["token", "session", "none"]) +def test_endpoint_auth(method, auth): + # Runs 4 x 3 = 12 test cases + response = make_request(method=method, auth_type=auth) + if auth == "none": + assert response.status_code == 401 + else: + assert response.status_code in (200, 201, 204) +``` + +--- + +### 3. Mocking + +#### monkeypatch -- Environment Variables and Attributes + +```python +def test_reads_api_key_from_env(monkeypatch): + monkeypatch.setenv("API_KEY", "test-key-12345") + config = load_config() + assert config.api_key == "test-key-12345" + + +def test_missing_api_key_raises(monkeypatch): + monkeypatch.delenv("API_KEY", raising=False) + with pytest.raises(ConfigError, match="API_KEY is required"): + load_config() + + +def test_override_attribute(monkeypatch): + monkeypatch.setattr("myapp.settings.MAX_RETRIES", 0) + assert retry_request(failing_url) is None # No retries attempted + + +def test_override_dict_item(monkeypatch): + monkeypatch.setitem(app_config, "timeout", 1) + assert app_config["timeout"] == 1 +``` + +#### unittest.mock.patch + +```python +from unittest.mock import patch, Mock, AsyncMock + + +@patch("myapp.services.payment.stripe.Charge.create") +def test_charge_customer(mock_charge): + mock_charge.return_value = Mock(id="ch_123", status="succeeded") + + result = process_payment(amount=1000, currency="usd", token="tok_visa") + + mock_charge.assert_called_once_with( + amount=1000, currency="usd", source="tok_visa" + ) + assert result.charge_id == "ch_123" + + +@patch("myapp.services.email.send_email") +@patch("myapp.services.user.UserRepository.find_by_id") +def test_send_welcome_email(mock_find, mock_send): + mock_find.return_value = User(id=1, email="new@example.com") + mock_send.return_value = True + + send_welcome(user_id=1) + + mock_send.assert_called_once_with( + to="new@example.com", template="welcome" + ) +``` + +#### responses Library for HTTP Mocking + +```python +import responses +import requests + + +@responses.activate +def test_fetch_user_from_api(): + responses.add( + responses.GET, + "https://api.example.com/users/1", + json={"id": 1, "name": "Alice"}, + status=200, + ) + + result = fetch_user(user_id=1) + + assert result["name"] == "Alice" + assert len(responses.calls) == 1 + assert responses.calls[0].request.url == "https://api.example.com/users/1" + + +@responses.activate +def test_api_timeout_handling(): + responses.add( + responses.GET, + "https://api.example.com/users/1", + body=requests.exceptions.ConnectionError("Connection timed out"), + ) + + with pytest.raises(ServiceUnavailableError): + fetch_user(user_id=1) +``` + +#### pytest-mock's mocker Fixture + +```python +def test_with_mocker(mocker): + mock_repo = mocker.patch("myapp.services.OrderRepository") + mock_repo.return_value.get_by_id.return_value = Order( + id=1, status="pending" + ) + + service = OrderService() + order = service.get_order(1) + + assert order.status == "pending" + mock_repo.return_value.get_by_id.assert_called_once_with(1) + + +def test_spy_on_method(mocker): + spy = mocker.spy(UserService, "validate_email") + + service = UserService() + service.register("alice@example.com") + + spy.assert_called_once_with(service, "alice@example.com") +``` + +--- + +### 4. Async Testing + +#### pytest-asyncio Basics + +```python +import pytest +import httpx + + +@pytest.mark.asyncio +async def test_async_fetch(): + async with httpx.AsyncClient() as client: + response = await client.get("https://httpbin.org/get") + assert response.status_code == 200 + + +@pytest.mark.asyncio +async def test_async_exception(): + with pytest.raises(ValueError, match="invalid"): + await validate_async_input("") +``` + +#### Async Fixtures + +```python +@pytest.fixture +async def async_db_session(): + session = AsyncSession(bind=async_engine) + await session.begin() + yield session + await session.rollback() + await session.close() + + +@pytest.mark.asyncio +async def test_async_query(async_db_session): + result = await async_db_session.execute( + select(User).where(User.active == True) + ) + users = result.scalars().all() + assert len(users) >= 0 +``` + +#### Configuring asyncio Mode + +In `pyproject.toml` or `pytest.ini`, set the default mode to avoid repeating the marker: + +```toml +# pyproject.toml +[tool.pytest.ini_options] +asyncio_mode = "auto" +``` + +With `asyncio_mode = "auto"`, any `async def test_*` function is automatically treated as async -- no `@pytest.mark.asyncio` needed. + +--- + +### 5. Test Organization + +#### conftest.py Hierarchy + +``` +tests/ +├── conftest.py # Session/global fixtures (db connection, app client) +├── unit/ +│ ├── conftest.py # Unit-specific fixtures (mocked services) +│ ├── test_models.py +│ └── test_utils.py +├── integration/ +│ ├── conftest.py # Integration fixtures (real db session, test server) +│ ├── test_api.py +│ └── test_repositories.py +└── e2e/ + ├── conftest.py # E2E fixtures (browser, full app) + └── test_workflows.py +``` + +Fixtures in a `conftest.py` are available to all tests in the same directory and below. No imports needed. + +#### Test Discovery + +pytest discovers tests by default based on these rules: +- Files matching `test_*.py` or `*_test.py` +- Classes prefixed with `Test` (no `__init__` method) +- Functions prefixed with `test_` + +Configure custom discovery in `pyproject.toml`: + +```toml +[tool.pytest.ini_options] +testpaths = ["tests"] +python_files = ["test_*.py"] +python_classes = ["Test*"] +python_functions = ["test_*"] +``` + +#### Markers + +```python +import pytest +import sys + +# Built-in markers +@pytest.mark.skip(reason="Not implemented yet") +def test_future_feature(): + pass + + +@pytest.mark.skipif( + sys.platform == "win32", reason="Unix-only functionality" +) +def test_unix_permissions(): + pass + + +@pytest.mark.xfail(reason="Known bug #1234, fix pending") +def test_known_broken(): + result = buggy_function() + assert result == "expected" +``` + +#### Custom Markers + +Register markers in `pyproject.toml` to avoid warnings: + +```toml +[tool.pytest.ini_options] +markers = [ + "slow: marks tests as slow (deselect with '-m \"not slow\"')", + "integration: marks integration tests requiring external services", + "smoke: critical path tests for quick validation", +] +``` + +```python +@pytest.mark.slow +def test_full_data_migration(): + migrate_all_records() # Takes 30+ seconds + assert count_records() == EXPECTED_TOTAL + + +@pytest.mark.smoke +def test_health_endpoint(client): + response = client.get("/health") + assert response.status_code == 200 +``` + +Run selectively: + +```bash +pytest -m "smoke" # Only smoke tests +pytest -m "not slow" # Skip slow tests +pytest -m "integration and not slow" # Integration but not slow +``` + +--- + +### 6. Coverage + +#### Basic Usage + +```bash +pytest --cov=src --cov-report=term-missing +pytest --cov=src --cov-report=html # Generates htmlcov/ +pytest --cov=src --cov-branch # Enable branch coverage +``` + +#### Configuration in pyproject.toml + +```toml +[tool.pytest.ini_options] +addopts = "--cov=src --cov-report=term-missing --cov-fail-under=80" + +[tool.coverage.run] +source = ["src"] +branch = true +omit = [ + "*/migrations/*", + "*/tests/*", + "*/__pycache__/*", + "*/conftest.py", +] + +[tool.coverage.report] +exclude_lines = [ + "pragma: no cover", + "def __repr__", + "if TYPE_CHECKING:", + "raise NotImplementedError", + "@overload", +] +fail_under = 80 +show_missing = true +``` + +#### .coveragerc Alternative + +If not using `pyproject.toml`, create `.coveragerc`: + +```ini +[run] +source = src +branch = true + +[report] +fail_under = 80 +show_missing = true +exclude_lines = + pragma: no cover + def __repr__ + if TYPE_CHECKING: +``` + +--- + +### 7. Assertions + +#### pytest.raises for Exceptions + +```python +def test_raises_value_error(): + with pytest.raises(ValueError) as exc_info: + parse_age("not-a-number") + assert "invalid literal" in str(exc_info.value) + + +def test_raises_with_match(): + with pytest.raises(PermissionError, match=r"User .+ lacks role 'admin'"): + authorize(user=viewer, required_role="admin") +``` + +#### pytest.approx for Floating Point + +```python +def test_circle_area(): + assert calculate_area(radius=5) == pytest.approx(78.5398, rel=1e-4) + + +def test_approx_list(): + result = distribute_evenly(total=100, buckets=3) + assert result == pytest.approx([33.33, 33.33, 33.34], abs=0.01) +``` + +#### Custom Assertion Helpers + +Build reusable assertion logic for domain-specific validation. + +```python +def assert_valid_api_response(response, expected_status=200): + """Reusable assertion for API responses.""" + assert response.status_code == expected_status, ( + f"Expected {expected_status}, got {response.status_code}: " + f"{response.text}" + ) + data = response.json() + assert "error" not in data, f"Unexpected error: {data['error']}" + return data + + +def test_create_user(client): + response = client.post("/users", json={"name": "Alice"}) + data = assert_valid_api_response(response, expected_status=201) + assert data["name"] == "Alice" + assert "id" in data +``` + +--- + ## Best Practices -1. Use fixtures for test setup -2. Parametrize for multiple test cases -3. Mock external dependencies -4. Use descriptive test names -5. Keep tests independent +1. **Name tests descriptively** -- Use `test_[function]_[scenario]_[expected]` so failures are self-explanatory without reading the test body. `test_parse_date_invalid_format_raises_valueerror` tells you everything. + +2. **Keep tests independent** -- Never rely on test execution order. Each test should set up its own state via fixtures and tear it down afterward. Shared mutable state between tests is the top cause of flaky suites. + +3. **One assertion focus per test** -- A test can have multiple `assert` statements, but they should all verify the same behavior. If you need to check two independent behaviors, write two tests. + +4. **Use fixtures over setup methods** -- Prefer composable fixtures over `setUp`/`tearDown` methods or `setup_function`. Fixtures are explicit about dependencies, reusable across files via `conftest.py`, and support scoping. + +5. **Mock at the boundary, not in the middle** -- Mock external services, databases, and network calls. Do not mock internal functions unless they are truly expensive. Over-mocking produces tests that pass but verify nothing. + +6. **Use `tmp_path` for file operations** -- pytest's built-in `tmp_path` fixture provides a unique temporary directory per test. Never write to the real filesystem in tests. + +7. **Pin randomness and time** -- When testing code that depends on randomness or the current time, use `random.seed()` or a time-freezing library to make tests deterministic. + +8. **Run the full suite in CI with branch coverage** -- Local development can use `pytest -x` for fast feedback (stop on first failure), but CI must run the full suite with `--cov-branch` to catch untested branches and regressions. + +--- ## Common Pitfalls -- **Shared state**: Use fresh fixtures -- **Over-mocking**: Only mock boundaries -- **Slow tests**: Use markers for slow tests +1. **Shared mutable fixtures** -- A module-scoped fixture returning a mutable object (list, dict, instance) gets modified by one test and breaks another. Return fresh copies or use function scope for mutable data. + +2. **Patching the wrong import path** -- `@patch("myapp.services.requests.get")` patches where `requests.get` is looked up, not where it is defined. If `services.py` does `from requests import get`, you must patch `myapp.services.get`, not `requests.get`. + +3. **Forgetting to await in async tests** -- Omitting `await` makes the test pass vacuously because it never actually runs the coroutine. Always `await` the function under test and use `@pytest.mark.asyncio`. + +4. **Tests that depend on execution order** -- If test B relies on side effects from test A, parallel test execution (pytest-xdist) and `--randomly` will expose the coupling immediately. Fix by making each test self-contained. + +5. **Asserting on mock call count without checking arguments** -- `mock.assert_called_once()` confirms the call count but not what was passed. Use `assert_called_once_with(...)` or inspect `mock.call_args` to verify the actual arguments. + +6. **Ignoring warnings as errors** -- Configure `filterwarnings = ["error"]` in `pyproject.toml` to catch deprecation warnings early. A passing test suite that emits 50 deprecation warnings is a time bomb. + +--- + +## Related Skills + +- `testing/vitest` -- JavaScript/TypeScript testing counterpart +- `languages/python` -- Python language patterns and idioms +- `methodology/test-driven-development` -- TDD workflow for writing tests first +- `devops/github-actions` — Running pytest in CI/CD pipelines diff --git a/.claude/skills/testing/pytest/references/fixture-patterns.md b/.claude/skills/testing/pytest/references/fixture-patterns.md new file mode 100644 index 0000000..c383f95 --- /dev/null +++ b/.claude/skills/testing/pytest/references/fixture-patterns.md @@ -0,0 +1,248 @@ +# pytest Fixture Patterns + +Catalog of reusable fixture patterns for common testing scenarios. + +## 1. Factory Fixture + +Create multiple instances with customizable defaults. + +```python +import pytest +from dataclasses import dataclass + +@pytest.fixture +def make_user(): + """Factory fixture: creates User instances with sensible defaults.""" + created = [] + + def _make_user( + name: str = "Test User", + email: str | None = None, + is_active: bool = True, + ): + if email is None: + email = f"user-{len(created)}@test.com" + user = User(name=name, email=email, is_active=is_active) + created.append(user) + return user + + yield _make_user + + # Cleanup: delete all created users + for user in created: + user.delete() +``` + +Usage: + +```python +def test_deactivate_user(make_user): + user = make_user(name="Alice", is_active=True) + user.deactivate() + assert not user.is_active +``` + +## 2. Database Session (SQLAlchemy) + +Transaction-isolated database session that rolls back after each test. + +```python +import pytest +from sqlalchemy import create_engine +from sqlalchemy.orm import sessionmaker + +@pytest.fixture(scope="session") +def engine(): + """Create a test database engine (once per test session).""" + engine = create_engine("postgresql://test:test@localhost:5432/test_db") + yield engine + engine.dispose() + +@pytest.fixture(scope="session") +def tables(engine): + """Create all tables once, drop after all tests.""" + Base.metadata.create_all(engine) + yield + Base.metadata.drop_all(engine) + +@pytest.fixture +def db_session(engine, tables): + """Provide a transactional database session that rolls back after each test.""" + connection = engine.connect() + transaction = connection.begin() + session = sessionmaker(bind=connection)() + + yield session + + session.close() + transaction.rollback() + connection.close() +``` + +## 3. Temporary Files and Directories + +```python +@pytest.fixture +def sample_config(tmp_path: Path) -> Path: + """Create a temporary config file with test content.""" + config = tmp_path / "config.yaml" + config.write_text( + """\ + database: + host: localhost + port: 5432 + debug: true + """ + ) + return config + +@pytest.fixture +def data_dir(tmp_path: Path) -> Path: + """Create a temporary directory structure for testing.""" + (tmp_path / "input").mkdir() + (tmp_path / "output").mkdir() + (tmp_path / "input" / "data.csv").write_text("id,name\n1,Alice\n2,Bob\n") + return tmp_path +``` + +## 4. Mock External Service + +```python +import pytest +from unittest.mock import AsyncMock, MagicMock, patch + +@pytest.fixture +def mock_http_client(): + """Mock an HTTP client with pre-configured responses.""" + client = MagicMock() + client.get.return_value = MagicMock( + status_code=200, + json=lambda: {"status": "ok"}, + ) + client.post.return_value = MagicMock( + status_code=201, + json=lambda: {"id": "new-123"}, + ) + return client + +@pytest.fixture +def mock_payment_gateway(): + """Mock a payment gateway service.""" + with patch("app.services.payment.PaymentGateway") as mock_cls: + instance = mock_cls.return_value + instance.charge.return_value = { + "transaction_id": "txn-test-123", + "status": "succeeded", + } + instance.refund.return_value = { + "refund_id": "ref-test-456", + "status": "refunded", + } + yield instance + +# Async version +@pytest.fixture +def mock_email_service(): + """Mock an async email service.""" + with patch("app.services.email.EmailService") as mock_cls: + instance = mock_cls.return_value + instance.send = AsyncMock(return_value={"message_id": "msg-test-789"}) + yield instance +``` + +## 5. Authenticated Test Client (FastAPI) + +```python +import pytest +from httpx import AsyncClient, ASGITransport +from app.main import app +from app.auth import create_access_token + +@pytest.fixture +def auth_token(): + """Generate a valid JWT token for testing.""" + return create_access_token( + data={"sub": "test-user-id", "role": "admin"}, + expires_minutes=60, + ) + +@pytest.fixture +def auth_headers(auth_token: str) -> dict[str, str]: + """HTTP headers with Bearer token.""" + return {"Authorization": f"Bearer {auth_token}"} + +@pytest.fixture +async def client(): + """Unauthenticated async test client.""" + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as c: + yield c + +@pytest.fixture +async def auth_client(auth_headers): + """Authenticated async test client.""" + transport = ASGITransport(app=app) + async with AsyncClient( + transport=transport, + base_url="http://test", + headers=auth_headers, + ) as c: + yield c +``` + +## 6. Environment Variables + +```python +@pytest.fixture +def env_vars(monkeypatch): + """Set environment variables for the test, automatically restored after.""" + monkeypatch.setenv("DATABASE_URL", "postgresql://test:test@localhost/test") + monkeypatch.setenv("SECRET_KEY", "test-secret-key") + monkeypatch.delenv("PRODUCTION_API_KEY", raising=False) +``` + +## 7. Freezing Time + +```python +@pytest.fixture +def frozen_time(): + """Freeze time to a specific moment.""" + fixed = datetime(2025, 1, 15, 12, 0, 0, tzinfo=timezone.utc) + with patch("app.services.datetime") as mock_dt: + mock_dt.now.return_value = fixed + mock_dt.utcnow.return_value = fixed + mock_dt.side_effect = lambda *a, **kw: datetime(*a, **kw) + yield fixed +``` + +Alternative: use `freezegun` library with `@freeze_time("2025-01-15 12:00:00")`. + +## 8. Parametrized Fixture + +```python +@pytest.fixture(params=["sqlite", "postgresql"]) +def db_url(request): + """Run tests against multiple database backends.""" + urls = { + "sqlite": "sqlite:///test.db", + "postgresql": "postgresql://test:test@localhost/test", + } + return urls[request.param] +``` + +## Fixture Scope Reference + +| Scope | Lifetime | Use For | +|-------|----------|---------| +| `function` (default) | Each test | Most fixtures, mutable state | +| `class` | Each test class | Shared setup for a class | +| `module` | Each test file | Expensive setup shared across file | +| `session` | Entire test run | Database engine, heavy resources | + +## Tips + +- Use `yield` (not `return`) when cleanup is needed after the test. +- Use `autouse=True` sparingly -- only for things every test needs. +- Keep fixtures small and composable -- combine them in tests, not in other fixtures. +- Use `monkeypatch` instead of `unittest.mock.patch` for env vars and attributes when possible. +- Name fixtures after what they provide, not what they do: `db_session` not `setup_database`. diff --git a/.claude/skills/testing/pytest/templates/conftest.py b/.claude/skills/testing/pytest/templates/conftest.py new file mode 100644 index 0000000..a82b7a6 --- /dev/null +++ b/.claude/skills/testing/pytest/templates/conftest.py @@ -0,0 +1,197 @@ +""" +Starter conftest.py -- common fixtures for pytest. + +Usage: + Place this file at the root of your tests/ directory. + pytest automatically discovers conftest.py and makes its fixtures + available to all tests in the same directory and below. +""" + +import os +from collections.abc import AsyncGenerator, Generator +from pathlib import Path +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +# --------------------------------------------------------------------------- +# If using FastAPI + httpx: +# from httpx import ASGITransport, AsyncClient +# from app.main import app + +# If using SQLAlchemy: +# from sqlalchemy import create_engine +# from sqlalchemy.orm import Session, sessionmaker +# from app.models import Base +# --------------------------------------------------------------------------- + + +# ========================================================================== +# Environment Variables +# ========================================================================== + + +@pytest.fixture(autouse=True) +def _test_env(monkeypatch: pytest.MonkeyPatch) -> None: + """Set safe default environment variables for all tests. + + autouse=True ensures this runs for every test automatically. + monkeypatch restores original values after each test. + """ + monkeypatch.setenv("APP_ENV", "test") + monkeypatch.setenv("DEBUG", "true") + monkeypatch.setenv("SECRET_KEY", "test-secret-not-for-production") + monkeypatch.setenv("DATABASE_URL", "sqlite:///test.db") + + # Remove any production secrets that should never leak into tests. + monkeypatch.delenv("PRODUCTION_API_KEY", raising=False) + monkeypatch.delenv("AWS_SECRET_ACCESS_KEY", raising=False) + + +# ========================================================================== +# Temporary Directory +# ========================================================================== + + +@pytest.fixture +def data_dir(tmp_path: Path) -> Path: + """Provide a temporary directory with input/output subdirectories.""" + (tmp_path / "input").mkdir() + (tmp_path / "output").mkdir() + return tmp_path + + +@pytest.fixture +def sample_file(tmp_path: Path) -> Path: + """Create a sample text file for testing file operations.""" + f = tmp_path / "sample.txt" + f.write_text("line 1\nline 2\nline 3\n") + return f + + +# ========================================================================== +# Database Session (SQLAlchemy) +# ========================================================================== + +# Uncomment this section if using SQLAlchemy. + +# TEST_DATABASE_URL = os.getenv( +# "TEST_DATABASE_URL", "postgresql://test:test@localhost:5432/test_db" +# ) +# +# +# @pytest.fixture(scope="session") +# def engine(): +# """Create database engine for the entire test session.""" +# eng = create_engine(TEST_DATABASE_URL) +# yield eng +# eng.dispose() +# +# +# @pytest.fixture(scope="session") +# def tables(engine): +# """Create tables at start of session, drop at end.""" +# Base.metadata.create_all(engine) +# yield +# Base.metadata.drop_all(engine) +# +# +# @pytest.fixture +# def db_session(engine, tables) -> Generator[Session, None, None]: +# """Transactional database session -- rolls back after each test.""" +# connection = engine.connect() +# transaction = connection.begin() +# session = sessionmaker(bind=connection)() +# +# yield session +# +# session.close() +# transaction.rollback() +# connection.close() + + +# ========================================================================== +# HTTP Test Client (FastAPI) +# ========================================================================== + +# Uncomment this section if using FastAPI. + +# @pytest.fixture +# async def client() -> AsyncGenerator[AsyncClient, None]: +# """Async HTTP client for testing API endpoints.""" +# transport = ASGITransport(app=app) +# async with AsyncClient(transport=transport, base_url="http://test") as c: +# yield c +# +# +# @pytest.fixture +# def auth_headers() -> dict[str, str]: +# """Authorization headers with a test JWT token.""" +# from app.auth import create_access_token +# token = create_access_token(data={"sub": "test-user", "role": "admin"}) +# return {"Authorization": f"Bearer {token}"} +# +# +# @pytest.fixture +# async def auth_client(auth_headers) -> AsyncGenerator[AsyncClient, None]: +# """Authenticated async HTTP client.""" +# transport = ASGITransport(app=app) +# async with AsyncClient( +# transport=transport, base_url="http://test", headers=auth_headers +# ) as c: +# yield c + + +# ========================================================================== +# Mock External Services +# ========================================================================== + + +@pytest.fixture +def mock_http_client() -> MagicMock: + """Generic mock HTTP client with default 200/201 responses.""" + client = MagicMock() + client.get.return_value = MagicMock( + status_code=200, + json=lambda: {"status": "ok"}, + ) + client.post.return_value = MagicMock( + status_code=201, + json=lambda: {"id": "new-123"}, + ) + return client + + +# @pytest.fixture +# def mock_email_service(): +# """Mock email service to prevent real emails in tests.""" +# with patch("app.services.email.send_email") as mock_send: +# mock_send.return_value = {"message_id": "test-msg-001"} +# yield mock_send + + +# ========================================================================== +# Factory Fixtures +# ========================================================================== + + +# @pytest.fixture +# def make_user(db_session): +# """Factory fixture: creates User instances with defaults.""" +# created = [] +# +# def _make_user( +# name: str = "Test User", +# email: str | None = None, +# is_active: bool = True, +# ): +# from app.models import User +# if email is None: +# email = f"user-{len(created)}@test.com" +# user = User(name=name, email=email, is_active=is_active) +# db_session.add(user) +# db_session.flush() +# created.append(user) +# return user +# +# return _make_user diff --git a/.claude/skills/testing/vitest/SKILL.md b/.claude/skills/testing/vitest/SKILL.md index c312341..7e811e3 100644 --- a/.claude/skills/testing/vitest/SKILL.md +++ b/.claude/skills/testing/vitest/SKILL.md @@ -1,89 +1,845 @@ +--- +name: vitest +description: > + Trigger this skill whenever writing, debugging, or refactoring JavaScript or TypeScript tests, or when Vitest mocking, coverage, or configuration are mentioned. Activate for any .test.ts, .test.tsx, .test.js, .spec.ts, .spec.js file, vitest.config.ts reference, or React component testing with Testing Library. Also use when the user asks about JS/TS test patterns, test organization, or vi.mock/vi.fn usage. +--- + # Vitest -## Description - -Modern JavaScript/TypeScript testing with Vitest including mocking and coverage. - ## When to Use - Testing JavaScript/TypeScript - React component testing - Unit and integration tests +## When NOT to Use + +- Python testing -- use the `testing/pytest` skill instead +- Projects that explicitly mandate Jest-only by convention with no Vitest dependency +- Non-JavaScript/TypeScript projects + --- ## Core Patterns -### Basic Tests +### 1. Test Structure + +#### describe / it / expect ```typescript import { describe, it, expect } from 'vitest'; +import { formatCurrency } from './format'; -describe('math', () => { - it('should add numbers', () => { - expect(1 + 1).toBe(2); +describe('formatCurrency', () => { + it('should format whole dollars', () => { + expect(formatCurrency(100)).toBe('$100.00'); }); - it('should throw on invalid input', () => { - expect(() => divide(1, 0)).toThrow('Division by zero'); + it('should format cents correctly', () => { + expect(formatCurrency(9.5)).toBe('$9.50'); + }); + + it('should handle zero', () => { + expect(formatCurrency(0)).toBe('$0.00'); + }); + + it('should throw on negative values', () => { + expect(() => formatCurrency(-5)).toThrow('Amount must be non-negative'); }); }); ``` -### Mocking +#### Lifecycle Hooks ```typescript -import { vi, describe, it, expect } from 'vitest'; +import { describe, it, expect, beforeEach, afterEach, beforeAll, afterAll } from 'vitest'; +import { Database } from './database'; -// Mock module -vi.mock('./api', () => ({ - fetchUser: vi.fn().mockResolvedValue({ id: 1 }) +describe('UserRepository', () => { + let db: Database; + + beforeAll(async () => { + // Runs once before all tests in this describe block + db = await Database.connect('test://localhost/testdb'); + await db.migrate(); + }); + + afterAll(async () => { + await db.disconnect(); + }); + + beforeEach(async () => { + // Runs before each test + await db.seed({ users: [{ id: 1, name: 'Alice' }] }); + }); + + afterEach(async () => { + await db.truncate('users'); + }); + + it('should find user by id', async () => { + const user = await db.users.findById(1); + expect(user).toEqual({ id: 1, name: 'Alice' }); + }); + + it('should return null for missing user', async () => { + const user = await db.users.findById(999); + expect(user).toBeNull(); + }); +}); +``` + +#### test.each for Parametrized Tests + +```typescript +import { describe, it, expect, test } from 'vitest'; +import { validateEmail } from './validators'; + +describe('validateEmail', () => { + test.each([ + { email: 'user@example.com', expected: true }, + { email: 'admin@test.org', expected: true }, + { email: 'name+tag@domain.co.uk', expected: true }, + ])('should accept valid email: $email', ({ email, expected }) => { + expect(validateEmail(email)).toBe(expected); + }); + + test.each([ + { email: '', reason: 'empty string' }, + { email: 'no-at-sign', reason: 'missing @' }, + { email: '@no-local.com', reason: 'missing local part' }, + { email: 'spaces in@email.com', reason: 'contains spaces' }, + ])('should reject invalid email ($reason): $email', ({ email }) => { + expect(validateEmail(email)).toBe(false); + }); +}); +``` + +#### Nested describe Blocks + +```typescript +describe('ShoppingCart', () => { + describe('when empty', () => { + it('should have zero total', () => { + const cart = new ShoppingCart(); + expect(cart.total()).toBe(0); + }); + + it('should have zero item count', () => { + const cart = new ShoppingCart(); + expect(cart.itemCount()).toBe(0); + }); + }); + + describe('with items', () => { + let cart: ShoppingCart; + + beforeEach(() => { + cart = new ShoppingCart(); + cart.add({ name: 'Widget', price: 9.99, quantity: 2 }); + cart.add({ name: 'Gadget', price: 24.99, quantity: 1 }); + }); + + it('should calculate total', () => { + expect(cart.total()).toBeCloseTo(44.97); + }); + + it('should count all items', () => { + expect(cart.itemCount()).toBe(3); + }); + }); +}); +``` + +--- + +### 2. Mocking + +#### vi.mock for Module Mocking + +```typescript +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { sendWelcomeEmail } from './onboarding'; + +// Mock the entire email module -- hoisted to the top of the file automatically +vi.mock('./email', () => ({ + sendEmail: vi.fn().mockResolvedValue({ messageId: 'msg-123' }), })); -// Mock function -const callback = vi.fn(); -callback('arg'); -expect(callback).toHaveBeenCalledWith('arg'); -``` +// Import AFTER vi.mock declaration +import { sendEmail } from './email'; -### Async Tests +describe('sendWelcomeEmail', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); -```typescript -it('should fetch data', async () => { - const data = await fetchData(); - expect(data).toEqual({ id: 1 }); -}); + it('should send email with welcome template', async () => { + await sendWelcomeEmail('alice@example.com'); -it('should reject on error', async () => { - await expect(fetchData()).rejects.toThrow('Error'); + expect(sendEmail).toHaveBeenCalledWith({ + to: 'alice@example.com', + template: 'welcome', + subject: 'Welcome to our platform!', + }); + }); + + it('should return the message id', async () => { + const result = await sendWelcomeEmail('alice@example.com'); + expect(result.messageId).toBe('msg-123'); + }); }); ``` -### React Testing +#### vi.fn for Function Spies ```typescript +import { describe, it, expect, vi } from 'vitest'; + +describe('EventEmitter', () => { + it('should call listener on emit', () => { + const emitter = new EventEmitter(); + const listener = vi.fn(); + + emitter.on('click', listener); + emitter.emit('click', { x: 10, y: 20 }); + + expect(listener).toHaveBeenCalledOnce(); + expect(listener).toHaveBeenCalledWith({ x: 10, y: 20 }); + }); + + it('should track multiple calls', () => { + const callback = vi.fn(); + + callback('first'); + callback('second'); + callback('third'); + + expect(callback).toHaveBeenCalledTimes(3); + expect(callback.mock.calls).toEqual([['first'], ['second'], ['third']]); + }); +}); +``` + +#### vi.spyOn + +```typescript +import { describe, it, expect, vi, afterEach } from 'vitest'; +import * as mathUtils from './math-utils'; + +describe('calculateTax', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('should use the tax rate function', () => { + const spy = vi.spyOn(mathUtils, 'getTaxRate').mockReturnValue(0.08); + + const result = calculateTax(100); + + expect(spy).toHaveBeenCalledWith(); + expect(result).toBe(8); + }); + + it('should spy without changing behavior', () => { + const spy = vi.spyOn(console, 'warn'); + + triggerDeprecationWarning(); + + expect(spy).toHaveBeenCalledWith( + expect.stringContaining('deprecated') + ); + }); +}); +``` + +#### mockResolvedValue / mockRejectedValue + +```typescript +import { describe, it, expect, vi } from 'vitest'; + +describe('UserService', () => { + it('should return user on successful fetch', async () => { + const fetchUser = vi.fn().mockResolvedValue({ id: 1, name: 'Alice' }); + + const user = await fetchUser(1); + expect(user).toEqual({ id: 1, name: 'Alice' }); + }); + + it('should throw on failed fetch', async () => { + const fetchUser = vi.fn().mockRejectedValue(new Error('User not found')); + + await expect(fetchUser(999)).rejects.toThrow('User not found'); + }); + + it('should return different values on successive calls', async () => { + const getToken = vi.fn() + .mockResolvedValueOnce('token-1') + .mockResolvedValueOnce('token-2') + .mockRejectedValueOnce(new Error('Expired')); + + expect(await getToken()).toBe('token-1'); + expect(await getToken()).toBe('token-2'); + await expect(getToken()).rejects.toThrow('Expired'); + }); +}); +``` + +#### MSW (Mock Service Worker) for API Mocking + +```typescript +import { describe, it, expect, beforeAll, afterAll, afterEach } from 'vitest'; +import { setupServer } from 'msw/node'; +import { http, HttpResponse } from 'msw'; +import { fetchUsers } from './api-client'; + +const server = setupServer( + http.get('https://api.example.com/users', () => { + return HttpResponse.json([ + { id: 1, name: 'Alice' }, + { id: 2, name: 'Bob' }, + ]); + }), + + http.post('https://api.example.com/users', async ({ request }) => { + const body = await request.json() as { name: string }; + return HttpResponse.json( + { id: 3, name: body.name }, + { status: 201 } + ); + }) +); + +beforeAll(() => server.listen({ onUnhandledRequest: 'error' })); +afterEach(() => server.resetHandlers()); +afterAll(() => server.close()); + +describe('API Client', () => { + it('should fetch users', async () => { + const users = await fetchUsers(); + expect(users).toHaveLength(2); + expect(users[0].name).toBe('Alice'); + }); + + it('should handle server errors', async () => { + server.use( + http.get('https://api.example.com/users', () => { + return HttpResponse.json( + { message: 'Internal Server Error' }, + { status: 500 } + ); + }) + ); + + await expect(fetchUsers()).rejects.toThrow('Server error'); + }); +}); +``` + +--- + +### 3. React Testing + +#### Render and Query + +```tsx +import { describe, it, expect } from 'vitest'; import { render, screen } from '@testing-library/react'; -import { userEvent } from '@testing-library/user-event'; +import { Greeting } from './Greeting'; -it('should handle click', async () => { - const onClick = vi.fn(); - render(); +describe('Greeting', () => { + it('should display the user name', () => { + render(); - await userEvent.click(screen.getByRole('button')); - expect(onClick).toHaveBeenCalled(); + // getBy* throws if not found -- use for elements that must exist + expect(screen.getByText('Hello, Alice!')).toBeInTheDocument(); + }); + + it('should not display admin badge for regular users', () => { + render(); + + // queryBy* returns null if not found -- use for asserting absence + expect(screen.queryByText('Admin')).not.toBeInTheDocument(); + }); + + it('should display admin badge for admins', () => { + render(); + expect(screen.getByText('Admin')).toBeInTheDocument(); + }); }); ``` +#### userEvent for Interactions + +```tsx +import { describe, it, expect, vi } from 'vitest'; +import { render, screen } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { LoginForm } from './LoginForm'; + +describe('LoginForm', () => { + it('should submit credentials', async () => { + const user = userEvent.setup(); + const onSubmit = vi.fn(); + render(); + + await user.type(screen.getByLabelText('Email'), 'alice@example.com'); + await user.type(screen.getByLabelText('Password'), 'secret123'); + await user.click(screen.getByRole('button', { name: 'Sign In' })); + + expect(onSubmit).toHaveBeenCalledWith({ + email: 'alice@example.com', + password: 'secret123', + }); + }); + + it('should show validation error on empty submit', async () => { + const user = userEvent.setup(); + render(); + + await user.click(screen.getByRole('button', { name: 'Sign In' })); + + expect(screen.getByText('Email is required')).toBeInTheDocument(); + }); + + it('should toggle password visibility', async () => { + const user = userEvent.setup(); + render(); + + const passwordInput = screen.getByLabelText('Password'); + expect(passwordInput).toHaveAttribute('type', 'password'); + + await user.click(screen.getByRole('button', { name: 'Show password' })); + expect(passwordInput).toHaveAttribute('type', 'text'); + }); +}); +``` + +#### findBy for Async Rendering and waitFor + +```tsx +import { describe, it, expect, vi } from 'vitest'; +import { render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { UserProfile } from './UserProfile'; + +describe('UserProfile', () => { + it('should load and display user data', async () => { + render(); + + // findBy* waits for the element to appear (async query) + const heading = await screen.findByRole('heading', { name: 'Alice' }); + expect(heading).toBeInTheDocument(); + }); + + it('should show loading state initially', () => { + render(); + expect(screen.getByText('Loading...')).toBeInTheDocument(); + }); + + it('should update after action', async () => { + const user = userEvent.setup(); + render(); + + await screen.findByRole('heading', { name: 'Alice' }); + await user.click(screen.getByRole('button', { name: 'Deactivate' })); + + await waitFor(() => { + expect(screen.getByText('Status: Inactive')).toBeInTheDocument(); + }); + }); +}); +``` + +#### Testing with Context Providers + +```tsx +import { describe, it, expect } from 'vitest'; +import { render, screen } from '@testing-library/react'; +import { ThemeProvider } from './ThemeContext'; +import { ThemedButton } from './ThemedButton'; + +function renderWithProviders(ui: React.ReactElement, options?: { theme?: 'light' | 'dark' }) { + const theme = options?.theme ?? 'light'; + return render( + + {ui} + + ); +} + +describe('ThemedButton', () => { + it('should apply light theme styles', () => { + renderWithProviders(Click me, { theme: 'light' }); + expect(screen.getByRole('button')).toHaveClass('btn-light'); + }); + + it('should apply dark theme styles', () => { + renderWithProviders(Click me, { theme: 'dark' }); + expect(screen.getByRole('button')).toHaveClass('btn-dark'); + }); +}); +``` + +--- + +### 4. Async Testing + +#### Promises and async/await + +```typescript +import { describe, it, expect } from 'vitest'; +import { fetchUser, processQueue } from './services'; + +describe('async operations', () => { + it('should resolve with user data', async () => { + const user = await fetchUser(1); + expect(user).toEqual({ id: 1, name: 'Alice' }); + }); + + it('should reject with descriptive error', async () => { + await expect(fetchUser(-1)).rejects.toThrow('Invalid user ID'); + }); + + it('should process all items in queue', async () => { + const results = await processQueue(['a', 'b', 'c']); + expect(results).toHaveLength(3); + expect(results.every((r) => r.status === 'done')).toBe(true); + }); +}); +``` + +#### Fake Timers + +```typescript +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { debounce } from './debounce'; + +describe('debounce', () => { + beforeEach(() => { + vi.useFakeTimers(); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + it('should not call function before delay', () => { + const fn = vi.fn(); + const debounced = debounce(fn, 300); + + debounced(); + vi.advanceTimersByTime(200); + + expect(fn).not.toHaveBeenCalled(); + }); + + it('should call function after delay', () => { + const fn = vi.fn(); + const debounced = debounce(fn, 300); + + debounced(); + vi.advanceTimersByTime(300); + + expect(fn).toHaveBeenCalledOnce(); + }); + + it('should reset timer on subsequent calls', () => { + const fn = vi.fn(); + const debounced = debounce(fn, 300); + + debounced(); + vi.advanceTimersByTime(200); + debounced(); // reset + vi.advanceTimersByTime(200); + + expect(fn).not.toHaveBeenCalled(); + + vi.advanceTimersByTime(100); + expect(fn).toHaveBeenCalledOnce(); + }); +}); +``` + +#### Fake Timers with Date + +```typescript +import { describe, it, expect, vi } from 'vitest'; +import { isExpired } from './token'; + +describe('isExpired', () => { + it('should detect expired tokens', () => { + vi.useFakeTimers(); + vi.setSystemTime(new Date('2025-06-15T12:00:00Z')); + + const token = { expiresAt: '2025-06-15T11:00:00Z' }; + expect(isExpired(token)).toBe(true); + + vi.useRealTimers(); + }); +}); +``` + +--- + +### 5. Snapshot Testing + +#### toMatchSnapshot + +```typescript +import { describe, it, expect } from 'vitest'; +import { render } from '@testing-library/react'; +import { Badge } from './Badge'; + +describe('Badge', () => { + it('should match snapshot for success variant', () => { + const { container } = render(Active); + expect(container.firstChild).toMatchSnapshot(); + }); +}); +``` + +#### toMatchInlineSnapshot + +Inline snapshots embed the expected value directly in the test file. Vitest updates them automatically on first run. + +```typescript +import { describe, it, expect } from 'vitest'; +import { formatError } from './errors'; + +describe('formatError', () => { + it('should format validation error', () => { + const error = formatError({ field: 'email', rule: 'required' }); + + expect(error).toMatchInlineSnapshot(` + { + "code": "VALIDATION_ERROR", + "field": "email", + "message": "email is required", + } + `); + }); +}); +``` + +#### When to Use Snapshots (and When Not To) + +**Use snapshots for:** +- Serialized output that is tedious to write by hand (large objects, rendered markup) +- Catching unintended changes in generated output +- Error message formatting + +**Do not use snapshots for:** +- Business logic assertions -- write explicit `expect(value).toBe(expected)` instead +- Frequently changing output -- snapshot churn leads to mindless updates +- Large component trees -- a small change deep in the tree makes the diff unreadable; test specific elements instead + +--- + +### 6. Coverage + +#### vitest.config.ts Coverage Settings + +```typescript +import { defineConfig } from 'vitest/config'; + +export default defineConfig({ + test: { + coverage: { + provider: 'v8', // or 'istanbul' + reporter: ['text', 'html', 'lcov'], + reportsDirectory: './coverage', + include: ['src/**/*.{ts,tsx}'], + exclude: [ + 'src/**/*.test.{ts,tsx}', + 'src/**/*.d.ts', + 'src/**/index.ts', // barrel files + 'src/test-utils/**', + ], + thresholds: { + statements: 80, + branches: 80, + functions: 80, + lines: 80, + }, + }, + }, +}); +``` + +#### Running Coverage + +```bash +vitest run --coverage # Run once with coverage +vitest --coverage # Watch mode with coverage +vitest run --coverage.provider=v8 # Override provider via CLI +``` + +#### Per-File Thresholds + +```typescript +// vitest.config.ts +export default defineConfig({ + test: { + coverage: { + provider: 'v8', + thresholds: { + // Global thresholds + statements: 80, + // Per-glob overrides for critical paths + 'src/auth/**': { + statements: 95, + branches: 95, + }, + }, + }, + }, +}); +``` + +--- + +### 7. Setup and Configuration + +#### vitest.config.ts Basics + +```typescript +import { defineConfig } from 'vitest/config'; +import react from '@vitejs/plugin-react'; + +export default defineConfig({ + plugins: [react()], + test: { + globals: true, // Use describe/it/expect without imports + environment: 'jsdom', // DOM environment for React (or 'happy-dom') + setupFiles: ['./src/test-setup.ts'], + include: ['src/**/*.test.{ts,tsx}'], + exclude: ['node_modules', 'dist', 'e2e'], + testTimeout: 10_000, + hookTimeout: 30_000, + }, + resolve: { + alias: { + '@': '/src', + }, + }, +}); +``` + +#### Setup File + +```typescript +// src/test-setup.ts +import '@testing-library/jest-dom/vitest'; +import { cleanup } from '@testing-library/react'; +import { afterEach } from 'vitest'; + +// Automatic cleanup after each test +afterEach(() => { + cleanup(); +}); +``` + +#### Workspace Configuration + +For monorepos with multiple packages: + +```typescript +// vitest.workspace.ts +import { defineWorkspace } from 'vitest/config'; + +export default defineWorkspace([ + { + extends: './vitest.config.ts', + test: { + name: 'ui', + include: ['packages/ui/**/*.test.{ts,tsx}'], + environment: 'jsdom', + }, + }, + { + extends: './vitest.config.ts', + test: { + name: 'api', + include: ['packages/api/**/*.test.ts'], + environment: 'node', + }, + }, +]); +``` + +#### Environment Per File + +Use a magic comment at the top of a test file to override the environment: + +```typescript +// @vitest-environment happy-dom + +import { describe, it, expect } from 'vitest'; + +describe('DOM-heavy tests', () => { + it('should create elements', () => { + const div = document.createElement('div'); + div.textContent = 'Hello'; + expect(div.textContent).toBe('Hello'); + }); +}); +``` + +#### Globals Mode + +When `globals: true` is set in config, you do not need to import `describe`, `it`, `expect`, `vi`, etc. Add the types to `tsconfig.json`: + +```json +{ + "compilerOptions": { + "types": ["vitest/globals"] + } +} +``` + +--- + ## Best Practices -1. Use describe blocks for grouping -2. Prefer async/await for async tests -3. Use userEvent over fireEvent -4. Mock at module boundaries -5. Clean up after tests +1. **Use `userEvent` over `fireEvent`** -- `userEvent` simulates real user behavior (focus, keystrokes, blur) while `fireEvent` dispatches raw DOM events. `userEvent` catches bugs that `fireEvent` misses, such as disabled buttons still receiving clicks. + +2. **Query by role and label, not test IDs** -- Prefer `getByRole('button', { name: 'Submit' })` and `getByLabelText('Email')` over `getByTestId('submit-btn')`. Accessible queries validate your markup and are resilient to refactors. + +3. **Clear mocks between tests** -- Call `vi.clearAllMocks()` in `beforeEach` or `vi.restoreAllMocks()` in `afterEach`. Leaked mock state between tests causes order-dependent failures that are painful to debug. + +4. **Keep tests focused on one behavior** -- Each `it` block should test a single user-observable behavior. If your test description contains "and", split it into two tests. + +5. **Avoid testing implementation details** -- Do not assert on component state, internal method calls, or private variables. Test what the user sees and what the component outputs. Implementation tests break on every refactor without catching real bugs. + +6. **Use MSW for network mocking over vi.mock on fetch** -- MSW intercepts at the network level, so your tests exercise the actual fetch/axios code paths. Mocking `fetch` directly skips serialization, headers, and error handling logic. + +7. **Colocate tests with source files** -- Place `Button.test.tsx` next to `Button.tsx`. This makes it obvious which files have tests and simplifies imports. Reserve a top-level `e2e/` folder only for end-to-end tests. + +8. **Run tests in watch mode during development** -- `vitest` (no flags) starts in watch mode and re-runs only affected tests on file change. Use `vitest run` in CI for a single full run with exit code. + +--- ## Common Pitfalls -- **Not awaiting async**: Always await promises -- **Stale mocks**: Clear mocks between tests -- **Testing implementation**: Test behavior +1. **Forgetting to await userEvent calls** -- Every `userEvent` method is async. Omitting `await` causes the assertion to run before the interaction completes, leading to false passes or intermittent failures. + +2. **vi.mock hoisting confusion** -- `vi.mock()` calls are hoisted to the top of the file. If you define a mock implementation that references a variable declared below the `vi.mock` call, it will be `undefined`. Use `vi.mock` with a factory function or move the variable above. + +3. **Not cleaning up after fake timers** -- Forgetting `vi.useRealTimers()` in `afterEach` causes subsequent tests to silently use fake timers, producing mysterious timeouts and passing tests that should fail. + +4. **Using `getBy` queries for elements that may not exist** -- `getByText('Error')` throws immediately if the element is absent. When asserting that something is NOT rendered, use `queryByText('Error')` which returns `null`. + +5. **Snapshot overuse** -- Developers update snapshots without reviewing the diff. Over time, snapshots become rubber stamps. Limit snapshots to serialized output and error formatting; use explicit assertions for behavior. + +6. **Testing third-party library internals** -- Do not test that React Router navigates correctly or that Zustand updates state. Test that your component renders the right thing after navigation or state change. Trust library authors; test your code. + +--- + +## Related Skills + +- `testing/pytest` -- Python testing counterpart +- `languages/typescript` -- TypeScript language patterns and strict typing +- `frameworks/react` -- React component patterns for component testing +- `methodology/test-driven-development` -- TDD workflow for writing tests first +- `devops/github-actions` — Running vitest in CI/CD pipelines diff --git a/.claude/skills/testing/vitest/references/mock-patterns.md b/.claude/skills/testing/vitest/references/mock-patterns.md new file mode 100644 index 0000000..69239b1 --- /dev/null +++ b/.claude/skills/testing/vitest/references/mock-patterns.md @@ -0,0 +1,242 @@ +# Vitest Mock Patterns + +Catalog of mocking patterns for common testing scenarios. + +## 1. Module Mock (Full) + +Replace an entire module with mock implementations. + +```typescript +import { describe, it, expect, vi } from "vitest"; + +// Mock the entire module BEFORE importing code that uses it. +vi.mock("@/services/payment", () => ({ + chargeCard: vi.fn().mockResolvedValue({ + transactionId: "txn-123", + status: "succeeded", + }), + refundCharge: vi.fn().mockResolvedValue({ + refundId: "ref-456", + status: "refunded", + }), +})); + +import { chargeCard } from "@/services/payment"; +import { checkout } from "@/services/checkout"; + +describe("checkout", () => { + it("should charge the card and return success", async () => { + const result = await checkout({ amount: 42, cardToken: "tok_test" }); + + expect(chargeCard).toHaveBeenCalledWith({ + amount: 42, + token: "tok_test", + }); + expect(result.status).toBe("succeeded"); + }); +}); +``` + +## 2. Partial Module Mock + +Mock only specific exports; keep the rest real. + +```typescript +import { describe, it, expect, vi } from "vitest"; + +vi.mock("@/utils/config", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + // Override only this one export + getFeatureFlag: vi.fn().mockReturnValue(true), + }; +}); + +import { getFeatureFlag, parseConfig } from "@/utils/config"; + +describe("with feature flag enabled", () => { + it("should use the new algorithm", () => { + // getFeatureFlag is mocked, parseConfig is real + expect(getFeatureFlag("new-algo")).toBe(true); + }); +}); +``` + +## 3. Manual Mock Reset / Per-Test Overrides + +```typescript +import { describe, it, expect, vi, beforeEach } from "vitest"; +import { fetchUser } from "@/api/users"; + +vi.mock("@/api/users"); + +// Type the mock for autocomplete +const mockFetchUser = vi.mocked(fetchUser); + +beforeEach(() => { + vi.resetAllMocks(); // Clear call history AND implementations +}); + +describe("user profile", () => { + it("shows user data on success", async () => { + mockFetchUser.mockResolvedValueOnce({ id: "1", name: "Alice" }); + // ...test + }); + + it("shows error on failure", async () => { + mockFetchUser.mockRejectedValueOnce(new Error("Network error")); + // ...test + }); +}); +``` + +## 4. API Mock with MSW (Mock Service Worker) + +Best for integration tests that should exercise real fetch/axios code. + +```typescript +// test/mocks/handlers.ts +import { http, HttpResponse } from "msw"; + +export const handlers = [ + http.get("/api/users/:id", ({ params }) => { + return HttpResponse.json({ id: params.id, name: "Alice", email: "alice@example.com" }); + }), + http.post("/api/users", async ({ request }) => { + const body = await request.json(); + return HttpResponse.json({ id: "new-1", ...body }, { status: 201 }); + }), +]; +``` + +```typescript +// test/mocks/server.ts +import { setupServer } from "msw/node"; +import { handlers } from "./handlers"; + +export const server = setupServer(...handlers); +``` + +```typescript +// test/setup.ts (referenced in vitest.config.ts setupFiles) +import { afterAll, afterEach, beforeAll } from "vitest"; +import { server } from "./mocks/server"; + +beforeAll(() => server.listen({ onUnhandledRequest: "error" })); +afterEach(() => server.resetHandlers()); +afterAll(() => server.close()); +``` + +```typescript +// Usage in tests -- override handlers per test +import { http, HttpResponse } from "msw"; +import { server } from "../mocks/server"; + +it("handles server error", async () => { + server.use( + http.get("/api/users/:id", () => { + return HttpResponse.json({ error: "Not found" }, { status: 404 }); + }), + ); + // ...test error handling +}); +``` + +## 5. Timer Mocks + +Control `setTimeout`, `setInterval`, `Date.now`. + +```typescript +beforeEach(() => vi.useFakeTimers()); +afterEach(() => vi.useRealTimers()); + +it("should call the function after the delay", () => { + const fn = vi.fn(); + const debounced = debounce(fn, 300); + + debounced(); + expect(fn).not.toHaveBeenCalled(); + + vi.advanceTimersByTime(300); + expect(fn).toHaveBeenCalledOnce(); +}); + +// Fake date: vi.setSystemTime(new Date("2025-01-15T12:00:00Z")) +``` + +## 6. Spy Patterns + +Observe calls without replacing implementation. + +```typescript +import { describe, it, expect, vi } from "vitest"; + +describe("logging", () => { + it("should log errors to console", () => { + const consoleSpy = vi.spyOn(console, "error").mockImplementation(() => {}); + + logError("something went wrong"); + + expect(consoleSpy).toHaveBeenCalledWith( + expect.stringContaining("something went wrong"), + ); + consoleSpy.mockRestore(); + }); +}); + +// Spy on object method without changing behavior +it("should call save", () => { + const repo = new UserRepository(); + const saveSpy = vi.spyOn(repo, "save"); + + repo.createUser({ name: "Alice" }); + + expect(saveSpy).toHaveBeenCalledOnce(); + expect(saveSpy).toHaveBeenCalledWith(expect.objectContaining({ name: "Alice" })); +}); +``` + +## 7. Global / Window Mocks + +```typescript +// Mock window.location +vi.spyOn(window, "location", "get").mockReturnValue({ ...window.location, pathname: "/dashboard" }); + +// Mock localStorage +const storage: Record = {}; +vi.spyOn(Storage.prototype, "getItem").mockImplementation((key) => storage[key] ?? null); +vi.spyOn(Storage.prototype, "setItem").mockImplementation((key, val) => { storage[key] = val; }); + +// Mock fetch (when not using MSW) +global.fetch = vi.fn().mockResolvedValue({ ok: true, json: () => Promise.resolve({ data: "test" }) }); +``` + +## 8. Class Mock + +```typescript +vi.mock("@/services/analytics", () => ({ + AnalyticsClient: vi.fn().mockImplementation(() => ({ + track: vi.fn(), + identify: vi.fn(), + flush: vi.fn().mockResolvedValue(undefined), + })), +})); +``` + +## Quick Reference: Mock Functions + +| Method | Purpose | +|--------|---------| +| `vi.fn()` | Create a standalone mock function | +| `vi.fn().mockReturnValue(x)` | Always return `x` | +| `vi.fn().mockReturnValueOnce(x)` | Return `x` once, then default | +| `vi.fn().mockResolvedValue(x)` | Return `Promise.resolve(x)` | +| `vi.fn().mockRejectedValue(e)` | Return `Promise.reject(e)` | +| `vi.fn().mockImplementation(fn)` | Use custom implementation | +| `vi.spyOn(obj, "method")` | Spy on existing method | +| `vi.mocked(fn)` | Type helper for mocked function | +| `vi.mock("module")` | Auto-mock all exports | +| `vi.resetAllMocks()` | Reset history and implementations | +| `vi.restoreAllMocks()` | Restore original implementations | +| `vi.clearAllMocks()` | Clear call history only | diff --git a/.claude/skills/testing/vitest/templates/vitest.config.ts b/.claude/skills/testing/vitest/templates/vitest.config.ts new file mode 100644 index 0000000..c9b1116 --- /dev/null +++ b/.claude/skills/testing/vitest/templates/vitest.config.ts @@ -0,0 +1,100 @@ +/// +import { defineConfig } from "vitest/config"; +import path from "node:path"; + +export default defineConfig({ + // ------------------------------------------------------------------------- + // Path aliases -- must match tsconfig.json "paths" + // ------------------------------------------------------------------------- + resolve: { + alias: { + "@": path.resolve(__dirname, "src"), + "@test": path.resolve(__dirname, "test"), + }, + }, + + test: { + // ----------------------------------------------------------------------- + // Environment + // ----------------------------------------------------------------------- + // "node" -- default, for backend / library code + // "jsdom" -- for code that accesses DOM APIs (React, etc.) + // "happy-dom" -- faster jsdom alternative + environment: "jsdom", + + // ----------------------------------------------------------------------- + // Globals + // ----------------------------------------------------------------------- + // Set to true to use describe/it/expect without importing from "vitest". + // Requires adding "vitest/globals" to tsconfig "types". + globals: true, + + // ----------------------------------------------------------------------- + // Setup files -- run before each test file + // ----------------------------------------------------------------------- + setupFiles: [ + "./test/setup.ts", + // "./test/mocks/server.ts", // MSW server setup + ], + + // ----------------------------------------------------------------------- + // File patterns + // ----------------------------------------------------------------------- + include: [ + "src/**/*.{test,spec}.{ts,tsx}", + "test/**/*.{test,spec}.{ts,tsx}", + ], + exclude: [ + "node_modules", + "dist", + "e2e/**", + ], + + // ----------------------------------------------------------------------- + // Coverage + // ----------------------------------------------------------------------- + coverage: { + provider: "v8", // or "istanbul" + reporter: ["text", "text-summary", "lcov", "json"], + reportsDirectory: "./coverage", + + include: ["src/**/*.{ts,tsx}"], + exclude: [ + "src/**/*.d.ts", + "src/**/*.test.{ts,tsx}", + "src/**/*.spec.{ts,tsx}", + "src/**/index.ts", // barrel files + "src/types/**", + ], + + // Minimum thresholds -- fail if coverage drops below these. + thresholds: { + statements: 80, + branches: 80, + functions: 80, + lines: 80, + }, + }, + + // ----------------------------------------------------------------------- + // Timeouts + // ----------------------------------------------------------------------- + testTimeout: 10_000, // 10s per test + hookTimeout: 10_000, // 10s per beforeEach/afterEach + + // ----------------------------------------------------------------------- + // Reporters + // ----------------------------------------------------------------------- + reporters: ["default"], + // For CI, add JUnit output: + // reporters: ["default", "junit"], + // outputFile: { junit: "./junit.xml" }, + + // ----------------------------------------------------------------------- + // Other options + // ----------------------------------------------------------------------- + // restoreMocks: true, // Automatically restore mocks after each test + // clearMocks: true, // Clear mock call history after each test + // mockReset: true, // Reset mocks (clear + remove implementations) + }, +}); diff --git a/README.md b/README.md index 126a718..49f93bb 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ A comprehensive toolkit for Claude Code to accelerate development workflows for - **20 Specialized Agents** - From planning to deployment - **27+ Slash Commands** - Workflow automation with flag support -- **30+ Skills** - Framework, language, methodology, and optimization expertise +- **38 Skills** - Framework, language, methodology, patterns, and optimization expertise (all with YAML frontmatter and bundled resources) - **7 Behavioral Modes** - Task-specific response optimization - **Command Flag System** - Combinable `--flag` syntax for customization - **Token Optimization** - 30-70% cost savings with compressed output modes @@ -29,11 +29,18 @@ A comprehensive toolkit for Claude Code to accelerate development workflows for ├── commands/ # 27+ workflow commands ├── modes/ # 7 behavioral mode definitions ├── mcp/ # MCP server configurations -└── skills/ # Framework, language, and methodology skills - ├── frameworks/ # FastAPI, Next.js, React, etc. +└── skills/ # 38 skills with YAML frontmatter & bundled resources + ├── api/ # OpenAPI specification patterns + ├── databases/ # PostgreSQL, MongoDB + ├── devops/ # Docker, GitHub Actions + ├── frameworks/ # FastAPI, Django, Next.js, React + ├── frontend/ # Tailwind CSS, shadcn/ui ├── languages/ # Python, TypeScript, JavaScript - ├── methodology/ # TDD, debugging, planning (14 skills) - └── optimization/ # Token efficiency patterns + ├── methodology/ # TDD, debugging, planning, review (14 skills) + ├── optimization/ # Token efficiency patterns + ├── patterns/ # Error handling, state, logging, caching, auth, API client + ├── security/ # OWASP security patterns + └── testing/ # pytest, vitest ``` ## Agents @@ -112,34 +119,55 @@ A comprehensive toolkit for Claude Code to accelerate development workflows for /spawn [task] # Launch parallel background task ``` -## Skills +## Skills (38 Total) + +Every skill includes YAML frontmatter for reliable triggering, "When to Use" / "When NOT to Use" sections, core patterns with code examples, best practices, common pitfalls, cross-references, and bundled resources (reference docs, templates, scripts). ### Languages -- Python, TypeScript, JavaScript +- **Python** — Type hints, async, dataclasses, Pydantic, decorators, pattern matching +- **TypeScript** — Advanced types, generics, Zod, discriminated unions, branded types +- **JavaScript** — ES6+, async patterns, Proxy/Reflect, generators, modules ### Frameworks -- FastAPI, Django, Next.js, React +- **FastAPI** — Routes, dependency injection, middleware, WebSocket, testing +- **Django** — ORM, views, migrations, DRF, signals, admin +- **Next.js** — App Router, server/client components, caching, middleware +- **React** — Hooks, custom hooks, context, Suspense, error boundaries, performance ### Databases -- PostgreSQL, MongoDB +- **PostgreSQL** — Schema, indexing (B-tree/GIN/GiST), migrations, CTEs, JSONB +- **MongoDB** — Schema design, aggregation pipelines, indexing, transactions ### DevOps -- Docker, GitHub Actions +- **Docker** — Multi-stage builds, Compose, security hardening, layer caching +- **GitHub Actions** — CI/CD, matrix strategy, reusable workflows, deployment ### Frontend -- Tailwind CSS, shadcn/ui +- **Tailwind CSS** — Responsive, dark mode, animations, theme customization +- **shadcn/ui** — Components, forms, data tables, theming, toast + +### API +- **OpenAPI** — 3.1 spec, pagination, versioning, error schemas, webhooks ### Security -- OWASP best practices +- **OWASP** — Top 10, auth, CORS, CSP, secret management, rate limiting ### Testing -- pytest, vitest +- **pytest** — Fixtures, parametrize, mocking, async, coverage +- **vitest** — React Testing Library, mocking, MSW, snapshots, configuration ### Optimization -- Token-efficient output patterns -- Sequential thinking methodology +- **Token-efficient** — Compressed output modes (30-70% cost savings) -### Methodology (Superpowers) +### Developer Patterns (New) +- **error-handling** — Custom errors, retry patterns, Result type, error boundaries +- **state-management** — React state, Zustand, TanStack Query, form state, URL state +- **logging** — Structured logging, log levels, correlation IDs, redaction +- **caching** — Memoization, HTTP cache, Redis, CDN, cache invalidation +- **api-client** — HTTP clients, interceptors, retry, type-safe clients +- **authentication** — JWT, OAuth2, sessions, RBAC, MFA, password hashing + +### Methodology (14 Skills) | Category | Skills | |----------|--------| @@ -147,6 +175,7 @@ A comprehensive toolkit for Claude Code to accelerate development workflows for | **Testing** | test-driven-development, verification-before-completion, testing-anti-patterns | | **Debugging** | systematic-debugging, root-cause-tracing, defense-in-depth | | **Collaboration** | dispatching-parallel-agents, requesting-code-review, receiving-code-review, finishing-development-branch | +| **Reasoning** | sequential-thinking | Key methodology principles: - **TDD Strict**: No production code without failing test first @@ -155,6 +184,16 @@ Key methodology principles: - **Bite-sized Tasks**: 2-5 minute increments with exact code - **Sequential Thinking**: Step-by-step reasoning with confidence scores +### Bundled Resources + +Skills include progressive-disclosure resources loaded on demand: + +| Resource Type | Purpose | Examples | +|---------------|---------|----------| +| **references/** | Cheat sheets, decision trees, pattern catalogs | OWASP Top 10, index decision tree, auth flows | +| **templates/** | Starter files, boilerplate, configs | OpenAPI spec, Dockerfile, CI workflows, conftest.py | +| **scripts/** | Executable helpers for deterministic tasks | Security audit scanner, OpenAPI validator | + ## Behavioral Modes Switch modes to optimize responses for different task types: @@ -307,16 +346,45 @@ Use $ARGUMENTS for command arguments. Create a new skill in `.claude/skills/category/skillname/SKILL.md`: -```markdown -# Skill Name +```yaml +--- +name: my-skill +description: > + What this skill does and when to trigger it. Be specific — list + contexts, keywords, and scenarios. 2-4 pushy sentences. +--- +``` -## Description -Brief description for matching. +```markdown +# My Skill + +Brief overview. + +## When to Use +- Scenario 1 +- Scenario 2 + +## When NOT to Use +- Anti-trigger scenario --- -## Patterns -Your patterns and examples here. +## Core Patterns +### Pattern Name +Code examples with good/bad comparisons. + +## Best Practices +## Common Pitfalls +## Related Skills +``` + +Optionally add bundled resources: +``` +my-skill/ +├── SKILL.md +├── references/ # Loaded into context on demand +├── scripts/ # Executed without loading into context +└── templates/ # Scaffolded into user's project ``` ## Workflow Chains