mirror of
https://github.com/duthaho/claudekit.git
synced 2026-08-05 03:00:21 +03:00
feat: enhanced the writing skills
This commit is contained in:
duthaho
@@ -0,0 +1,64 @@
|
||||
---
|
||||
name: databases
|
||||
description: >
|
||||
Use when working with PostgreSQL, MongoDB, or Redis — including schema design, queries, indexing, migrations, connection pooling, caching layers, or any database operation. Also activate for keywords like SQL, aggregation pipeline, BSON, ioredis, alembic, prisma migrate, django migrate, EXPLAIN ANALYZE, ORM configuration, or NoSQL data modeling.
|
||||
---
|
||||
|
||||
# Databases
|
||||
|
||||
## When to Use
|
||||
|
||||
- PostgreSQL database operations, SQL query optimization, schema design
|
||||
- JSONB document storage, full-text search, window functions, CTEs
|
||||
- MongoDB document modeling, aggregation pipelines, semi-structured data
|
||||
- Redis caching, session storage, rate limiting, pub/sub, job queues, distributed locks
|
||||
- Database migrations — adding/modifying tables, columns, indexes, constraints
|
||||
- Resolving migration conflicts, rolling back failed migrations
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- Simple key-value caching within a single process — use `functools.lru_cache` or `Map`
|
||||
- File-based storage that doesn't need a database engine
|
||||
- Static data or configuration that belongs in environment variables
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Topic | Reference | Key tools |
|
||||
|-------|-----------|-----------|
|
||||
| PostgreSQL | `references/postgresql.md` | SQL, SQLAlchemy, Prisma, EXPLAIN ANALYZE, pg_stat_statements |
|
||||
| MongoDB | `references/mongodb.md` | Aggregation, Mongoose, Motor, document schemas, ESR indexing |
|
||||
| Redis | `references/redis.md` | Caching, pub/sub, ioredis, BullMQ, session storage, distributed locks |
|
||||
| Migrations | `references/migrations.md` | Alembic, Prisma Migrate, Django migrations, rollback strategies |
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use parameterized queries everywhere.** Never concatenate user input into SQL strings.
|
||||
2. **Design schema around access patterns.** Ask "how will I read this?" before "how does this relate?" Embed data fetched together (MongoDB); normalize data accessed independently (PostgreSQL).
|
||||
3. **Index foreign keys and query fields.** PostgreSQL doesn't auto-index FK child columns. MongoDB queries without indexes trigger full collection scans.
|
||||
4. **Use appropriate consistency levels.** `TIMESTAMPTZ` over `TIMESTAMP` (PostgreSQL). `w: "majority"` for durable writes (MongoDB). TTLs on every Redis cache key.
|
||||
5. **Monitor query performance.** `pg_stat_statements` (PostgreSQL), `db.setProfilingLevel(1)` (MongoDB), connection pool metrics (all).
|
||||
6. **Use bulk/batch operations.** `bulkWrite` (MongoDB), `COPY` (PostgreSQL), pipelines (Redis) for high-throughput writes.
|
||||
7. **Never edit deployed migrations.** Create a new migration instead of modifying one already applied.
|
||||
8. **Test rollback paths.** Always verify your downgrade/rollback strategy before deploying schema changes.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **N+1 queries from ORM lazy loading.** Use eager loading (`joinedload`, `select_related`, `$lookup` with caution).
|
||||
2. **Table locks during migrations.** Use `CREATE INDEX CONCURRENTLY` (PostgreSQL). Batch backfills for large tables.
|
||||
3. **Unbounded growth.** Dead tuples from UPDATE-heavy workloads (PostgreSQL). Arrays exceeding 16MB document limit (MongoDB). Redis keys without TTLs.
|
||||
4. **OFFSET pagination on large datasets.** Use keyset/cursor pagination instead.
|
||||
5. **Connection exhaustion.** Use connection pools (PgBouncer, application-level pools). Never open per-request connections.
|
||||
6. **Cache stampede.** When a popular Redis key expires, many requests hit the DB simultaneously. Use distributed locks or stale-while-revalidate.
|
||||
7. **Running `migrate reset` in production.** This drops all data.
|
||||
|
||||
---
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `backend-frameworks` — Framework-specific ORM integration
|
||||
- `error-handling` — Database error handling patterns
|
||||
- `logging` — Query logging and slow query detection
|
||||
@@ -1,237 +0,0 @@
|
||||
# MongoDB Schema Design Patterns
|
||||
|
||||
Quick reference for embedding vs referencing decisions and common schema patterns.
|
||||
|
||||
## Embedding vs Referencing Decision Tree
|
||||
|
||||
```
|
||||
What is the relationship cardinality?
|
||||
|
|
||||
+-- One-to-Few (< 50 items)?
|
||||
| --> EMBED in parent document
|
||||
| Example: user.addresses, post.tags
|
||||
|
|
||||
+-- One-to-Many (50 - 1000s)?
|
||||
| |
|
||||
| +-- Child data always accessed with parent?
|
||||
| | --> EMBED (but watch 16 MB doc limit)
|
||||
| |
|
||||
| +-- Child data accessed independently?
|
||||
| | --> REFERENCE (store child _id in parent array)
|
||||
| |
|
||||
| +-- Need atomic updates across parent + children?
|
||||
| --> EMBED
|
||||
|
|
||||
+-- One-to-Millions?
|
||||
| --> REFERENCE from child to parent
|
||||
| Example: log_entry.host_id (not host.log_entry_ids)
|
||||
|
|
||||
+-- Many-to-Many?
|
||||
--> REFERENCE with array of _ids on one or both sides
|
||||
Example: student.course_ids[], course.student_ids[]
|
||||
```
|
||||
|
||||
## Decision Factors
|
||||
|
||||
| Factor | Favor Embedding | Favor Referencing |
|
||||
|--------|----------------|-------------------|
|
||||
| **Read pattern** | Always read together | Read independently |
|
||||
| **Write pattern** | Infrequent child updates | Frequent child updates |
|
||||
| **Data size** | Small, bounded children | Large or growing children |
|
||||
| **Atomicity** | Need single-doc transactions | Can tolerate multi-doc txn |
|
||||
| **Duplication** | OK to denormalize | Must avoid duplication |
|
||||
| **Cardinality** | Few items | Many/unbounded items |
|
||||
| **Document size** | Well under 16 MB limit | Approaching 16 MB |
|
||||
|
||||
## Pattern Catalog
|
||||
|
||||
### 1. Subset Pattern
|
||||
|
||||
**Problem**: Document is large but reads only need a few fields from embedded data.
|
||||
|
||||
**Solution**: Embed a subset; keep full data in a separate collection.
|
||||
|
||||
```javascript
|
||||
// products collection - fast reads for listing pages
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
name: "Widget",
|
||||
price: 29.99,
|
||||
// Only the 10 most recent reviews (subset)
|
||||
recent_reviews: [
|
||||
{ user: "alice", rating: 5, text: "Great!", date: ISODate("...") }
|
||||
],
|
||||
review_count: 247
|
||||
}
|
||||
|
||||
// reviews collection - full review data
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
product_id: ObjectId("..."),
|
||||
user: "alice",
|
||||
rating: 5,
|
||||
text: "Great!",
|
||||
date: ISODate("..."),
|
||||
helpful_votes: 12
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Product pages, user profiles, any "preview + detail" pattern.
|
||||
|
||||
### 2. Computed Pattern
|
||||
|
||||
**Problem**: Expensive aggregation queries run repeatedly on the same data.
|
||||
|
||||
**Solution**: Pre-compute and store the result, update on write.
|
||||
|
||||
```javascript
|
||||
// movies collection
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Example Movie",
|
||||
// Pre-computed from screenings collection
|
||||
computed: {
|
||||
total_revenue: 1250000,
|
||||
avg_rating: 4.2,
|
||||
rating_count: 843,
|
||||
last_computed: ISODate("2025-01-15T00:00:00Z")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Update strategy**: On each new rating, increment count and recalculate average. Or use a background job for less time-sensitive data.
|
||||
|
||||
**When to use**: Dashboards, leaderboards, summary statistics.
|
||||
|
||||
### 3. Bucket Pattern
|
||||
|
||||
**Problem**: Many small, time-series documents create overhead (indexes, storage per doc).
|
||||
|
||||
**Solution**: Group related data into fixed-size buckets.
|
||||
|
||||
```javascript
|
||||
// sensor_readings collection - one doc per sensor per hour
|
||||
{
|
||||
sensor_id: "sensor-42",
|
||||
bucket_start: ISODate("2025-01-15T14:00:00Z"),
|
||||
bucket_end: ISODate("2025-01-15T14:59:59Z"),
|
||||
count: 60,
|
||||
readings: [
|
||||
{ ts: ISODate("2025-01-15T14:00:00Z"), temp: 22.1, humidity: 45 },
|
||||
{ ts: ISODate("2025-01-15T14:01:00Z"), temp: 22.3, humidity: 44 }
|
||||
// ... up to 60 readings per bucket
|
||||
],
|
||||
// Pre-computed aggregates for the bucket
|
||||
summary: {
|
||||
avg_temp: 22.4,
|
||||
min_temp: 21.8,
|
||||
max_temp: 23.1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Bucket sizing**: Choose a size that balances doc count reduction vs update frequency. Common choices: 1 hour, 1 day, 100 events.
|
||||
|
||||
**When to use**: IoT, time-series, event logging, analytics.
|
||||
|
||||
### 4. Outlier Pattern
|
||||
|
||||
**Problem**: A few documents have vastly more data than the norm (e.g., a viral post with millions of likes).
|
||||
|
||||
**Solution**: Flag outliers and overflow into separate documents.
|
||||
|
||||
```javascript
|
||||
// books collection - normal case
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Normal Book",
|
||||
customers_purchased: ["user1", "user2", "user3"],
|
||||
has_overflow: false
|
||||
}
|
||||
|
||||
// books collection - outlier (bestseller)
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Bestseller",
|
||||
customers_purchased: ["user1", "user2", /* ... first 1000 */],
|
||||
has_overflow: true
|
||||
}
|
||||
|
||||
// book_purchases_overflow collection
|
||||
{
|
||||
book_id: ObjectId("..."),
|
||||
page: 2,
|
||||
customers_purchased: ["user1001", "user1002", /* ... next 1000 */]
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Social media (viral posts), e-commerce (bestsellers), any data with power-law distribution.
|
||||
|
||||
### 5. Extended Reference Pattern
|
||||
|
||||
**Problem**: Frequent joins (lookups) to get a few fields from a referenced document.
|
||||
|
||||
**Solution**: Copy the most-accessed fields into the referencing document.
|
||||
|
||||
```javascript
|
||||
// orders collection
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
date: ISODate("..."),
|
||||
customer_id: ObjectId("..."),
|
||||
// Extended reference - copied fields for fast reads
|
||||
customer_name: "Alice Smith",
|
||||
customer_email: "alice@example.com",
|
||||
items: [
|
||||
{
|
||||
product_id: ObjectId("..."),
|
||||
product_name: "Widget", // copied
|
||||
price: 29.99, // copied (snapshot at time of order)
|
||||
quantity: 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Trade-off**: Stale data is acceptable (order snapshots price at purchase time). For data that must be current, keep only the reference.
|
||||
|
||||
**When to use**: Orders (snapshot pricing), notifications (snapshot user name), audit logs.
|
||||
|
||||
### 6. Polymorphic Pattern
|
||||
|
||||
**Problem**: Objects share some fields but differ in others (e.g., different product types).
|
||||
|
||||
**Solution**: Store in a single collection with a type discriminator.
|
||||
|
||||
```javascript
|
||||
// vehicles collection
|
||||
{ type: "car", make: "Toyota", doors: 4, trunk_size_liters: 450 }
|
||||
{ type: "truck", make: "Ford", doors: 2, payload_kg: 5000 }
|
||||
{ type: "motorcycle", make: "Harley", engine_cc: 1200 }
|
||||
```
|
||||
|
||||
**Index strategy**: Index common fields. Use partial indexes for type-specific fields.
|
||||
|
||||
```javascript
|
||||
db.vehicles.createIndex(
|
||||
{ payload_kg: 1 },
|
||||
{ partialFilterExpression: { type: "truck" } }
|
||||
);
|
||||
```
|
||||
|
||||
**When to use**: Product catalogs, content management (articles, videos, images), mixed event streams.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Mistake | Problem | Fix |
|
||||
|---------|---------|-----|
|
||||
| Unbounded array growth | Document exceeds 16 MB | Use bucket or outlier pattern |
|
||||
| Deep nesting (> 3 levels) | Hard to query and index | Flatten or reference |
|
||||
| Normalizing everything | Too many lookups, slow reads | Embed when read together |
|
||||
| Embedding large blobs | Wastes RAM in working set | Store in GridFS or S3 |
|
||||
| No schema validation | Inconsistent data over time | Use JSON Schema validation |
|
||||
| Indexing every field | Slow writes, wasted space | Index based on query patterns |
|
||||
|
||||
## Schema Validation
|
||||
|
||||
Use `db.createCollection()` with `$jsonSchema` validator to enforce structure. Set `validationLevel: "moderate"` to apply only on inserts and updates (not existing docs).
|
||||
@@ -1,173 +0,0 @@
|
||||
# PostgreSQL Index Decision Tree
|
||||
|
||||
Quick reference for choosing the right index type.
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```
|
||||
What are you querying?
|
||||
|
|
||||
+-- Equality (=) or Range (<, >, BETWEEN, ORDER BY)?
|
||||
| |
|
||||
| +-- On a single scalar column?
|
||||
| | --> B-tree (default)
|
||||
| |
|
||||
| +-- On a timestamp/date column with append-only inserts?
|
||||
| | --> BRIN (much smaller than B-tree)
|
||||
| |
|
||||
| +-- Need the index to also return columns without table lookup?
|
||||
| --> Covering Index (B-tree with INCLUDE)
|
||||
|
|
||||
+-- Array containment (@>, &&) or JSONB queries?
|
||||
| --> GIN
|
||||
|
|
||||
+-- Full-text search (tsvector, @@)?
|
||||
| --> GIN
|
||||
|
|
||||
+-- Geometric/spatial data (points, polygons, PostGIS)?
|
||||
| --> GiST
|
||||
|
|
||||
+-- Range types (int4range, tsrange, overlaps)?
|
||||
| --> GiST
|
||||
|
|
||||
+-- Nearest-neighbor / distance queries (KNN)?
|
||||
| --> GiST (or SP-GiST for partitioned space)
|
||||
|
|
||||
+-- Only a subset of rows match your WHERE clause?
|
||||
| --> Partial Index (any type + WHERE filter)
|
||||
|
|
||||
+-- Trigram similarity (LIKE '%pattern%', pg_trgm)?
|
||||
| --> GIN with pg_trgm (or GiST for smaller, slower)
|
||||
|
|
||||
+-- Hash equality only (= but never range)?
|
||||
--> Hash index (rarely better than B-tree in practice)
|
||||
```
|
||||
|
||||
## Index Type Comparison
|
||||
|
||||
| Type | Best For | Operators | Size | Write Cost | Notes |
|
||||
|------|----------|-----------|------|------------|-------|
|
||||
| **B-tree** | Equality, range, sorting | `= < > <= >= BETWEEN IN IS NULL` | Medium | Low | Default. Covers 90% of cases. |
|
||||
| **GIN** | Multi-valued data | `@> && @@ ? ?& ?|` | Large | High (slow updates) | Best for arrays, JSONB, full-text. Use `fastupdate=on`. |
|
||||
| **GiST** | Spatial, ranges, nearest-neighbor | `<< >> && @> <@ <->` | Medium | Medium | Lossy for some types. Supports KNN. |
|
||||
| **SP-GiST** | Partitioned search spaces | Same as GiST | Medium | Medium | Good for phone numbers, IP addresses, non-balanced trees. |
|
||||
| **BRIN** | Large sequential/append-only tables | `= < > <= >=` | Tiny | Very Low | 1000x smaller than B-tree. Only effective when physical order correlates with column values. |
|
||||
| **Hash** | Equality only | `=` | Medium | Low | WAL-logged since PG10. Rarely outperforms B-tree. |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Covering Index (Index-Only Scans)
|
||||
|
||||
Avoid heap lookups by including extra columns:
|
||||
|
||||
```sql
|
||||
-- Query: SELECT email, name FROM users WHERE email = ?
|
||||
CREATE INDEX idx_users_email_covering
|
||||
ON users (email) INCLUDE (name);
|
||||
```
|
||||
|
||||
### Partial Index (Filtered)
|
||||
|
||||
Index only the rows you actually query:
|
||||
|
||||
```sql
|
||||
-- Only index active orders (skip 95% of rows)
|
||||
CREATE INDEX idx_orders_active
|
||||
ON orders (created_at)
|
||||
WHERE status = 'active';
|
||||
```
|
||||
|
||||
### Composite Index (Multi-Column)
|
||||
|
||||
Column order matters -- put equality columns first, range columns last:
|
||||
|
||||
```sql
|
||||
-- Query: WHERE tenant_id = ? AND created_at > ?
|
||||
CREATE INDEX idx_events_tenant_date
|
||||
ON events (tenant_id, created_at);
|
||||
```
|
||||
|
||||
### Expression Index
|
||||
|
||||
Index a computed value:
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_users_lower_email
|
||||
ON users (lower(email));
|
||||
```
|
||||
|
||||
### GIN for JSONB
|
||||
|
||||
```sql
|
||||
-- Index all keys and values in a JSONB column
|
||||
CREATE INDEX idx_metadata_gin
|
||||
ON products USING gin (metadata jsonb_path_ops);
|
||||
|
||||
-- Supports: metadata @> '{"color": "red"}'
|
||||
```
|
||||
|
||||
### GiST for Range Overlap
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_reservations_during
|
||||
ON reservations USING gist (during);
|
||||
|
||||
-- Supports: WHERE during && '[2025-01-01, 2025-01-31]'::daterange
|
||||
```
|
||||
|
||||
### BRIN for Time-Series
|
||||
|
||||
```sql
|
||||
-- Table has millions of rows inserted in timestamp order
|
||||
CREATE INDEX idx_logs_ts_brin
|
||||
ON logs USING brin (created_at)
|
||||
WITH (pages_per_range = 32);
|
||||
```
|
||||
|
||||
## Sizing Rules of Thumb
|
||||
|
||||
| Table Rows | B-tree Size | BRIN Size | GIN Size |
|
||||
|------------|-------------|-----------|----------|
|
||||
| 1M | ~20 MB | ~50 KB | ~30 MB |
|
||||
| 10M | ~200 MB | ~500 KB | ~300 MB |
|
||||
| 100M | ~2 GB | ~5 MB | ~3 GB |
|
||||
|
||||
## Diagnostic Queries
|
||||
|
||||
```sql
|
||||
-- Check if an index is being used
|
||||
EXPLAIN (ANALYZE, BUFFERS) SELECT ...;
|
||||
|
||||
-- Find unused indexes
|
||||
SELECT indexrelname, idx_scan
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE idx_scan = 0
|
||||
ORDER BY pg_relation_size(indexrelid) DESC;
|
||||
|
||||
-- Check index size
|
||||
SELECT pg_size_pretty(pg_relation_size('idx_name'));
|
||||
|
||||
-- Index bloat estimate
|
||||
SELECT * FROM pgstatindex('idx_name');
|
||||
```
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Mistake | Why It Hurts |
|
||||
|---------|-------------|
|
||||
| Indexing every column | Slows writes, wastes disk, confuses planner |
|
||||
| Wrong column order in composite | Index cannot be used for the query |
|
||||
| GIN on tiny tables | Overhead exceeds benefit |
|
||||
| B-tree on low-cardinality columns | Planner prefers seq scan anyway |
|
||||
| Missing `CONCURRENTLY` on production | Locks the table during index build |
|
||||
| Forgetting `ANALYZE` after bulk load | Planner uses stale statistics |
|
||||
|
||||
## Safe Index Creation
|
||||
|
||||
```sql
|
||||
-- Non-blocking index creation (no table lock)
|
||||
CREATE INDEX CONCURRENTLY idx_name ON table (column);
|
||||
|
||||
-- Always run ANALYZE after bulk operations
|
||||
ANALYZE table;
|
||||
```
|
||||
@@ -1,143 +0,0 @@
|
||||
-- =============================================================================
|
||||
-- Migration: [DESCRIPTION]
|
||||
-- Created: [DATE]
|
||||
-- Author: [AUTHOR]
|
||||
-- Ticket: [TICKET-ID]
|
||||
-- =============================================================================
|
||||
--
|
||||
-- SAFETY CHECKLIST (review before running):
|
||||
-- [ ] Tested on staging with production-size data
|
||||
-- [ ] Backward compatible with current application code
|
||||
-- [ ] No exclusive locks on large tables during peak hours
|
||||
-- [ ] Rollback (DOWN) section tested independently
|
||||
-- [ ] Estimated run time: ___
|
||||
-- [ ] Estimated lock duration: ___
|
||||
--
|
||||
|
||||
-- ============================================================
|
||||
-- UP MIGRATION
|
||||
-- ============================================================
|
||||
|
||||
BEGIN;
|
||||
|
||||
-- Set a statement timeout to prevent long-running locks.
|
||||
-- Adjust as needed; remove for data-only migrations.
|
||||
SET LOCAL lock_timeout = '5s';
|
||||
SET LOCAL statement_timeout = '30s';
|
||||
|
||||
-- ------------------------------------
|
||||
-- 1. Schema changes
|
||||
-- ------------------------------------
|
||||
|
||||
-- Add new table
|
||||
-- CREATE TABLE IF NOT EXISTS example (
|
||||
-- id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
-- name text NOT NULL,
|
||||
-- created_at timestamptz NOT NULL DEFAULT now(),
|
||||
-- updated_at timestamptz NOT NULL DEFAULT now()
|
||||
-- );
|
||||
|
||||
-- Add column (safe: does not rewrite table)
|
||||
-- ALTER TABLE example ADD COLUMN IF NOT EXISTS description text;
|
||||
|
||||
-- Add column with default (PG 11+: does not rewrite table)
|
||||
-- ALTER TABLE example ADD COLUMN IF NOT EXISTS is_active boolean NOT NULL DEFAULT true;
|
||||
|
||||
-- Rename column (safe: metadata-only change)
|
||||
-- ALTER TABLE example RENAME COLUMN old_name TO new_name;
|
||||
|
||||
-- ------------------------------------
|
||||
-- 2. Constraints
|
||||
-- ------------------------------------
|
||||
|
||||
-- Add NOT NULL (requires all existing rows to satisfy it)
|
||||
-- ALTER TABLE example ALTER COLUMN name SET NOT NULL;
|
||||
|
||||
-- Add check constraint (NOT VALID avoids full table scan, then VALIDATE separately)
|
||||
-- ALTER TABLE example ADD CONSTRAINT chk_example_name CHECK (name <> '') NOT VALID;
|
||||
-- ALTER TABLE example VALIDATE CONSTRAINT chk_example_name;
|
||||
|
||||
-- Add foreign key (NOT VALID + VALIDATE pattern to avoid long locks)
|
||||
-- ALTER TABLE example ADD CONSTRAINT fk_example_parent
|
||||
-- FOREIGN KEY (parent_id) REFERENCES parent(id) NOT VALID;
|
||||
-- ALTER TABLE example VALIDATE CONSTRAINT fk_example_parent;
|
||||
|
||||
-- ------------------------------------
|
||||
-- 3. Indexes (use CONCURRENTLY outside transaction)
|
||||
-- ------------------------------------
|
||||
|
||||
-- NOTE: CREATE INDEX CONCURRENTLY cannot run inside a transaction.
|
||||
-- Run these statements separately after committing the transaction above.
|
||||
--
|
||||
-- CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_example_name
|
||||
-- ON example (name);
|
||||
--
|
||||
-- CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_example_created_at
|
||||
-- ON example USING brin (created_at);
|
||||
|
||||
-- ------------------------------------
|
||||
-- 4. Data migration
|
||||
-- ------------------------------------
|
||||
|
||||
-- Backfill in batches to avoid long transactions:
|
||||
-- UPDATE example SET description = 'default' WHERE description IS NULL;
|
||||
--
|
||||
-- For large tables, batch with:
|
||||
-- DO $$
|
||||
-- DECLARE
|
||||
-- batch_size int := 10000;
|
||||
-- rows_updated int;
|
||||
-- BEGIN
|
||||
-- LOOP
|
||||
-- UPDATE example
|
||||
-- SET description = 'default'
|
||||
-- WHERE id IN (
|
||||
-- SELECT id FROM example
|
||||
-- WHERE description IS NULL
|
||||
-- LIMIT batch_size
|
||||
-- FOR UPDATE SKIP LOCKED
|
||||
-- );
|
||||
-- GET DIAGNOSTICS rows_updated = ROW_COUNT;
|
||||
-- EXIT WHEN rows_updated = 0;
|
||||
-- RAISE NOTICE 'Updated % rows', rows_updated;
|
||||
-- COMMIT;
|
||||
-- END LOOP;
|
||||
-- END $$;
|
||||
|
||||
-- ------------------------------------
|
||||
-- 5. Permissions
|
||||
-- ------------------------------------
|
||||
|
||||
-- GRANT SELECT, INSERT, UPDATE ON example TO app_role;
|
||||
-- GRANT USAGE ON SEQUENCE example_id_seq TO app_role;
|
||||
|
||||
COMMIT;
|
||||
|
||||
|
||||
-- ============================================================
|
||||
-- DOWN MIGRATION (rollback)
|
||||
-- ============================================================
|
||||
-- Run this section to undo the UP migration.
|
||||
-- Test this independently before deploying the UP migration.
|
||||
|
||||
-- BEGIN;
|
||||
--
|
||||
-- -- Reverse data migration
|
||||
-- -- UPDATE example SET description = NULL;
|
||||
--
|
||||
-- -- Drop constraints
|
||||
-- -- ALTER TABLE example DROP CONSTRAINT IF EXISTS chk_example_name;
|
||||
-- -- ALTER TABLE example DROP CONSTRAINT IF EXISTS fk_example_parent;
|
||||
--
|
||||
-- -- Drop columns
|
||||
-- -- ALTER TABLE example DROP COLUMN IF EXISTS description;
|
||||
-- -- ALTER TABLE example DROP COLUMN IF EXISTS is_active;
|
||||
--
|
||||
-- -- Drop tables
|
||||
-- -- DROP TABLE IF EXISTS example;
|
||||
--
|
||||
-- COMMIT;
|
||||
--
|
||||
-- -- Drop indexes (outside transaction)
|
||||
-- -- DROP INDEX CONCURRENTLY IF EXISTS idx_example_name;
|
||||
-- -- DROP INDEX CONCURRENTLY IF EXISTS idx_example_created_at;
|
||||
@@ -0,0 +1,312 @@
|
||||
# Databases — Migration Patterns
|
||||
|
||||
|
||||
# Database Migrations
|
||||
|
||||
## When to Use
|
||||
|
||||
- Adding or modifying database tables/columns
|
||||
- Creating indexes or constraints
|
||||
- Running migrations in development, staging, or production
|
||||
- Resolving migration conflicts in a team
|
||||
- Rolling back a failed migration
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- Query optimization without schema changes — use `postgresql` skill
|
||||
- Initial database design from scratch — use `postgresql` or `mongodb` skill
|
||||
- ORM configuration without migrations — use framework-specific skill
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| I need... | Go to |
|
||||
|-----------|-------|
|
||||
| Alembic (FastAPI/SQLAlchemy) | SS Alembic below |
|
||||
| Prisma (NestJS/Express) | SS Prisma below |
|
||||
| Django migrations | SS Django below |
|
||||
| Safe production patterns | SS Production Safety below |
|
||||
| Rollback strategies | SS Rollbacks below |
|
||||
|
||||
---
|
||||
|
||||
## Alembic (Python / SQLAlchemy)
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
pip install alembic
|
||||
alembic init migrations
|
||||
```
|
||||
|
||||
```python
|
||||
# migrations/env.py — configure target metadata
|
||||
from src.models import Base
|
||||
target_metadata = Base.metadata
|
||||
```
|
||||
|
||||
### Create a migration
|
||||
|
||||
```bash
|
||||
# Auto-generate from model changes
|
||||
alembic revision --autogenerate -m "add orders table"
|
||||
|
||||
# Manual migration (for data migrations or complex changes)
|
||||
alembic revision -m "backfill order status"
|
||||
```
|
||||
|
||||
### Migration file
|
||||
|
||||
```python
|
||||
# migrations/versions/003_add_orders_table.py
|
||||
"""add orders table"""
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = '003'
|
||||
down_revision = '002'
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
'orders',
|
||||
sa.Column('id', sa.UUID(), primary_key=True, server_default=sa.text('gen_random_uuid()')),
|
||||
sa.Column('user_id', sa.UUID(), sa.ForeignKey('users.id', ondelete='CASCADE'), nullable=False),
|
||||
sa.Column('total', sa.Numeric(10, 2), nullable=False),
|
||||
sa.Column('status', sa.String(20), nullable=False, server_default='pending'),
|
||||
sa.Column('created_at', sa.DateTime(timezone=True), server_default=sa.func.now()),
|
||||
)
|
||||
op.create_index('ix_orders_user_id', 'orders', ['user_id'])
|
||||
op.create_index('ix_orders_created_at', 'orders', ['created_at'])
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table('orders')
|
||||
```
|
||||
|
||||
### Run migrations
|
||||
|
||||
```bash
|
||||
# Apply all pending
|
||||
alembic upgrade head
|
||||
|
||||
# Apply one step
|
||||
alembic upgrade +1
|
||||
|
||||
# Check current state
|
||||
alembic current
|
||||
|
||||
# Check for pending migrations
|
||||
alembic check
|
||||
|
||||
# View migration history
|
||||
alembic history --verbose
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Prisma (TypeScript / NestJS / Express)
|
||||
|
||||
### Create a migration
|
||||
|
||||
```bash
|
||||
# Generate migration from schema changes
|
||||
npx prisma migrate dev --name add_orders_table
|
||||
|
||||
# Apply in production (no interactive prompts)
|
||||
npx prisma migrate deploy
|
||||
|
||||
# Check status
|
||||
npx prisma migrate status
|
||||
```
|
||||
|
||||
### Schema change
|
||||
|
||||
```prisma
|
||||
// prisma/schema.prisma
|
||||
model Order {
|
||||
id String @id @default(uuid())
|
||||
userId String
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
total Decimal @db.Decimal(10, 2)
|
||||
status String @default("pending")
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
@@index([userId])
|
||||
@@index([createdAt])
|
||||
}
|
||||
```
|
||||
|
||||
### Generated migration SQL
|
||||
|
||||
```sql
|
||||
-- prisma/migrations/20260417_add_orders_table/migration.sql
|
||||
CREATE TABLE "Order" (
|
||||
"id" TEXT NOT NULL DEFAULT gen_random_uuid(),
|
||||
"userId" TEXT NOT NULL,
|
||||
"total" DECIMAL(10,2) NOT NULL,
|
||||
"status" TEXT NOT NULL DEFAULT 'pending',
|
||||
"createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
CONSTRAINT "Order_pkey" PRIMARY KEY ("id")
|
||||
);
|
||||
|
||||
CREATE INDEX "Order_userId_idx" ON "Order"("userId");
|
||||
CREATE INDEX "Order_createdAt_idx" ON "Order"("createdAt");
|
||||
|
||||
ALTER TABLE "Order" ADD CONSTRAINT "Order_userId_fkey"
|
||||
FOREIGN KEY ("userId") REFERENCES "User"("id") ON DELETE CASCADE;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Django
|
||||
|
||||
### Create and apply
|
||||
|
||||
```bash
|
||||
# Auto-generate from model changes
|
||||
python manage.py makemigrations app_name
|
||||
|
||||
# Apply
|
||||
python manage.py migrate
|
||||
|
||||
# Check for pending
|
||||
python manage.py showmigrations
|
||||
|
||||
# SQL preview (don't execute)
|
||||
python manage.py sqlmigrate app_name 0003
|
||||
```
|
||||
|
||||
### Data migration
|
||||
|
||||
```python
|
||||
# app/migrations/0004_backfill_order_status.py
|
||||
from django.db import migrations
|
||||
|
||||
def backfill_status(apps, schema_editor):
|
||||
Order = apps.get_model('orders', 'Order')
|
||||
Order.objects.filter(status='').update(status='pending')
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [('orders', '0003_add_orders')]
|
||||
operations = [migrations.RunPython(backfill_status, migrations.RunPython.noop)]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Production Safety
|
||||
|
||||
### Golden rules
|
||||
|
||||
1. **Never drop columns in the same deploy as removing code references.** Remove code first, deploy, then drop column in next migration.
|
||||
2. **Add columns as nullable or with defaults.** `NOT NULL` without a default locks the table during backfill on large tables.
|
||||
3. **Create indexes concurrently** (PostgreSQL):
|
||||
```sql
|
||||
CREATE INDEX CONCURRENTLY ix_orders_status ON orders(status);
|
||||
```
|
||||
4. **Test migrations against a production-size dataset** before deploying.
|
||||
5. **Always have a rollback plan** — either a `downgrade()` function or a manual SQL script.
|
||||
|
||||
### Safe column addition pattern
|
||||
|
||||
```python
|
||||
# Step 1: Add nullable column (fast, no lock)
|
||||
op.add_column('users', sa.Column('phone', sa.String(20), nullable=True))
|
||||
|
||||
# Step 2: Backfill in batches (separate migration or script)
|
||||
# Don't do UPDATE users SET phone = '...' on millions of rows at once
|
||||
|
||||
# Step 3: Add NOT NULL constraint (after backfill confirms all rows filled)
|
||||
op.alter_column('users', 'phone', nullable=False)
|
||||
```
|
||||
|
||||
### Safe column rename pattern
|
||||
|
||||
```
|
||||
Deploy 1: Add new column, write to both old and new
|
||||
Deploy 2: Backfill new column from old, read from new
|
||||
Deploy 3: Stop writing to old column
|
||||
Deploy 4: Drop old column
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollbacks
|
||||
|
||||
### Alembic
|
||||
|
||||
```bash
|
||||
# Rollback one step
|
||||
alembic downgrade -1
|
||||
|
||||
# Rollback to specific revision
|
||||
alembic downgrade 002
|
||||
|
||||
# Rollback to base (dangerous — drops everything)
|
||||
alembic downgrade base
|
||||
```
|
||||
|
||||
### Prisma
|
||||
|
||||
Prisma doesn't have built-in rollback. Options:
|
||||
- Apply a new migration that reverses the change
|
||||
- Manually run SQL: `npx prisma db execute --file rollback.sql`
|
||||
- Restore from database backup
|
||||
|
||||
### Django
|
||||
|
||||
```bash
|
||||
# Rollback to specific migration
|
||||
python manage.py migrate app_name 0002
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Team Workflow
|
||||
|
||||
### Resolving migration conflicts
|
||||
|
||||
When two developers create migrations from the same parent:
|
||||
|
||||
**Alembic:**
|
||||
```bash
|
||||
# Developer A and B both branched from revision 002
|
||||
# Alembic detects multiple heads
|
||||
alembic heads # shows 003a and 003b
|
||||
alembic merge -m "merge migrations" 003a 003b
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
**Prisma:**
|
||||
```bash
|
||||
# Reset and re-apply (dev only)
|
||||
npx prisma migrate reset
|
||||
# Or resolve manually by editing the migration SQL
|
||||
```
|
||||
|
||||
**Django:**
|
||||
```bash
|
||||
# Django auto-detects and asks to merge
|
||||
python manage.py makemigrations --merge
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **Running `migrate reset` in production.** This drops all data. Only use in development.
|
||||
2. **Editing already-applied migrations.** Never modify a migration that's been deployed. Create a new migration instead.
|
||||
3. **Forgetting indexes.** Add indexes for foreign keys and frequently-queried columns in the same migration.
|
||||
4. **Large table locks.** `ALTER TABLE` with `NOT NULL` or `ADD COLUMN DEFAULT` can lock large tables. Use batched backfills.
|
||||
5. **Not testing downgrade.** Always test your rollback path before deploying.
|
||||
6. **Circular foreign keys.** Use `sa.ForeignKey` with `use_alter=True` in Alembic to handle circular deps.
|
||||
|
||||
---
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `postgresql` — Database design, query optimization, indexing strategies
|
||||
- `fastapi` — SQLAlchemy async patterns with FastAPI
|
||||
- `nestjs` — Prisma integration with NestJS
|
||||
- `django` — Django ORM models and migrations
|
||||
- `docker` — Running migration containers in CI/CD
|
||||
+5
-8
@@ -1,8 +1,5 @@
|
||||
---
|
||||
name: mongodb
|
||||
description: >
|
||||
Use this skill whenever working with MongoDB, document databases, or NoSQL data modeling. Trigger on keywords like MongoDB, Mongo, document database, aggregation pipeline, collection, embedded documents, or BSON. Also applies when designing document schemas, building aggregation queries, handling unstructured or semi-structured data, or migrating from relational to document-based storage.
|
||||
---
|
||||
# Databases — MongoDB Patterns
|
||||
|
||||
|
||||
# MongoDB
|
||||
|
||||
@@ -574,6 +571,6 @@ db.orders.updateMany(
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `databases/postgresql` - Relational database patterns for structured data with complex relationships
|
||||
- `patterns/caching` - Caching strategies to reduce database load
|
||||
- `patterns/logging` - Logging patterns for query debugging and monitoring
|
||||
- `postgresql` - Relational database patterns for structured data with complex relationships
|
||||
- `caching` - Caching strategies to reduce database load
|
||||
- `logging` - Logging patterns for query debugging and monitoring
|
||||
+5
-8
@@ -1,8 +1,5 @@
|
||||
---
|
||||
name: postgresql
|
||||
description: >
|
||||
Use this skill whenever working with PostgreSQL databases, writing SQL queries, designing schemas, or optimizing database performance. Trigger on keywords like PostgreSQL, Postgres, SQL query, schema design, indexing, migrations, EXPLAIN ANALYZE, connection pooling, or any relational database operation. Also applies when debugging slow queries, setting up database tables, or working with ORMs that target PostgreSQL.
|
||||
---
|
||||
# Databases — PostgreSQL Patterns
|
||||
|
||||
|
||||
# PostgreSQL
|
||||
|
||||
@@ -607,6 +604,6 @@ SELECT * FROM orders WHERE id > 100000 ORDER BY id LIMIT 20;
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `databases/mongodb` - Document-based database patterns for non-relational data
|
||||
- `patterns/caching` - Caching strategies to reduce database load
|
||||
- `patterns/logging` - Logging patterns for query debugging and monitoring
|
||||
- `mongodb` - Document-based database patterns for non-relational data
|
||||
- `caching` - Caching strategies to reduce database load
|
||||
- `logging` - Logging patterns for query debugging and monitoring
|
||||
@@ -0,0 +1,279 @@
|
||||
# Databases — Redis Patterns
|
||||
|
||||
|
||||
# Redis
|
||||
|
||||
## When to Use
|
||||
|
||||
- Caching database queries or API responses
|
||||
- Session storage for web applications
|
||||
- Rate limiting (distributed across instances)
|
||||
- Job/task queues (BullMQ, Celery)
|
||||
- Pub/sub messaging between services
|
||||
- Distributed locks
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- **Primary data storage** — Redis is a cache/broker, not a database of record
|
||||
- **Complex queries** — use PostgreSQL for relational queries
|
||||
- **Large blobs** — use S3/R2 for file storage
|
||||
- **In-memory caching only** — use `functools.lru_cache` or `Map` for single-process caches
|
||||
|
||||
---
|
||||
|
||||
## Python (redis-py / FastAPI)
|
||||
|
||||
### Connection
|
||||
|
||||
```python
|
||||
# src/core/redis.py
|
||||
import redis.asyncio as redis
|
||||
|
||||
pool = redis.ConnectionPool.from_url(
|
||||
"redis://localhost:6379/0",
|
||||
max_connections=20,
|
||||
decode_responses=True,
|
||||
)
|
||||
|
||||
async def get_redis() -> redis.Redis:
|
||||
return redis.Redis(connection_pool=pool)
|
||||
```
|
||||
|
||||
### Cache-aside pattern
|
||||
|
||||
```python
|
||||
import json
|
||||
from datetime import timedelta
|
||||
|
||||
async def get_user_cached(user_id: str, db: AsyncSession) -> User:
|
||||
r = await get_redis()
|
||||
cache_key = f"user:{user_id}"
|
||||
|
||||
# Check cache
|
||||
cached = await r.get(cache_key)
|
||||
if cached:
|
||||
return User(**json.loads(cached))
|
||||
|
||||
# Cache miss — fetch from DB
|
||||
user = await db.get(User, user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
# Store in cache with TTL
|
||||
await r.setex(cache_key, timedelta(minutes=15), json.dumps(user.to_dict()))
|
||||
return user
|
||||
```
|
||||
|
||||
### Cache invalidation
|
||||
|
||||
```python
|
||||
async def update_user(user_id: str, data: UpdateUserRequest, db: AsyncSession) -> User:
|
||||
user = await db.get(User, user_id)
|
||||
for key, value in data.dict(exclude_unset=True).items():
|
||||
setattr(user, key, value)
|
||||
await db.commit()
|
||||
|
||||
# Invalidate cache
|
||||
r = await get_redis()
|
||||
await r.delete(f"user:{user_id}")
|
||||
|
||||
return user
|
||||
```
|
||||
|
||||
### Rate limiting
|
||||
|
||||
```python
|
||||
from fastapi import Request, HTTPException
|
||||
|
||||
async def rate_limit(request: Request, limit: int = 100, window: int = 900):
|
||||
r = await get_redis()
|
||||
key = f"rate:{request.client.host}"
|
||||
current = await r.incr(key)
|
||||
if current == 1:
|
||||
await r.expire(key, window)
|
||||
if current > limit:
|
||||
raise HTTPException(status_code=429, detail="Rate limit exceeded")
|
||||
```
|
||||
|
||||
### Session storage
|
||||
|
||||
```python
|
||||
import secrets
|
||||
|
||||
async def create_session(user_id: str) -> str:
|
||||
r = await get_redis()
|
||||
session_id = secrets.token_urlsafe(32)
|
||||
await r.setex(f"session:{session_id}", timedelta(hours=24), user_id)
|
||||
return session_id
|
||||
|
||||
async def get_session(session_id: str) -> str | None:
|
||||
r = await get_redis()
|
||||
return await r.get(f"session:{session_id}")
|
||||
|
||||
async def delete_session(session_id: str):
|
||||
r = await get_redis()
|
||||
await r.delete(f"session:{session_id}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## TypeScript (ioredis / NestJS / Express)
|
||||
|
||||
### Connection
|
||||
|
||||
```typescript
|
||||
// src/core/redis.ts
|
||||
import Redis from 'ioredis';
|
||||
|
||||
export const redis = new Redis(process.env.REDIS_URL ?? 'redis://localhost:6379', {
|
||||
maxRetriesPerRequest: 3,
|
||||
lazyConnect: true,
|
||||
});
|
||||
```
|
||||
|
||||
### NestJS module
|
||||
|
||||
```typescript
|
||||
// src/cache/cache.module.ts
|
||||
import { Global, Module } from '@nestjs/common';
|
||||
import { CacheService } from './cache.service';
|
||||
|
||||
@Global()
|
||||
@Module({
|
||||
providers: [CacheService],
|
||||
exports: [CacheService],
|
||||
})
|
||||
export class CacheModule {}
|
||||
```
|
||||
|
||||
```typescript
|
||||
// src/cache/cache.service.ts
|
||||
import { Injectable, OnModuleDestroy } from '@nestjs/common';
|
||||
import Redis from 'ioredis';
|
||||
|
||||
@Injectable()
|
||||
export class CacheService implements OnModuleDestroy {
|
||||
private readonly redis = new Redis(process.env.REDIS_URL!);
|
||||
|
||||
async get<T>(key: string): Promise<T | null> {
|
||||
const data = await this.redis.get(key);
|
||||
return data ? JSON.parse(data) : null;
|
||||
}
|
||||
|
||||
async set(key: string, value: unknown, ttlSeconds: number): Promise<void> {
|
||||
await this.redis.setex(key, ttlSeconds, JSON.stringify(value));
|
||||
}
|
||||
|
||||
async del(key: string): Promise<void> {
|
||||
await this.redis.del(key);
|
||||
}
|
||||
|
||||
async onModuleDestroy() {
|
||||
await this.redis.quit();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Cache-aside in service
|
||||
|
||||
```typescript
|
||||
@Injectable()
|
||||
export class UsersService {
|
||||
constructor(
|
||||
private readonly prisma: PrismaService,
|
||||
private readonly cache: CacheService,
|
||||
) {}
|
||||
|
||||
async findOne(id: string): Promise<User> {
|
||||
// Check cache
|
||||
const cached = await this.cache.get<User>(`user:${id}`);
|
||||
if (cached) return cached;
|
||||
|
||||
// Cache miss
|
||||
const user = await this.prisma.user.findUnique({ where: { id } });
|
||||
if (!user) throw new NotFoundException(`User ${id} not found`);
|
||||
|
||||
// Store with 15min TTL
|
||||
await this.cache.set(`user:${id}`, user, 900);
|
||||
return user;
|
||||
}
|
||||
|
||||
async update(id: string, dto: UpdateUserDto): Promise<User> {
|
||||
const user = await this.prisma.user.update({ where: { id }, data: dto });
|
||||
await this.cache.del(`user:${id}`); // Invalidate
|
||||
return user;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pub/Sub
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
# Publisher
|
||||
async def publish_event(channel: str, event: dict):
|
||||
r = await get_redis()
|
||||
await r.publish(channel, json.dumps(event))
|
||||
|
||||
# Subscriber
|
||||
async def subscribe_events(channel: str):
|
||||
r = await get_redis()
|
||||
pubsub = r.pubsub()
|
||||
await pubsub.subscribe(channel)
|
||||
async for message in pubsub.listen():
|
||||
if message['type'] == 'message':
|
||||
yield json.loads(message['data'])
|
||||
```
|
||||
|
||||
### TypeScript
|
||||
|
||||
```typescript
|
||||
// Publisher
|
||||
const pub = new Redis(process.env.REDIS_URL!);
|
||||
await pub.publish('orders', JSON.stringify({ type: 'created', orderId: '123' }));
|
||||
|
||||
// Subscriber (separate connection required)
|
||||
const sub = new Redis(process.env.REDIS_URL!);
|
||||
sub.subscribe('orders');
|
||||
sub.on('message', (channel, message) => {
|
||||
const event = JSON.parse(message);
|
||||
console.log(`[${channel}]`, event);
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Naming Conventions
|
||||
|
||||
```
|
||||
entity:id → user:abc123
|
||||
entity:id:field → user:abc123:orders
|
||||
rate:ip → rate:192.168.1.1
|
||||
session:token → session:abc123def
|
||||
lock:resource → lock:order-processing
|
||||
queue:name → queue:email-notifications
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **Not setting TTLs.** Every cache key should have an expiration. Unbounded caches exhaust memory.
|
||||
2. **Cache stampede.** When a popular key expires, many requests hit the DB simultaneously. Use distributed locks or stale-while-revalidate.
|
||||
3. **Using the same connection for pub/sub.** Subscribers can't run other commands. Use a dedicated connection.
|
||||
4. **Storing large objects.** Redis is fast for small values. Keep values under 1MB; for larger data, store a pointer to S3.
|
||||
5. **Not handling connection failures.** Redis connections drop. Use retry logic and connection pools.
|
||||
6. **Forgetting to invalidate.** When data changes, delete the cache key. Stale cache is worse than no cache.
|
||||
|
||||
---
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `caching` — HTTP caching, CDN, memoization (framework-agnostic patterns)
|
||||
- `background-jobs` — BullMQ/Celery use Redis as broker
|
||||
- `fastapi` — Redis integration with FastAPI dependency injection
|
||||
- `nestjs` — Redis caching module in NestJS
|
||||
- `docker` — Running Redis in Docker Compose for development
|
||||
Reference in New Issue
Block a user