mirror of
https://github.com/duthaho/claudekit.git
synced 2026-07-22 04:31:00 +03:00
feat: adding new skills, including testing patterns and methodologies, along with bundled resources for better usability.
This commit is contained in:
duthaho
@@ -1,73 +1,579 @@
|
||||
---
|
||||
name: mongodb
|
||||
description: >
|
||||
Use this skill whenever working with MongoDB, document databases, or NoSQL data modeling. Trigger on keywords like MongoDB, Mongo, document database, aggregation pipeline, collection, embedded documents, or BSON. Also applies when designing document schemas, building aggregation queries, handling unstructured or semi-structured data, or migrating from relational to document-based storage.
|
||||
---
|
||||
|
||||
# MongoDB
|
||||
|
||||
## Description
|
||||
|
||||
MongoDB patterns including document design, queries, and aggregation.
|
||||
|
||||
## When to Use
|
||||
|
||||
- MongoDB database operations
|
||||
- Document-based data modeling
|
||||
- Aggregation pipelines
|
||||
- Semi-structured or polymorphic data that varies per record
|
||||
- Rapid prototyping where schema flexibility accelerates iteration
|
||||
- Event logging, IoT telemetry, or content management systems
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- Relational-heavy data models with complex joins and foreign key constraints
|
||||
- SQL-only projects where the entire stack is built around relational databases
|
||||
- Simple key-value storage where Redis or a lightweight store is more appropriate
|
||||
- Financial systems requiring multi-table ACID transactions as the norm
|
||||
|
||||
---
|
||||
|
||||
## Core Patterns
|
||||
|
||||
### Document Operations
|
||||
### 1. Schema Design
|
||||
|
||||
```javascript
|
||||
// Insert
|
||||
db.users.insertOne({
|
||||
email: 'user@example.com',
|
||||
name: 'John',
|
||||
createdAt: new Date()
|
||||
});
|
||||
The central decision in MongoDB modeling is **embed vs. reference**.
|
||||
|
||||
// Find
|
||||
db.users.find({ active: true }).sort({ createdAt: -1 }).limit(20);
|
||||
**Decision tree:**
|
||||
|
||||
// Update
|
||||
db.users.updateOne(
|
||||
{ _id: ObjectId('...') },
|
||||
{ $set: { name: 'Jane' } }
|
||||
);
|
||||
```
|
||||
Does the child data belong to exactly one parent?
|
||||
YES --> Is the child array unbounded (could grow to thousands)?
|
||||
YES --> Reference (separate collection)
|
||||
NO --> Embed
|
||||
NO --> Is it a many-to-many relationship?
|
||||
YES --> Reference (with array of ObjectIds on one or both sides)
|
||||
NO --> Reference
|
||||
```
|
||||
|
||||
### Aggregation
|
||||
**Embedding pattern -- best for data that is read together:**
|
||||
|
||||
```javascript
|
||||
// User with embedded address and preferences
|
||||
// Good: one read fetches everything the profile page needs
|
||||
db.users.insertOne({
|
||||
email: "user@example.com",
|
||||
name: "Alice Chen",
|
||||
address: {
|
||||
street: "123 Main St",
|
||||
city: "Portland",
|
||||
state: "OR",
|
||||
zip: "97201"
|
||||
},
|
||||
preferences: {
|
||||
theme: "dark",
|
||||
language: "en",
|
||||
notifications: { email: true, push: false }
|
||||
},
|
||||
createdAt: new Date()
|
||||
});
|
||||
```
|
||||
|
||||
**Referencing pattern -- best for independent or unbounded data:**
|
||||
|
||||
```javascript
|
||||
// Orders reference the user by ID
|
||||
// Good: orders grow unboundedly, accessed independently
|
||||
db.orders.insertOne({
|
||||
userId: ObjectId("6651a..."),
|
||||
status: "shipped",
|
||||
totalCents: 4999,
|
||||
items: [
|
||||
{ sku: "WIDGET-001", name: "Blue Widget", qty: 2, priceCents: 1999 },
|
||||
{ sku: "GADGET-010", name: "Mini Gadget", qty: 1, priceCents: 1001 }
|
||||
],
|
||||
placedAt: new Date()
|
||||
});
|
||||
```
|
||||
|
||||
**Denormalization pattern -- duplicate data to avoid frequent lookups:**
|
||||
|
||||
```javascript
|
||||
// Store author name directly on the post (denormalized from users)
|
||||
// Trade-off: faster reads, but updates to user name require updating all posts
|
||||
db.posts.insertOne({
|
||||
title: "Getting Started with MongoDB",
|
||||
body: "...",
|
||||
author: {
|
||||
_id: ObjectId("6651a..."),
|
||||
name: "Alice Chen" // denormalized -- must be updated if name changes
|
||||
},
|
||||
tags: ["mongodb", "tutorial"],
|
||||
publishedAt: new Date()
|
||||
});
|
||||
```
|
||||
|
||||
**Polymorphic pattern -- different shapes in one collection:**
|
||||
|
||||
```javascript
|
||||
// Events collection stores different event types
|
||||
db.events.insertMany([
|
||||
{
|
||||
type: "page_view",
|
||||
userId: ObjectId("6651a..."),
|
||||
url: "/products/widget",
|
||||
timestamp: new Date()
|
||||
},
|
||||
{
|
||||
type: "purchase",
|
||||
userId: ObjectId("6651a..."),
|
||||
orderId: ObjectId("6651b..."),
|
||||
totalCents: 4999,
|
||||
timestamp: new Date()
|
||||
}
|
||||
]);
|
||||
// Use a discriminator field (type) and query by it
|
||||
```
|
||||
|
||||
**Schema validation -- enforce structure at the database level:**
|
||||
|
||||
```javascript
|
||||
db.createCollection("users", {
|
||||
validator: {
|
||||
$jsonSchema: {
|
||||
bsonType: "object",
|
||||
required: ["email", "name", "createdAt"],
|
||||
properties: {
|
||||
email: {
|
||||
bsonType: "string",
|
||||
pattern: "^.+@.+\\..+$",
|
||||
description: "Must be a valid email"
|
||||
},
|
||||
name: {
|
||||
bsonType: "string",
|
||||
minLength: 1
|
||||
},
|
||||
role: {
|
||||
enum: ["admin", "editor", "viewer"],
|
||||
description: "Must be a valid role"
|
||||
},
|
||||
createdAt: { bsonType: "date" }
|
||||
}
|
||||
}
|
||||
},
|
||||
validationLevel: "strict",
|
||||
validationAction: "error"
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Aggregation Pipeline
|
||||
|
||||
Build complex data transformations as a sequence of stages.
|
||||
|
||||
```javascript
|
||||
// Revenue report: total and average spend per user, last 30 days
|
||||
db.orders.aggregate([
|
||||
{ $match: { status: 'completed' } },
|
||||
{ $group: {
|
||||
_id: '$userId',
|
||||
totalSpent: { $sum: '$amount' },
|
||||
orderCount: { $count: {} }
|
||||
// Stage 1: filter to recent delivered orders
|
||||
{ $match: {
|
||||
status: "delivered",
|
||||
placedAt: { $gte: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000) }
|
||||
}},
|
||||
{ $sort: { totalSpent: -1 } }
|
||||
|
||||
// Stage 2: group by user
|
||||
{ $group: {
|
||||
_id: "$userId",
|
||||
totalSpent: { $sum: "$totalCents" },
|
||||
orderCount: { $sum: 1 },
|
||||
avgOrderValue: { $avg: "$totalCents" }
|
||||
}},
|
||||
|
||||
// Stage 3: sort by spend
|
||||
{ $sort: { totalSpent: -1 } },
|
||||
|
||||
// Stage 4: limit to top 10
|
||||
{ $limit: 10 },
|
||||
|
||||
// Stage 5: join user details
|
||||
{ $lookup: {
|
||||
from: "users",
|
||||
localField: "_id",
|
||||
foreignField: "_id",
|
||||
as: "user"
|
||||
}},
|
||||
|
||||
// Stage 6: flatten the joined array
|
||||
{ $unwind: "$user" },
|
||||
|
||||
// Stage 7: reshape output
|
||||
{ $project: {
|
||||
_id: 0,
|
||||
userName: "$user.name",
|
||||
email: "$user.email",
|
||||
totalSpent: 1,
|
||||
orderCount: 1,
|
||||
avgOrderValue: { $round: ["$avgOrderValue", 0] }
|
||||
}}
|
||||
]);
|
||||
```
|
||||
|
||||
### Indexes
|
||||
**$unwind -- flatten arrays into individual documents:**
|
||||
|
||||
```javascript
|
||||
// Single field
|
||||
// Expand order items to analyze product-level metrics
|
||||
db.orders.aggregate([
|
||||
{ $unwind: "$items" },
|
||||
{ $group: {
|
||||
_id: "$items.sku",
|
||||
totalQty: { $sum: "$items.qty" },
|
||||
totalRevenue: { $sum: { $multiply: ["$items.qty", "$items.priceCents"] } }
|
||||
}},
|
||||
{ $sort: { totalRevenue: -1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
**$lookup with pipeline -- filtered/correlated joins:**
|
||||
|
||||
```javascript
|
||||
// For each user, get their 3 most recent orders
|
||||
db.users.aggregate([
|
||||
{ $lookup: {
|
||||
from: "orders",
|
||||
let: { uid: "$_id" },
|
||||
pipeline: [
|
||||
{ $match: { $expr: { $eq: ["$userId", "$$uid"] } } },
|
||||
{ $sort: { placedAt: -1 } },
|
||||
{ $limit: 3 },
|
||||
{ $project: { status: 1, totalCents: 1, placedAt: 1 } }
|
||||
],
|
||||
as: "recentOrders"
|
||||
}}
|
||||
]);
|
||||
```
|
||||
|
||||
**$facet -- run multiple aggregations in parallel:**
|
||||
|
||||
```javascript
|
||||
// Dashboard: get summary stats and top products in one query
|
||||
db.orders.aggregate([
|
||||
{ $match: { status: "delivered" } },
|
||||
{ $facet: {
|
||||
summary: [
|
||||
{ $group: {
|
||||
_id: null,
|
||||
totalRevenue: { $sum: "$totalCents" },
|
||||
totalOrders: { $sum: 1 }
|
||||
}}
|
||||
],
|
||||
topProducts: [
|
||||
{ $unwind: "$items" },
|
||||
{ $group: { _id: "$items.sku", sold: { $sum: "$items.qty" } } },
|
||||
{ $sort: { sold: -1 } },
|
||||
{ $limit: 5 }
|
||||
],
|
||||
monthlyTrend: [
|
||||
{ $group: {
|
||||
_id: { $dateToString: { format: "%Y-%m", date: "$placedAt" } },
|
||||
revenue: { $sum: "$totalCents" }
|
||||
}},
|
||||
{ $sort: { _id: 1 } }
|
||||
]
|
||||
}}
|
||||
]);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Index Strategies
|
||||
|
||||
```javascript
|
||||
// Single field index -- most common
|
||||
db.users.createIndex({ email: 1 }, { unique: true });
|
||||
|
||||
// Compound
|
||||
db.posts.createIndex({ userId: 1, createdAt: -1 });
|
||||
// Compound index -- order matters, follows the ESR rule:
|
||||
// Equality fields first, Sort fields next, Range fields last
|
||||
db.orders.createIndex({ status: 1, placedAt: -1 });
|
||||
// Supports: find({status: "pending"}).sort({placedAt: -1})
|
||||
// Also supports: find({status: "pending"}) alone (prefix)
|
||||
|
||||
// Multikey index -- automatically indexes each array element
|
||||
db.posts.createIndex({ tags: 1 });
|
||||
// Supports: find({ tags: "mongodb" })
|
||||
|
||||
// Text index -- basic full-text search
|
||||
db.posts.createIndex(
|
||||
{ title: "text", body: "text" },
|
||||
{ weights: { title: 10, body: 1 }, name: "posts_text_search" }
|
||||
);
|
||||
// Usage:
|
||||
db.posts.find(
|
||||
{ $text: { $search: "mongodb aggregation" } },
|
||||
{ score: { $meta: "textScore" } }
|
||||
).sort({ score: { $meta: "textScore" } });
|
||||
|
||||
// TTL index -- auto-delete documents after expiry
|
||||
db.sessions.createIndex(
|
||||
{ expiresAt: 1 },
|
||||
{ expireAfterSeconds: 0 } // delete when expiresAt is in the past
|
||||
);
|
||||
// Documents must have a Date field; they are removed by a background task ~every 60s
|
||||
|
||||
// Partial index -- only index documents matching a filter
|
||||
db.orders.createIndex(
|
||||
{ placedAt: -1 },
|
||||
{ partialFilterExpression: { status: "pending" } }
|
||||
);
|
||||
// Smaller index; only used when the query includes the filter condition
|
||||
|
||||
// Wildcard index -- for querying arbitrary keys in a sub-document
|
||||
db.products.createIndex({ "attributes.$**": 1 });
|
||||
// Supports: find({ "attributes.color": "red" }) without knowing keys in advance
|
||||
|
||||
// Collation -- case-insensitive sorting and matching
|
||||
db.users.createIndex(
|
||||
{ name: 1 },
|
||||
{ collation: { locale: "en", strength: 2 } }
|
||||
);
|
||||
```
|
||||
|
||||
**The ESR rule for compound indexes:** order fields by **E**quality, **S**ort, **R**ange. This produces the most efficient index scans.
|
||||
|
||||
```javascript
|
||||
// Query: find active orders for a user, sorted by date, in a date range
|
||||
// Equality: userId, status
|
||||
// Sort: placedAt
|
||||
// Range: placedAt (but sort and range on same field -- sort wins)
|
||||
db.orders.createIndex({ userId: 1, status: 1, placedAt: -1 });
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Transactions
|
||||
|
||||
Multi-document transactions work across collections (requires replica set or sharded cluster).
|
||||
|
||||
```javascript
|
||||
const session = client.startSession();
|
||||
|
||||
try {
|
||||
session.startTransaction({
|
||||
readConcern: { level: "snapshot" },
|
||||
writeConcern: { w: "majority" },
|
||||
readPreference: "primary"
|
||||
});
|
||||
|
||||
const accounts = client.db("bank").collection("accounts");
|
||||
|
||||
// Transfer $50 from account A to account B
|
||||
const fromAccount = await accounts.findOne(
|
||||
{ _id: "account-A" },
|
||||
{ session }
|
||||
);
|
||||
|
||||
if (fromAccount.balanceCents < 5000) {
|
||||
await session.abortTransaction();
|
||||
throw new Error("Insufficient funds");
|
||||
}
|
||||
|
||||
await accounts.updateOne(
|
||||
{ _id: "account-A" },
|
||||
{ $inc: { balanceCents: -5000 } },
|
||||
{ session }
|
||||
);
|
||||
|
||||
await accounts.updateOne(
|
||||
{ _id: "account-B" },
|
||||
{ $inc: { balanceCents: 5000 } },
|
||||
{ session }
|
||||
);
|
||||
|
||||
// Record the transfer in a separate collection -- still in the same tx
|
||||
await client.db("bank").collection("transfers").insertOne({
|
||||
from: "account-A",
|
||||
to: "account-B",
|
||||
amountCents: 5000,
|
||||
timestamp: new Date()
|
||||
}, { session });
|
||||
|
||||
await session.commitTransaction();
|
||||
} catch (error) {
|
||||
await session.abortTransaction();
|
||||
throw error;
|
||||
} finally {
|
||||
await session.endSession();
|
||||
}
|
||||
```
|
||||
|
||||
**Guidelines:**
|
||||
- Keep transactions short -- they hold locks and consume resources
|
||||
- Design your schema to minimize the need for multi-document transactions
|
||||
- Transactions have a default 60-second timeout (`maxTimeMS`)
|
||||
- Retryable writes (`retryWrites=true` in connection string) handle transient errors automatically
|
||||
|
||||
---
|
||||
|
||||
### 5. Change Streams
|
||||
|
||||
Watch for real-time changes to collections, databases, or the entire deployment.
|
||||
|
||||
```javascript
|
||||
// Watch a single collection for inserts and updates
|
||||
const pipeline = [
|
||||
{ $match: {
|
||||
operationType: { $in: ["insert", "update"] },
|
||||
"fullDocument.status": "urgent"
|
||||
}}
|
||||
];
|
||||
|
||||
const changeStream = db.collection("tickets").watch(pipeline, {
|
||||
fullDocument: "updateLookup" // include the full document on updates
|
||||
});
|
||||
|
||||
changeStream.on("change", (change) => {
|
||||
console.log("Change detected:", change.operationType);
|
||||
console.log("Document:", change.fullDocument);
|
||||
console.log("Resume token:", change.resumeToken);
|
||||
|
||||
// Process the change (e.g., send notification, update cache)
|
||||
notifyTeam(change.fullDocument);
|
||||
});
|
||||
|
||||
// Handle errors and resume from last known position
|
||||
changeStream.on("error", (error) => {
|
||||
console.error("Change stream error:", error);
|
||||
// Reconnect using the stored resume token
|
||||
});
|
||||
```
|
||||
|
||||
**Resumable pattern for production:**
|
||||
|
||||
```javascript
|
||||
let resumeToken = await loadResumeTokenFromStorage();
|
||||
|
||||
async function watchWithResume(collection) {
|
||||
const options = { fullDocument: "updateLookup" };
|
||||
if (resumeToken) {
|
||||
options.resumeAfter = resumeToken;
|
||||
}
|
||||
|
||||
const stream = collection.watch([], options);
|
||||
|
||||
stream.on("change", async (change) => {
|
||||
// Process change
|
||||
await handleChange(change);
|
||||
|
||||
// Persist resume token so we can recover after restart
|
||||
resumeToken = change._id;
|
||||
await saveResumeTokenToStorage(resumeToken);
|
||||
});
|
||||
|
||||
stream.on("error", async () => {
|
||||
// Wait and reconnect
|
||||
await new Promise(r => setTimeout(r, 5000));
|
||||
watchWithResume(collection);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
**Use cases:** real-time dashboards, cache invalidation, event-driven architectures, syncing data to search indexes (e.g., Elasticsearch).
|
||||
|
||||
---
|
||||
|
||||
### 6. Performance
|
||||
|
||||
#### Reading explain() output
|
||||
|
||||
```javascript
|
||||
// Run explain to see the query plan
|
||||
db.orders.find({
|
||||
userId: ObjectId("6651a..."),
|
||||
status: "pending"
|
||||
}).sort({ placedAt: -1 }).explain("executionStats");
|
||||
```
|
||||
|
||||
**Key fields in executionStats:**
|
||||
|
||||
| Field | What to look for |
|
||||
|-------|-----------------|
|
||||
| `winningPlan.stage` | `IXSCAN` good, `COLLSCAN` bad (full collection scan) |
|
||||
| `totalKeysExamined` | Should be close to `nReturned` (no wasted index scans) |
|
||||
| `totalDocsExamined` | Should be close to `nReturned` (no wasted document reads) |
|
||||
| `executionTimeMillis` | Overall query time |
|
||||
| `rejectedPlans` | Shows alternatives the optimizer considered |
|
||||
|
||||
**Covered queries -- answered entirely from the index:**
|
||||
|
||||
```javascript
|
||||
// Create an index that covers the query
|
||||
db.orders.createIndex({ userId: 1, status: 1, totalCents: 1 });
|
||||
|
||||
// This query only needs fields in the index -- no document fetch
|
||||
db.orders.find(
|
||||
{ userId: ObjectId("6651a..."), status: "delivered" },
|
||||
{ _id: 0, totalCents: 1 } // projection must exclude _id and only include indexed fields
|
||||
);
|
||||
// explain() will show: "totalDocsExamined": 0
|
||||
```
|
||||
|
||||
**Projection optimization -- fetch only what you need:**
|
||||
|
||||
```javascript
|
||||
// BAD: fetches entire document including large body field
|
||||
const posts = await db.posts.find({ author: userId }).toArray();
|
||||
|
||||
// GOOD: only fetch fields needed for the list view
|
||||
const posts = await db.posts.find(
|
||||
{ author: userId },
|
||||
{ projection: { title: 1, publishedAt: 1, tags: 1 } }
|
||||
).toArray();
|
||||
```
|
||||
|
||||
**Bulk operations for write-heavy workloads:**
|
||||
|
||||
```javascript
|
||||
const bulk = db.products.initializeUnorderedBulkOp();
|
||||
|
||||
for (const update of priceUpdates) {
|
||||
bulk.find({ sku: update.sku })
|
||||
.updateOne({ $set: { priceCents: update.newPrice, updatedAt: new Date() } });
|
||||
}
|
||||
|
||||
const result = await bulk.execute();
|
||||
console.log(`Modified: ${result.nModified}, Errors: ${result.getWriteErrorCount()}`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. Embed frequently accessed data
|
||||
2. Use references for large/independent data
|
||||
3. Create indexes for query patterns
|
||||
4. Use aggregation for complex queries
|
||||
5. Avoid unbounded arrays
|
||||
1. **Design schema around query patterns, not data relationships.** Ask "how will I read this data?" before "how does this data relate?" Embed data that is always fetched together; reference data accessed independently.
|
||||
|
||||
2. **Use the ESR rule for compound indexes.** Order index fields by Equality, Sort, Range. This maximizes the index's usefulness and minimizes keys examined.
|
||||
|
||||
3. **Set read/write concerns appropriately.** Use `w: "majority"` and `readConcern: "majority"` for data that must survive failovers. Use `w: 1` for non-critical writes where speed matters more than durability.
|
||||
|
||||
4. **Use projection to limit returned fields.** Transferring large documents over the network when you only need two fields wastes bandwidth and memory. Always project.
|
||||
|
||||
5. **Avoid unbounded array growth.** An embedded array that can grow to thousands of elements bloats the document (16 MB max) and degrades performance. Move to a separate collection with a reference when the array exceeds ~100 elements.
|
||||
|
||||
6. **Use bulk operations for batch writes.** Individual `insertOne` or `updateOne` calls in a loop are slow. Batch them with `bulkWrite` or `initializeUnorderedBulkOp` for 10-50x throughput improvement.
|
||||
|
||||
7. **Enable retryable writes.** Add `retryWrites=true` to your connection string. This handles transient network errors and primary elections automatically without application-level retry logic.
|
||||
|
||||
8. **Monitor with database profiler and serverStatus.** Use `db.setProfilingLevel(1, { slowms: 100 })` to log slow queries. Check `db.serverStatus().opcounters` and `db.serverStatus().connections` for overall health.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Unbounded arrays**: Limit array size
|
||||
- **Missing indexes**: Analyze query patterns
|
||||
- **Over-embedding**: Consider data access patterns
|
||||
1. **Treating MongoDB like a relational database.** Normalizing everything into separate collections and using `$lookup` for every query defeats the purpose. If you need heavy joins, PostgreSQL is likely a better fit. Design for embedding first.
|
||||
|
||||
2. **Missing indexes on query fields.** Every `find()`, `$match`, and `sort()` should be backed by an index. Use `db.collection.getIndexes()` and `explain()` to verify. A `COLLSCAN` on a large collection is almost always a bug.
|
||||
|
||||
3. **Ignoring the 16 MB document size limit.** Embedding unbounded arrays (comments, logs, events) will eventually hit this wall, crashing writes. Use the bucket pattern (fixed-size sub-documents) or reference a separate collection.
|
||||
|
||||
4. **Not using readPreference for read-heavy workloads.** By default all reads go to the primary. For analytics or non-critical reads, use `readPreference: "secondaryPreferred"` to distribute load across replicas.
|
||||
|
||||
5. **Forgetting that updates replace matched array elements, not all of them.** Using `$set` on a matched array element with positional `$` only updates the first match. Use `$[]` for all elements or `$[<identifier>]` with `arrayFilters` for conditional updates:
|
||||
|
||||
```javascript
|
||||
// Update price for a specific item in all orders
|
||||
db.orders.updateMany(
|
||||
{ "items.sku": "WIDGET-001" },
|
||||
{ $set: { "items.$[item].priceCents": 2499 } },
|
||||
{ arrayFilters: [{ "item.sku": "WIDGET-001" }] }
|
||||
);
|
||||
```
|
||||
|
||||
6. **Running aggregation pipelines without early $match.** Always filter as early as possible in the pipeline. A `$group` or `$unwind` before `$match` processes the entire collection unnecessarily. Put `$match` first to leverage indexes and reduce documents flowing through subsequent stages.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `databases/postgresql` - Relational database patterns for structured data with complex relationships
|
||||
- `patterns/caching` - Caching strategies to reduce database load
|
||||
- `patterns/logging` - Logging patterns for query debugging and monitoring
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
# MongoDB Schema Design Patterns
|
||||
|
||||
Quick reference for embedding vs referencing decisions and common schema patterns.
|
||||
|
||||
## Embedding vs Referencing Decision Tree
|
||||
|
||||
```
|
||||
What is the relationship cardinality?
|
||||
|
|
||||
+-- One-to-Few (< 50 items)?
|
||||
| --> EMBED in parent document
|
||||
| Example: user.addresses, post.tags
|
||||
|
|
||||
+-- One-to-Many (50 - 1000s)?
|
||||
| |
|
||||
| +-- Child data always accessed with parent?
|
||||
| | --> EMBED (but watch 16 MB doc limit)
|
||||
| |
|
||||
| +-- Child data accessed independently?
|
||||
| | --> REFERENCE (store child _id in parent array)
|
||||
| |
|
||||
| +-- Need atomic updates across parent + children?
|
||||
| --> EMBED
|
||||
|
|
||||
+-- One-to-Millions?
|
||||
| --> REFERENCE from child to parent
|
||||
| Example: log_entry.host_id (not host.log_entry_ids)
|
||||
|
|
||||
+-- Many-to-Many?
|
||||
--> REFERENCE with array of _ids on one or both sides
|
||||
Example: student.course_ids[], course.student_ids[]
|
||||
```
|
||||
|
||||
## Decision Factors
|
||||
|
||||
| Factor | Favor Embedding | Favor Referencing |
|
||||
|--------|----------------|-------------------|
|
||||
| **Read pattern** | Always read together | Read independently |
|
||||
| **Write pattern** | Infrequent child updates | Frequent child updates |
|
||||
| **Data size** | Small, bounded children | Large or growing children |
|
||||
| **Atomicity** | Need single-doc transactions | Can tolerate multi-doc txn |
|
||||
| **Duplication** | OK to denormalize | Must avoid duplication |
|
||||
| **Cardinality** | Few items | Many/unbounded items |
|
||||
| **Document size** | Well under 16 MB limit | Approaching 16 MB |
|
||||
|
||||
## Pattern Catalog
|
||||
|
||||
### 1. Subset Pattern
|
||||
|
||||
**Problem**: Document is large but reads only need a few fields from embedded data.
|
||||
|
||||
**Solution**: Embed a subset; keep full data in a separate collection.
|
||||
|
||||
```javascript
|
||||
// products collection - fast reads for listing pages
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
name: "Widget",
|
||||
price: 29.99,
|
||||
// Only the 10 most recent reviews (subset)
|
||||
recent_reviews: [
|
||||
{ user: "alice", rating: 5, text: "Great!", date: ISODate("...") }
|
||||
],
|
||||
review_count: 247
|
||||
}
|
||||
|
||||
// reviews collection - full review data
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
product_id: ObjectId("..."),
|
||||
user: "alice",
|
||||
rating: 5,
|
||||
text: "Great!",
|
||||
date: ISODate("..."),
|
||||
helpful_votes: 12
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Product pages, user profiles, any "preview + detail" pattern.
|
||||
|
||||
### 2. Computed Pattern
|
||||
|
||||
**Problem**: Expensive aggregation queries run repeatedly on the same data.
|
||||
|
||||
**Solution**: Pre-compute and store the result, update on write.
|
||||
|
||||
```javascript
|
||||
// movies collection
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Example Movie",
|
||||
// Pre-computed from screenings collection
|
||||
computed: {
|
||||
total_revenue: 1250000,
|
||||
avg_rating: 4.2,
|
||||
rating_count: 843,
|
||||
last_computed: ISODate("2025-01-15T00:00:00Z")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Update strategy**: On each new rating, increment count and recalculate average. Or use a background job for less time-sensitive data.
|
||||
|
||||
**When to use**: Dashboards, leaderboards, summary statistics.
|
||||
|
||||
### 3. Bucket Pattern
|
||||
|
||||
**Problem**: Many small, time-series documents create overhead (indexes, storage per doc).
|
||||
|
||||
**Solution**: Group related data into fixed-size buckets.
|
||||
|
||||
```javascript
|
||||
// sensor_readings collection - one doc per sensor per hour
|
||||
{
|
||||
sensor_id: "sensor-42",
|
||||
bucket_start: ISODate("2025-01-15T14:00:00Z"),
|
||||
bucket_end: ISODate("2025-01-15T14:59:59Z"),
|
||||
count: 60,
|
||||
readings: [
|
||||
{ ts: ISODate("2025-01-15T14:00:00Z"), temp: 22.1, humidity: 45 },
|
||||
{ ts: ISODate("2025-01-15T14:01:00Z"), temp: 22.3, humidity: 44 }
|
||||
// ... up to 60 readings per bucket
|
||||
],
|
||||
// Pre-computed aggregates for the bucket
|
||||
summary: {
|
||||
avg_temp: 22.4,
|
||||
min_temp: 21.8,
|
||||
max_temp: 23.1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Bucket sizing**: Choose a size that balances doc count reduction vs update frequency. Common choices: 1 hour, 1 day, 100 events.
|
||||
|
||||
**When to use**: IoT, time-series, event logging, analytics.
|
||||
|
||||
### 4. Outlier Pattern
|
||||
|
||||
**Problem**: A few documents have vastly more data than the norm (e.g., a viral post with millions of likes).
|
||||
|
||||
**Solution**: Flag outliers and overflow into separate documents.
|
||||
|
||||
```javascript
|
||||
// books collection - normal case
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Normal Book",
|
||||
customers_purchased: ["user1", "user2", "user3"],
|
||||
has_overflow: false
|
||||
}
|
||||
|
||||
// books collection - outlier (bestseller)
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Bestseller",
|
||||
customers_purchased: ["user1", "user2", /* ... first 1000 */],
|
||||
has_overflow: true
|
||||
}
|
||||
|
||||
// book_purchases_overflow collection
|
||||
{
|
||||
book_id: ObjectId("..."),
|
||||
page: 2,
|
||||
customers_purchased: ["user1001", "user1002", /* ... next 1000 */]
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Social media (viral posts), e-commerce (bestsellers), any data with power-law distribution.
|
||||
|
||||
### 5. Extended Reference Pattern
|
||||
|
||||
**Problem**: Frequent joins (lookups) to get a few fields from a referenced document.
|
||||
|
||||
**Solution**: Copy the most-accessed fields into the referencing document.
|
||||
|
||||
```javascript
|
||||
// orders collection
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
date: ISODate("..."),
|
||||
customer_id: ObjectId("..."),
|
||||
// Extended reference - copied fields for fast reads
|
||||
customer_name: "Alice Smith",
|
||||
customer_email: "alice@example.com",
|
||||
items: [
|
||||
{
|
||||
product_id: ObjectId("..."),
|
||||
product_name: "Widget", // copied
|
||||
price: 29.99, // copied (snapshot at time of order)
|
||||
quantity: 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Trade-off**: Stale data is acceptable (order snapshots price at purchase time). For data that must be current, keep only the reference.
|
||||
|
||||
**When to use**: Orders (snapshot pricing), notifications (snapshot user name), audit logs.
|
||||
|
||||
### 6. Polymorphic Pattern
|
||||
|
||||
**Problem**: Objects share some fields but differ in others (e.g., different product types).
|
||||
|
||||
**Solution**: Store in a single collection with a type discriminator.
|
||||
|
||||
```javascript
|
||||
// vehicles collection
|
||||
{ type: "car", make: "Toyota", doors: 4, trunk_size_liters: 450 }
|
||||
{ type: "truck", make: "Ford", doors: 2, payload_kg: 5000 }
|
||||
{ type: "motorcycle", make: "Harley", engine_cc: 1200 }
|
||||
```
|
||||
|
||||
**Index strategy**: Index common fields. Use partial indexes for type-specific fields.
|
||||
|
||||
```javascript
|
||||
db.vehicles.createIndex(
|
||||
{ payload_kg: 1 },
|
||||
{ partialFilterExpression: { type: "truck" } }
|
||||
);
|
||||
```
|
||||
|
||||
**When to use**: Product catalogs, content management (articles, videos, images), mixed event streams.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Mistake | Problem | Fix |
|
||||
|---------|---------|-----|
|
||||
| Unbounded array growth | Document exceeds 16 MB | Use bucket or outlier pattern |
|
||||
| Deep nesting (> 3 levels) | Hard to query and index | Flatten or reference |
|
||||
| Normalizing everything | Too many lookups, slow reads | Embed when read together |
|
||||
| Embedding large blobs | Wastes RAM in working set | Store in GridFS or S3 |
|
||||
| No schema validation | Inconsistent data over time | Use JSON Schema validation |
|
||||
| Indexing every field | Slow writes, wasted space | Index based on query patterns |
|
||||
|
||||
## Schema Validation
|
||||
|
||||
Use `db.createCollection()` with `$jsonSchema` validator to enforce structure. Set `validationLevel: "moderate"` to apply only on inserts and updates (not existing docs).
|
||||
@@ -1,69 +1,612 @@
|
||||
---
|
||||
name: postgresql
|
||||
description: >
|
||||
Use this skill whenever working with PostgreSQL databases, writing SQL queries, designing schemas, or optimizing database performance. Trigger on keywords like PostgreSQL, Postgres, SQL query, schema design, indexing, migrations, EXPLAIN ANALYZE, connection pooling, or any relational database operation. Also applies when debugging slow queries, setting up database tables, or working with ORMs that target PostgreSQL.
|
||||
---
|
||||
|
||||
# PostgreSQL
|
||||
|
||||
## Description
|
||||
|
||||
PostgreSQL database patterns including queries, indexing, and optimization.
|
||||
|
||||
## When to Use
|
||||
|
||||
- PostgreSQL database operations
|
||||
- SQL query optimization
|
||||
- Schema design
|
||||
- Schema design and migrations
|
||||
- JSONB document storage within a relational model
|
||||
- Full-text search without a dedicated search engine
|
||||
- Complex analytical queries with window functions and CTEs
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- NoSQL-only projects where no relational database is involved
|
||||
- In-memory databases like Redis or SQLite used purely for caching or ephemeral storage
|
||||
- File-based storage scenarios that do not require a database engine
|
||||
|
||||
---
|
||||
|
||||
## Core Patterns
|
||||
|
||||
### Basic Queries
|
||||
### 1. Schema Design
|
||||
|
||||
Design tables with explicit constraints, proper types, and clear relationships.
|
||||
|
||||
```sql
|
||||
-- Select with filtering
|
||||
SELECT id, name, email
|
||||
FROM users
|
||||
WHERE active = true
|
||||
ORDER BY created_at DESC
|
||||
LIMIT 20 OFFSET 0;
|
||||
-- Enums for constrained value sets
|
||||
CREATE TYPE user_role AS ENUM ('admin', 'editor', 'viewer');
|
||||
CREATE TYPE order_status AS ENUM ('pending', 'processing', 'shipped', 'delivered', 'cancelled');
|
||||
|
||||
-- Join
|
||||
SELECT u.*, COUNT(p.id) as post_count
|
||||
-- Composite types for reusable structures
|
||||
CREATE TYPE address AS (
|
||||
street TEXT,
|
||||
city TEXT,
|
||||
state TEXT,
|
||||
zip VARCHAR(10)
|
||||
);
|
||||
|
||||
-- Users table with constraints
|
||||
CREATE TABLE users (
|
||||
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
email TEXT NOT NULL UNIQUE,
|
||||
name TEXT NOT NULL CHECK (char_length(name) >= 1),
|
||||
role user_role NOT NULL DEFAULT 'viewer',
|
||||
metadata JSONB DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Organizations with self-referencing hierarchy
|
||||
CREATE TABLE organizations (
|
||||
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
parent_id BIGINT REFERENCES organizations(id) ON DELETE SET NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Membership join table with composite primary key
|
||||
CREATE TABLE org_memberships (
|
||||
user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
org_id BIGINT NOT NULL REFERENCES organizations(id) ON DELETE CASCADE,
|
||||
role user_role NOT NULL DEFAULT 'viewer',
|
||||
joined_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (user_id, org_id)
|
||||
);
|
||||
|
||||
-- Orders with foreign keys, check constraints, and enum status
|
||||
CREATE TABLE orders (
|
||||
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||||
status order_status NOT NULL DEFAULT 'pending',
|
||||
total_cents BIGINT NOT NULL CHECK (total_cents >= 0),
|
||||
shipping address,
|
||||
items JSONB NOT NULL DEFAULT '[]',
|
||||
placed_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Auto-update updated_at with a trigger
|
||||
CREATE OR REPLACE FUNCTION set_updated_at()
|
||||
RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
NEW.updated_at = now();
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
CREATE TRIGGER trg_users_updated_at
|
||||
BEFORE UPDATE ON users
|
||||
FOR EACH ROW EXECUTE FUNCTION set_updated_at();
|
||||
```
|
||||
|
||||
**Key principles:**
|
||||
- Use `BIGINT GENERATED ALWAYS AS IDENTITY` over `SERIAL` for new projects
|
||||
- Use `TIMESTAMPTZ` (not `TIMESTAMP`) to store times with timezone awareness
|
||||
- Prefer `TEXT` over `VARCHAR(n)` unless a hard length limit is business-critical
|
||||
- Add `ON DELETE` actions on every foreign key (CASCADE, RESTRICT, or SET NULL)
|
||||
- Use `CHECK` constraints for business rules that live at the data level
|
||||
|
||||
---
|
||||
|
||||
### 2. Index Strategy
|
||||
|
||||
Choose the right index type based on your query patterns.
|
||||
|
||||
**Decision guide:**
|
||||
|
||||
| Query Pattern | Index Type | Example |
|
||||
|---------------|-----------|---------|
|
||||
| Equality (`=`) and range (`<`, `>`, `BETWEEN`) | B-tree (default) | `WHERE created_at > '2025-01-01'` |
|
||||
| Array containment (`@>`), JSONB queries | GIN | `WHERE tags @> '{postgres}'` |
|
||||
| Full-text search (`@@`) | GIN | `WHERE to_tsvector(body) @@ query` |
|
||||
| Geometry, range overlap | GiST | `WHERE location <-> point '(40.7,-74.0)' < 0.01` |
|
||||
| Filtered subset of rows | Partial | `WHERE active = true` |
|
||||
| Index-only scans (no heap lookup) | Covering (INCLUDE) | Frequently selected columns |
|
||||
|
||||
```sql
|
||||
-- B-tree: default, good for equality and range
|
||||
CREATE INDEX idx_orders_placed_at ON orders(placed_at DESC);
|
||||
CREATE INDEX idx_orders_user_status ON orders(user_id, status);
|
||||
|
||||
-- GIN: arrays and JSONB containment
|
||||
CREATE INDEX idx_users_metadata ON users USING GIN (metadata);
|
||||
CREATE INDEX idx_orders_items ON orders USING GIN (items jsonb_path_ops);
|
||||
|
||||
-- GIN: full-text search
|
||||
ALTER TABLE articles ADD COLUMN search_vector tsvector
|
||||
GENERATED ALWAYS AS (
|
||||
setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
|
||||
setweight(to_tsvector('english', coalesce(body, '')), 'B')
|
||||
) STORED;
|
||||
|
||||
CREATE INDEX idx_articles_search ON articles USING GIN (search_vector);
|
||||
|
||||
-- Full-text search query
|
||||
SELECT id, title, ts_rank(search_vector, query) AS rank
|
||||
FROM articles, plainto_tsquery('english', 'database optimization') AS query
|
||||
WHERE search_vector @@ query
|
||||
ORDER BY rank DESC
|
||||
LIMIT 20;
|
||||
|
||||
-- GiST: geometry and range types
|
||||
CREATE INDEX idx_events_duration ON events USING GiST (
|
||||
tstzrange(starts_at, ends_at)
|
||||
);
|
||||
|
||||
-- Find overlapping events
|
||||
SELECT * FROM events
|
||||
WHERE tstzrange(starts_at, ends_at) && tstzrange('2025-06-01', '2025-06-02');
|
||||
|
||||
-- Partial index: only index rows you actually query
|
||||
CREATE INDEX idx_orders_pending ON orders(placed_at)
|
||||
WHERE status = 'pending';
|
||||
|
||||
-- Covering index: avoids heap lookup for common queries
|
||||
CREATE INDEX idx_users_email_covering ON users(email)
|
||||
INCLUDE (name, role);
|
||||
|
||||
-- This query can now be answered entirely from the index
|
||||
SELECT name, role FROM users WHERE email = 'user@example.com';
|
||||
```
|
||||
|
||||
**When to add an index:** Run `EXPLAIN ANALYZE` first. Add an index when you see sequential scans on large tables with selective WHERE clauses. Do not index columns with very low cardinality (e.g., a boolean on a small table) unless combined with other columns.
|
||||
|
||||
---
|
||||
|
||||
### 3. Query Optimization
|
||||
|
||||
#### Reading EXPLAIN ANALYZE
|
||||
|
||||
```sql
|
||||
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
|
||||
SELECT u.name, COUNT(o.id) AS order_count
|
||||
FROM users u
|
||||
LEFT JOIN posts p ON p.user_id = u.id
|
||||
GROUP BY u.id;
|
||||
JOIN orders o ON o.user_id = u.id
|
||||
WHERE o.placed_at > now() - INTERVAL '30 days'
|
||||
GROUP BY u.id, u.name
|
||||
ORDER BY order_count DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
### Indexes
|
||||
**What to look for in the output:**
|
||||
- **Seq Scan on large tables** -- add an index or rewrite the WHERE clause
|
||||
- **Nested Loop with high row counts** -- consider a Hash Join (may need more `work_mem`)
|
||||
- **actual rows far exceeding estimated rows** -- run `ANALYZE tablename` to update statistics
|
||||
- **Buffers: shared read** large numbers -- data not cached, check `shared_buffers` sizing
|
||||
- **Sort Method: external merge** -- increase `work_mem` for this query
|
||||
|
||||
#### Common Query Rewrites
|
||||
|
||||
```sql
|
||||
-- Single column index
|
||||
CREATE INDEX idx_users_email ON users(email);
|
||||
-- BAD: correlated subquery runs once per row
|
||||
SELECT u.name,
|
||||
(SELECT COUNT(*) FROM orders o WHERE o.user_id = u.id) AS order_count
|
||||
FROM users u;
|
||||
|
||||
-- Composite index
|
||||
CREATE INDEX idx_posts_user_date ON posts(user_id, created_at DESC);
|
||||
-- GOOD: single pass with JOIN + GROUP BY
|
||||
SELECT u.name, COUNT(o.id) AS order_count
|
||||
FROM users u
|
||||
LEFT JOIN orders o ON o.user_id = u.id
|
||||
GROUP BY u.id, u.name;
|
||||
|
||||
-- Partial index
|
||||
CREATE INDEX idx_active_users ON users(email) WHERE active = true;
|
||||
-- BAD: OR on different columns defeats index usage
|
||||
SELECT * FROM orders WHERE user_id = 5 OR status = 'pending';
|
||||
|
||||
-- GOOD: UNION ALL lets each branch use its own index
|
||||
SELECT * FROM orders WHERE user_id = 5
|
||||
UNION ALL
|
||||
SELECT * FROM orders WHERE status = 'pending' AND user_id != 5;
|
||||
|
||||
-- BAD: function call on indexed column prevents index use
|
||||
SELECT * FROM users WHERE LOWER(email) = 'user@example.com';
|
||||
|
||||
-- GOOD: expression index or use citext
|
||||
CREATE INDEX idx_users_email_lower ON users(LOWER(email));
|
||||
-- or better: define email as CITEXT type
|
||||
|
||||
-- Avoiding N+1: fetch users and their latest order in one query
|
||||
SELECT DISTINCT ON (u.id)
|
||||
u.id, u.name, o.id AS latest_order_id, o.total_cents, o.placed_at
|
||||
FROM users u
|
||||
LEFT JOIN orders o ON o.user_id = u.id
|
||||
ORDER BY u.id, o.placed_at DESC;
|
||||
```
|
||||
|
||||
### Migrations
|
||||
---
|
||||
|
||||
### 4. Migrations
|
||||
|
||||
Follow the up/down pattern and plan for zero-downtime deployments.
|
||||
|
||||
```sql
|
||||
-- Add column with default
|
||||
ALTER TABLE users ADD COLUMN role VARCHAR(50) DEFAULT 'user';
|
||||
-- ============================================
|
||||
-- Migration: 20250601_001_add_user_preferences
|
||||
-- ============================================
|
||||
|
||||
-- Add constraint
|
||||
ALTER TABLE users ADD CONSTRAINT unique_email UNIQUE (email);
|
||||
-- UP
|
||||
ALTER TABLE users ADD COLUMN preferences JSONB DEFAULT '{}';
|
||||
|
||||
-- Create index CONCURRENTLY to avoid locking the table
|
||||
CREATE INDEX CONCURRENTLY idx_users_preferences
|
||||
ON users USING GIN (preferences);
|
||||
|
||||
-- DOWN
|
||||
DROP INDEX IF EXISTS idx_users_preferences;
|
||||
ALTER TABLE users DROP COLUMN IF EXISTS preferences;
|
||||
```
|
||||
|
||||
**Safe vs unsafe operations:**
|
||||
|
||||
| Operation | Safe? | Notes |
|
||||
|-----------|-------|-------|
|
||||
| ADD COLUMN (nullable or with volatile default) | Yes | Instant in PG 11+ with non-volatile default too |
|
||||
| ADD COLUMN NOT NULL without default | No | Fails if rows exist; add nullable first, backfill, then set NOT NULL |
|
||||
| DROP COLUMN | Mostly | Quick, but ORM queries may break if they SELECT * |
|
||||
| RENAME COLUMN | Dangerous | Breaks all queries referencing old name; use a transition period |
|
||||
| ADD INDEX | Safe with CONCURRENTLY | Without CONCURRENTLY, locks writes for duration |
|
||||
| ADD CONSTRAINT (CHECK/FK) | Careful | Use NOT VALID then VALIDATE CONSTRAINT in two steps |
|
||||
| Change column type | Dangerous | Rewrites entire table; use a new column + migration instead |
|
||||
|
||||
```sql
|
||||
-- Zero-downtime: add NOT NULL constraint safely
|
||||
-- Step 1: add column as nullable
|
||||
ALTER TABLE users ADD COLUMN phone TEXT;
|
||||
|
||||
-- Step 2: backfill in batches
|
||||
UPDATE users SET phone = '' WHERE phone IS NULL AND id BETWEEN 1 AND 10000;
|
||||
UPDATE users SET phone = '' WHERE phone IS NULL AND id BETWEEN 10001 AND 20000;
|
||||
-- ... continue in batches
|
||||
|
||||
-- Step 3: add constraint without full table lock
|
||||
ALTER TABLE users ADD CONSTRAINT users_phone_not_null
|
||||
CHECK (phone IS NOT NULL) NOT VALID;
|
||||
|
||||
-- Step 4: validate (scans table but allows concurrent writes)
|
||||
ALTER TABLE users VALIDATE CONSTRAINT users_phone_not_null;
|
||||
|
||||
-- Step 5: optionally convert to proper NOT NULL
|
||||
ALTER TABLE users ALTER COLUMN phone SET NOT NULL;
|
||||
ALTER TABLE users DROP CONSTRAINT users_phone_not_null;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. JSON/JSONB
|
||||
|
||||
Use JSONB for semi-structured data that lives alongside relational columns.
|
||||
|
||||
**When to use JSONB:**
|
||||
- User preferences, settings, or metadata with varying keys
|
||||
- API response caching or event payloads
|
||||
- Flexible attributes that differ per row
|
||||
|
||||
**When NOT to use JSONB:**
|
||||
- Data you regularly JOIN on or use in WHERE clauses across tables -- normalize it
|
||||
- Data that has a fixed, well-known schema -- use proper columns
|
||||
|
||||
```sql
|
||||
-- Querying JSONB: operators
|
||||
-- -> returns JSONB element (keeps type)
|
||||
-- ->> returns TEXT value
|
||||
-- @> containment (left contains right)
|
||||
-- ? key exists
|
||||
|
||||
-- Get a nested value
|
||||
SELECT
|
||||
metadata->>'department' AS department,
|
||||
metadata->'settings'->>'theme' AS theme
|
||||
FROM users
|
||||
WHERE metadata @> '{"role": "admin"}';
|
||||
|
||||
-- Check if a key exists
|
||||
SELECT * FROM users WHERE metadata ? 'avatar_url';
|
||||
|
||||
-- Query inside JSONB arrays
|
||||
SELECT * FROM orders
|
||||
WHERE items @> '[{"sku": "WIDGET-001"}]';
|
||||
|
||||
-- Update a nested JSONB field
|
||||
UPDATE users
|
||||
SET metadata = jsonb_set(metadata, '{settings,notifications}', '"email"')
|
||||
WHERE id = 42;
|
||||
|
||||
-- Remove a key
|
||||
UPDATE users
|
||||
SET metadata = metadata - 'deprecated_field'
|
||||
WHERE metadata ? 'deprecated_field';
|
||||
|
||||
-- Aggregate JSONB: expand array elements into rows
|
||||
SELECT o.id, item->>'sku' AS sku, (item->>'qty')::int AS qty
|
||||
FROM orders o, jsonb_array_elements(o.items) AS item
|
||||
WHERE o.status = 'pending';
|
||||
|
||||
-- Index strategies for JSONB
|
||||
-- General containment queries: GIN with jsonb_ops (default)
|
||||
CREATE INDEX idx_users_metadata_gin ON users USING GIN (metadata);
|
||||
|
||||
-- Containment-only queries (smaller, faster index): jsonb_path_ops
|
||||
CREATE INDEX idx_orders_items_path ON orders USING GIN (items jsonb_path_ops);
|
||||
|
||||
-- Specific key lookups: expression index on extracted value
|
||||
CREATE INDEX idx_users_department ON users ((metadata->>'department'));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. CTEs and Window Functions
|
||||
|
||||
#### Common Table Expressions (CTEs)
|
||||
|
||||
```sql
|
||||
-- Readable multi-step query with CTEs
|
||||
WITH monthly_revenue AS (
|
||||
SELECT
|
||||
date_trunc('month', placed_at) AS month,
|
||||
SUM(total_cents) AS revenue_cents
|
||||
FROM orders
|
||||
WHERE status = 'delivered'
|
||||
GROUP BY 1
|
||||
),
|
||||
revenue_with_growth AS (
|
||||
SELECT
|
||||
month,
|
||||
revenue_cents,
|
||||
LAG(revenue_cents) OVER (ORDER BY month) AS prev_month,
|
||||
ROUND(
|
||||
100.0 * (revenue_cents - LAG(revenue_cents) OVER (ORDER BY month))
|
||||
/ NULLIF(LAG(revenue_cents) OVER (ORDER BY month), 0),
|
||||
1
|
||||
) AS growth_pct
|
||||
FROM monthly_revenue
|
||||
)
|
||||
SELECT * FROM revenue_with_growth ORDER BY month DESC;
|
||||
|
||||
-- Recursive CTE: org hierarchy tree
|
||||
WITH RECURSIVE org_tree AS (
|
||||
-- Base case: top-level orgs
|
||||
SELECT id, name, parent_id, 0 AS depth, name::TEXT AS path
|
||||
FROM organizations
|
||||
WHERE parent_id IS NULL
|
||||
|
||||
UNION ALL
|
||||
|
||||
-- Recursive step
|
||||
SELECT o.id, o.name, o.parent_id, t.depth + 1, t.path || ' > ' || o.name
|
||||
FROM organizations o
|
||||
JOIN org_tree t ON o.parent_id = t.id
|
||||
)
|
||||
SELECT * FROM org_tree ORDER BY path;
|
||||
```
|
||||
|
||||
#### Window Functions
|
||||
|
||||
```sql
|
||||
-- ROW_NUMBER: assign rank within a partition
|
||||
SELECT
|
||||
user_id,
|
||||
id AS order_id,
|
||||
total_cents,
|
||||
ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY placed_at DESC) AS rn
|
||||
FROM orders;
|
||||
|
||||
-- Get each user's most recent order
|
||||
SELECT * FROM (
|
||||
SELECT
|
||||
o.*,
|
||||
ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY placed_at DESC) AS rn
|
||||
FROM orders o
|
||||
) sub WHERE rn = 1;
|
||||
|
||||
-- LAG/LEAD: compare with previous/next row
|
||||
SELECT
|
||||
placed_at::date AS order_date,
|
||||
total_cents,
|
||||
LAG(total_cents) OVER (ORDER BY placed_at) AS prev_order_total,
|
||||
total_cents - LAG(total_cents) OVER (ORDER BY placed_at) AS diff
|
||||
FROM orders
|
||||
WHERE user_id = 42;
|
||||
|
||||
-- Running total
|
||||
SELECT
|
||||
placed_at::date AS order_date,
|
||||
total_cents,
|
||||
SUM(total_cents) OVER (
|
||||
ORDER BY placed_at
|
||||
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
|
||||
) AS running_total
|
||||
FROM orders
|
||||
WHERE user_id = 42;
|
||||
|
||||
-- NTILE: divide rows into equal buckets (e.g., quartiles)
|
||||
SELECT
|
||||
user_id,
|
||||
SUM(total_cents) AS lifetime_spend,
|
||||
NTILE(4) OVER (ORDER BY SUM(total_cents) DESC) AS spend_quartile
|
||||
FROM orders
|
||||
GROUP BY user_id;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Transaction Isolation
|
||||
|
||||
PostgreSQL supports four isolation levels. The two most commonly used:
|
||||
|
||||
| Level | Dirty Read | Non-Repeatable Read | Phantom Read | Use Case |
|
||||
|-------|-----------|-------------------|-------------|----------|
|
||||
| READ COMMITTED (default) | No | Possible | Possible | Most OLTP workloads |
|
||||
| REPEATABLE READ | No | No | No (in PG) | Reports, consistent snapshots |
|
||||
| SERIALIZABLE | No | No | No | Financial transactions, inventory |
|
||||
|
||||
```sql
|
||||
-- Default: READ COMMITTED
|
||||
-- Each statement sees the latest committed data
|
||||
BEGIN;
|
||||
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
|
||||
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
|
||||
COMMIT;
|
||||
|
||||
-- SERIALIZABLE: full isolation, detects write conflicts
|
||||
BEGIN ISOLATION LEVEL SERIALIZABLE;
|
||||
-- Read current inventory
|
||||
SELECT quantity FROM inventory WHERE sku = 'WIDGET-001';
|
||||
-- Decrement if sufficient (PG will abort if concurrent tx conflicts)
|
||||
UPDATE inventory SET quantity = quantity - 1 WHERE sku = 'WIDGET-001';
|
||||
COMMIT;
|
||||
-- If another SERIALIZABLE tx modified the same row, one will get:
|
||||
-- ERROR: could not serialize access due to concurrent update
|
||||
-- Your application must retry on serialization failure (SQLSTATE 40001)
|
||||
|
||||
-- Advisory locks for application-level coordination
|
||||
SELECT pg_advisory_xact_lock(hashtext('process-user-' || '42'));
|
||||
-- Lock is held until transaction ends; no table-level contention
|
||||
```
|
||||
|
||||
**Guidelines:**
|
||||
- Use READ COMMITTED for general CRUD operations
|
||||
- Use SERIALIZABLE when correctness requires that concurrent transactions behave as if run sequentially (e.g., balance transfers, seat reservations)
|
||||
- Always implement retry logic for serialization failures
|
||||
- Keep transactions as short as possible to reduce contention
|
||||
|
||||
---
|
||||
|
||||
### 8. Connection Pooling
|
||||
|
||||
Direct PostgreSQL connections are expensive (~1-10 MB RAM each). Use a pooler.
|
||||
|
||||
**PgBouncer configuration (pgbouncer.ini):**
|
||||
|
||||
```ini
|
||||
[databases]
|
||||
myapp = host=127.0.0.1 port=5432 dbname=myapp
|
||||
|
||||
[pgbouncer]
|
||||
listen_addr = 127.0.0.1
|
||||
listen_port = 6432
|
||||
auth_type = scram-sha-256
|
||||
auth_file = /etc/pgbouncer/userlist.txt
|
||||
|
||||
; Pool mode: transaction is best for most web apps
|
||||
pool_mode = transaction
|
||||
|
||||
; Sizing: start conservative, tune with monitoring
|
||||
default_pool_size = 20
|
||||
max_client_conn = 200
|
||||
min_pool_size = 5
|
||||
reserve_pool_size = 5
|
||||
reserve_pool_timeout = 3
|
||||
|
||||
; Timeouts
|
||||
server_idle_timeout = 300
|
||||
client_idle_timeout = 60
|
||||
query_timeout = 30
|
||||
```
|
||||
|
||||
**Pool sizing formula:**
|
||||
|
||||
```
|
||||
optimal_pool_size = ((2 * cpu_cores) + effective_disk_spindles)
|
||||
```
|
||||
|
||||
For a 4-core SSD server: `(2 * 4) + 1 = 9` connections is a good starting point. More connections does not mean more throughput -- too many causes contention.
|
||||
|
||||
**Pool modes:**
|
||||
|
||||
| Mode | Description | Caveats |
|
||||
|------|-------------|---------|
|
||||
| `transaction` | Connection returned after each transaction | Cannot use session-level features (LISTEN/NOTIFY, prepared statements, temp tables) |
|
||||
| `session` | Connection held for entire client session | Fewer pooling benefits; use only when session features needed |
|
||||
| `statement` | Connection returned after each statement | No multi-statement transactions; rarely used |
|
||||
|
||||
**Application-level pooling (Python example with asyncpg):**
|
||||
|
||||
```python
|
||||
import asyncpg
|
||||
|
||||
pool = await asyncpg.create_pool(
|
||||
dsn="postgresql://user:pass@localhost:6432/myapp",
|
||||
min_size=5,
|
||||
max_size=20,
|
||||
max_inactive_connection_lifetime=300,
|
||||
command_timeout=30,
|
||||
)
|
||||
|
||||
async with pool.acquire() as conn:
|
||||
rows = await conn.fetch("SELECT * FROM users WHERE active = true")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. Use indexes for filtered/sorted columns
|
||||
2. Use EXPLAIN ANALYZE for slow queries
|
||||
3. Avoid SELECT * in production
|
||||
4. Use transactions for multiple operations
|
||||
5. Use connection pooling
|
||||
1. **Use parameterized queries everywhere.** Never concatenate user input into SQL strings. ORMs and query builders handle this, but verify in raw SQL contexts.
|
||||
|
||||
2. **Run ANALYZE after bulk data changes.** The query planner relies on statistics. After large imports or deletes, run `ANALYZE tablename` to update them.
|
||||
|
||||
3. **Prefer BIGINT for primary keys.** INTEGER (max ~2.1 billion) can be exhausted sooner than expected in high-write systems. BIGINT costs 4 extra bytes per row but avoids a painful migration later.
|
||||
|
||||
4. **Store money as integers (cents).** Floating-point arithmetic causes rounding errors. Use `BIGINT` for cents or `NUMERIC(19,4)` if sub-cent precision is needed.
|
||||
|
||||
5. **Add indexes for foreign keys.** PostgreSQL does not automatically index the child side of a foreign key. Without it, DELETE on the parent table triggers a sequential scan on the child.
|
||||
|
||||
6. **Use TIMESTAMPTZ, not TIMESTAMP.** `TIMESTAMP WITHOUT TIME ZONE` silently drops timezone info. Always use `TIMESTAMPTZ` and let the application control display timezone.
|
||||
|
||||
7. **Set statement_timeout for web requests.** Prevent runaway queries from holding connections: `SET statement_timeout = '5s';` at session start, or configure per-role in PostgreSQL.
|
||||
|
||||
8. **Monitor with pg_stat_statements.** Enable this extension to track query performance over time. The top queries by `total_exec_time` are your optimization targets.
|
||||
|
||||
```sql
|
||||
-- Find slowest queries
|
||||
SELECT
|
||||
calls,
|
||||
round(total_exec_time::numeric, 1) AS total_ms,
|
||||
round(mean_exec_time::numeric, 1) AS mean_ms,
|
||||
query
|
||||
FROM pg_stat_statements
|
||||
ORDER BY total_exec_time DESC
|
||||
LIMIT 10;
|
||||
```
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **N+1 queries**: Use JOINs or batch loading
|
||||
- **Missing indexes**: Add indexes for WHERE/ORDER BY
|
||||
- **Large transactions**: Keep transactions short
|
||||
1. **N+1 queries from ORM lazy loading.** Loading a list of users and then accessing `user.orders` in a loop generates one query per user. Use eager loading (`joinedload` in SQLAlchemy, `select_related` in Django) or batch the query with a JOIN.
|
||||
|
||||
2. **Locking the table during migrations.** `ALTER TABLE ... ADD COLUMN NOT NULL DEFAULT 'x'` is safe in PG 11+, but `CREATE INDEX` without `CONCURRENTLY` locks writes. Always use `CREATE INDEX CONCURRENTLY` in production migrations.
|
||||
|
||||
3. **Bloated tables from UPDATE-heavy workloads.** PostgreSQL MVCC creates dead tuples on every UPDATE. If autovacuum cannot keep up, table size and query times grow. Monitor `pg_stat_user_tables.n_dead_tup` and tune autovacuum settings for hot tables.
|
||||
|
||||
4. **Using OFFSET for pagination on large datasets.** `OFFSET 100000` forces PG to scan and discard 100,000 rows. Use keyset pagination instead:
|
||||
|
||||
```sql
|
||||
-- BAD: slow for deep pages
|
||||
SELECT * FROM orders ORDER BY id LIMIT 20 OFFSET 100000;
|
||||
|
||||
-- GOOD: keyset pagination
|
||||
SELECT * FROM orders WHERE id > 100000 ORDER BY id LIMIT 20;
|
||||
```
|
||||
|
||||
5. **Ignoring connection limits.** Each PostgreSQL connection consumes RAM. Opening hundreds of direct connections (e.g., one per serverless function invocation) will exhaust `max_connections` and crash the server. Always use PgBouncer or an application-level pool.
|
||||
|
||||
6. **Storing large blobs in the database.** Files over a few KB should go in object storage (S3, R2). Store the URL/key in PostgreSQL. Large `bytea` or `TEXT` columns bloat the table, slow backups, and waste shared_buffers cache.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `databases/mongodb` - Document-based database patterns for non-relational data
|
||||
- `patterns/caching` - Caching strategies to reduce database load
|
||||
- `patterns/logging` - Logging patterns for query debugging and monitoring
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
# PostgreSQL Index Decision Tree
|
||||
|
||||
Quick reference for choosing the right index type.
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```
|
||||
What are you querying?
|
||||
|
|
||||
+-- Equality (=) or Range (<, >, BETWEEN, ORDER BY)?
|
||||
| |
|
||||
| +-- On a single scalar column?
|
||||
| | --> B-tree (default)
|
||||
| |
|
||||
| +-- On a timestamp/date column with append-only inserts?
|
||||
| | --> BRIN (much smaller than B-tree)
|
||||
| |
|
||||
| +-- Need the index to also return columns without table lookup?
|
||||
| --> Covering Index (B-tree with INCLUDE)
|
||||
|
|
||||
+-- Array containment (@>, &&) or JSONB queries?
|
||||
| --> GIN
|
||||
|
|
||||
+-- Full-text search (tsvector, @@)?
|
||||
| --> GIN
|
||||
|
|
||||
+-- Geometric/spatial data (points, polygons, PostGIS)?
|
||||
| --> GiST
|
||||
|
|
||||
+-- Range types (int4range, tsrange, overlaps)?
|
||||
| --> GiST
|
||||
|
|
||||
+-- Nearest-neighbor / distance queries (KNN)?
|
||||
| --> GiST (or SP-GiST for partitioned space)
|
||||
|
|
||||
+-- Only a subset of rows match your WHERE clause?
|
||||
| --> Partial Index (any type + WHERE filter)
|
||||
|
|
||||
+-- Trigram similarity (LIKE '%pattern%', pg_trgm)?
|
||||
| --> GIN with pg_trgm (or GiST for smaller, slower)
|
||||
|
|
||||
+-- Hash equality only (= but never range)?
|
||||
--> Hash index (rarely better than B-tree in practice)
|
||||
```
|
||||
|
||||
## Index Type Comparison
|
||||
|
||||
| Type | Best For | Operators | Size | Write Cost | Notes |
|
||||
|------|----------|-----------|------|------------|-------|
|
||||
| **B-tree** | Equality, range, sorting | `= < > <= >= BETWEEN IN IS NULL` | Medium | Low | Default. Covers 90% of cases. |
|
||||
| **GIN** | Multi-valued data | `@> && @@ ? ?& ?|` | Large | High (slow updates) | Best for arrays, JSONB, full-text. Use `fastupdate=on`. |
|
||||
| **GiST** | Spatial, ranges, nearest-neighbor | `<< >> && @> <@ <->` | Medium | Medium | Lossy for some types. Supports KNN. |
|
||||
| **SP-GiST** | Partitioned search spaces | Same as GiST | Medium | Medium | Good for phone numbers, IP addresses, non-balanced trees. |
|
||||
| **BRIN** | Large sequential/append-only tables | `= < > <= >=` | Tiny | Very Low | 1000x smaller than B-tree. Only effective when physical order correlates with column values. |
|
||||
| **Hash** | Equality only | `=` | Medium | Low | WAL-logged since PG10. Rarely outperforms B-tree. |
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Covering Index (Index-Only Scans)
|
||||
|
||||
Avoid heap lookups by including extra columns:
|
||||
|
||||
```sql
|
||||
-- Query: SELECT email, name FROM users WHERE email = ?
|
||||
CREATE INDEX idx_users_email_covering
|
||||
ON users (email) INCLUDE (name);
|
||||
```
|
||||
|
||||
### Partial Index (Filtered)
|
||||
|
||||
Index only the rows you actually query:
|
||||
|
||||
```sql
|
||||
-- Only index active orders (skip 95% of rows)
|
||||
CREATE INDEX idx_orders_active
|
||||
ON orders (created_at)
|
||||
WHERE status = 'active';
|
||||
```
|
||||
|
||||
### Composite Index (Multi-Column)
|
||||
|
||||
Column order matters -- put equality columns first, range columns last:
|
||||
|
||||
```sql
|
||||
-- Query: WHERE tenant_id = ? AND created_at > ?
|
||||
CREATE INDEX idx_events_tenant_date
|
||||
ON events (tenant_id, created_at);
|
||||
```
|
||||
|
||||
### Expression Index
|
||||
|
||||
Index a computed value:
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_users_lower_email
|
||||
ON users (lower(email));
|
||||
```
|
||||
|
||||
### GIN for JSONB
|
||||
|
||||
```sql
|
||||
-- Index all keys and values in a JSONB column
|
||||
CREATE INDEX idx_metadata_gin
|
||||
ON products USING gin (metadata jsonb_path_ops);
|
||||
|
||||
-- Supports: metadata @> '{"color": "red"}'
|
||||
```
|
||||
|
||||
### GiST for Range Overlap
|
||||
|
||||
```sql
|
||||
CREATE INDEX idx_reservations_during
|
||||
ON reservations USING gist (during);
|
||||
|
||||
-- Supports: WHERE during && '[2025-01-01, 2025-01-31]'::daterange
|
||||
```
|
||||
|
||||
### BRIN for Time-Series
|
||||
|
||||
```sql
|
||||
-- Table has millions of rows inserted in timestamp order
|
||||
CREATE INDEX idx_logs_ts_brin
|
||||
ON logs USING brin (created_at)
|
||||
WITH (pages_per_range = 32);
|
||||
```
|
||||
|
||||
## Sizing Rules of Thumb
|
||||
|
||||
| Table Rows | B-tree Size | BRIN Size | GIN Size |
|
||||
|------------|-------------|-----------|----------|
|
||||
| 1M | ~20 MB | ~50 KB | ~30 MB |
|
||||
| 10M | ~200 MB | ~500 KB | ~300 MB |
|
||||
| 100M | ~2 GB | ~5 MB | ~3 GB |
|
||||
|
||||
## Diagnostic Queries
|
||||
|
||||
```sql
|
||||
-- Check if an index is being used
|
||||
EXPLAIN (ANALYZE, BUFFERS) SELECT ...;
|
||||
|
||||
-- Find unused indexes
|
||||
SELECT indexrelname, idx_scan
|
||||
FROM pg_stat_user_indexes
|
||||
WHERE idx_scan = 0
|
||||
ORDER BY pg_relation_size(indexrelid) DESC;
|
||||
|
||||
-- Check index size
|
||||
SELECT pg_size_pretty(pg_relation_size('idx_name'));
|
||||
|
||||
-- Index bloat estimate
|
||||
SELECT * FROM pgstatindex('idx_name');
|
||||
```
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Mistake | Why It Hurts |
|
||||
|---------|-------------|
|
||||
| Indexing every column | Slows writes, wastes disk, confuses planner |
|
||||
| Wrong column order in composite | Index cannot be used for the query |
|
||||
| GIN on tiny tables | Overhead exceeds benefit |
|
||||
| B-tree on low-cardinality columns | Planner prefers seq scan anyway |
|
||||
| Missing `CONCURRENTLY` on production | Locks the table during index build |
|
||||
| Forgetting `ANALYZE` after bulk load | Planner uses stale statistics |
|
||||
|
||||
## Safe Index Creation
|
||||
|
||||
```sql
|
||||
-- Non-blocking index creation (no table lock)
|
||||
CREATE INDEX CONCURRENTLY idx_name ON table (column);
|
||||
|
||||
-- Always run ANALYZE after bulk operations
|
||||
ANALYZE table;
|
||||
```
|
||||
@@ -0,0 +1,143 @@
|
||||
-- =============================================================================
|
||||
-- Migration: [DESCRIPTION]
|
||||
-- Created: [DATE]
|
||||
-- Author: [AUTHOR]
|
||||
-- Ticket: [TICKET-ID]
|
||||
-- =============================================================================
|
||||
--
|
||||
-- SAFETY CHECKLIST (review before running):
|
||||
-- [ ] Tested on staging with production-size data
|
||||
-- [ ] Backward compatible with current application code
|
||||
-- [ ] No exclusive locks on large tables during peak hours
|
||||
-- [ ] Rollback (DOWN) section tested independently
|
||||
-- [ ] Estimated run time: ___
|
||||
-- [ ] Estimated lock duration: ___
|
||||
--
|
||||
|
||||
-- ============================================================
|
||||
-- UP MIGRATION
|
||||
-- ============================================================
|
||||
|
||||
BEGIN;
|
||||
|
||||
-- Set a statement timeout to prevent long-running locks.
|
||||
-- Adjust as needed; remove for data-only migrations.
|
||||
SET LOCAL lock_timeout = '5s';
|
||||
SET LOCAL statement_timeout = '30s';
|
||||
|
||||
-- ------------------------------------
|
||||
-- 1. Schema changes
|
||||
-- ------------------------------------
|
||||
|
||||
-- Add new table
|
||||
-- CREATE TABLE IF NOT EXISTS example (
|
||||
-- id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
-- name text NOT NULL,
|
||||
-- created_at timestamptz NOT NULL DEFAULT now(),
|
||||
-- updated_at timestamptz NOT NULL DEFAULT now()
|
||||
-- );
|
||||
|
||||
-- Add column (safe: does not rewrite table)
|
||||
-- ALTER TABLE example ADD COLUMN IF NOT EXISTS description text;
|
||||
|
||||
-- Add column with default (PG 11+: does not rewrite table)
|
||||
-- ALTER TABLE example ADD COLUMN IF NOT EXISTS is_active boolean NOT NULL DEFAULT true;
|
||||
|
||||
-- Rename column (safe: metadata-only change)
|
||||
-- ALTER TABLE example RENAME COLUMN old_name TO new_name;
|
||||
|
||||
-- ------------------------------------
|
||||
-- 2. Constraints
|
||||
-- ------------------------------------
|
||||
|
||||
-- Add NOT NULL (requires all existing rows to satisfy it)
|
||||
-- ALTER TABLE example ALTER COLUMN name SET NOT NULL;
|
||||
|
||||
-- Add check constraint (NOT VALID avoids full table scan, then VALIDATE separately)
|
||||
-- ALTER TABLE example ADD CONSTRAINT chk_example_name CHECK (name <> '') NOT VALID;
|
||||
-- ALTER TABLE example VALIDATE CONSTRAINT chk_example_name;
|
||||
|
||||
-- Add foreign key (NOT VALID + VALIDATE pattern to avoid long locks)
|
||||
-- ALTER TABLE example ADD CONSTRAINT fk_example_parent
|
||||
-- FOREIGN KEY (parent_id) REFERENCES parent(id) NOT VALID;
|
||||
-- ALTER TABLE example VALIDATE CONSTRAINT fk_example_parent;
|
||||
|
||||
-- ------------------------------------
|
||||
-- 3. Indexes (use CONCURRENTLY outside transaction)
|
||||
-- ------------------------------------
|
||||
|
||||
-- NOTE: CREATE INDEX CONCURRENTLY cannot run inside a transaction.
|
||||
-- Run these statements separately after committing the transaction above.
|
||||
--
|
||||
-- CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_example_name
|
||||
-- ON example (name);
|
||||
--
|
||||
-- CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_example_created_at
|
||||
-- ON example USING brin (created_at);
|
||||
|
||||
-- ------------------------------------
|
||||
-- 4. Data migration
|
||||
-- ------------------------------------
|
||||
|
||||
-- Backfill in batches to avoid long transactions:
|
||||
-- UPDATE example SET description = 'default' WHERE description IS NULL;
|
||||
--
|
||||
-- For large tables, batch with:
|
||||
-- DO $$
|
||||
-- DECLARE
|
||||
-- batch_size int := 10000;
|
||||
-- rows_updated int;
|
||||
-- BEGIN
|
||||
-- LOOP
|
||||
-- UPDATE example
|
||||
-- SET description = 'default'
|
||||
-- WHERE id IN (
|
||||
-- SELECT id FROM example
|
||||
-- WHERE description IS NULL
|
||||
-- LIMIT batch_size
|
||||
-- FOR UPDATE SKIP LOCKED
|
||||
-- );
|
||||
-- GET DIAGNOSTICS rows_updated = ROW_COUNT;
|
||||
-- EXIT WHEN rows_updated = 0;
|
||||
-- RAISE NOTICE 'Updated % rows', rows_updated;
|
||||
-- COMMIT;
|
||||
-- END LOOP;
|
||||
-- END $$;
|
||||
|
||||
-- ------------------------------------
|
||||
-- 5. Permissions
|
||||
-- ------------------------------------
|
||||
|
||||
-- GRANT SELECT, INSERT, UPDATE ON example TO app_role;
|
||||
-- GRANT USAGE ON SEQUENCE example_id_seq TO app_role;
|
||||
|
||||
COMMIT;
|
||||
|
||||
|
||||
-- ============================================================
|
||||
-- DOWN MIGRATION (rollback)
|
||||
-- ============================================================
|
||||
-- Run this section to undo the UP migration.
|
||||
-- Test this independently before deploying the UP migration.
|
||||
|
||||
-- BEGIN;
|
||||
--
|
||||
-- -- Reverse data migration
|
||||
-- -- UPDATE example SET description = NULL;
|
||||
--
|
||||
-- -- Drop constraints
|
||||
-- -- ALTER TABLE example DROP CONSTRAINT IF EXISTS chk_example_name;
|
||||
-- -- ALTER TABLE example DROP CONSTRAINT IF EXISTS fk_example_parent;
|
||||
--
|
||||
-- -- Drop columns
|
||||
-- -- ALTER TABLE example DROP COLUMN IF EXISTS description;
|
||||
-- -- ALTER TABLE example DROP COLUMN IF EXISTS is_active;
|
||||
--
|
||||
-- -- Drop tables
|
||||
-- -- DROP TABLE IF EXISTS example;
|
||||
--
|
||||
-- COMMIT;
|
||||
--
|
||||
-- -- Drop indexes (outside transaction)
|
||||
-- -- DROP INDEX CONCURRENTLY IF EXISTS idx_example_name;
|
||||
-- -- DROP INDEX CONCURRENTLY IF EXISTS idx_example_created_at;
|
||||
Reference in New Issue
Block a user