mirror of
https://github.com/duthaho/claudekit.git
synced 2026-09-03 08:40:52 +03:00
feat: adding new skills, including testing patterns and methodologies, along with bundled resources for better usability.
This commit is contained in:
duthaho
@@ -1,73 +1,579 @@
|
||||
---
|
||||
name: mongodb
|
||||
description: >
|
||||
Use this skill whenever working with MongoDB, document databases, or NoSQL data modeling. Trigger on keywords like MongoDB, Mongo, document database, aggregation pipeline, collection, embedded documents, or BSON. Also applies when designing document schemas, building aggregation queries, handling unstructured or semi-structured data, or migrating from relational to document-based storage.
|
||||
---
|
||||
|
||||
# MongoDB
|
||||
|
||||
## Description
|
||||
|
||||
MongoDB patterns including document design, queries, and aggregation.
|
||||
|
||||
## When to Use
|
||||
|
||||
- MongoDB database operations
|
||||
- Document-based data modeling
|
||||
- Aggregation pipelines
|
||||
- Semi-structured or polymorphic data that varies per record
|
||||
- Rapid prototyping where schema flexibility accelerates iteration
|
||||
- Event logging, IoT telemetry, or content management systems
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- Relational-heavy data models with complex joins and foreign key constraints
|
||||
- SQL-only projects where the entire stack is built around relational databases
|
||||
- Simple key-value storage where Redis or a lightweight store is more appropriate
|
||||
- Financial systems requiring multi-table ACID transactions as the norm
|
||||
|
||||
---
|
||||
|
||||
## Core Patterns
|
||||
|
||||
### Document Operations
|
||||
### 1. Schema Design
|
||||
|
||||
```javascript
|
||||
// Insert
|
||||
db.users.insertOne({
|
||||
email: 'user@example.com',
|
||||
name: 'John',
|
||||
createdAt: new Date()
|
||||
});
|
||||
The central decision in MongoDB modeling is **embed vs. reference**.
|
||||
|
||||
// Find
|
||||
db.users.find({ active: true }).sort({ createdAt: -1 }).limit(20);
|
||||
**Decision tree:**
|
||||
|
||||
// Update
|
||||
db.users.updateOne(
|
||||
{ _id: ObjectId('...') },
|
||||
{ $set: { name: 'Jane' } }
|
||||
);
|
||||
```
|
||||
Does the child data belong to exactly one parent?
|
||||
YES --> Is the child array unbounded (could grow to thousands)?
|
||||
YES --> Reference (separate collection)
|
||||
NO --> Embed
|
||||
NO --> Is it a many-to-many relationship?
|
||||
YES --> Reference (with array of ObjectIds on one or both sides)
|
||||
NO --> Reference
|
||||
```
|
||||
|
||||
### Aggregation
|
||||
**Embedding pattern -- best for data that is read together:**
|
||||
|
||||
```javascript
|
||||
// User with embedded address and preferences
|
||||
// Good: one read fetches everything the profile page needs
|
||||
db.users.insertOne({
|
||||
email: "user@example.com",
|
||||
name: "Alice Chen",
|
||||
address: {
|
||||
street: "123 Main St",
|
||||
city: "Portland",
|
||||
state: "OR",
|
||||
zip: "97201"
|
||||
},
|
||||
preferences: {
|
||||
theme: "dark",
|
||||
language: "en",
|
||||
notifications: { email: true, push: false }
|
||||
},
|
||||
createdAt: new Date()
|
||||
});
|
||||
```
|
||||
|
||||
**Referencing pattern -- best for independent or unbounded data:**
|
||||
|
||||
```javascript
|
||||
// Orders reference the user by ID
|
||||
// Good: orders grow unboundedly, accessed independently
|
||||
db.orders.insertOne({
|
||||
userId: ObjectId("6651a..."),
|
||||
status: "shipped",
|
||||
totalCents: 4999,
|
||||
items: [
|
||||
{ sku: "WIDGET-001", name: "Blue Widget", qty: 2, priceCents: 1999 },
|
||||
{ sku: "GADGET-010", name: "Mini Gadget", qty: 1, priceCents: 1001 }
|
||||
],
|
||||
placedAt: new Date()
|
||||
});
|
||||
```
|
||||
|
||||
**Denormalization pattern -- duplicate data to avoid frequent lookups:**
|
||||
|
||||
```javascript
|
||||
// Store author name directly on the post (denormalized from users)
|
||||
// Trade-off: faster reads, but updates to user name require updating all posts
|
||||
db.posts.insertOne({
|
||||
title: "Getting Started with MongoDB",
|
||||
body: "...",
|
||||
author: {
|
||||
_id: ObjectId("6651a..."),
|
||||
name: "Alice Chen" // denormalized -- must be updated if name changes
|
||||
},
|
||||
tags: ["mongodb", "tutorial"],
|
||||
publishedAt: new Date()
|
||||
});
|
||||
```
|
||||
|
||||
**Polymorphic pattern -- different shapes in one collection:**
|
||||
|
||||
```javascript
|
||||
// Events collection stores different event types
|
||||
db.events.insertMany([
|
||||
{
|
||||
type: "page_view",
|
||||
userId: ObjectId("6651a..."),
|
||||
url: "/products/widget",
|
||||
timestamp: new Date()
|
||||
},
|
||||
{
|
||||
type: "purchase",
|
||||
userId: ObjectId("6651a..."),
|
||||
orderId: ObjectId("6651b..."),
|
||||
totalCents: 4999,
|
||||
timestamp: new Date()
|
||||
}
|
||||
]);
|
||||
// Use a discriminator field (type) and query by it
|
||||
```
|
||||
|
||||
**Schema validation -- enforce structure at the database level:**
|
||||
|
||||
```javascript
|
||||
db.createCollection("users", {
|
||||
validator: {
|
||||
$jsonSchema: {
|
||||
bsonType: "object",
|
||||
required: ["email", "name", "createdAt"],
|
||||
properties: {
|
||||
email: {
|
||||
bsonType: "string",
|
||||
pattern: "^.+@.+\\..+$",
|
||||
description: "Must be a valid email"
|
||||
},
|
||||
name: {
|
||||
bsonType: "string",
|
||||
minLength: 1
|
||||
},
|
||||
role: {
|
||||
enum: ["admin", "editor", "viewer"],
|
||||
description: "Must be a valid role"
|
||||
},
|
||||
createdAt: { bsonType: "date" }
|
||||
}
|
||||
}
|
||||
},
|
||||
validationLevel: "strict",
|
||||
validationAction: "error"
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Aggregation Pipeline
|
||||
|
||||
Build complex data transformations as a sequence of stages.
|
||||
|
||||
```javascript
|
||||
// Revenue report: total and average spend per user, last 30 days
|
||||
db.orders.aggregate([
|
||||
{ $match: { status: 'completed' } },
|
||||
{ $group: {
|
||||
_id: '$userId',
|
||||
totalSpent: { $sum: '$amount' },
|
||||
orderCount: { $count: {} }
|
||||
// Stage 1: filter to recent delivered orders
|
||||
{ $match: {
|
||||
status: "delivered",
|
||||
placedAt: { $gte: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000) }
|
||||
}},
|
||||
{ $sort: { totalSpent: -1 } }
|
||||
|
||||
// Stage 2: group by user
|
||||
{ $group: {
|
||||
_id: "$userId",
|
||||
totalSpent: { $sum: "$totalCents" },
|
||||
orderCount: { $sum: 1 },
|
||||
avgOrderValue: { $avg: "$totalCents" }
|
||||
}},
|
||||
|
||||
// Stage 3: sort by spend
|
||||
{ $sort: { totalSpent: -1 } },
|
||||
|
||||
// Stage 4: limit to top 10
|
||||
{ $limit: 10 },
|
||||
|
||||
// Stage 5: join user details
|
||||
{ $lookup: {
|
||||
from: "users",
|
||||
localField: "_id",
|
||||
foreignField: "_id",
|
||||
as: "user"
|
||||
}},
|
||||
|
||||
// Stage 6: flatten the joined array
|
||||
{ $unwind: "$user" },
|
||||
|
||||
// Stage 7: reshape output
|
||||
{ $project: {
|
||||
_id: 0,
|
||||
userName: "$user.name",
|
||||
email: "$user.email",
|
||||
totalSpent: 1,
|
||||
orderCount: 1,
|
||||
avgOrderValue: { $round: ["$avgOrderValue", 0] }
|
||||
}}
|
||||
]);
|
||||
```
|
||||
|
||||
### Indexes
|
||||
**$unwind -- flatten arrays into individual documents:**
|
||||
|
||||
```javascript
|
||||
// Single field
|
||||
// Expand order items to analyze product-level metrics
|
||||
db.orders.aggregate([
|
||||
{ $unwind: "$items" },
|
||||
{ $group: {
|
||||
_id: "$items.sku",
|
||||
totalQty: { $sum: "$items.qty" },
|
||||
totalRevenue: { $sum: { $multiply: ["$items.qty", "$items.priceCents"] } }
|
||||
}},
|
||||
{ $sort: { totalRevenue: -1 } }
|
||||
]);
|
||||
```
|
||||
|
||||
**$lookup with pipeline -- filtered/correlated joins:**
|
||||
|
||||
```javascript
|
||||
// For each user, get their 3 most recent orders
|
||||
db.users.aggregate([
|
||||
{ $lookup: {
|
||||
from: "orders",
|
||||
let: { uid: "$_id" },
|
||||
pipeline: [
|
||||
{ $match: { $expr: { $eq: ["$userId", "$$uid"] } } },
|
||||
{ $sort: { placedAt: -1 } },
|
||||
{ $limit: 3 },
|
||||
{ $project: { status: 1, totalCents: 1, placedAt: 1 } }
|
||||
],
|
||||
as: "recentOrders"
|
||||
}}
|
||||
]);
|
||||
```
|
||||
|
||||
**$facet -- run multiple aggregations in parallel:**
|
||||
|
||||
```javascript
|
||||
// Dashboard: get summary stats and top products in one query
|
||||
db.orders.aggregate([
|
||||
{ $match: { status: "delivered" } },
|
||||
{ $facet: {
|
||||
summary: [
|
||||
{ $group: {
|
||||
_id: null,
|
||||
totalRevenue: { $sum: "$totalCents" },
|
||||
totalOrders: { $sum: 1 }
|
||||
}}
|
||||
],
|
||||
topProducts: [
|
||||
{ $unwind: "$items" },
|
||||
{ $group: { _id: "$items.sku", sold: { $sum: "$items.qty" } } },
|
||||
{ $sort: { sold: -1 } },
|
||||
{ $limit: 5 }
|
||||
],
|
||||
monthlyTrend: [
|
||||
{ $group: {
|
||||
_id: { $dateToString: { format: "%Y-%m", date: "$placedAt" } },
|
||||
revenue: { $sum: "$totalCents" }
|
||||
}},
|
||||
{ $sort: { _id: 1 } }
|
||||
]
|
||||
}}
|
||||
]);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Index Strategies
|
||||
|
||||
```javascript
|
||||
// Single field index -- most common
|
||||
db.users.createIndex({ email: 1 }, { unique: true });
|
||||
|
||||
// Compound
|
||||
db.posts.createIndex({ userId: 1, createdAt: -1 });
|
||||
// Compound index -- order matters, follows the ESR rule:
|
||||
// Equality fields first, Sort fields next, Range fields last
|
||||
db.orders.createIndex({ status: 1, placedAt: -1 });
|
||||
// Supports: find({status: "pending"}).sort({placedAt: -1})
|
||||
// Also supports: find({status: "pending"}) alone (prefix)
|
||||
|
||||
// Multikey index -- automatically indexes each array element
|
||||
db.posts.createIndex({ tags: 1 });
|
||||
// Supports: find({ tags: "mongodb" })
|
||||
|
||||
// Text index -- basic full-text search
|
||||
db.posts.createIndex(
|
||||
{ title: "text", body: "text" },
|
||||
{ weights: { title: 10, body: 1 }, name: "posts_text_search" }
|
||||
);
|
||||
// Usage:
|
||||
db.posts.find(
|
||||
{ $text: { $search: "mongodb aggregation" } },
|
||||
{ score: { $meta: "textScore" } }
|
||||
).sort({ score: { $meta: "textScore" } });
|
||||
|
||||
// TTL index -- auto-delete documents after expiry
|
||||
db.sessions.createIndex(
|
||||
{ expiresAt: 1 },
|
||||
{ expireAfterSeconds: 0 } // delete when expiresAt is in the past
|
||||
);
|
||||
// Documents must have a Date field; they are removed by a background task ~every 60s
|
||||
|
||||
// Partial index -- only index documents matching a filter
|
||||
db.orders.createIndex(
|
||||
{ placedAt: -1 },
|
||||
{ partialFilterExpression: { status: "pending" } }
|
||||
);
|
||||
// Smaller index; only used when the query includes the filter condition
|
||||
|
||||
// Wildcard index -- for querying arbitrary keys in a sub-document
|
||||
db.products.createIndex({ "attributes.$**": 1 });
|
||||
// Supports: find({ "attributes.color": "red" }) without knowing keys in advance
|
||||
|
||||
// Collation -- case-insensitive sorting and matching
|
||||
db.users.createIndex(
|
||||
{ name: 1 },
|
||||
{ collation: { locale: "en", strength: 2 } }
|
||||
);
|
||||
```
|
||||
|
||||
**The ESR rule for compound indexes:** order fields by **E**quality, **S**ort, **R**ange. This produces the most efficient index scans.
|
||||
|
||||
```javascript
|
||||
// Query: find active orders for a user, sorted by date, in a date range
|
||||
// Equality: userId, status
|
||||
// Sort: placedAt
|
||||
// Range: placedAt (but sort and range on same field -- sort wins)
|
||||
db.orders.createIndex({ userId: 1, status: 1, placedAt: -1 });
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Transactions
|
||||
|
||||
Multi-document transactions work across collections (requires replica set or sharded cluster).
|
||||
|
||||
```javascript
|
||||
const session = client.startSession();
|
||||
|
||||
try {
|
||||
session.startTransaction({
|
||||
readConcern: { level: "snapshot" },
|
||||
writeConcern: { w: "majority" },
|
||||
readPreference: "primary"
|
||||
});
|
||||
|
||||
const accounts = client.db("bank").collection("accounts");
|
||||
|
||||
// Transfer $50 from account A to account B
|
||||
const fromAccount = await accounts.findOne(
|
||||
{ _id: "account-A" },
|
||||
{ session }
|
||||
);
|
||||
|
||||
if (fromAccount.balanceCents < 5000) {
|
||||
await session.abortTransaction();
|
||||
throw new Error("Insufficient funds");
|
||||
}
|
||||
|
||||
await accounts.updateOne(
|
||||
{ _id: "account-A" },
|
||||
{ $inc: { balanceCents: -5000 } },
|
||||
{ session }
|
||||
);
|
||||
|
||||
await accounts.updateOne(
|
||||
{ _id: "account-B" },
|
||||
{ $inc: { balanceCents: 5000 } },
|
||||
{ session }
|
||||
);
|
||||
|
||||
// Record the transfer in a separate collection -- still in the same tx
|
||||
await client.db("bank").collection("transfers").insertOne({
|
||||
from: "account-A",
|
||||
to: "account-B",
|
||||
amountCents: 5000,
|
||||
timestamp: new Date()
|
||||
}, { session });
|
||||
|
||||
await session.commitTransaction();
|
||||
} catch (error) {
|
||||
await session.abortTransaction();
|
||||
throw error;
|
||||
} finally {
|
||||
await session.endSession();
|
||||
}
|
||||
```
|
||||
|
||||
**Guidelines:**
|
||||
- Keep transactions short -- they hold locks and consume resources
|
||||
- Design your schema to minimize the need for multi-document transactions
|
||||
- Transactions have a default 60-second timeout (`maxTimeMS`)
|
||||
- Retryable writes (`retryWrites=true` in connection string) handle transient errors automatically
|
||||
|
||||
---
|
||||
|
||||
### 5. Change Streams
|
||||
|
||||
Watch for real-time changes to collections, databases, or the entire deployment.
|
||||
|
||||
```javascript
|
||||
// Watch a single collection for inserts and updates
|
||||
const pipeline = [
|
||||
{ $match: {
|
||||
operationType: { $in: ["insert", "update"] },
|
||||
"fullDocument.status": "urgent"
|
||||
}}
|
||||
];
|
||||
|
||||
const changeStream = db.collection("tickets").watch(pipeline, {
|
||||
fullDocument: "updateLookup" // include the full document on updates
|
||||
});
|
||||
|
||||
changeStream.on("change", (change) => {
|
||||
console.log("Change detected:", change.operationType);
|
||||
console.log("Document:", change.fullDocument);
|
||||
console.log("Resume token:", change.resumeToken);
|
||||
|
||||
// Process the change (e.g., send notification, update cache)
|
||||
notifyTeam(change.fullDocument);
|
||||
});
|
||||
|
||||
// Handle errors and resume from last known position
|
||||
changeStream.on("error", (error) => {
|
||||
console.error("Change stream error:", error);
|
||||
// Reconnect using the stored resume token
|
||||
});
|
||||
```
|
||||
|
||||
**Resumable pattern for production:**
|
||||
|
||||
```javascript
|
||||
let resumeToken = await loadResumeTokenFromStorage();
|
||||
|
||||
async function watchWithResume(collection) {
|
||||
const options = { fullDocument: "updateLookup" };
|
||||
if (resumeToken) {
|
||||
options.resumeAfter = resumeToken;
|
||||
}
|
||||
|
||||
const stream = collection.watch([], options);
|
||||
|
||||
stream.on("change", async (change) => {
|
||||
// Process change
|
||||
await handleChange(change);
|
||||
|
||||
// Persist resume token so we can recover after restart
|
||||
resumeToken = change._id;
|
||||
await saveResumeTokenToStorage(resumeToken);
|
||||
});
|
||||
|
||||
stream.on("error", async () => {
|
||||
// Wait and reconnect
|
||||
await new Promise(r => setTimeout(r, 5000));
|
||||
watchWithResume(collection);
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
**Use cases:** real-time dashboards, cache invalidation, event-driven architectures, syncing data to search indexes (e.g., Elasticsearch).
|
||||
|
||||
---
|
||||
|
||||
### 6. Performance
|
||||
|
||||
#### Reading explain() output
|
||||
|
||||
```javascript
|
||||
// Run explain to see the query plan
|
||||
db.orders.find({
|
||||
userId: ObjectId("6651a..."),
|
||||
status: "pending"
|
||||
}).sort({ placedAt: -1 }).explain("executionStats");
|
||||
```
|
||||
|
||||
**Key fields in executionStats:**
|
||||
|
||||
| Field | What to look for |
|
||||
|-------|-----------------|
|
||||
| `winningPlan.stage` | `IXSCAN` good, `COLLSCAN` bad (full collection scan) |
|
||||
| `totalKeysExamined` | Should be close to `nReturned` (no wasted index scans) |
|
||||
| `totalDocsExamined` | Should be close to `nReturned` (no wasted document reads) |
|
||||
| `executionTimeMillis` | Overall query time |
|
||||
| `rejectedPlans` | Shows alternatives the optimizer considered |
|
||||
|
||||
**Covered queries -- answered entirely from the index:**
|
||||
|
||||
```javascript
|
||||
// Create an index that covers the query
|
||||
db.orders.createIndex({ userId: 1, status: 1, totalCents: 1 });
|
||||
|
||||
// This query only needs fields in the index -- no document fetch
|
||||
db.orders.find(
|
||||
{ userId: ObjectId("6651a..."), status: "delivered" },
|
||||
{ _id: 0, totalCents: 1 } // projection must exclude _id and only include indexed fields
|
||||
);
|
||||
// explain() will show: "totalDocsExamined": 0
|
||||
```
|
||||
|
||||
**Projection optimization -- fetch only what you need:**
|
||||
|
||||
```javascript
|
||||
// BAD: fetches entire document including large body field
|
||||
const posts = await db.posts.find({ author: userId }).toArray();
|
||||
|
||||
// GOOD: only fetch fields needed for the list view
|
||||
const posts = await db.posts.find(
|
||||
{ author: userId },
|
||||
{ projection: { title: 1, publishedAt: 1, tags: 1 } }
|
||||
).toArray();
|
||||
```
|
||||
|
||||
**Bulk operations for write-heavy workloads:**
|
||||
|
||||
```javascript
|
||||
const bulk = db.products.initializeUnorderedBulkOp();
|
||||
|
||||
for (const update of priceUpdates) {
|
||||
bulk.find({ sku: update.sku })
|
||||
.updateOne({ $set: { priceCents: update.newPrice, updatedAt: new Date() } });
|
||||
}
|
||||
|
||||
const result = await bulk.execute();
|
||||
console.log(`Modified: ${result.nModified}, Errors: ${result.getWriteErrorCount()}`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. Embed frequently accessed data
|
||||
2. Use references for large/independent data
|
||||
3. Create indexes for query patterns
|
||||
4. Use aggregation for complex queries
|
||||
5. Avoid unbounded arrays
|
||||
1. **Design schema around query patterns, not data relationships.** Ask "how will I read this data?" before "how does this data relate?" Embed data that is always fetched together; reference data accessed independently.
|
||||
|
||||
2. **Use the ESR rule for compound indexes.** Order index fields by Equality, Sort, Range. This maximizes the index's usefulness and minimizes keys examined.
|
||||
|
||||
3. **Set read/write concerns appropriately.** Use `w: "majority"` and `readConcern: "majority"` for data that must survive failovers. Use `w: 1` for non-critical writes where speed matters more than durability.
|
||||
|
||||
4. **Use projection to limit returned fields.** Transferring large documents over the network when you only need two fields wastes bandwidth and memory. Always project.
|
||||
|
||||
5. **Avoid unbounded array growth.** An embedded array that can grow to thousands of elements bloats the document (16 MB max) and degrades performance. Move to a separate collection with a reference when the array exceeds ~100 elements.
|
||||
|
||||
6. **Use bulk operations for batch writes.** Individual `insertOne` or `updateOne` calls in a loop are slow. Batch them with `bulkWrite` or `initializeUnorderedBulkOp` for 10-50x throughput improvement.
|
||||
|
||||
7. **Enable retryable writes.** Add `retryWrites=true` to your connection string. This handles transient network errors and primary elections automatically without application-level retry logic.
|
||||
|
||||
8. **Monitor with database profiler and serverStatus.** Use `db.setProfilingLevel(1, { slowms: 100 })` to log slow queries. Check `db.serverStatus().opcounters` and `db.serverStatus().connections` for overall health.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Unbounded arrays**: Limit array size
|
||||
- **Missing indexes**: Analyze query patterns
|
||||
- **Over-embedding**: Consider data access patterns
|
||||
1. **Treating MongoDB like a relational database.** Normalizing everything into separate collections and using `$lookup` for every query defeats the purpose. If you need heavy joins, PostgreSQL is likely a better fit. Design for embedding first.
|
||||
|
||||
2. **Missing indexes on query fields.** Every `find()`, `$match`, and `sort()` should be backed by an index. Use `db.collection.getIndexes()` and `explain()` to verify. A `COLLSCAN` on a large collection is almost always a bug.
|
||||
|
||||
3. **Ignoring the 16 MB document size limit.** Embedding unbounded arrays (comments, logs, events) will eventually hit this wall, crashing writes. Use the bucket pattern (fixed-size sub-documents) or reference a separate collection.
|
||||
|
||||
4. **Not using readPreference for read-heavy workloads.** By default all reads go to the primary. For analytics or non-critical reads, use `readPreference: "secondaryPreferred"` to distribute load across replicas.
|
||||
|
||||
5. **Forgetting that updates replace matched array elements, not all of them.** Using `$set` on a matched array element with positional `$` only updates the first match. Use `$[]` for all elements or `$[<identifier>]` with `arrayFilters` for conditional updates:
|
||||
|
||||
```javascript
|
||||
// Update price for a specific item in all orders
|
||||
db.orders.updateMany(
|
||||
{ "items.sku": "WIDGET-001" },
|
||||
{ $set: { "items.$[item].priceCents": 2499 } },
|
||||
{ arrayFilters: [{ "item.sku": "WIDGET-001" }] }
|
||||
);
|
||||
```
|
||||
|
||||
6. **Running aggregation pipelines without early $match.** Always filter as early as possible in the pipeline. A `$group` or `$unwind` before `$match` processes the entire collection unnecessarily. Put `$match` first to leverage indexes and reduce documents flowing through subsequent stages.
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `databases/postgresql` - Relational database patterns for structured data with complex relationships
|
||||
- `patterns/caching` - Caching strategies to reduce database load
|
||||
- `patterns/logging` - Logging patterns for query debugging and monitoring
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
# MongoDB Schema Design Patterns
|
||||
|
||||
Quick reference for embedding vs referencing decisions and common schema patterns.
|
||||
|
||||
## Embedding vs Referencing Decision Tree
|
||||
|
||||
```
|
||||
What is the relationship cardinality?
|
||||
|
|
||||
+-- One-to-Few (< 50 items)?
|
||||
| --> EMBED in parent document
|
||||
| Example: user.addresses, post.tags
|
||||
|
|
||||
+-- One-to-Many (50 - 1000s)?
|
||||
| |
|
||||
| +-- Child data always accessed with parent?
|
||||
| | --> EMBED (but watch 16 MB doc limit)
|
||||
| |
|
||||
| +-- Child data accessed independently?
|
||||
| | --> REFERENCE (store child _id in parent array)
|
||||
| |
|
||||
| +-- Need atomic updates across parent + children?
|
||||
| --> EMBED
|
||||
|
|
||||
+-- One-to-Millions?
|
||||
| --> REFERENCE from child to parent
|
||||
| Example: log_entry.host_id (not host.log_entry_ids)
|
||||
|
|
||||
+-- Many-to-Many?
|
||||
--> REFERENCE with array of _ids on one or both sides
|
||||
Example: student.course_ids[], course.student_ids[]
|
||||
```
|
||||
|
||||
## Decision Factors
|
||||
|
||||
| Factor | Favor Embedding | Favor Referencing |
|
||||
|--------|----------------|-------------------|
|
||||
| **Read pattern** | Always read together | Read independently |
|
||||
| **Write pattern** | Infrequent child updates | Frequent child updates |
|
||||
| **Data size** | Small, bounded children | Large or growing children |
|
||||
| **Atomicity** | Need single-doc transactions | Can tolerate multi-doc txn |
|
||||
| **Duplication** | OK to denormalize | Must avoid duplication |
|
||||
| **Cardinality** | Few items | Many/unbounded items |
|
||||
| **Document size** | Well under 16 MB limit | Approaching 16 MB |
|
||||
|
||||
## Pattern Catalog
|
||||
|
||||
### 1. Subset Pattern
|
||||
|
||||
**Problem**: Document is large but reads only need a few fields from embedded data.
|
||||
|
||||
**Solution**: Embed a subset; keep full data in a separate collection.
|
||||
|
||||
```javascript
|
||||
// products collection - fast reads for listing pages
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
name: "Widget",
|
||||
price: 29.99,
|
||||
// Only the 10 most recent reviews (subset)
|
||||
recent_reviews: [
|
||||
{ user: "alice", rating: 5, text: "Great!", date: ISODate("...") }
|
||||
],
|
||||
review_count: 247
|
||||
}
|
||||
|
||||
// reviews collection - full review data
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
product_id: ObjectId("..."),
|
||||
user: "alice",
|
||||
rating: 5,
|
||||
text: "Great!",
|
||||
date: ISODate("..."),
|
||||
helpful_votes: 12
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Product pages, user profiles, any "preview + detail" pattern.
|
||||
|
||||
### 2. Computed Pattern
|
||||
|
||||
**Problem**: Expensive aggregation queries run repeatedly on the same data.
|
||||
|
||||
**Solution**: Pre-compute and store the result, update on write.
|
||||
|
||||
```javascript
|
||||
// movies collection
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Example Movie",
|
||||
// Pre-computed from screenings collection
|
||||
computed: {
|
||||
total_revenue: 1250000,
|
||||
avg_rating: 4.2,
|
||||
rating_count: 843,
|
||||
last_computed: ISODate("2025-01-15T00:00:00Z")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Update strategy**: On each new rating, increment count and recalculate average. Or use a background job for less time-sensitive data.
|
||||
|
||||
**When to use**: Dashboards, leaderboards, summary statistics.
|
||||
|
||||
### 3. Bucket Pattern
|
||||
|
||||
**Problem**: Many small, time-series documents create overhead (indexes, storage per doc).
|
||||
|
||||
**Solution**: Group related data into fixed-size buckets.
|
||||
|
||||
```javascript
|
||||
// sensor_readings collection - one doc per sensor per hour
|
||||
{
|
||||
sensor_id: "sensor-42",
|
||||
bucket_start: ISODate("2025-01-15T14:00:00Z"),
|
||||
bucket_end: ISODate("2025-01-15T14:59:59Z"),
|
||||
count: 60,
|
||||
readings: [
|
||||
{ ts: ISODate("2025-01-15T14:00:00Z"), temp: 22.1, humidity: 45 },
|
||||
{ ts: ISODate("2025-01-15T14:01:00Z"), temp: 22.3, humidity: 44 }
|
||||
// ... up to 60 readings per bucket
|
||||
],
|
||||
// Pre-computed aggregates for the bucket
|
||||
summary: {
|
||||
avg_temp: 22.4,
|
||||
min_temp: 21.8,
|
||||
max_temp: 23.1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Bucket sizing**: Choose a size that balances doc count reduction vs update frequency. Common choices: 1 hour, 1 day, 100 events.
|
||||
|
||||
**When to use**: IoT, time-series, event logging, analytics.
|
||||
|
||||
### 4. Outlier Pattern
|
||||
|
||||
**Problem**: A few documents have vastly more data than the norm (e.g., a viral post with millions of likes).
|
||||
|
||||
**Solution**: Flag outliers and overflow into separate documents.
|
||||
|
||||
```javascript
|
||||
// books collection - normal case
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Normal Book",
|
||||
customers_purchased: ["user1", "user2", "user3"],
|
||||
has_overflow: false
|
||||
}
|
||||
|
||||
// books collection - outlier (bestseller)
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
title: "Bestseller",
|
||||
customers_purchased: ["user1", "user2", /* ... first 1000 */],
|
||||
has_overflow: true
|
||||
}
|
||||
|
||||
// book_purchases_overflow collection
|
||||
{
|
||||
book_id: ObjectId("..."),
|
||||
page: 2,
|
||||
customers_purchased: ["user1001", "user1002", /* ... next 1000 */]
|
||||
}
|
||||
```
|
||||
|
||||
**When to use**: Social media (viral posts), e-commerce (bestsellers), any data with power-law distribution.
|
||||
|
||||
### 5. Extended Reference Pattern
|
||||
|
||||
**Problem**: Frequent joins (lookups) to get a few fields from a referenced document.
|
||||
|
||||
**Solution**: Copy the most-accessed fields into the referencing document.
|
||||
|
||||
```javascript
|
||||
// orders collection
|
||||
{
|
||||
_id: ObjectId("..."),
|
||||
date: ISODate("..."),
|
||||
customer_id: ObjectId("..."),
|
||||
// Extended reference - copied fields for fast reads
|
||||
customer_name: "Alice Smith",
|
||||
customer_email: "alice@example.com",
|
||||
items: [
|
||||
{
|
||||
product_id: ObjectId("..."),
|
||||
product_name: "Widget", // copied
|
||||
price: 29.99, // copied (snapshot at time of order)
|
||||
quantity: 2
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Trade-off**: Stale data is acceptable (order snapshots price at purchase time). For data that must be current, keep only the reference.
|
||||
|
||||
**When to use**: Orders (snapshot pricing), notifications (snapshot user name), audit logs.
|
||||
|
||||
### 6. Polymorphic Pattern
|
||||
|
||||
**Problem**: Objects share some fields but differ in others (e.g., different product types).
|
||||
|
||||
**Solution**: Store in a single collection with a type discriminator.
|
||||
|
||||
```javascript
|
||||
// vehicles collection
|
||||
{ type: "car", make: "Toyota", doors: 4, trunk_size_liters: 450 }
|
||||
{ type: "truck", make: "Ford", doors: 2, payload_kg: 5000 }
|
||||
{ type: "motorcycle", make: "Harley", engine_cc: 1200 }
|
||||
```
|
||||
|
||||
**Index strategy**: Index common fields. Use partial indexes for type-specific fields.
|
||||
|
||||
```javascript
|
||||
db.vehicles.createIndex(
|
||||
{ payload_kg: 1 },
|
||||
{ partialFilterExpression: { type: "truck" } }
|
||||
);
|
||||
```
|
||||
|
||||
**When to use**: Product catalogs, content management (articles, videos, images), mixed event streams.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
| Mistake | Problem | Fix |
|
||||
|---------|---------|-----|
|
||||
| Unbounded array growth | Document exceeds 16 MB | Use bucket or outlier pattern |
|
||||
| Deep nesting (> 3 levels) | Hard to query and index | Flatten or reference |
|
||||
| Normalizing everything | Too many lookups, slow reads | Embed when read together |
|
||||
| Embedding large blobs | Wastes RAM in working set | Store in GridFS or S3 |
|
||||
| No schema validation | Inconsistent data over time | Use JSON Schema validation |
|
||||
| Indexing every field | Slow writes, wasted space | Index based on query patterns |
|
||||
|
||||
## Schema Validation
|
||||
|
||||
Use `db.createCollection()` with `$jsonSchema` validator to enforce structure. Set `validationLevel: "moderate"` to apply only on inserts and updates (not existing docs).
|
||||
Reference in New Issue
Block a user