What is the aggregation pipeline?
A list of stages that documents flow through in order, each transforming the stream, like Unix pipes:
$matchfilters,$groupaggregates,$sortorders,$projectreshapes. It is how MongoDB does what SQL does withGROUP BY, joins and computed columns, and it runs on the server.Node.jsconst { MongoClient } = require('mongodb'); const client = new MongoClient('mongodb://localhost:27017'); async function main() { const orders = client.db('shop').collection('orders'); await orders.insertMany([ { customer: 'asha', total: 30, status: 'paid' }, { customer: 'ravi', total: 20, status: 'paid' }, { customer: 'asha', total: 15, status: 'paid' }, { customer: 'ravi', total: 50, status: 'cancelled' }, ]); const rows = await orders.aggregate([ { $match: { status: 'paid' } }, { $group: { _id: '$customer', spent: { $sum: '$total' } } }, { $sort: { spent: -1 } }, ]).toArray(); console.log(rows); } main().finally(() => client.close());Which aggregation stages should you know?
Stage What it does SQL $matchfilters documents WHERE, HAVING $groupgroups and aggregates GROUP BY $project,$setpicks or computes fields SELECT list $sort,$skip,$limitorders and pages ORDER BY, LIMIT $lookupjoins another collection LEFT OUTER JOIN $unwindone document per array element $countcounts documents COUNT(*) $facetseveral pipelines on one input $out,$mergewrites results to a collection INSERT INTO ... SELECT When do you use
aggregateinstead offind?findreturns stored documents: filtered, projected, sorted and limited. Useaggregatewhen the result is computed: totals and counts per group, joins, reshaped or unwound documents, or several results in one call. A pipeline that starts with$matchand$sortcan use the same indexes asfind, so the choice is about what you need back, not speed.How do you write SQL's
GROUP BY ... HAVINGin MongoDB?Node.jsconst { MongoClient } = require('mongodb'); const client = new MongoClient('mongodb://localhost:27017'); async function main() { const sales = client.db('shop').collection('sales'); await sales.insertMany([ { city: 'Pune', amount: 10 }, { city: 'Pune', amount: 50 }, { city: 'Delhi', amount: 30 }, { city: 'Delhi', amount: 20 }, { city: 'Pune', amount: 40 }, ]); const rows = await sales.aggregate([ { $group: { _id: '$city', total: { $sum: '$amount' }, orders: { $sum: 1 } } }, { $match: { total: { $gt: 60 } } }, ]).toArray(); for (const r of rows) console.log(r._id, r.total, r.orders); } main().finally(() => client.close());It prints
Pune 100 3.$groupplays the part ofGROUP BY, and a$matchafter it playsHAVING, filtering on the computedtotal, so Delhi's 50 is dropped. A$matchbefore the$groupis theWHERE.{ $sum: 1 }counts documents; since MongoDB 5.0,$count: {}inside$groupdoes the same. See the SQL side in GROUP BY and HAVING.What happens to documents with an empty array in
$unwind?Node.jsconst { MongoClient } = require('mongodb'); const client = new MongoClient('mongodb://localhost:27017'); async function main() { const orders = client.db('shop').collection('orders'); await orders.insertMany([ { no: 1, items: ['a', 'b'] }, { no: 2, items: [] }, { no: 3 }, ]); const show = (rows) => rows.map((r) => `${r.no}:${r.items ?? 'none'}`).join(' '); console.log(show(await orders.aggregate([{ $sort: { no: 1 } }, { $unwind: '$items' }]).toArray())); console.log(show(await orders.aggregate([ { $sort: { no: 1 } }, { $unwind: { path: '$items', preserveNullAndEmptyArrays: true } }, ]).toArray())); } main().finally(() => client.close());It prints
1:a 1:b, then1:a 1:b 2:none 3:none.$unwindoutputs one document per array element, so a document whose array is empty, null or missing produces nothing and quietly drops out, a classic bug in reports.preserveNullAndEmptyArrays: truekeeps those documents, without the field.How do you join two collections in MongoDB?
With
$lookup, which works like a left outer join: every input document gets an array of matching documents from the other collection, empty when nothing matches. Add$unwindfor one row per match. A join costs more than reading one embedded document, so design the schema so frequent reads need none.Node.jsconst { MongoClient } = require('mongodb'); const client = new MongoClient('mongodb://localhost:27017'); async function main() { const db = client.db('shop'); await db.collection('customers').insertMany([{ _id: 1, name: 'Asha' }, { _id: 2, name: 'Ben' }]); await db.collection('orders').insertMany([ { customerId: 1, total: 30 }, { customerId: 1, total: 20 }, ]); const rows = await db.collection('customers').aggregate([ { $lookup: { from: 'orders', localField: '_id', foreignField: 'customerId', as: 'orders' } }, { $sort: { name: 1 } }, ]).toArray(); for (const c of rows) console.log(c.name, c.orders.length); } main().finally(() => client.close());What is the difference between
$projectand$addFields?$projectoutputs only the fields you list (plus_id), or everything except the ones you exclude, and can compute new fields.$addFields, also named$setsince 4.2, keeps every existing field and adds or overwrites the ones you name. Use$setto add a computed field mid-pipeline, and$projector$unsetat the end to shape the output.{ $set: { total: { $multiply: ['$price', '$qty'] } } } { $project: { _id: 0, name: 1, total: 1 } }Why should
$matchcome early in a pipeline?Stages run in order, so filtering first gives every later stage fewer documents, and only a
$matchor$sortat the start of the pipeline can use an index. The optimizer moves some stages for you (a$matchthat does not depend on computed fields moves ahead of a$project), but write the efficient order yourself and confirm it withexplain.How do you get the top N documents per group?
Since MongoDB 5.2,
$topNinside$groupdoes it in one stage ($topfor one,$firstNfor the first N in input order). On older versions,$sortfirst, then$groupwith$firstfor the top one, or$pushfollowed by$slicefor N.Node.jsconst { MongoClient } = require('mongodb'); const client = new MongoClient('mongodb://localhost:27017'); async function main() { const sales = client.db('shop').collection('sales'); await sales.insertMany([ { city: 'Pune', amount: 10 }, { city: 'Pune', amount: 50 }, { city: 'Delhi', amount: 30 }, { city: 'Delhi', amount: 20 }, { city: 'Pune', amount: 40 }, ]); const rows = await sales.aggregate([ { $group: { _id: '$city', best: { $topN: { n: 2, sortBy: { amount: -1 }, output: '$amount' } } } }, { $sort: { _id: 1 } }, ]).toArray(); for (const r of rows) console.log(r._id, r.best); } main().finally(() => client.close());How do you return a page of results and the total count in one query?
With
$facet, which runs several sub-pipelines on the same input and returns one document with an array per facet. Put the$matchbefore it: stages inside$facetcannot use indexes, and the single result document must stay under 16 MB.Node.jsconst { MongoClient } = require('mongodb'); const client = new MongoClient('mongodb://localhost:27017'); async function main() { const products = client.db('shop').collection('products'); await products.insertMany([ { name: 'pen', price: 5 }, { name: 'cup', price: 12 }, { name: 'lamp', price: 30 }, { name: 'bag', price: 40 }, ]); const [result] = await products.aggregate([ { $match: { price: { $lt: 40 } } }, { $facet: { total: [{ $count: 'n' }], page: [{ $sort: { price: 1 } }, { $limit: 2 }, { $project: { _id: 0, name: 1 } }], } }, ]).toArray(); console.log(result.total[0].n, result.page.map((p) => p.name)); } main().finally(() => client.close());What are the memory limits of the aggregation pipeline?
Each blocking stage, such as
$groupor$sort, may use 100 MB of RAM. Beyond that it spills to temporary files on disk if disk use is allowed, which is the default since MongoDB 6.0 (allowDiskUseByDefault); before 6.0 you passedallowDiskUse: trueor the pipeline failed. Each output document is still limited to 16 MB, which matters for$groupwith$pushand for$facet.What is the difference between
$outand$merge?Both write the pipeline's results to a collection and must be the last stage.
$outreplaces the target collection completely.$merge(4.2+) merges into it: you choose what happens when a document matches (replace,merge,keepExisting,failor a pipeline) and when it does not (insert,discard,fail). That makes$mergethe tool for incremental materialized views, such as daily totals refreshed every hour.
MongoDB Aggregation Interview Questions
The aggregation pipeline: its stages, $group, $unwind, joins with $lookup, top N per group, $facet and the memory limits.