Builds onEmbeddings Vector Search & HNSW K-Nearest Neighbors Gradient Boosting Model Evaluation & Metrics Class Imbalance & Resampling Hypothesis Testing

Lensa ML

Design a Recommendation Feed

Step 1 of 6: Why Not Score Every Item?

The latency budget forces retrieval before ranking

Latency

31 ms

Recall

0.87

Feed P@10

0.72

Categories

4.5

Test days

6

Design for 10M users · 1M items · 1,000 req/s · 100 ms budget
Recall, precision and categories measured on the toy data

Items in the catalog (M)1M

Ranker cost per item0.10 ms

Latency budget per request100 ms

Feed requests per second at peak1,000

Registered users10M

Score every item

100 s

Funnel

31 ms

Ranker cores, all items

100k

Ranker cores, funnel

20

Scoring all 1M items with the ranker takes 100 s per request, 1,000 times the budget, and 1,000 requests a second would keep 100k cores busy. Retrieval first cuts the catalog to 200 candidates, items worth scoring carefully, so the whole request takes 31 ms.

Assumptions

Retrieval time (ANN search + user tower)10 ms

One item-item similarity in the re-ranker1.0 µs

Ranker cost	0.10 ms per item	Time on one core. Cores busy = requests/s × items scored × cost
Toy world	300 users · 1,000 items	Hidden 8-dimensional tastes plus noise, 8 categories, seed 7. Recall, precision and diversity are measured here, not on 1M items
Relevant items	true top 20	Each user's 20 highest true affinities, the answer key for recall and precision
Two-tower model	rank-2 SVD	The best 2-D dot-product fit to the true affinities. A model trained on clicks would do worse
Ranker noise	σ = 0.16 × √(0.1 ms ÷ cost)	Simulated: a costlier ranker sees more features, so its score is closer to the true affinity
Re-rank pool	top 100	MMR re-orders only the top of the ranked list
Feedback loop	K = 10 · 40 rounds	Simulated, seed 11. The prior counts as K impressions
Targets	recall ≥ 0.8 · P@10 ≥ 0.6 · ≥ 4 categories · ≤ 14 days	Set for this brief, not industry standards
Not modelled	feature fetch, caching, position bias	Retrieval time stands in for network and feature lookups

All constants are assumptions for this brief, set 2026-10-25.

Drag Items in the catalog to 100M and Ranker cost per item to its maximum. The full-scoring bar grows to hours while the funnel stays near the budget, because the funnel's cost depends on the candidate count N, not on catalog size. The cores readout is requests per second × items scored × cost per item, the compute a fleet would keep busy at peak. A tenfold larger catalog multiplies it by ten for full scoring and leaves the funnel unchanged.

Step 2 of 6: Candidate Generation

Two towers and a dot product find the top N

Latency

31 ms

Recall

0.87

Feed P@10

0.72

Categories

4.5

Test days

6

Design for 10M users · 1M items · 1,000 req/s · 100 ms budget
Recall, precision and categories measured on the toy data

retrieved (top 200)relevant, foundrelevant, missedwhite arrow: user vector

Candidates retrieved (N)200

Example user#1 of 300

Recall, all users

0.87

User #1 found

20 of 20

Share of toy catalog

20%

Ranking time

20 ms

The two-tower model learns one vector per user and one per item, so that their dot product predicts affinity. Retrieval keeps every item past the dashed line, the top 200 by dot product. Here it finds 87% of each user's 20 relevant items (measured on the toy data), because 2 dimensions cannot hold all 8 hidden tastes.

The vectors are a rank-2 SVD of the true affinities, the best any 2-D two-tower model could do. On a 1M-item catalog the same N is a far smaller share, so recall would be lower.

Assumptions

Retrieval time (ANN search + user tower)10 ms

One item-item similarity in the re-ranker1.0 µs

Ranker cost	0.10 ms per item	Time on one core. Cores busy = requests/s × items scored × cost
Toy world	300 users · 1,000 items	Hidden 8-dimensional tastes plus noise, 8 categories, seed 7. Recall, precision and diversity are measured here, not on 1M items
Relevant items	true top 20	Each user's 20 highest true affinities, the answer key for recall and precision
Two-tower model	rank-2 SVD	The best 2-D dot-product fit to the true affinities. A model trained on clicks would do worse
Ranker noise	σ = 0.16 × √(0.1 ms ÷ cost)	Simulated: a costlier ranker sees more features, so its score is closer to the true affinity
Re-rank pool	top 100	MMR re-orders only the top of the ranked list
Feedback loop	K = 10 · 40 rounds	Simulated, seed 11. The prior counts as K impressions
Targets	recall ≥ 0.8 · P@10 ≥ 0.6 · ≥ 4 categories · ≤ 14 days	Set for this brief, not industry standards
Not modelled	feature fetch, caching, position bias	Retrieval time stands in for network and feature lookups

All constants are assumptions for this brief, set 2026-10-25.

Move Example user and watch the dashed boundary turn with the white arrow. Every item past the line has a higher dot product with that user's vector than any item before it, which is why retrieval can use an approximate nearest-neighbour index instead of scoring items one by one. Rose rings are relevant items the 2-D vectors place behind the line. Adding dimensions would catch more of them; adding candidates catches them at ranking cost.

Step 3 of 6: Ranking

A richer score per item, paid for per candidate

Latency

31 ms

Recall

0.87

Feed P@10

0.72

Categories

4.5

Test days

6

Design for 10M users · 1M items · 1,000 req/s · 100 ms budget
Recall, precision and categories measured on the toy data

Ranker cost per item (cheap to rich)0.10 ms

Example user#1 of 300

Ranker noise σ (simulated)

0.160

Precision@10, all users

0.80

Ranking time

20 ms

Request total

31 ms

The ranker scores each candidate with richer features than retrieval can afford. In this simulation its score is the true affinity plus noise that shrinks as cost rises. At 0.10 ms per item, precision@10, the share of the top 10 that are relevant, is 0.80 on the toy data. Recall caps it: an item retrieval missed cannot be ranked.

Assumptions

Retrieval time (ANN search + user tower)10 ms

One item-item similarity in the re-ranker1.0 µs

Ranker cost	0.10 ms per item	Time on one core. Cores busy = requests/s × items scored × cost
Toy world	300 users · 1,000 items	Hidden 8-dimensional tastes plus noise, 8 categories, seed 7. Recall, precision and diversity are measured here, not on 1M items
Relevant items	true top 20	Each user's 20 highest true affinities, the answer key for recall and precision
Two-tower model	rank-2 SVD	The best 2-D dot-product fit to the true affinities. A model trained on clicks would do worse
Ranker noise	σ = 0.16 × √(0.1 ms ÷ cost)	Simulated: a costlier ranker sees more features, so its score is closer to the true affinity
Re-rank pool	top 100	MMR re-orders only the top of the ranked list
Feedback loop	K = 10 · 40 rounds	Simulated, seed 11. The prior counts as K impressions
Targets	recall ≥ 0.8 · P@10 ≥ 0.6 · ≥ 4 categories · ≤ 14 days	Set for this brief, not industry standards
Not modelled	feature fetch, caching, position bias	Retrieval time stands in for network and feature lookups

All constants are assumptions for this brief, set 2026-10-25.

Drag Ranker cost per item from 0.01 to 0.5 ms and watch the cloud tighten around the diagonal. Each point is one candidate: its true click probability across, the ranker's score up. Points on the diagonal are scored perfectly. Teal rings mark the user's 20 relevant items; any that fall below the dashed line miss the feed. Precision rises fast at first and then flattens, while ranking time grows in a straight line with cost.

Step 4 of 6: Re-ranking for the Feed

Trade a little relevance for variety with MMR

Latency

31 ms

Recall

0.87

Feed P@10

0.72

Categories

4.5

Test days

6

Design for 10M users · 1M items · 1,000 req/s · 100 ms budget
Recall, precision and categories measured on the toy data

Relevance weight λ (1 = rank order, 0 = most diverse)0.70

Items per feed (k)10

Example user#1 of 300

Categories in top 10

4.5

Long-tail share

7.6%

Relevance lost

2.5%

Feed P@10

0.72

Maximal marginal relevance (MMR) builds the feed one slot at a time. Each pick maximises λ × relevance − (1 − λ) × its highest similarity to the items already picked. Here categories rise from 3.6 to 4.5 for 2.5% relevance lost. The long-tail share, items outside the top 20% by popularity, moves only from 8.5% to 7.6%: category diversity is not the same as reaching less popular items.

Assumptions

Retrieval time (ANN search + user tower)10 ms

One item-item similarity in the re-ranker1.0 µs

Ranker cost	0.10 ms per item	Time on one core. Cores busy = requests/s × items scored × cost
Toy world	300 users · 1,000 items	Hidden 8-dimensional tastes plus noise, 8 categories, seed 7. Recall, precision and diversity are measured here, not on 1M items
Relevant items	true top 20	Each user's 20 highest true affinities, the answer key for recall and precision
Two-tower model	rank-2 SVD	The best 2-D dot-product fit to the true affinities. A model trained on clicks would do worse
Ranker noise	σ = 0.16 × √(0.1 ms ÷ cost)	Simulated: a costlier ranker sees more features, so its score is closer to the true affinity
Re-rank pool	top 100	MMR re-orders only the top of the ranked list
Feedback loop	K = 10 · 40 rounds	Simulated, seed 11. The prior counts as K impressions
Targets	recall ≥ 0.8 · P@10 ≥ 0.6 · ≥ 4 categories · ≤ 14 days	Set for this brief, not industry standards
Not modelled	feature fetch, caching, position bias	Retrieval time stands in for network and feature lookups

All constants are assumptions for this brief, set 2026-10-25.

Set λ to 0 and then back to 1. At 0 the feed takes the most dissimilar items it can find, so categories climb towards 8 and relevance collapses. The curve shows the useful range: between about 0.6 and 0.8, categories rise faster than relevance falls. Similarity here is the cosine between hidden item vectors. In production it often comes from item embeddings or simple rules, such as at most two items from one creator.

Step 5 of 6: The Feedback Loop

Clicks only come from what the feed shows

Latency

31 ms

Recall

0.87

Feed P@10

0.72

Categories

4.5

Test days

6

Design for 10M users · 1M items · 1,000 req/s · 100 ms budget
Recall, precision and categories measured on the toy data

Round40

Exploration rate ε (share of slots given to random items)10%

CTR, round 40

42.1%

Impression Gini

0.89

Catalog ever shown

100%

CTR with ε = 0

46.2%

Giving 10% of slots to random items costs 4.2% of CTR in round 40 and shows 100% of the catalog instead of 8% (simulated). The Gini coefficient of impressions, 0 when every item is shown equally and 1 when one item gets them all, falls from 0.97 to 0.89. The next model trains on those wider logs.

Simulated. Each estimate starts at the two-tower prediction and updates from that user's clicks: (clicks + 10 × prior) ÷ (impressions + 10). P(click) is the true affinity.

Assumptions

Retrieval time (ANN search + user tower)10 ms

One item-item similarity in the re-ranker1.0 µs

Ranker cost	0.10 ms per item	Time on one core. Cores busy = requests/s × items scored × cost
Toy world	300 users · 1,000 items	Hidden 8-dimensional tastes plus noise, 8 categories, seed 7. Recall, precision and diversity are measured here, not on 1M items
Relevant items	true top 20	Each user's 20 highest true affinities, the answer key for recall and precision
Two-tower model	rank-2 SVD	The best 2-D dot-product fit to the true affinities. A model trained on clicks would do worse
Ranker noise	σ = 0.16 × √(0.1 ms ÷ cost)	Simulated: a costlier ranker sees more features, so its score is closer to the true affinity
Re-rank pool	top 100	MMR re-orders only the top of the ranked list
Feedback loop	K = 10 · 40 rounds	Simulated, seed 11. The prior counts as K impressions
Targets	recall ≥ 0.8 · P@10 ≥ 0.6 · ≥ 4 categories · ≤ 14 days	Set for this brief, not industry standards
Not modelled	feature fetch, caching, position bias	Retrieval time stands in for network and feature lookups

All constants are assumptions for this brief, set 2026-10-25.

Press Play at ε = 0, then again at 10%. Without exploration, coverage stalls below 10% of the catalog in this simulation, because an item the model never shows can never be clicked, and its estimate never changes. Exploration is the cheapest fix: a small share of slots shows items the model is unsure about. Bandit methods choose those slots more carefully than random picks, and off-policy corrections reweight the logged clicks when the next model is trained.

Step 6 of 6: Offline vs Online, and Review

A/B test length and the design against the brief

Latency

31 ms

Recall

0.87

Feed P@10

0.72

Categories

4.5

Test days

6

Design for 10M users · 1M items · 1,000 req/s · 100 ms budget
Recall, precision and categories measured on the toy data

✓Latency retrieval + rank + re-rank ≤ 100 ms31 ms

✓Recall ≥ 0.8, toy data0.87

✓Feed precision@10 ≥ 0.6, toy data0.72

✓Diversity ≥ 4 categories, toy data4.5

✓Test length ≤ 14 days6 days

Baseline rate p₁ (share of users who click)5.0%

Minimum detectable effect (relative lift)2.0%

Significance level α (two-sided)0.05

Power (1 − β)80%

Daily users (share of registered users)3M (30%)

Traffic share in the test (both arms)10%

n = (1.96 + 0.84)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂ − p₁)², p₂ = 5.10%
days = 2 × 752,700 ÷ 300,000 a day = 5.0

Users per arm

753k

Test users per day

300k

Test days

6

Every target is met: 31 ms per request, recall 0.87 and feed precision 0.72 on the toy data, 4.5 categories per feed, and a 6-day test. Offline metrics such as recall and precision are computed on logged data, which only holds what the old model showed. The online A/B test splits live users between the old and new design and is the final check.

Assumptions

Retrieval time (ANN search + user tower)10 ms

One item-item similarity in the re-ranker1.0 µs

Ranker cost	0.10 ms per item	Time on one core. Cores busy = requests/s × items scored × cost
Toy world	300 users · 1,000 items	Hidden 8-dimensional tastes plus noise, 8 categories, seed 7. Recall, precision and diversity are measured here, not on 1M items
Relevant items	true top 20	Each user's 20 highest true affinities, the answer key for recall and precision
Two-tower model	rank-2 SVD	The best 2-D dot-product fit to the true affinities. A model trained on clicks would do worse
Ranker noise	σ = 0.16 × √(0.1 ms ÷ cost)	Simulated: a costlier ranker sees more features, so its score is closer to the true affinity
Re-rank pool	top 100	MMR re-orders only the top of the ranked list
Feedback loop	K = 10 · 40 rounds	Simulated, seed 11. The prior counts as K impressions
Targets	recall ≥ 0.8 · P@10 ≥ 0.6 · ≥ 4 categories · ≤ 14 days	Set for this brief, not industry standards
Not modelled	feature fetch, caching, position bias	Retrieval time stands in for network and feature lookups

All constants are assumptions for this brief, set 2026-10-25.

Lower Minimum detectable effect from 2% to 1% and the test length roughly quadruples, since the sample size grows with one over the squared difference. Raising the traffic share shortens the test but puts more users on an unproven design. Each user counts once, in one arm, so a test that runs for days also has to span weekly patterns. Offline recall and precision are cheap to compute and pick candidates; the A/B test decides which one ships.

How is a recommendation feed designed?

A recommendation feed picks a handful of items for each user from a catalog far too large to score in full. Production systems split the work into stages: retrieval narrows millions of items to a few hundred candidates, a ranker scores those candidates carefully, and a re-ranker reorders the top of the list for variety and business rules. Each stage spends its share of a latency budget, usually around 100 ms per request.

This case study designs a feed for 10 million users and 1 million items at 1,000 requests per second. Latency, compute and A/B test length are exact arithmetic. Recall, precision and diversity are measured on a seeded toy world of 300 users and 1,000 items whose true tastes are known, so every measurement can be checked against the answer key. The ranker's accuracy and the feedback loop are simulations, labelled as such on the page.

Key ideas

A ranker that is affordable for a few hundred items is far too slow for a million, so feeds retrieve candidates first and rank only those.
A two-tower model gives users and items vectors whose dot product predicts affinity, which turns retrieval into a nearest-neighbour search.
Recall at retrieval caps everything after it: an item that is not retrieved cannot be ranked or shown.
Ranker cost per item multiplies by the candidate count, so richer rankers and more candidates compete for the same latency budget.
Maximal marginal relevance trades a little relevance for variety; category diversity does not by itself reach less popular items.
A feed only collects clicks on what it shows, so without exploration the logs, and the next model, narrow to a small part of the catalog.
Offline metrics choose which designs to test; an A/B test sized for the smallest effect worth detecting decides which design ships.

Common questions

Why do recommendation systems use two stages, retrieval and ranking?

Scoring every item with a rich model costs catalog size × cost per item: 1 million items at 0.1 ms each is 100 seconds per request. Retrieval uses a cheap model whose scores can be searched with an index, so it returns a few hundred candidates in milliseconds. The rich ranker then spends its budget only on those. YouTube's 2016 system described this split, with candidate generation feeding a separate ranking network.

What is a two-tower model?

Two neural networks, one for users and one for items, each map their input to a vector of the same size. The predicted affinity is the dot product of the two vectors. Item vectors can be computed ahead of time and stored in a vector index, so at request time only the user vector is computed and the index returns the items with the largest dot products.

How long should a recommendation A/B test run?

Long enough to reach the sample size for the smallest effect worth detecting. With a 5% baseline click rate, a 2% relative lift, α = 0.05 and 80% power, each arm needs about 753,000 users. At 300,000 test users a day that is 6 days. In practice it is common to run whole weeks, so weekday and weekend behaviour are both included.

Why do recommendation feeds keep showing popular items?

Clicks only come from items the feed shows, and the model learns only from those clicks. Items shown early collect evidence and keep their place, while items never shown keep whatever estimate they started with. In this case study's simulation, a feed with no exploration ever shows under 10% of the catalog. Giving a small share of slots to exploration costs some click-through rate and widens what the next model can learn from.

Related explainers

Embeddings — Turning words and categories into vectors
Vector Search & HNSW — Finding nearest embeddings fast with HNSW
K-Nearest Neighbors — Classification by proximity and distance metrics
Gradient Boosting — Sequential residual fitting for powerful predictions
Model Evaluation & Metrics — Precision, recall, F1, ROC curves, and regression metrics
Class Imbalance & Resampling — SMOTE, class weights, and metrics beyond accuracy
Hypothesis Testing — Make principled decisions from data using null hypotheses, p-values, and significance levels

References

Last updated October 25, 2026