Design ML Systems That Work in Production, Not Just Notebooks
ML system design interviews test whether you understand the full production lifecycle of a machine learning system — from data ingestion and feature engineering through model training, serving, monitoring, and retraining. Most candidates overweight the modeling component and underweight the infrastructure. The candidates who impress interview panels can explain why a feature store matters and when to use real-time vs. batch inference.
Lead with the problem framing and training/serving split. Define the feedback loop and monitoring strategy before deep-diving into model architecture — that's what separates ML engineers from ML researchers in these interviews.
Of ML models never reach production due to infrastructure gaps
Gartner researchOf ML production failures are caused by training-serving skew
Industry survey dataMedian base salary for Senior ML Engineers at growth-stage tech companies
Levels.fyi dataIs this guide for you?
Use this Good fit if you…
- ✓You're targeting Senior ML Engineer, Applied Scientist, or ML Platform roles
- ✓You've built models but haven't designed end-to-end ML systems
- ✓Your ML system design rounds stall after the modeling discussion
Skip Not the right fit if…
- ✗You're targeting pure research roles where system design isn't evaluated
- ✗You're focused on data engineering without an ML component
- ✗You're already converting ML system design rounds consistently
The playbook
Five things to do, in order.
Frame the ML problem before the model
What are you predicting, what are you optimizing, what's the feedback loop? "We're predicting next-item purchase, optimizing for 7-day revenue, and feedback comes from purchase events with 24h delay." That frame drives every downstream decision.
Separate the training and serving architectures
Training pipeline: data ingestion → feature engineering → model training → evaluation → versioning. Serving: feature retrieval → model inference → output post-processing → logging. These are different systems with different constraints.
Design the feature store explicitly
Decide which features are batch (computed offline) and which are real-time (user context, session data). Explain the consistency requirement — training-serving skew is the most common ML production failure.
Define your model monitoring strategy
Data drift, concept drift, prediction distribution shift, business metric degradation. Know which signals trigger automated retraining vs manual review. Most system design answers skip this entirely.
Discuss the retraining trigger and cadence
Schedule-based vs performance-triggered retraining. "We retrain weekly on a rolling 90-day window, with an automated performance check that triggers emergency retraining if AUC drops below 0.82." Specificity here shows production experience.
See the transformation
"I'd use a recommendation model with collaborative filtering and serve it via an API."
"For a real-time product recommendation system at 10M DAU, I'd use a two-stage retrieval + ranking architecture. ANN retrieval (Faiss) narrows 10M items to 500 candidates in <10ms; a LightGBM ranker scores candidates using 50 user + item features from Redis feature store. Training pipeline runs daily in Spark on 90-day event window. Monitoring: CTR degradation >15% triggers alert; AUC <0.80 triggers retraining. Training-serving skew managed by shared feature computation layer."
Questions people ask
How do I prepare for ML system design if I work primarily on research?
Study production ML case studies from Uber, Netflix, Airbnb engineering blogs. Focus on the parts you don't do: feature stores, model serving latency, A/B testing infrastructure for models, and monitoring.
When should I choose real-time vs batch inference?
Real-time when the feature freshness matters for prediction quality (e.g., session context, recent behavior). Batch when predictions can be precomputed and freshness requirements are loose (e.g., daily email personalization). Lead with the latency and freshness requirements, not the model type.
Ready to put this into practice?
Get personalized coaching for your ML & AI Engineering job search — resume, interviews, and offer strategy tailored to you.
