AI System Design Basics for Working Professionals
Building a model is maybe 20% of the job. The other 80% is everything around it — getting data in, serving predictions fast enough, and knowing when the system is quietly breaking. Here are the building blocks I walk every mentee through.
1. Data flow comes first
Before models, map how data moves: sources → ingestion → storage → features. Decide early whether you need batch (periodic jobs, tolerant of delay) or streaming (real-time, event-driven). This single choice cascades through the entire architecture.
2. Features are the contract
The most common production bug isn't the model — it's training/serving skew, where features are computed differently offline and online. A feature store (or at least shared feature code) keeps both paths consistent. Treat feature definitions as a versioned contract.
3. Serving: match the pattern to the need
- Online / real-time — low-latency API behind a request (fraud checks, recommendations).
- Batch — score millions of rows on a schedule (churn scores, nightly ranking).
- Streaming — continuous scoring on an event pipeline.
Always state the latency budget explicitly. "P99 under 100ms" changes your model, hardware, and caching decisions.
4. Monitoring is not optional
A model that was 95% accurate at launch can silently rot. Monitor three layers:
- System — latency, error rate, throughput.
- Data — input drift, missing features, schema changes.
- Model — prediction distribution and, where you have labels, live accuracy.
5. Close the loop
Design retraining from day one: how labels arrive, how often you retrain, and how a new model is validated before it replaces the old one. Shadow deployments and canary rollouts let you ship safely.
A good AI system is judged not when it launches, but six months later — when the data has shifted and it's still trustworthy.
Master these five blocks and you can reason about almost any AI system design question — in interviews and, more importantly, in production.