A model in a notebook creates zero value. Value comes from a model that serves predictions reliably, cheaply, and correctly — and keeps doing so as data shifts. This is software and systems engineering with a few ML-specific twists, which means your existing strengths (APIs, nginx, OAuth, containers, CI/CD) transfer almost wholesale. This is the phase where you can differentiate fastest from people who can only train models.

Experiment tracking & reproducibility

Before deployment comes the discipline of knowing what you trained and how. Log every run’s parameters, metrics, code version, data version, and artifacts.

• MLflow (open-source, self-hostable) or Weights & Biases (hosted, great UI) for tracking and the model registry.

• Reproducibility hygiene: pin dependencies, set random seeds, version data (DVC or dataset hashes), and capture the git SHA with each run.

import mlflow
with mlflow.start_run():
    mlflow.log_params({"lr": 1e-3, "depth": 6})
    mlflow.log_metric("val_auc", auc)
    mlflow.sklearn.log_model(model, "model")     # versioned artifact
    mlflow.set_tag("git_sha", current_git_sha())

Packaging & serving

Inference patterns

Serving frameworks

Wrap the model in an API (FastAPI/Flask) for full control, or use a dedicated server (TorchServe, Triton, BentoML, KServe) for batching, versioning, and metrics out of the box. Export to ONNX for a framework-agnostic, optimized runtime.

from fastapi import FastAPI
from pydantic import BaseModel
import torch

app = FastAPI()
model = torch.jit.load("model.pt").eval()        # TorchScript artifact

class Req(BaseModel):
    features: list[float]

@app.post("/predict")
def predict(req: Req):
    x = torch.tensor([req.features])
    with torch.no_grad():
        prob = model(x).softmax(-1)[0].tolist()
    return {"probabilities": prob}

@app.get("/health")
def health():                                     # liveness/readiness probe
    return {"status": "ok"}

Containerization

Docker is the unit of deployment. For ML, mind image size (multi-stage builds), CUDA/driver compatibility for GPU images, and pinning the exact inference dependencies. The same container runs in CI, staging, and production — eliminating environment skew.

Versioning & registries

A model registry (MLflow Registry, W&B, SageMaker) is the source of truth: it stores versioned model artifacts with stage labels (staging/production), lineage (which run/data produced it), and enables one-click rollback. Treat model promotion like a deploy: reviewed, gated, and reversible.

Monitoring

Once live, a model needs more monitoring than ordinary services, because it can fail silently.

• Operational: latency, throughput, error rate, resource use (the usual SRE signals).

• Data drift: input distribution shifts from training (monitor feature statistics; tests like PSI or KS).

• Concept drift: the input\(\to\)output relationship changes (the world moved; yesterday’s patterns no longer hold).

• Performance: the real metric (accuracy/AUC) once ground-truth labels arrive — often delayed, so use proxies meanwhile.

• Prediction monitoring: output distribution, confidence, and (for LLMs) safety/quality checks.

Optimization for production

Making models fast and cheap is often as valuable as making them accurate.

• Quantization: use int8/fp16 instead of fp32 — big speed/memory wins with small accuracy loss (post-training or quantization-aware).

• Distillation: train a small “student” to mimic a large “teacher” — much smaller, nearly as good.

• Pruning: remove redundant weights/structures.

• Graph/runtime optimization: ONNX Runtime, TensorRT (fuse ops, pick optimal kernels).

• Batching: dynamic batching of concurrent requests raises throughput dramatically (tune against your latency SLO).

Feature stores & pipelines

A feature store (e.g. Feast) centralizes feature definitions and serves them consistently to both training (offline) and serving (online) — the standard cure for training/serving skew. You do not need one for small projects, but understand the concept: “compute a feature once, use it everywhere, consistently.”

CI/CD for ML

Extend familiar CI/CD with ML stages:

Add: data validation (Great Expectations), automated retraining triggers (on drift or schedule), a quality gate (don’t deploy if the candidate underperforms the incumbent on a held-out set), and progressive rollout (canary / shadow / A-B, connecting back to Phase 1).

Checkpoint project

Phase 7 self-check

• Explain why ML systems version data + code + model, not just code.

• Choose an inference pattern (batch/online/streaming/edge) for a given use case.

• Stand up a containerized prediction API with health checks.

• Distinguish data drift, concept drift, and training/serving skew, and say how you’d detect each.

• Pick an optimization (quantization/distillation/batching) for a latency or cost target.

• Describe a CI/CD pipeline for ML with a quality gate and a retraining trigger.