Subscription churn scoring with scikit-learn and MLflow

The scenario

Imagine a subscription app whose product team wants to understand which accounts may stop renewing. A churn score could help prioritize product research or a customer-success review. This example would begin with an offline model and a clearly defined prediction window, rather than automatically sending retention offers.

Define the question before choosing a model

The team would first agree what counts as churn and when a prediction is made. For example, a snapshot at the beginning of a renewal period could use only activity available before that snapshot. A cancellation recorded afterwards belongs to the label, not to the input features.

A dataset would include activity recency, feature usage, account tenure, and plan information where appropriate. Each feature would have an availability timestamp. Accounts with incomplete observation windows would need explicit treatment so that a still-active account is not accidentally labelled as a retained one too early.

Compare a simple baseline on a future period

A chronological holdout would better resemble a future release than a random split across all snapshots. Repeated snapshots from the same account would require additional checks to prevent identity leakage. Preprocessing would be fitted only on training data and packaged with the estimator in a scikit-learn Pipeline.

Begin with a transparent baseline such as logistic regression, then compare a more flexible model only if it improves the chosen validation measure. Precision at a review-team capacity, recall, and calibration answer different questions. MLflow could record the dataset version, feature definitions, parameters, and evaluation artifacts for each candidate.

Release scores without automating the decision

The first scoring run could be a scheduled batch. FastAPI would become useful if a product genuinely needs on-demand scoring; it would accept a versioned feature schema and return the model version with each result. Missing features would produce a clear fallback instead of a plausible-looking default score.

A shadow rollout would compare score distributions and feature availability before exposing a review queue. Later monitoring would compare predictions with outcomes once labels mature. A model release should be reversible independently of the application release.

What to validate

• Every feature existed at prediction time.
• The evaluation period occurs after the training period.
• Saved preprocessing and model artifacts reproduce a score.
• Reviewers understand the score’s limits and can ignore it.

The tradeoff

A churn score is not evidence that an intervention causes retention. Product decisions would still need a separate experiment, and a simple inactivity rule may be more useful when there is little labelled history.