The $100 Mistake: Saving Customer Relationships with Cost-Sensitive ML
In the world of subscription-based businesses, there is a fundamental law of economics: acquiring a new customer is 5 to 25 times more expensive than retaining an existing one. When a customer leaves (churns), it’s not just a loss of future revenue—it’s a marketing failure that requires a heavy investment to fix.
In this project, we explored how to move beyond standard "accuracy" to focus on what matters most: the bottom line.
The Strategy: Putting a Dollar Value on Mistakes
Most machine learning models treat every error the same. But in the real world, missing a customer who is about to leave is much costlier than sending a discount code to someone who was planning to stay.
We designed a custom loss function based on a 10:1 cost ratio:
False Negative ($100): You predict a customer will stay, but they leave. Cost: Loss of customer + high acquisition cost for a replacement.
False Positive ($10): You predict a customer will leave, so you offer them a $10 retention incentive. If they were staying anyway, you "wasted" $10.
By optimizing for cost rather than just accuracy, our models became more "aggressive." We identified an optimal classification threshold of 11% (for our blackbox model)—meaning if there is even an 11% chance a customer might leave, it makes financial sense to intervene.
The Data & Technical Pipeline
We utilized the Telco Customer Churn dataset (7,043 records) to build our solution. Our preprocessing was designed to ensure the highest possible model stability:
Cleaning: We fixed datatype inconsistencies and handled missing values in the
TotalChargescolumn.Transformation: We used One-Hot Encoding for categorical variables (like contract type) and StandardScaler for continuous variables (like tenure and monthly charges).
Split: An 80/20 train-validation split ensured our results were generalizable.
The Showdown: Glass-box vs. Blackbox
We wanted to see if complex "Blackbox" models would significantly outperform interpretable "Glass-box" models.
| Model Category | Champion Model | Performance (Min Loss) | Key Takeaway |
| Glass-box | Scaled Logistic Regression | $7,090 | Feature scaling was the secret sauce for stable L2 regularization. |
| Blackbox | Optimized XGBoost | $7,090 | Captured non-linear patterns that linear models missed. |
Interestingly, both models arrived at the same minimum cost, but the path there was different. We also discovered that training with Surrogate Loss (Log Loss) consistently outperformed standard Mean Squared Error for this binary task.
Why Do They Leave? (Insights via SHAP)
Predicting churn is great, but understanding it is better. We used SHAP (SHapley Additive exPlanations) to peek inside the XGBoost model and identify the biggest churn drivers.
Who Stays? Customers with Two-Year Contracts and high tenure are the most loyal. Long-term commitments are the strongest defense against churn.
Who Leaves? Customers using Fiber Optic internet and those paying via Electronic Check were at significantly higher risk. This suggests potential issues with service reliability or friction in the payment experience.
The "Noise": Factors like gender or having device protection had almost no impact on the final prediction.
The Bottom Line: Real Financial Impact
By shifting our focus from "Standard F1-Score" to "Minimum Business Cost," the results were clear:
Financial Savings: We saved $2,940 (approx. 29%) compared to a traditional balanced approach.
High Sensitivity: Our final model achieved a 95% recall. In other words, we caught 19 out of every 20 potential churners before they walked out the door.
Final Thoughts
Data science is most powerful when it speaks the language of the business. By aligning our model's objective with the actual financial penalties of churn, we created a tool that doesn't just predict the future—it protects the company's revenue.
Dataset Source: Kaggle Telco Churn Further Reading: The Value of Keeping the Right Customers (HBR)
