The Most Expensive Data Science Mistake I’ve Witnessed in My Career

Why true success in machine learning goes beyond optimizing a single metric

Image by Clker-Free-Vector-Images from Pixabay

“The new credit model didn’t perform as expected,” he said.

My colleagues from another team came out from the “war room” looking dejected.

This was my first time witnessing a data science “mistake” in a business context, so I didn’t quite understand the scale of it.

As I would learn over the next six months, this was a very costly error from both a financial and operational perspective, because of the downstream impact and reverberating effects it had across the company.

This issue impacted every team and we all worked tirelessly for months after:

• Analysts worked on metrics to track portfolio health,
• Data scientists worked on diagnosing the issue and patching it,
• Engineers worked with data scientists to deploy the new model,
• Product marketers worked on campaigns and messaging,
• Operational folks worked on customer outreach and education on repayment.

The first sign of trouble was much higher default rates relative to expectations, but this was just the…