Bridging the Gap: From ML Metrics to Business Value in Data Projects

As people who operate in the data practice, we inevitably find ourselves having to agree on what success means in our projects and what actually drives value. A major misconception among data professionals, consultants and engineers, is that the problem of business value is a problem for your clients. The truth is that, understanding how value gets created and how success ought to be measured is as much of an analytics problem as it is a business one. If data professionals cannot identify how value is created or clearly articulate what distinguishes a successful outcome from an unsuccessful one, then it is very hard to actually solve problems in any meaningful way.

One of the reasons for this difficulty is the difference in the types of metrics we use in both the domains of analytics and business. For the sake of simplicity, we can call these ML metrics and Business metrics. Suppose you are working on building an improved model to help generate demand forecasts for a business, and you need to identify how much value is being driven by these better forecasts. One way to approach the problem would be to use a series of ML metrics such as the Symmetric mean absolute percentage error (sMAPE), Mean Absolute Error (MAE), or Bias. You could then compare how your new and improved model, bettered previous forecasting mechanisms by observing the deltas across each of these metrics. Now to a data professional, such comparisons and techniques for determining the value of the developed solution aren’t in the least bit objectionable, and are generally thought to be scientific and objective measures of value. Furthermore, these metrics are quite easy to measure for data scientists.

However, if we take a step back and try to adopt a customer’s perspective, we find ourselves facing difficulties in translating success across these metrics, into business performance. In other words, there is an issue in attempting to translate ML metrics, into business metrics. You can imagine presenting your metrics to a product owner or stakeholder in the business, only to be asked, “so how much money does this save me each year?”. Such questions do not come about due to some primitiveness in the minds of non-data geeks, but are simply an attempt to cut through the noise and understand how the delivered solution actually creates value for the business.

It should be noted that business metrics are often very difficult measure, and require some very complex calculations. In the case of demand forecasting, if you were to try to measure the cost savings that would come about due to better forecasts, you would have to identify all the possible ways in which more accurate forecasts would
make business operations more efficient, for example: allowing outlets to have sufficient inventory, have sufficient trucks or vehicles for order deliveries, and to enable better rostering. When producing more efficient rosters for example, you would need to break down how forecasts were converted into a roster, and then identify how many hours were being saved. Hence, trying to accurately frame success through a series of business metrics that are specific to your problem, for example: labour savings or order profitability, is a process which requires a solid understanding of your client’s business and value chain, such that you can then present the benefits of your solution in the context of that business. This process is tedious, but is necessary if you care about driving real value.

None of this is to say that ML metrics are useless and ought to be discarded whenever one is operating in a business context. ML metrics have a very important role in validating the outcomes of any and all ML development. However, imagine that instead of simply compiling a range of ML metrics, and setting some benchmarks for how much better you’d like the model bias to be at the end of development, you decided to measure success by how much you could help your clients save, by translating improved forecasting accuracy into dollars saved. This slight change, would allow you to make significant progress in conveying to your clients that you understand what value means to them and what a successful outcome looks like.