Data science models are only useful if woven into the product roadmap strategically. And before they can be integrated, they need cross-departmental buy-in. That’s why for Charley Frazier, head of data and machine learning at Homebound Technologies, building successful products often requires expressing evaluation metrics as business outcomes. Frazier said that capturing the support of important stakeholders who aren’t fluent in data science language requires translating regression terminology into more general, results-oriented terms.
“Your CEO doesn’t care about an ‘area under the curve (AUC)’ of .83, but she will be excited to fund future projects if you improve conversion rates by five percent,” he said.
Evan Lodge, head of product at Vivian Health, notes another challenge — end-user wariness of new technology, and customers’ comfort level with handing over important decision-making to algorithms. For the data scientists who work with the details and build the models, it is clear that the numbers can perform more consistently than humans. However, “it’s hard to get people to trust a system for which the inputs and algorithms are hard to describe,” he said.
Below, the two data science leaders further explain how they empower their teams and consequently, their companies.
Describe a project you recently worked on that incorporated data science in its core functionality. What was the product designed to do?
Our machine learning team has recently built a suite of home valuation models to predict expected site acquisition cost, expected build costs, and expected sell price of the developed home. These three metrics allow us to determine the feasibility of acquiring a property.
Internally, we built a robust property data asset containing more than 500 data points for more than 10 million properties, enabling us to build highly predictive machine learning models. We can use the output of these models to identify the best new markets to enter, the optimal development areas within those markets, and tens of thousands of attractive properties to acquire within these development areas.
What are some of the unique challenges posed to product managers when building AI, machine learning or other data science technologies directly into a product?
First, connecting evaluation metrics to a business outcome. Your CEO doesn’t care about an AUC of .83, but she will be excited to fund future projects if you improve conversion rates by 5 percent.
Second, uncertainty if your product will ultimately be successful. It can often be difficult or impossible to know if your product will be successful until a model can be built, deployed, and evaluated in the wild.
Third, maintaining high-quality models and predictions in an ever-changing production environment. Training machine learning models is only a tiny fraction of successful data science projects, so the need to measure and maintain (or hopefully improve) performance over time is imperative.
How can product managers structure their projects to account for these challenges?
There are a few ways to mitigate these challenges for successful product development.
One method is to accelerate the path to production! Reduce the investment it takes to build new products by establishing core functionality that all data scientists can leverage. Investing in feature stores, utility libraries, deployment frameworks and model management tools will enable your teams to deploy models in days versus months. Work these platform improvements into your roadmap.
It is also important to empower your data scientists with as much business context as possible. They can do more than code! Ensure they know the metrics or outcomes you are trying to drive, solicit ideas for new products, and bring them to your cross functional meetings. You won’t be disappointed with the results.
Finally, work backward from your team’s goals. It’s often tricky to estimate how long data science projects will take. Set milestones along the way, pivot or iterate when you start seeing diminishing results, and constantly (re)prioritize your roadmap to help keep your team on track.
Describe a project you recently worked on that incorporated data science in its core functionality. What was the product designed to do?
Vivian now uses machine learning to predict which applicants are most likely to respond to, and get hired for, specific jobs. Inputs into the model include applicant site behavior, job requirements, job preferences and more. This model was recently used to improve an offering that allows employers to search our database and proactively reach out to healthcare applicants. Using machine learning in this way also allows us to drastically improve positive response rates from applicants.
What are some of the unique challenges posed to product managers when building AI, machine learning or other data science technologies directly into a product?
Convincing clients to trust their livelihood with an algorithm is challenging. While the numbers prove that our algorithms perform better than humans, it’s hard to get people to trust a system for which the inputs and algorithms are hard to describe. Machine learning can often seem like a black box, and it can be a challenge to design these products so that they can be trusted by our users. Designers become very important in helping to overcome this communication barrier. For this specific feature, adding visual queues that highlight points of explanation for results is key.
How can product managers structure their projects to account for these challenges?
The first step to making any feature like this successful is close collaboration with customers. At Vivian, we bring customers into the product development process as soon as possible. Once we understand what needs to be communicated, data and design teams figure out how to address concerns visually. Collaboration between data and design teams is the key to successfully helping customers understand and trust the results of machine learning output.
