How to scope a predictive analytics or machine learning project

Name the decision a prediction would change

A forecast or prediction is valuable when it helps someone make a better decision. Before commissioning machine learning development services, identify that decision, the available information and what a wrong prediction would cost in extra work or missed opportunities.

The first comparison should include the business's current method. A complex model needs to show a useful improvement over a simpler rule or an existing planning process.

Read the source: Google: Machine Learning Crash Course, Overfitting.

Define what the prediction changes

Machine learning consulting should establish the outcome, who uses it and the action it may influence. Decide how quickly the information must arrive and which decisions still need a person.

For predictive analytics consulting, ask whether the proposed result is a forecast, a ranking, a warning or an explanation of past activity. Those outputs have different acceptance tests. If you need a forecast, compare predicted outcomes against what actually happened in a later period.

Check whether the records support the question

Review the available history, missing values and the way outcomes were recorded. Determine whether earlier decisions changed which results the business can observe. Keep information that would not be available at prediction time out of the input.

Demand forecasting development should account for how the business defines demand and the events that change the meaning of its history. Sales made while an item was available may tell a different story from unmet requests during a shortage.

Evaluate on information the model has not learned

Google’s machine learning guidance explains how a model can perform well on examples it learned from and poorly on new ones. Ask the supplier to describe the separate examples used to evaluate the result.

For an illustrative forecasting test, use an earlier period to develop the approach and a later, withheld period to compare its predictions with the observed result. Choose a method that reflects how the system will actually be used, and document what the comparison cannot establish.

Match the measure to the action

Recommendation system development should examine whether the suggestions help users make a suitable choice. Check suggestions for less common items and set a useful default for new users and products.

Anomaly detection development should balance useful warnings against investigation load. A system that flags nearly everything may identify unusual records while giving the team more work than it removes. Agree how alerts will be reviewed and what makes one worth acting on.

Plan for changing conditions

After launch, monitor both prediction quality and the resulting business process. New products, changed customer behavior and different data collection can alter the conditions the model learned.

Assign an owner for reviewing performance, correcting records and deciding when the system needs a change. Keep a practical fallback and enough documentation to explain how a prediction was produced.

AI App Studio can help assess a prediction problem and build a useful first test. Book a call with the decision, existing method and available records.