Have you ever come across a scenario where a team of data scientists are hard at work building an AI model to solve a business problem within a large organization? What follows is a series of back and forths where the team accesses the data, analyzes it, identifies data quality issues, cleans it, and builds the required AI model functionality. However, despite all the investment of resources, time and money, the model remains inaccurate. Sound familiar?

An analysis of this problem revealed a glaring shortcoming – DATA QUALITY! The team believes that this whole AI model process would have had an impact with a faster turnaround time if a full data quality report had been shared with them. While many data scientists would have faced a dilemma like this, what lags behind is the acceptance that data has always been an underestimated factor when working with AI.

The importance of data quality

Going from our first example, if the system was fed with poor quality or inaccurate data, the outcome of real-time decisions made based on that inaccuracy was serious. One of the reasons data issues are discovered through trial and error is that there is a lack of automated tools and methods for AI developers to assess data quality, keep a log of all the changes applied to the data and to write programs to solve each problem found. The challenge is always to effectively examine multiple data sources, analyze the relevant data, and then transform it into the required model.

In the age of artificial intelligence and automated decisions, data quality is critical and a prerequisite for building a strong automated system. Data quality must be integrity, accurate, valid, consistent, and today’s systems must be aware of the potential issues they may face due to the lack of robust data. Data scientists also need to build a systematic study to improve data quality before moving to the modeling stage.

The problem of data bias

Data bias is typically an error that occurs in machine learning in which some data is weighted or represented more than others, misrepresenting a model and causing a biased result, error, or low precision. It also implies that the data is the oxygen needed for the model to do its job accurately. Data bias can be of several types, some of which include:

– Sample bias: when the data does not reflect the real environment of the model
– Measurement bias: occurs when there is a discrepancy between data collected for training and real-world data
– Association bias: typically occurs when data from a machine learning model reinforces cultural bias

As data scientists try to solve problems through manual analysis, it continues to be an extremely time-consuming and difficult process, which delays the development of AI models. This requires automating and creating algorithms to evaluate data in different ways, suggest recommendations to improve data quality, and automatically generate code to execute those recommendations.

Make data a first-class citizen in the AI ​​journey

It is well established that data is the backbone of an AI model and critical to its success. Now is the time for industry and academia to raise the bar and elevate data quality to a first-class citizen and build an accepting ecosystem.

– B2B: commercial products in the market must ensure that their products include data quality for the AI ​​matrix so that their customers can effectively use these methods to improve data
– Organizations or researchers should make their APIs or toolkits available to general developers and student communities for more hands-on experience with first-hand problems and understanding how to solve them
– More importantly, it is essential to engage academia to include topics such as data quality, data preparation, data lifecycle as part of its core curriculum in their AI course to train and develop the right talents

AI has its life cycle and sometimes AI fails. However, one can avoid problems and failures with AI if it is built on a solid foundation of data quality. It’s time for CIOs to kickstart the conversations and bring data quality to the table and encourage the need for strong data to avoid gaps in their AI journey.

Views are personal. The author is IBM Distinguished Engineer and Lead-Data and AI Platforms, IBM Research India.