Data Professionals Spend Two-fifths of Time Fighting Bad Data

Data professionals spend a staggering 40 percent of their time evaluating or checking data quality, according to a new report from Monte Carlo, which also found that poor data quality impacts a quarter (26%) of company revenue.

The State of Data Quality 2022

“The state of data quality 2022” study was conducted by Wakefield Research, which surveyed more than 300 data professionals about the number of data incidents and how long it takes to detect and resolve them, their impact on the business, and trends in the life of the data engineer.

As explained by Monte Caro, the majority (75%) of respondents take four or more hours to detect a data quality incident; half said it takes an average of nine hours to resolve a data issue once identified.

Moreover, with more data to handle, data pipelines growing more complex, and larger teams, 58 percent said the total number of data incidents has increased “somewhat or greatly” over the past year.

Fun fact: Data engineers at organizations with more than 500 people were more likely to spend more time answering data questions. Unsurprisingly, almost every organization surveyed (90%) said they will invest or plan to invest in improving their data quality over the next six months.

“The first step to improving data quality and trust is measuring it, starting with the number and type of incidents, and setting baselines on response rates and data downtime. Don't be surprised to find the number of incidents is greater than expected after putting in place an observability platform,” said Shane Murray, the field CTO at Monte Carlo in the report.

“Scaling access to data across a large organization requires standards for data certification and reliability, to ensure your democratized data is also trustworthy,” he said.

The survey was conducted by polling the 300 data engineers working in the US over two weeks in April and May, using an email invitation and an online survey.

Of course, it should be noted that Monte Carlo bills itself as a data reliability company that touts a data reliability platform designed to monitor and offer alerts for missing or inaccurate data, so it would benefit from having you think that.

The full report can be accessed here (pdf).

Image credit: iStockphoto/brebca