We Need to Stop Lying About Being Data-driven

Every company is in a race to become data-driven. With human movement hampered by an increasing list of restrictions and markets recalibrating themselves to a new normal, data offers the only refuge for getting better insights.

The benefits of a data-driven organization have been clear for several years. Yet, when you ask data scientists and data workers, you feel becoming data-driven is still a near impossible goal.

Why? Rob Woollen, chief technology officer and co-founder of Sigma Computing blames poor design, bad data quality, difficult data access and the disappointment of self-service business intelligence (BI).

Design for humans

The first challenge is design. While advanced data tools fill the market, many data workers still rely on spreadsheets.

“Interestingly, our research shows that 93% of domain experts, and even 88% of data experts, use Excel or Google Sheets for data analysis,” said Woollen. 

He noted that design plays a critical role in creating a tool that is powerful enough to satisfy the needs of data experts and easy enough for business users.

“This requires a tremendous amount of empathy, especially when users come from such a wide variety of backgrounds and experience levels. User experience and interface designers really need to understand people before they design, or even use, a product,” he added.

Sigma, Woollen alleges, takes a “very human-centric approach.” “We know that people are smart, curious, and capable, and it is our job to provide a tool that helps them shine.”

Such an approach can help “free data teams and analysts from report factory hell, so they can leverage the power of data for complex, innovative, and fulfilling initiatives,” he added.

Loss of data trust

Data scientists and data workers know that poor quality is a perennial problem. Often this is because of outdated reports, unclear definitions, and non-standardized metrics. 

But it also creates a culture that questions data quality, even when the right tools are deployed. It makes data analytics a burden as analysts have to verify and double check the results (often manually). 

“Data quality is a problem for nearly every company. In fact, according to O’Reilly’s ‘State of Data Quality in 2020’ report, only 20% of organizations publish data provenance and data lineage,” said Woollen.

Here, cloud offers the answer. “Many data quality issues can be solved with a cloud data stack and modern data governance,” said Woollen.

Another reason is that data capturing is still a very manual process, often involving adding numbers onto spreadsheets. Many companies have had "spreadsheet sprawl" because their only choice for enabling anyone outside the data or BI team to analyze data has been through Excel or Google Sheets.

“So, they've been stuck trying to govern those and verify their accuracy,” said Woollen.

The idea of verifying data is viewed as a burden also suggests there is a cultural issue that may require a shift in mindset.

“Data and analytics drive the most important decisions in any company. Leveraging your data must be a strategic differentiator, and it is important for leaders to establish that importance in the company culture, not view it as a burden.”

Data politics of control

Data workers often complain that they do not have access to all the data in an organization. And sometimes, it becomes a challenge as other departments may challenge data ownership.

Woollen feels that this kind of thinking needs to go away.

“The keepers of data need to embrace the idea of providing governed data access to everyone. This may be challenging, especially for those that have lived through the wild west days of data, but providing access doesn’t have to mean losing control,” said Woollen.

Sigma Computing is helping to tackle this issue with cloud. “With Sigma, data never leaves the cloud data warehouse and we offer a full suite of security features, which allow admins to manage access roles, permissions, and keep sensitive information safe; while still allowing everyone to explore the data that is relevant to them.”

The self-service disappointment

The trough of disappointment that previous self-service BI tools created is an immense barrier.

Here, Woollen quotes his customer Chris Lambert, chief technology officer at Payload. Lambert reportedly noted that the Sigma [solution] “is an order of magnitude less work than our old tool, which required us to learn a proprietary coding language, and prevented us from being agile.”

Lambert added that Sigma Computing helped them to run reports from day one “because it was so easy.”

One reason Woollen’s company puts a lot of emphasis on better self-service capabilities is that it does not readily see AI and machine learning as the future of BI. In this way, Sigma Computing differs from many in the same field.

“We believe that there is no substitute for humans - their knowledge, experience, and empathy — and they must remain at the center of data analysis. We believe the future of analytics and business intelligence is a community-driven approach, where everyone gets to apply their unique expertise to the data agenda and analysis process firsthand. That is the data-driven dream realized.”

Leadership, culture variables

While many companies want to become data driven, Woollen warns against making it an ambiguous goal.

“Our research shows that 39% of domain experts do not even know what it means to be “data-driven.” Now that the technology is available, we need to work on that last mile — the cultural shift,” said Woollen.

This shift includes providing access to data and analytical tools and training them with the knowledge to use them. 

“They must be empowered with the skills, data access, and tools they need to actively participate in the analytics and business intelligence process. Their business will be better off for it,” said Woollen.

However, he also advised companies not to look for blueprints for building the right data-driven culture. Every company is unique, as is its culture.

“Best practices can certainly be replicated from company to company; but business leaders should be less concerned with copying another company’s culture, and more interested in cultivating a unique culture of their own that values data-driven decision-making, celebrates curiosity, and supports data literacy.”

Photo credit: iStockphoto/SIphotography