Data Professionals Struggling With Rising Requirements
Organization requirements for data products are growing faster than team size, according to a new study by data engineering firm Ascend.io, which found that almost all (96%) of data professionals are at or over capacity.
This was the conclusion of the 2021 DataAware Pulse Survey, which looked at the work capacity and priorities of data teams, including data analysts, data scientists, data engineers, and enterprise architects.
Conducted in Q2 this year, the latest iteration of the annual survey is currently in its second year and polled more than 400 U.S.-based data professionals.
Limited team capacity the problem
Though 74% indicated their organization’s need for data products is growing faster than their team size, the situation is particularly acute for data engineer respondents, where 81% agreed to this sentiment.
However, the majority (79%) of respondents also indicated that their infrastructure and systems can scale to meet their increased data volume processing needs. This underscores that the problem with scale is focused on team capacity and less on technology capacity.
Additional pointers include:
- The vast majority (93%) of respondents anticipate the number of data pipelines in their organization to increase between now and the end of the year.
- 56% project the number of data pipelines to increase by more than 50%.
- Data scientists are 2.4 times more likely than other roles to anticipate significant growth in pipeline demands.
Overcoming bottlenecks through automation
The most common bottlenecks across teams include the maintenance of existing and legacy data systems (39%), data system setup and prototyping (30%), and having to ask for access to data or systems (26%).
Among the respondents, data scientists reported the greatest need to ask for access to data at 39%. The issue of access is hardly new – and is a commonly mentioned barrier given governance considerations and disparate or siloed data sources.
Of course, data teams are not resting on their laurels but are turning to technology solutions to overcome their bandwidth limitations and do more at a quicker pace.
When asked how they plan to increase bandwidth across their team, more than half of respondents say they plan to buy new products or tools (53%) or implement automation technology (53%). Other strategies to increase bandwidth include hiring more staff (47%) and re-platforming and retiring legacy technologies (30%).
No-code doesn’t mean no hassle
Despite the importance of automation, only 4% of data professionals prefer a no-code user interface. Explaining this apparent contradiction, Sean Knapp, the chief executive officer and founder of Ascend.io agreed that the proliferation of low- and no-code technology can increase the bandwidth of teams.
However, he cautioned that many also suffer from inherent limitations due to their lack of support for customized code for more complex business logic. So, while no-code tools can significantly speed up 95% of the job, the result is that the last 5% of work might be impossible to complete.
Unsurprisingly, Knapp suggested that data professionals would do best with no-code tools that offer their preferred programming language of choice – alluding to his firm’s data engineering platform.
“Data pipelines are fueling nearly every data-driven initiative across the business. However, as innovations at the infrastructure layer continue to enable processing of greater volumes and velocities of data, businesses face a new scaling challenge: how to enable their teams to achieve more, and faster,” said Knapp
“Our research shows that team sizes are not scaling at a fast enough rate to keep up with the needs of the business. Combined with our data that highlights almost every data professional today is already at capacity, this leaves little room for strategic work and innovation,” he said.
Paul Mah is the editor of DSAITrends. A former system administrator, programmer, and IT lecturer, he enjoys writing both code and prose. You can reach him at [email protected].
Image credit: iStockphoto/imtmphoto
