Data Lessons From Twilio
Data volume is at an all-time high, and businesses are striving to do more with their data, according to Twilio in its latest annual Customer Data Platform report. And as organizations focus on operational efficiency with an eye on the uncertain road ahead, this has a bearing on how they collect and process customer data.
Twilio should know a thing or two about data – it processed some 1.11 trillion API calls on the Twilio Segment Platform last August alone. We take a closer look at its report for data lessons that can benefit businesses.
Quality over quantity
Around the world, the “growth at all costs” mentality is coming to an end as funding dries up amid soaring inflation. Twilio says this attitude is being mirrored in how businesses are now approaching their data architecture, with them shifting away from vanity metrics that look impressive, but don’t relate to meaningful business goals or key performance indicators (KPI).
There is a spot of good news, however. Pointing to how some point solutions on its platform have higher growth than even established market leaders such as Snowflake and Facebook, Twilio argues that opportunities exist for products that fill specific, business-critical needs despite economic uncertainty and dwindling budgets.
Twilio predicts businesses will continue to get more efficient and intelligent with their customer data by prioritizing business outcomes over sheer volume.
To be clear, we know that data professionals spend a staggering 40 percent of their time evaluating or checking data quality. The sheer volume and velocity of data by which they are continually created necessitate good data management systems and talent to separate the wheat from the chaff.
More organizations are turning to data warehouses
Half (53%) of Twilio Segment customers now connect to a data warehouse, just behind analytics as a destination. According to the authors of the report: “The rapid acceleration in volume and complexity of customer data has created a need for tools to help businesses manage that data. With a cloud warehouse, companies have ultimate flexibility in how they store and later query data.”
Of course, Twilio would have you run their CDP alongside your data warehouse to give non-technical teams access to real-time data without the inherent bottlenecks of going through data engineers.
“Setting up and maintaining a data warehouse… can be difficult, as non-technical teams are forced to rely heavily on data engineers to query, filter, and forward data on their behalf. This can lead to an operational bottleneck or ‘stale’ data by the time it reaches the point of activation,” they noted.
This is where a lakehouse storage solution might make a difference. As I wrote this week in “Simplifying Data And AI With a Data Lakehouse”, a data lakehouse puts a metadata layer in front of the raw file storage across disparate systems, giving organizations the ability to keep track of database tables and data files, including versioning.
Client applications can then be built to talk to this layer for self-service analytics with high manageability – without adding to operational overheads.
Trust and transparency remain vital
Citing the adoption of privacy features on its CDP platform, Twilio pointed to the trend of consumers asking for more transparency and respect for their personal data. Specifically, businesses today are processing user deletion requests to comply with data privacy regulations like GPDR or CCPA (California Consumer Privacy Act).
“Looking at the data, it’s evident that our customers have been committed to their ongoing efforts to uphold user privacy. In 2022, we successfully processed over 23 million user deletion requests on the Twilio Segment platform, a 69% increase over last year,” wrote the authors.
As every data professional knows, deleting data is easier said than done. This is due to the fragmentation in modern data stacks and a lack of interoperability that makes organization-wide management of datasets a pipe dream. Based on the Twilio report, this is an area that data professionals might want to pay particular attention to, either with the adoption of a solution that can manage data deletion from a centralized location, or manually implemented.
For the latter, I would caution that a heavy focus on in-house tooling can adversely affect talent retention. This is because data engineers often choose opportunities that allow them to expand their experience with industry-standard tooling, as I wrote last month.
You can access the report titled “The CDP Report 2023” here.
Paul Mah is the editor of DSAITrends. A former system administrator, programmer, and IT lecturer, he enjoys writing both code and prose. You can reach him at [email protected].
Image credit: iStockphoto/tiero
