top of page
Search

Is it the Data Team's Fault?

Writer's picture: Martin IgnatovskiMartin Ignatovski

Stop blaming your data team.
Stop blaming your data team.

Every day, if not more often than that, we hear about organizations committing to using data to make crucial decisions. Building appropriate data infrastructure and architecture allows data analytics and data science teams to gain valuable insights that can be used internally within the organization and as a value proposition for the customer the organization serves. Depending on the organization, and its strategy, the data teams are structured differently. Some prefer data teams to be centralized in one data operations group (DataOps), while others focus on a decentralized approach. In the decentralized approach, each organizational function has its own data resources. Each of those structures has its own pros and cons, which I can debate on; however, data team structures are not the scope or aim of this post. 


What I want to discuss are existing trends of the perception of data engineers by their organizations, and the stress they endure at their jobs. While doing research and continuing my education on all things data, I came across a comprehensive survey completed by 600 data engineers. Of the 600 surveyed data engineers, 100 were in some type of a management-level role. The survey asked various questions related to how organizational leadership sees the data teams, the stress that comes with the job, obstacles for data engineers, and other questions. There are some jarring trends that will be presented in this post. 


On the question “How often do you get data requests with unreasonable expectations?” 18.5% of the respondents answered all the time, 41.7% of the respondents answered often, 29.7% answered sometimes, while 9% and approximately 1.2% answered rarely and never, respectively. Looking at this data, and the graph below, we can obsere the lack of knowledge of data operations processes by the organization, thus placing data requests with unreasonable expectations more often than not. How do we address this problem? Is it a failure of the data team to educate the organization of what is reasonable and what is not, or is it the organization’s responsibility to learn more about creating reasonable expectations.


Frequency of unreasonable data requests
Frequency of unreasonable data requests.

Finally, another interesting question asked in the 2021 survey was “How often, if ever, does the data engineering team at your company receive the blame when things go wrong with the company’s data and analytics?”. The answers provide valuable insight into how the data teams are perceived within an organization. From the 600 respondents, 20.7% responded they were blamed all the time, 42.3% were blamed often, 23.7% were blamed sometimes, 12% were blamed rarely, and 1.3% were never blamed. From the data, and the graph below, we can see that the organization blames the data team often or all the time in 63% of the cases. Is it possible that data teams are that bad across organizations from various industries, with various revenues, and various numbers of employees? Or is it simply a lack of appropriate organizational strategy, data education, and potentially - data governance? 


Frequency of blame thrown at the data team.
Frequency of blame thrown at the data team.

The two trends mentioned above indicate problems on how organizations perceive the data functions, potential issues with existing data structures, and lack of education regarding data team capabilities. One way to start solving for the problems raised by the two above mentioned trends is to start an open dialogue on what works well and what doesn’t. It is crucial for organizations to set appropriate data strategy, align on organizational needs, and create operational structure that allows for data teams to perform at their best, while not feeling burned out at their job, or blamed for the work they perform! Only then, organizations and their customers will greatly benefit from the data insights provided by the data teams.


Note:

Data representations in this post are based on a 2021 survey of 600 data engineers. The survey was retrieved from data.world. No additional data has been collected outside of the provided survey.

6 views0 comments

Recent Posts

See All

Comments


bottom of page