State of Data Science 2021: Becoming “Essential,” Though Untapped Potential Remains

In last year’s State of Data Science report, we looked at how the field of data science was moving from hype toward maturity. It was perhaps inevitable that a profession dubbed the “sexiest job of the 21st century” back in 2012 would find itself at the center of a hype bubble in subsequent years. But as last year’s State of Data Science report showed, data scientists and the organizations they work for have found effective ways to collaborate to create real, measurable value. In doing so, the field has moved away from being a shiny new object—much desired but not always effectively utilized—and toward becoming a crucial part of businesses’ decision-making infrastructure.

Among the many new concepts introduced to us throughout 2020 was the idea of the “essential worker.” What jobs or roles are so integral to the functioning of a business that the company wouldn’t be able to operate without them? The idea of “essentialness” provides a helpful framework to consider how effectively data science has integrated with an organization’s most important operations.

Amid speculation about whether the COVID-19 pandemic (and resulting economic challenges) would dampen demand for what has been a red-hot field over the past few years, Anaconda asked respondents for the 2021 State of Data Science Survey about the impact they saw. 50% said that their organization’s investment in data science either stayed the same or increased due to the pandemic, with 37% stating they saw a decrease. For those that saw an increase in investment during this time, examples of how this manifested included an increased budget for data science, active hiring, and additional projects or expedited project timelines. This mirrors trends we’ve seen in our own Anaconda usage data, where between March 2020 and February 2021, we saw 4.6 billion package downloads, a 48% increase from the previous year.

The fact that half of our survey respondents saw steady or increased investment from their organizations during the past year shows that data science demonstrates real value in business settings. This is further reflected in this year’s State of Data Science survey finding that 53% of respondents said many or all business decisions in their organizations are based on insights interpreted by its data scientists.

These results suggest that data science is becoming “essential” to organizations, but also indicate that there is still room for growth in this area. After all, 37% of respondents said their organizations decreased investment in data science due to the pandemic. Some companies likely needed to make cuts across the board during the difficult economic circumstances of the past year. But in areas where cutbacks might have been avoided, what factors could help explain why data science was an area of disinvestment?

One likely explanation is that these organizations still have a large communication and understanding gap between the business and data science teams, decreasing the effectiveness of data scientists’ work. For instance, when asked about the factors that limit their ability to impact business decisions, 33% of respondents pointed to quickly shifting business priorities, 25% noted decision-makers at their organizations struggle with data literacy, and 11% of respondents said they were limited by the fact that they couldn’t demonstrate business impact. 25% pointed to insufficient resources for effective analysis as a root cause, which on the surface may seem unrelated. Yet, even this points to a gap between business and data science leaders in understanding what constitutes adequate tools.

Based on the report’s findings, it appears both sides have work to do in order to bridge this gap and collaborate more effectively. From the business perspective, leaders should invest time in improving their data literacy to better understand the role and work of their organization’s data scientists. In this year’s State of Data Science report, only 36% of respondents said their organization’s decision-makers were very data literate and understood the stories told by visualizations and models. Encouragingly, 52% said that their organization’s decision-makers were mostly data literate but needed some coaching on the stories told by visualizations and models. Nevertheless, there is room for further growth in this area, which will benefit the business as a whole.

From the other perspective, data scientists should also work to improve their understanding of business and any relevant domain knowledge to operate more efficiently and successfully. Asked “What important skills and knowledge are missing in the data science/ML area of your organization,” nearly a quarter of respondents in this year’s survey listed “business knowledge.” “Business knowledge” also ranked relatively low on questions about what data science students learn in school. Communication skills are another area for improvement, with 22% of respondents saying they are missing from the data science/ML area of their organization—data scientists must learn to be good storytellers who can explain their findings in a clear and compelling way. Developing these soft skills will help data scientists positively impact their organizations and become “essential” to everyday operations. With a 15% increase in the number of respondents from Generation Z in this year’s survey compared to 2020, there’s a clear opportunity to more effectively incorporate these skills into the education curriculum, so that students are better prepared to enter the field as professionals. However, more experienced data scientists should also invest their time in further developing soft skills, too, as part of their ongoing professional development.

It’s never been a more exciting time to be part of the data science community, and we’re inspired to continue doing our part to help advance and support data scientists from all walks of life here at Anaconda. While there’s still untapped potential, it’s clear that data scientists are well on their way to becoming essential parts of their organizations, providing business value, and pushing the field forward into new areas of development. We can’t wait to see the trends and changes that unfold over the next year!

To view the full 2021 State of Data Science report, including findings about how data scientists spend their time, which programming languages are most popular, and what concerns practitioners today, click here.