The Data Science and AI Market May Be Ready for Recalibration

The Data Science and AI Market May Be Ready for Recalibration

Data Scientist Was Supposed to Be “The Most Attractive Job of the 21ste One Hundred Years.” The question of whether Harvard Business Review’s famous 2012 wisdom is holding up. However, data on data scientists, and the related roles of data architects and data analysts, is starting to sound the alarm.

The personal part of Harvard Business Review’s wisdom is whether you really enjoy finding and cleaning data, building and debugging data pipelines and integration code, and building and improving machine learning models. In this to-do list, in that order, data scientists spend most of their time.

Some people are really attracted to data-driven jobs by the job description, while growth in demand and salaries attract others. While the dark sides of the job description itself are unheard of, the growth and salary portion hasn’t faced significant challenges. However, the situation could change: Data scientist jobs are still in high demand, but they are not immune to market turmoil.

mixed signals

At the beginning of 2022, the first sign appeared that something might change. As an IEEE Spectrum analysis of data published by online recruitment firm Dice in 2021 showed, salaries in artificial intelligence and machine learning have fallen despite wages in the US tech sector increasing by 7% on average. However, as IEEE Spectrum notes, competition for machine learning, natural language processing, and artificial intelligence experts has eased, with average salaries dropping 2.1%, 7.8%, and 8.9%, respectively.

This is the first time this has happened in recent years, as median US salaries for software engineers with experience in machine learning, for example, jumped 22% in 2019 compared to 2018, then increased 3.1% in 2020. At the same time, the demand for data scientist roles shows no sign of abating, quite the contrary.

Developer recruitment platforms indicate an increased demand for data science IT skills. The latest report from DevSkiller developer screening and interviewing platform recorded a 295% increase in the number of data science-related tasks assigned to candidates during the interview process in 2021.

CodinGame and CoderPad’s 2022 “Tech Hiring” survey identified data science as a profession in which demand far outstrips supply, along with DevOps and machine learning specialists. Therefore, employers will need to reassess the salaries and benefits they offer their employees if they are to stay competitive.

wave of hairstyles for workers

In addition, 2021 was marked by the so-called “Great Resignation” phenomenon, a period in which everyone rethinks everything, including their careers. In theory, the fact that a portion of the workforce is redefining its course and goals and/or leaving should increase demand and wages — analyzes have begun about why data scientists are resigning and what employers can do to hold them.

Then came layoffs, including layoffs of data scientists, data engineers, and data analysts. As LinkedIn’s analysis of the latest round of layoffs indicates, the turbulent year in the tech industry has been marked by daily announcements of layoffs, hiring freezes and job openings.

About 17,000 workers at more than 70 tech startups around the world were laid off in May, a 350% increase from April. This is the largest number of jobs lost in the sector since May 2020, at the height of the pandemic. In addition, tech giants like Netflix and PayPal are also cutting jobs, while Uber, Lyft, Snap and Meta have slowed hiring. According to data shared by, a website that tracks tech layoffs, layoffs affect between 7% and 33% of the workforce of the companies tracked. Looking at the data for each company, it can be seen that these positions are also data-dependent.

Examination of layoff data from fintech Klarna and the insurance startup policy Genius, for example, shows that the roles of data scientist, engineer, and data analyst are affected at both the junior and senior levels. At both companies, these roles account for about 4% of layoffs.

Measuring the consequences of automation

What should we think of these contradictory signals? It appears that the demand for data science-related tasks remains strong, but salaries are declining and these roles are not immune to layoffs. Each of these signals comes with its own context and effects.

As Michele Marian, director of marketing for Dice, told IEEE Spectrum, various factors are likely contributing to the decline in machine learning and AI salaries, one of which is that more techies are learning and mastering these skills. “As the number of talent increases over time, employers may have to pay at least a little bit less, as the skills are easier to find. We have seen this happen through a series of certifications and other highly specialized technical skills,” she said.

This seems like a reasonable conclusion. However, for data science and machine learning, there may be something else at play. Data scientists and machine learning experts are not only competing with each other, but also increasingly with automation. As Hong Kong quantitative portfolio manager Peter Yuen points out, quantitative analysts have seen it all before.

After learning that top AI researchers have earned salaries in the $1 million range, Peter Yuen writes that “this should be interpreted more accurately as a continuation of a long trend of ‘high-tech’ workers to code themselves to lose their jobs amid a global crisis. Skilled labor abundance “.

“Judging by the experience of three generations of quantums in automating financial markets,” Peter Yuen wrote, “the automation of AI-based practitioners in many industries probably won’t be for just ten years.” After that, he adds, “A small group of elite AI practitioners will have made it to an executive or owner position, while the rest will remain in middle-paying jobs tasked with monitoring and nurturing their creativity.

We may already be in the early stages of this cycle, as evidenced by developments such as AutoML and ready-to-use machine learning model libraries. If history is anything to go by, it will likely pass what Peter Yuen described as well, which will inevitably lead to questions about how displaced workers can “climb the heap”.

AI bubble burst

However, one can probably assume that data scientists won’t have to worry too much about this in the near future. After all, another frequently cited fact about data science projects is that nearly 80% of them always fail for a number of reasons.

Zillow’s case particularly illustrates the failure of data science. Zillow’s business relied heavily on the data science team to build accurate predictive models for home buying. It turns out that the models weren’t accurate. As a result, the company’s stock fell more than 30% in five days, the CEO blamed the data science team, and 25% of employees were laid off.

Whether or not the data science team was wrong about Zillow is up for debate. As for the recent layoffs, they are likely to be seen as part of a larger shift in the economy rather than the failure of data science teams per se. As Data Science Central community editor Kurt Cagle writes, there is talk of an impending AI winter, reminiscent of the 1970s when funding for AI projects dried up completely.

Kurt Cagle believes that while an AI winter is unlikely, an AI fall can be expected as the overheating of venture capital in this area subsides. The AI ​​winter of the 1970s was largely due to the technology not being up to the task and not having enough digital data for mining.

Today, the available computing power is much greater and the amount of data is on the rise. According to Kurt Cagle, the problem may be that we are approaching the limits of the currently used neural network structures.

Like many others, Kurt Cagle points out the shortcomings of the thinking system which is to say that “deep learning will be able to do it all”. This criticism seems valid, and the incorporation of currently neglected methods could advance the field. However, let’s not forget that the technological aspect is not the only important aspect here.

Perhaps recent history can enlighten us: What can we learn from the history of software development and the Internet? In some ways, where we are today is reminiscent of the days of the dotcom bubble: increased capital availability, excessive speculation, unrealistic expectations, and massive valuations. Today we may be heading towards an AI bubble bursting. This doesn’t mean that data science roles will lose their appeal overnight or that what you do is worthless. After all, software engineers are still in demand for all the advancements and automation that software engineering has experienced over the past few decades. But this likely means that recalibration is needed and expectations should be managed accordingly.


Leave a Comment

Your email address will not be published. Required fields are marked *