Poor data collaboration ‘holding back’ 95% of enterprises deploying AI, says Databricks survey
David Wyatt of Databricks
A survey commissioned by a provider of unified analytics, Databricks, reveals that only one in three Artificial Intelligence (AI) projects are succeeding.
Perhaps more importantly, it is taking businesses more than six months to go from concept to production. Organisations are hindered at multiple stages of the process when bringing AI projects to production.
Databricks commissioned IDG Research to conduct analysis of the AI, machine learning and deep learning landscape in large enterprises. The survey looked to sample the experiences of those working in specific senior data engineering and data science roles at companies with more than 1,000 employees. The audience of 200 people was split equally between the United States and Western Europe.
According to 95% of European respondents, collaboration between data engineering and data science teams is a challenging issue. Nearly all respondents cite data-related challenges when moving AI projects to production. IT executives point to unified analytics as a solution for these challenges with 90% of respondents saying the approach of unifying data science and data engineering across the machine learning lifecycle will conquer the AI dilemma.
The research, commissioned by Databricks through CIO/IDG Research Services, shows that more 93% of organisations surveyed are investing in technology to help with data prep and data exploration/modeling, including data processing, data streaming, machine learning and/or deep learning tools. As a result, organisations are using an average of seven different tools for data prep and modeling.
Based on these results, it is not a surprise that European organisations cited technology as their most common obstacle when moving AI projects to production.
Additional results of the survey speak to the complexity and organisational confusion being creating as companies pursue AI initiatives:
-
- The average number of AI projects considered completely successful within enterprises is around 39% according to those surveyed in Europe, compared to an average of 35% of projects in the US.
-
- 98% of the surveyed believe preparation and aggregation of large data sets in a timely fashion is a major challenge;
-
- 96% of respondents found data exploration and iterative model training challenging;
-
- 90% cited the deployment of models to production quickly and reliably as a significant challenge
David Wyatt, vice president and general manager EMEA, Databricks, commented on the survey’s results: “Getting AI right is challenging, and one of the biggest hurdles to success is how teams collaborate around data. The research shows how difficult and time consuming it can be to turn raw data into valuable insights for the business.”
“With unified analytics, enterprises can bring together their people, processes and technology to deliver results faster – not only does this make projects more efficient, it increases the chances that these projects can succeed in meeting their objectives over time.”
So, what will help these organisations conquer the AI dilemma? The surveyed executives said they need end-to-end solutions that combine data processing with machine learning capabilities. These streamlined solutions would simplify workflows, improve efficiency and ultimately accelerate business value.
In fact, nearly 80% of executives surveyed said they highly valued the notion of a unified analytics platform. Unified analytics makes AI more achievable for enterprise organisations by unifying data processing and AI technologies. Unified analytics solutions provide collaboration capabilities for data scientists and data engineers to work effectively across the entire AI development-to-production lifecycle.
With more than 90% of large companies facing data-related challenges and increasing complexity driven by an explosion of machine learning tools, the need for platforms and processes that can remove technology and organisational silos is more pronounced than ever. Unified analytics provides an ideal approach for companies facing modern AI implementation barriers.
Databricks accelerates innovation by unifying data science, engineering, and business. Through a fully managed, cloud-based service built by the original creators of Apache Spark, the Databricks unified analytics platform lowers the barrier for enterprises to innovate with AI and accelerates their innovation.
To view the complete survey results within the “Conquer the AI Dilemma by unifying data science and engineering” report, click here
Comment on this article below or via Twitter @IoTGN