Cloudera, a company that provides data management and analytics software, announced native GPU acceleration for Apache Spark 4.1 in Cloudera Data Engineering, enabled by the NVIDIA CUDA-X library, cuDF. The NVIDIA cuDF plug-in for Apache Spark will support the Cloudera Anywhere Cloud, which is designed to run Spark workloads across different environments without requiring organizations to rewrite PySpark or SQL code. The capability is intended to help data teams prepare data for analytics and AI while reducing cloud infrastructure costs across hybrid environments.
As organizations expand AI initiatives, data preparation can affect the time required to complete analytics and AI workloads. Large-scale Spark workloads can take hours to complete, increasing cloud compute costs. By embedding GPU acceleration into Cloudera Data Engineering, organizations can use GPU processing with existing Spark applications without changing code or operational workflows.
GPU acceleration for enterprise Spark workloads
Apache Spark is used in enterprise data pipelines. With GPU acceleration built into Cloudera Data Engineering, organizations can increase processing performance while maintaining the security and governance required for production workloads. Using NVIDIA cuDF for Spark workloads, Cloudera will provide up to 4x workload acceleration on NVIDIA GPUs compared with CPU infrastructure, reducing data processing times.
Together, Cloudera Data Engineering and NVIDIA CUDA-X libraries will provide GPU acceleration for Apache Spark 4.1 workloads without code changes, support ETL and data preparation for analytics and AI, and reduce cloud infrastructure costs through shorter compute runtimes. The capability also includes deployment without manual driver configuration, security and governance through the Cloudera Unified Data Fabric, and support across public cloud, private cloud, sovereign cloud, and on-premises environments.
Cloudera also extends the capability across hybrid environments rather than limiting GPU acceleration to a single cloud provider. Organizations can run Spark workloads on premises, in the cloud, or at the edge while maintaining the same governance and operational model.
Spark processing and cloud infrastructure costs
Preparing data for analytics and AI requires processing resources, making the cost and duration of Spark workloads relevant to data engineering teams. Data from Cloudera's The Great Re-Architecture Survey found that 84% of respondents said AI workloads have caused infrastructure costs to increase. GPU acceleration for Spark is intended to reduce processing time and, as a result, the compute resources required to run those workloads.
"For many organizations, AI isn't limited by models. It's limited by how quickly they can turn raw data into trusted, usable insights," said Leo Brunnick, Chief Product Officer at Cloudera. "Accelerating Spark inside Cloudera Data Engineering helps remove that bottleneck, allowing customers to move from data preparation to analytics and AI faster while keeping governance, security, and operational consistency at the center of their strategy."
"The fastest path to accelerating AI deployments is the one that aligns with how enterprises already operate today," said Pat Lee, vice president, Strategic Enterprise Partnerships at NVIDIA. "With NVIDIA AI infrastructure and CUDA-X libraries now native to Cloudera Data Engineering, enterprises can lower costs and dramatically speed up Apache Spark pipelines without changing a single line of PySpark or SQL code, turning business data into a foundation for AI."
The GPU acceleration capability for Apache Spark will be available in Cloudera Data Engineering as part of Cloudera Anywhere Cloud, announced at EVOLVE Singapore on August 20, 2026. Additional demonstrations and technical sessions will be featured at NVIDIA GTC Berlin and Cloudera EVOLVE New York later this year.
