Databricks opened a R&D center in Amsterdam earlier this year to leverage the database engineering talent there and to strengthen their collaboration with CWI. The San Francisco based company leads the development of open source software Apache Spark, the most widely used software tool to analyze large amounts of data. Databricks offers Spark as a cloud service to companies and organizations, such that those customers are able to run their data analysis in a managed environment efficiently and effectively.
Databricks funds research in the Database Architectures research group of CWI, which previously developed the well-known database systems VectorWise and MonetDB. The CWI researchers work on database techniques that Databricks is interested to incorporate. The CWI methods allow users to analyze large amounts of data, not limited to tables, but also including (social) networks with growing and changing data.
Enrichment of the data science ecosystem
Peter Boncz, senior researcher in the Database Architectures research group, coordinates the collaboration with Databricks. Boncz: "The arrival of Databricks to Amsterdam enriches the local data science ecosystem, and underlines the reputation of the CWI in the field of big data technology."
Boncz sees opportunities for the fundamental research at CWI: "For CWI, the collaboration provides the chance to look behind the scenes with Databricks. We will gain insight in the great diversity of data analysis problems that users encounter, and can thus discover some of the open questions in the field of data analysis. Also, professors from Berkeley and Stanford are involved in Databricks, so new scientific collaborations will arise."
Faster and more scalable
“Databricks is excited about growing our R&D presence in Amsterdam and doing some of the most innovative engineering work in big data analytics and data science,” says Ram Sriharsha, interim site manager for the new Databricks R&D center in Amsterdam. "The collaboration with CWI and the expertise of CWI in high-performance databases was decisive in the selection of our new location. With this collaboration, we aim to make Spark and Databricks faster and more scalable."
