Matei Zaharia

Berkeley

Chief Technology Officer and Co-Founder at Databricks

CTO at @Databricks and CS prof at @UCBerkeley. Working on data+AI, including @ApacheSpark, @DeltaLakeOSS, @MLflow, https://t.co/94gROE5Xa0. https://t.co/nmRYAKG0LZ

Featured in: Favicon

databricks.com Favicon

businessinsider.com Favicon

arxiv.org

oup.com

mit.edu

acm.org

linux.com

biorxiv.org Favicon

analyticsweek.com Favicon

academic.oup.com + 2 more

Articles

WARP: An Efficient Engine for Multi-Vector Retrieval

2 months ago | arxiv.org | Jan Luca |Matei Zaharia |Christopher Potts |Gustavo Alonso

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
AI Agent Systems: Modular Engineering for Reliable Enterprise AI Applications

Nov 13, 2024 | databricks.com | Naveen Rao |Matei Zaharia |Patrick Wendell |Eric Peter

Monolithic to ModularThe proof of concept (POC) of any new technology often starts with large, monolithic units that are difficult to characterize. By definition, POCs are designed to show that a technology works without considering issues around extensibility, maintenance, and quality. However, once technologies achieve maturity and are deployed widely, these needs drive product development to be broken down into smaller, more manageable units.
The Long Context RAG Capabilities of OpenAI o1 and Google Gemini

Oct 8, 2024 | databricks.com | Quinn Leng |Jacob P. Portes |Sam Havens |Matei Zaharia

Retrieval Augmented Generation (RAG) is the top use case for Databricks customers who want to customize AI workflows on their own data. The pace of large language model releases is incredibly fast, and many of our customers are looking for up-to-date guidance on how to build the best RAG pipelines. In a previous blog post, we ran over 2,000 long context RAG experiments on 13 popular open source and commercial LLMs to uncover their performance on various domain-specific datasets.
Generating Coding Tests for LLMs: A Focus on Spark SQL

Oct 2, 2024 | databricks.com | Linqing Liu |Matthew Hayes |Ritendra Datta |Matei Zaharia

IntroductionApplying Large Language Models (LLMs) for code generation is becoming increasingly prevalent, as it helps you code faster and smarter. A primary concern with LLM-generated code is its correctness. Most open-source coding benchmarks are designed to evaluate general coding skills. But, in enterprise environments, the LLMs must be capable not only of general programming but also of utilizing domain-specific libraries and tools, such as MLflow and Spark SQL.
Long Context RAG Performance of LLMs

Aug 12, 2024 | databricks.com | Quinn Leng |Jacob P. Portes |Sam Havens |Matei Zaharia

Retrieval Augmented Generation (RAG) is the most widely adopted generative AI use case among our customers. RAG enhances the accuracy of LLMs by retrieving information from external sources such as unstructured documents or structured data.
Enhancing LLM-as-a-Judge with Grading Notes

Jul 24, 2024 | databricks.com | Yi Liu |Matei Zaharia |Ritendra Datta |Justin Kim

Evaluating long-form LLM outputs quickly and accurately is critical for rapid AI development. As a result, many developers wish to deploy LLM-as-judge methods that work without human ratings. However, common LLM-as-a-judge methods still have major limitations, especially in tasks requiring specialized domain knowledge. For example, coding on Databricks requires understanding APIs that are not well-represented in the LLMs’ training data.
Open Sourcing Unity Catalog

Jun 13, 2024 | databricks.com | Matei Zaharia |Ali Ghodsi |Reynold Xin |Arsalan Tavakoli-Shiraji

We are excited to announce that we are open sourcing Unity Catalog, the industry’s first open source catalog for data and AI governance across clouds, data formats, and data platforms. Here are the most important pillars of the Unity Catalog vision:Open source API and implementation: It is built on OpenAPI spec and an open source server implementation under Apache 2.0 license. It is also compatible with Apache Hive's metastore API and Apache Iceberg's REST catalog API.
How Delta Sharing Enables Secure End-to-End Collaboration

Jun 4, 2024 | databricks.com | Bilal Obeidat |Bhavin Kukadia |Giselle Goicochea |Matei Zaharia

In today's digital landscape, secure data sharing is critical to operational efficiency and innovation. Databricks and the Linux Foundation developed Delta Sharing as the first open source approach to data sharing across data, analytics and AI. Databricks provides secure data exchange, facilitating seamless sharing across platforms, clouds and regions. Enterprises of all sizes trust Delta Sharing, which supports a broad spectrum of applications and diverse data formats.
Delta Sharing: Secure End-to-End Data Sharing Solution

May 24, 2024 | databricks.com | Bilal Obeidat |Bhavin Kukadia |Giselle Goicochea |Matei Zaharia

In today's digital landscape, secure data sharing is critical to operational efficiency and innovation. Databricks and the Linux Foundation developed Delta Sharing as the first open source approach to data sharing across data, analytics and AI. Databricks provides secure data exchange, facilitating seamless sharing across platforms, clouds and regions. Enterprises of all sizes trust Delta Sharing, which supports a broad spectrum of applications and diverse data formats.
How AI in Business is Revolutionized by Data Intelligence

Apr 17, 2024 | databricks.com | Adam Conway |Matei Zaharia |Scott Starbird |Rafi Kurlansik

AI is everywhere; on our cell phones, on our computers, and frequently in the headlines. But behind all the headlines, the use of AI in business has become irreplaceable and there's no sign of it going anywhere in the future. So, how will the future of data intelligence affect AI for businesses?