{"id":2991,"date":"2024-03-29T06:27:14","date_gmt":"2024-03-29T06:27:14","guid":{"rendered":"https:\/\/favtutor.com\/articles\/?p=2991"},"modified":"2024-03-29T07:34:53","modified_gmt":"2024-03-29T07:34:53","slug":"databricks-dbrx-benchmarks","status":"publish","type":"post","link":"https:\/\/favtutor.com\/articles\/databricks-dbrx-benchmarks\/","title":{"rendered":"DBRX, An Open-Source LLM by Databricks Beats GPT 3.5"},"content":{"rendered":"\n<p>The company behind DBRX said that it is the world\u2019s most powerful open-source AI mode. Let\u2019s look at how it was built.<\/p>\n\n\n\n<p><strong>Highlights:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Databricks recently introduced DBRX, an open general-purpose LLM claimed to be the world\u2019s most powerful open-source AI model.<\/li>\n\n\n\n<li>It outperforms OpenAI\u2019s GPT-3.5 as well as existing open-source LLMs like Llama 2 70B and Mixtral-8x7B on standard industry benchmarks.<\/li>\n\n\n\n<li>It is freely available for research and commercial use through GitHub and HuggingFace.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Meet DBRX, The New LLM in Market<\/strong><\/h2>\n\n\n\n<p><strong>DBRX is an open and general-purpose LLM built by Databricks to encourage customers to migrate away from commercial alternatives.<\/strong><\/p>\n\n\n\n<p>The team at Databricks spent roughly $10 million and two months training the new AI model.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"jeg_video_container jeg_video_content\"><iframe title=\"DBRX: A New Standard for Open Source LLMs\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/XjuGU6w8td4?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<p>DBRX is a transformer-based decoder-only LLM that is trained using next-token prediction. It uses a fine-grained mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input. It has been pre-trained on 12T tokens of text and code data.<\/p>\n\n\n\n<p>Ali Ghodsi, co-founder and CEO of Databricks, <a href=\"https:\/\/www.databricks.com\/company\/newsroom\/press-releases\/databricks-launches-dbrx-new-standard-efficient-open-source-models\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">spoke about how their vision<\/a> translated into DBRX:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p>&#8220;At Databricks, our vision has always been to democratize data and AI. We&#8217;re doing that by delivering data intelligence to every enterprise \u2014 helping them understand and use their private data to build their own AI systems. DBRX is the result of that aim.&#8221;<\/p>\n<cite>Ali Ghodsi<\/cite><\/blockquote>\n\n\n\n<p>DBRX uses the MoE architecture, a type of neural network that divides the learning process among multiple specialized subnetworks known as \u201cexperts.\u201d Each expert is proficient in a specific aspect of the designated task. A \u201cgating network\u201d decides how to allocate the input data among the experts optimally.<\/p>\n\n\n\n<p>Compared to other similar open MoE models like Mixtral and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. It has 16 experts and chooses 4, while Mixtral and Grok-1 have 8 experts and choose 2. This provides 65x more possible combinations of experts and this helps improve model quality.<\/p>\n\n\n\n<p>It was trained on a network of 3072 NVIDIA H100s interconnected via 3.2Tbps Infiniband. The development of DBRX, spanning pre-training, post-training, evaluation, red-teaming, and refinement, occurred over three months.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Why is DBRX open-source?<\/strong><\/h3>\n\n\n\n<p>Recently, <a href=\"https:\/\/favtutor.com\/articles\/grok-1-setup\/\">Grok by xAI is also made open-source<\/a>. By open-sourcing DBRX, Databricks is contributing to a growing movement that challenges the secretive approach of major companies in the current generative AI boom. <\/p>\n\n\n\n<p>While OpenAI and Google keep the code for their GPT-4 and Gemini large language models closely guarded, rivals like Meta have released their models to foster innovation among researchers, entrepreneurs, startups, and established businesses.<\/p>\n\n\n\n<p><strong>Databricks aims to be transparent about the creation process of its open-source model, a contrast to Meta&#8217;s approach with its Llama 2 model. With open-source models like this becoming available, the pace of AI development is expected to remain brisk.<\/strong><\/p>\n\n\n\n<p>Databricks has a particular motivation for its openness. While tech giants like Google have swiftly implemented new AI solutions in the past year, Ghodsi notes that many large companies in various sectors have yet to adopt the technology widely for their data.<\/p>\n\n\n\n<p>The aim is to assist companies in finance, healthcare, and other fields, that desire ChatGPT-like tools but are hesitant to entrust sensitive data to the cloud.<\/p>\n\n\n\n<p>\u201c<em>We call it data intelligence\u2014the intelligence to understand your own data,<\/em>\u201d Ghodsi explains. Databricks will either tailor DBRX for a client or develop a customized model from scratch to suit their business needs. For major corporations, the investment in creating a platform like DBRX is justified, he asserts. \u201c<em>That\u2019s the big business opportunity for us.<\/em>\u201d<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Comparing DBRX to other models<\/strong><\/h3>\n\n\n\n<p>DBRX outperforms existing open-source LLMs like Llama 2 70B and Mixtral-8x7B on standard industry benchmarks, such as language understanding (MMLU), programming (HumanEval), and math (GSM8K). The figure below shows a comparison between Databricks&#8217; LLM and other open-source LLMs.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"879\" height=\"579\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-with-other-open-source-models.png\" alt=\"DBRX with other open source models\" class=\"wp-image-2993\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-with-other-open-source-models.png 879w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-with-other-open-source-models-300x198.png 300w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-with-other-open-source-models-768x506.png 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-with-other-open-source-models-750x494.png 750w\" sizes=\"(max-width: 879px) 100vw, 879px\" \/><\/figure>\n<\/div>\n\n\n<p>It also outperforms GPT-3.5 on the same benchmarks as seen in the figure below:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"510\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-Comparison-with-GPT-3.5-1024x510.png\" alt=\"DBRX comparsion with GPT 3.5\" class=\"wp-image-2992\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-Comparison-with-GPT-3.5-1024x510.png 1024w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-Comparison-with-GPT-3.5-300x150.png 300w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-Comparison-with-GPT-3.5-768x383.png 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-Comparison-with-GPT-3.5-360x180.png 360w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-Comparison-with-GPT-3.5-750x374.png 750w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-Comparison-with-GPT-3.5-1140x568.png 1140w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/03\/DBRX-Comparison-with-GPT-3.5.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>It outperforms its rivals on several key benchmarks:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Language Understanding:<\/strong> DBRX achieves a score of 73.7%, surpassing GPT-3.5 (70.0%), Llama 2-70B (69.8%), Mixtral (71.4%), and Grok-1 (73.0%).<\/li>\n\n\n\n<li><strong>Programming:<\/strong> It demonstrates a significant lead with a score of 70.1%, compared to GPT-3.5\u2019s 48.1%, Llama 2-70B\u2019s 32.3%, Mixtral\u2019s 54.8%, and Grok-1\u2019s 63.2%.<\/li>\n\n\n\n<li><strong>Math:<\/strong> It achieves a score of 66.9%, edging out GPT-3.5 (57.1%), Llama 2-70B (54.1%), Mixtral (61.1%), and Grok-1 (62.9%).<\/li>\n<\/ul>\n\n\n\n<p><strong>DBRX also claims that for SQL-related tasks, it has surpassed GPT-3.5 Turbo and is challenging GPT-4 Turbo. It is also a leading model among open models and GPT-3.5 Turbo on Retrieval Augmented Generation (RAG) tasks.<\/strong><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Availability of DBRX<\/strong><\/h3>\n\n\n\n<p>DBRX is freely accessible for both research and commercial purposes on open-source collaboration platforms like GitHub and HuggingFace.<\/p>\n\n\n\n<p>It can be accessed through <a href=\"https:\/\/github.com\/databricks\/dbrx\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">GitHub<\/a>. It can also be accessed through <a href=\"https:\/\/huggingface.co\/databricks\" data-type=\"link\" data-id=\"https:\/\/huggingface.co\/databricks\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">HuggingFace<\/a>. Users can access and interact with DBRX hosted on HuggingFace for free.<\/p>\n\n\n\n<p>Developers can use this new openly available model released under an open license to build on top of the work done by Databricks. Developers can use its long context abilities in RAG systems and build custom DBRX models on their data today on the Databricks platform.<\/p>\n\n\n\n<p>The open-source LLM can be accessed on AWS and Google Cloud, as well as directly on Microsoft Azure through Azure Databricks. Additionally, it is expected to be available through the NVIDIA API Catalog and supported on the <a href=\"https:\/\/favtutor.com\/articles\/nvidia-nim-llm-deployment\/\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">NVIDIA NIM<\/a> inference microservice.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p>Databricks&#8217; introduction of DBRX marks a significant milestone in the world of open-source LLM models, showcasing superior performance across various benchmarks. By making it open-source, Databricks is contributing to a growing movement that challenges the secretive approach of major companies in the current generative AI boom.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Meet the latest open-source AI model by Databricks, known as DBRX that can beat various other LLMs. Find out how to access DBRX?<\/p>\n","protected":false},"author":18,"featured_media":2996,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jnews-multi-image_gallery":[],"jnews_single_post":null,"jnews_primary_category":{"id":"","hide":""},"footnotes":""},"categories":[57],"tags":[56,137,59,63,72],"class_list":["post-2991","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-ai","tag-dbrx","tag-generative-ai","tag-gpt","tag-llm"],"_links":{"self":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/2991","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/comments?post=2991"}],"version-history":[{"count":4,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/2991\/revisions"}],"predecessor-version":[{"id":2999,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/2991\/revisions\/2999"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/media\/2996"}],"wp:attachment":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/media?parent=2991"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/categories?post=2991"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/tags?post=2991"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}