{"id":3749,"date":"2024-04-16T09:51:03","date_gmt":"2024-04-16T09:51:03","guid":{"rendered":"https:\/\/favtutor.com\/articles\/?p=3749"},"modified":"2024-04-16T09:51:04","modified_gmt":"2024-04-16T09:51:04","slug":"reka-core-benchmarks","status":"publish","type":"post","link":"https:\/\/favtutor.com\/articles\/reka-core-benchmarks\/","title":{"rendered":"Reka Core LLM Outperforms Claude 3, Gemini, and GPT-4"},"content":{"rendered":"\n<p>Reka Core outperforms Claude 3 Opus, Gemini Ultra, and GPT-4 on various benchmarks for video and text understanding. So, how powerful is the Reka Core? What are its groundbreaking features? Let\u2019s get right into it!<\/p>\n\n\n\n<p><strong>Highlights:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Reka announces Reka Core, its latest addition to its family of multimodal language models.<\/li>\n\n\n\n<li>Comes with several features such as a 128K Context Window, Multilingual Capabilities and Reasoning.<\/li>\n\n\n\n<li>Available for use in API Access and Reka Playground.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Reka Core Multimodal Language Model Explained<\/strong><\/h2>\n\n\n\n<p>San Francisco-based AI startup Reka, founded by researchers from Google DeepMind and Meta, introduced their latest multimodal language model called <a href=\"https:\/\/www.reka.ai\/news\/reka-core-our-frontier-class-multimodal-language-model\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Reka Core<\/a>.<\/p>\n\n\n\n<div align=center><blockquote class=\"twitter-tweet\" data-media-max-width=\"560\"><p lang=\"en\" dir=\"ltr\">Meet Reka Core, our best and most capable multimodal language model yet. \ud83d\udd2e<br><br>It\u2019s been a busy few months training this model and we are glad to finally ship it! \ud83d\udcaa<br><br>Core has a lot of capabilities, and one of them is understanding video &#8212; let\u2019s see what Core thinks of the 3 body\u2026 <a href=\"https:\/\/t.co\/5ESvog35e9\" target=\"_blank\">pic.twitter.com\/5ESvog35e9<\/a><\/p>&mdash; Reka (@RekaAILabs) <a href=\"https:\/\/twitter.com\/RekaAILabs\/status\/1779894622334189592?ref_src=twsrc%5Etfw\" target=\"_blank\" rel=\"noopener\">April 15, 2024<\/a><\/blockquote> <script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/div>\n\n\n\n<p>For the past few months <a href=\"https:\/\/favtutor.com\/articles\/claude-3-benchmarks-comparison\/\">Anthropic\u2019s Claude 3 model family<\/a>, <a href=\"https:\/\/favtutor.com\/articles\/google-gemini-1-5-pro-features\/\">Google\u2019s Gemini 1.5 Pro<\/a>, and <a href=\"https:\/\/favtutor.com\/articles\/gpt-4-turbo-vision-use-cases\/\">Open AI\u2019s GPT-4 Turbo Vision<\/a> have made quite an impact on the Gen AI market. And now we have another competitor in the field courtesy of Reka.<\/p>\n\n\n\n<p><strong>Reka Core is the latest addition to Reka\u2019s family of leading multimodal models. It is described as the &#8220;largest and most capable model&#8221; offered by the company, and thousands of GPUs are used in its training process.<\/strong><\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><em>\u201cWe introduce Reka Core, Flash, and Edge, a series of powerful multimodal language models trained from scratch by Reka. Reka models are able to process and reason with text, images, video, and audio inputs\u201d <\/em><\/p>\n<\/blockquote>\n\n\n\n<p>Reka Edge and Flash are the other models in the family having parameter sizes of 7B and 21B respectively. They are state-of-the-art models based on their compute class and are also described as \u2018dense models\u2019. <\/p>\n\n\n\n<p>However, the latest family member, Reka Core is a highly powerful language model. Core approaches today\u2019s leading frontier models such as OpenAI\u2019s GPT-4, Anthropic\u2019s Claude 3 Opus, and Google\u2019s Gemini 1.5 Pro on both automatic evaluations and blind human evaluations.<\/p>\n\n\n\n<p>Core offers exceptional value considering its size and capabilities when considering the overall cost of ownership. Largely new use cases are made possible by the combination of Core&#8217;s capabilities and deployment flexibility.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>How Can You Access It?<\/strong><\/h3>\n\n\n\n<p>Reka Core is available today via <a href=\"https:\/\/platform.reka.ai\/onboarding\" data-type=\"link\" data-id=\"https:\/\/platform.reka.ai\/onboarding\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">API<\/a>, on-premise, or on-device deployment options. <a href=\"https:\/\/chat.reka.ai\/chat\" data-type=\"link\" data-id=\"https:\/\/chat.reka.ai\/chat\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Click here <\/a>to get redirected to Reka Playground where you will have access to Reka Edge, Flash, and Core models. All you have to do is log in or set up a Reka Account and you are good to go to have conversations with Reka Core.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Looking at the Benchmarks<\/strong><\/h3>\n\n\n\n<p>Reka Core has highly competitive benchmarks across several metrics compared to GPT-4, Claude 3 Opus, and Gemini 1.5 Pro.<\/p>\n\n\n\n<p><strong>Core performs similarly to GPT-4V on MMMU, beats Claude-3 Opus on Reka\u2019s multimodal human assessment carried out by an impartial third party, and beats Gemini Ultra on video tasks. When it comes to language tasks, Core performs comparably to other frontier models using reliable benchmarks.<\/strong><\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"722\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/rekacoreblog3-1-1024x722.jpg\" alt=\"Reka Core Benchmarks\" class=\"wp-image-3753\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/rekacoreblog3-1-1024x722.jpg 1024w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/rekacoreblog3-1-300x212.jpg 300w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/rekacoreblog3-1-768x542.jpg 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/rekacoreblog3-1-120x86.jpg 120w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/rekacoreblog3-1-750x529.jpg 750w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/rekacoreblog3-1-1140x804.jpg 1140w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/rekacoreblog3-1.jpg 1500w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n<p>Core outperforms GPT4-V on picture question-answering benchmarks (e.g., MMMU, VQAv2). In contrast, Core is ranked as the second most popular model on multimodal chat, surpassing models like Claude 3 Opus.<\/p>\n\n\n\n<p>In terms of text benchmarks, Core outperforms GPT4-0613 on human evaluation in addition to performing competitively against other frontier models on several well-known benchmarks (MMLU, GSM8K, etc.). Core does better than Gemini Ultra when it comes to answering video questions (Perception Test).<\/p>\n\n\n\n<p>Here\u2019s another benchmark called ELO that shows rankings on Human Evaluation for Multimodal. A higher ELO score represents better performance. You can see that Reka defeats Claude Opus and Gemini Pro 1.0, but it has still work left to do in catching up with GPT-4V.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"575\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-522-1-1024x575.png\" alt=\"Reka Core ELO Benchmark\" class=\"wp-image-3755\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-522-1-1024x575.png 1024w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-522-1-300x168.png 300w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-522-1-768x431.png 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-522-1-750x421.png 750w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-522-1-1140x640.png 1140w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-522-1.png 1154w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n<\/div>\n\n\n<p>Overall, we get the idea that this is a highly powerful model based on its competitive benchmarks, and developers should integrate this AI model in their day-to-day Gen AI tasks such as Multimodal capabilities, coding-related tasks, reasoning tasks, and general knowledge questions.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>Comprehending Reka Core\u2019s Architecture<\/strong><\/h3>\n\n\n\n<p>The overall architecture of Reka Core is a modular encoder-decoder architecture that accepts inputs in the forms of text, images, video, and audio. The model only allows text outputs at this time. <\/p>\n\n\n\n<p>The core Transformer model makes use of SwiGLU, Grouped Query Attention, Rotary positional embeddings, and RMSNorm, and is built on the &#8216;Noam&#8217; architecture.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"371\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-523-1-1024x371.png\" alt=\"Reka Architecture\" class=\"wp-image-3756\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-523-1-1024x371.png 1024w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-523-1-300x109.png 300w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-523-1-768x279.png 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-523-1-750x272.png 750w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-523-1-1140x413.png 1140w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-523-1.png 1183w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>This architecture bears a resemblance to the PaLM architecture, excluding the parallel layers. A 100K sentence piece vocabulary based on tiktoken (e.g., GPT-4 tokenizer) is used by Reka Flash and Edge. <\/p>\n\n\n\n<p>Sentinel tokens are added by Reka for particular use situations like tool use and masking spans. A multi-stage curriculum with varying mixture distributions, context lengths, and objectives is used in pretraining.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>The Training Data<\/strong><\/h3>\n\n\n\n<p>With a dataset knowledge cutoff of November 2023, Reka Core&#8217;s training data is a combination of proprietary\/licensed and publicly available datasets. Text, photos, videos, and audio snippets make up the dataset that the model learned from. <\/p>\n\n\n\n<p>The training datasets for Reka Flash and Reka Edge were roughly 5 trillion and 4.5 trillion thoroughly deduplicated and filtered language tokens, respectively.<\/p>\n\n\n\n<p>30% of the pretraining data are STEM-related, and about 25% are code-related. A quarter or so of the data comes via web crawling. Reka&#8217;s data has a 10% mathematical component. It is important to note that Reka Core has not finished training and is still improving<\/p>\n\n\n\n<p>Thirty-two different languages are tier-weighted (about based on frequency in the wild) and make up 15% of Reka Core&#8217;s expressly (and purposefully) multilingual pretraining data. In addition to these expressly up-weighted languages, Reka Core received training in all 110 languages found in the multilingual Wikipedia.<\/p>\n\n\n\n<p>The multimodal training data consists of substantial sets of web pages, documents, videos, and photos. Quality, diversity, and scale are carefully optimized for the selected data mixture.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"234\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-525-1024x234.png\" alt=\"Multimodal training data\" class=\"wp-image-3757\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-525-1024x234.png 1024w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-525-300x69.png 300w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-525-768x176.png 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-525-750x172.png 750w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/04\/Screenshot-525.png 1079w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Reka Core\u2019s Groundbreaking Features<\/strong><\/h2>\n\n\n\n<p>Reka Core comes with several capabilities and features that users from the developer community can highly make use of. Let\u2019s take a look at all of them:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Multimodal (image and video) understanding:<\/strong> Reka Core is more than just a large-scale frontier language model. It is one of only two commercially available full multimodal solutions with a robust contextualized understanding of images, videos, and sounds.<\/li>\n\n\n\n<li><strong>128K context window:<\/strong> Reka Core is capable of ingesting and precisely and accurately recalling much more information. Reka&#8217;s baseline models have an 8K context length. Long context models namely Reka Flash and Reka Core have a 128K context window for retrieval and lengthy document tasks. Every model supports the context in which it is used by passing needle-in-the-haystack (passkey retrieval). These tests appear to support the extrapolation of their 128K context length to 256K context length, but not higher.<\/li>\n\n\n\n<li><strong>Reasoning:<\/strong> Because of Core&#8217;s exceptional thinking skills in both English and maths, it can be used for complicated problems requiring in-depth analysis. \u00a0Even though it has catching up to do with OpenAI in reasoning capabilities, it still is a key AI model in the generative AI market.<\/li>\n\n\n\n<li><strong>Coding and agentic workflow:<\/strong> One of the best code generators is Core. When paired with other abilities, its coding skills can strengthen agentic workflows. So, developers go ahead and ask your DSA-related doubts in several programming languages and topics.<\/li>\n\n\n\n<li><strong>Multilingual:<\/strong> 32 different language sets of text were used to pretrain Core. In addition to speaking various Asian and European languages fluently, it speaks English. We already mentioned before that it received training in 110 languages found in the multilingual Wikipedia.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p>Reka Core is the latest advancement in the field of Generative AI. Its highly competitive benchmarks place it on a level field compared to the top-notch AI companies in today\u2019s world. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Comparing the benchmarks of the Reka Core multimodal model with different LLMs. Also, how to access Reka Core?<\/p>\n","protected":false},"author":15,"featured_media":3764,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jnews-multi-image_gallery":[],"jnews_single_post":null,"jnews_primary_category":{"id":"","hide":""},"footnotes":""},"categories":[57],"tags":[56,157,64,59,91,72,185,186],"class_list":["post-3749","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-ai","tag-claude-3","tag-gemini","tag-generative-ai","tag-gpt-4-2","tag-llm","tag-reka","tag-reka-core"],"_links":{"self":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/3749","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/comments?post=3749"}],"version-history":[{"count":9,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/3749\/revisions"}],"predecessor-version":[{"id":3765,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/3749\/revisions\/3765"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/media\/3764"}],"wp:attachment":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/media?parent=3749"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/categories?post=3749"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/tags?post=3749"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}