{"id":5348,"date":"2024-06-01T19:09:56","date_gmt":"2024-06-01T19:09:56","guid":{"rendered":"https:\/\/favtutor.com\/articles\/?p=5348"},"modified":"2024-06-01T19:10:35","modified_gmt":"2024-06-01T19:10:35","slug":"new-autocoder-llm-beats-gpt-4o","status":"publish","type":"post","link":"https:\/\/favtutor.com\/articles\/new-autocoder-llm-beats-gpt-4o\/","title":{"rendered":"New AutoCoder LLM, An Open-Source AI Coding Assistant Beats GPT-4o"},"content":{"rendered":"\n<p><strong>AutoCoder, a brand-new open-source LLM for coding, has shocked the world by defeating OpenAI\u2019s GPT-4 Turbo and GPT-4o (Omni), across benchmarks related to coding.<\/strong><\/p>\n\n\n\n<p><strong>Highlights:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>AutoCoder, the latest AI based Coding Assistant LLM, has defeated GPT-4o in coding benchmark.<\/li>\n\n\n\n<li>Built on the AIEV-Instruct Architecture which helps in downloading coding dependencies without needing any further assistance.<\/li>\n\n\n\n<li>The model is open-source and can be run locally.<\/li>\n<\/ul>\n\n\n\n<p>It\u2019s been just a few days since GPT-4o was launched and it took the generative AI world by storm with its natural language processing and strong vision capabilities. Many developers around the world have enjoyed GPT-4o\u2019s capabilities in developing AI systems, running and debugging complex codes, and much more.<\/p>\n\n\n\n<p>Now, just days later, we already have an AI coding Assistant that has surpassed GPT-4o in coding benchmarks. AutoCoder is here to completely shock the world in this ever-evolving Generative AI industry.<\/p>\n\n\n\n<p>In this article, we will look into AutoCoder\u2019s brand-new open-source technology and analyze this state-of-the-art coding assistant in much detail. So, let\u2019s explore right away!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>AutoCoder: The Latest AI Coding Assistant<\/strong><\/h2>\n\n\n\n<p><strong>Autocoder, a new large language model, has surpassed GPT 4 Turbo and GPT-4o Omni in benchmark tests. Its broad range of applications is enhanced by providing an adaptable code translator that permits the installation of external packages.<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-embed aligncenter is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"jeg_video_container jeg_video_content\"><iframe title=\"NEW AutoCoder LLM Beats GPT-4o! Best Opensource Coding LLM!\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/IXcF83xsmCU?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<p>AutoCoder comes in two versions: 6.7 billion and 33 billion. <strong>The 33 billion version has beaten GPT-4 Turbo and GPT-4o in the Human Eval Benchmark.<\/strong><\/p>\n\n\n\n<p>By installing necessary packages automatically and allowing for selective interpreter use, Autocoder performs better than other models. This is what developers are looking for nowadays, a coding assistant that can install all dependencies without any instructions. This saves up a lot of time for hectic coding tasks and development projects.<\/p>\n\n\n\n<p><strong>AutoCoder comes with a unique training methodology termed AIEV Instruct (Instruction Tuning with Agent-Interaction and Execution-Verified). This technique combines agent interactions with external code execution and verification. This is a highly innovative technical collaboration that takes the coding potential to very high limits.<\/strong><\/p>\n\n\n\n<p>This allows AutoCoder, whenever the user wants to run the code, to install the necessary packages automatically and try running the code until it determines there are no problems.<\/p>\n\n\n\n<p>This open code interpreter offers freedom in code verification by executing all generated Python code without requiring user input. Overall, this LLM is the perfect example of an AI agent doing the coding tasks for you automatically, without requiring any assistance at all!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>How can you Access it?<\/strong><\/h2>\n\n\n\n<p><strong>AutoCoder is currently Open-Source and is accessible to all.&nbsp; You can download AutoCoder\u2019s model weights from <a href=\"https:\/\/huggingface.co\/redponike\/AutoCoder-GGUF\" data-type=\"link\" data-id=\"https:\/\/huggingface.co\/redponike\/AutoCoder-GGUF\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">Hugging Face<\/a> and run it locally using LM Studio.<\/strong><\/p>\n\n\n\n<p>See this video below, to understand how you can setup AutoCoder Locally!<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"jeg_video_container jeg_video_content\"><iframe title=\"Install AutoCoder Locally - Beats GPT-4o Code Interpreter\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/x3BN3yVVecs?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>AIEV-Instruct: The Overall Architecture<\/strong><\/h2>\n\n\n\n<p><strong>The overall architecture of AutoCoder is built upon AIEV-Instruct, a novel method for creating high-quality large code datasets. <\/strong>Through agent interactions, it simulates programmers developing code and running unit tests while guaranteeing documentation accuracy&nbsp;while working with an external code executor.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><em>\u201cCompared to previous large-scale code dataset generation methods, AIEV-INSTRUCT reduces dependence on proprietary large models and provides execution-validated code dataset.\u201d<\/em><\/p>\n<cite>Bin Lei, primary author, in <a href=\"https:\/\/arxiv.org\/pdf\/2405.14906\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/arxiv.org\/pdf\/2405.14906\" rel=\"noreferrer noopener nofollow\">Autocoder&#8217;s research article<\/a>.<\/cite><\/blockquote>\n\n\n\n<p><strong>The architecture comprises a Teaching Stage and a Self-Learning Stage, which lessens the annotation process&#8217;s dependency on expensive closed-source models. Let\u2019s take a look at these two stages in depth.<\/strong><\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"1232\" height=\"666\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-720-1.png\" alt=\"AIEV-Instruct Architecture\" class=\"wp-image-5351\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-720-1.png 1232w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-720-1-768x415.png 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-720-1-750x405.png 750w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-720-1-1140x616.png 1140w\" sizes=\"(max-width: 1232px) 100vw, 1232px\" \/><figcaption class=\"wp-element-caption\">Source: <a href=\"https:\/\/arxiv.org\/pdf\/2405.14906\" target=\"_blank\" data-type=\"link\" data-id=\"https:\/\/arxiv.org\/pdf\/2405.14906\" rel=\"noreferrer noopener nofollow\">Research Paper<\/a><\/figcaption><\/figure>\n<\/div>\n\n\n<h3 class=\"wp-block-heading\"><strong>The Teaching Stage<\/strong><\/h3>\n\n\n\n<p><strong>During the Teaching Stage, GPT-4 Turbo is used as the teacher model to supplement and rectify the open-source code snippets used in the model. This stage is composed of 4 phases:<\/strong><\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>1. Initialization<\/strong><\/h4>\n\n\n\n<p>Two tasks are allocated to GPT-4 Turbo: <strong>programmer and questioner.<\/strong><\/p>\n\n\n\n<p>Ensuring diversity in the generated data can prevent convergence to a particular discussion template and produce a more uniform probability distribution. The dialogue messages start as an empty list and are used to hold data during the process.<\/p>\n\n\n\n<p>The dialogue will eventually be recorded in numerous rounds on this list and will be included as a single data entry to the final dataset.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>2. Proposing the Question<\/strong><\/h4>\n\n\n\n<p>To run OSS-Instruct, GPT-4 Turbo is used. A problem description and a targeted solution are created, including a code snippet based on the open-source code fragment.<\/p>\n\n\n\n<p>The primary purpose of GPT-4 Turbo here is to supply certain Unit Tests. These unit tests further guarantee the accuracy of the code in the dataset. The problem description, the solution, and the unit tests are successively appended with the dialogue messages initialized in the preceding stage.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>3. Execution Feedback<\/strong><\/h4>\n\n\n\n<p>The researchers used multiple rounds of execution feedback to check the generated code, thereby improving the dataset&#8217;s quality.<\/p>\n\n\n\n<p>The code snippet created in the second stage was first entered into the Code Interpreter by the researchers. The dialogue messages append the comprehensive Stderr output in the event of an execution error. In the meantime, the questioner receives this Stderr data and uses it to create a natural language description.<\/p>\n\n\n\n<p>The programmer will then continue to alter the code, receiving additional inquiries in natural&nbsp;language descriptions&nbsp;and the Stderr. The dialogue messages will continue this process, appending the new code that it generates.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>4. Termination<\/strong><\/h4>\n\n\n\n<p>Lastly, the model also uses the Code Interpreter to run the code generated by the programmer. If the program executes successfully, the Stdout is appended to the dialogue messages. This completes the analysis of one data entry<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>The Self-Learning Stage<\/strong><\/h3>\n\n\n\n<p><strong>During the Self-Learning Stage, AutoCoder refines its understanding through continuous interaction and feedback.<\/strong><\/p>\n\n\n\n<p>The Self-Learning Stage differs from the Teaching Stage in that the student model takes the place of the original teacher model in this stage. The entire execution feedback process is carried out by the student model, which is designated as the questioner and coder.<\/p>\n\n\n\n<p>Overall, this approach ensures that AutoCoder generates accurate and reliable code, setting it apart from other models in the market.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The Shocking Benchmarks<\/strong><\/h2>\n\n\n\n<p>AutoCoder has completely shocked the world with its powerful benchmarks.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"948\" height=\"642\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-719.png\" alt=\"Benchmarks AutoCoder\" class=\"wp-image-5350\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-719.png 948w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-719-768x520.png 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-719-750x508.png 750w\" sizes=\"(max-width: 948px) 100vw, 948px\" \/><\/figure>\n<\/div>\n\n\n<p><strong>According to the trial results, AutoCoder achieved remarkable results in Java, C++, and Rust, with Pass@1 scores of 61.4%, 68.9%, and 60.8%, respectively. Its performance was only surpassed by a few models (like CodeQwen1.5-Chat) in the other three languages.<\/strong> This illustrates how strong AutoCoder&#8217;s multilingual code creation skills are.<\/p>\n\n\n\n<p>It is superior to OpenAI&#8217;s GPT-4 Turbo and GPT-4 Omni in the HumanEval benchmark, and it performs exceptionally well in coding jobs. <strong>Its test accuracy on the HumanEval base dataset is higher than that of GPT-4 Turbo (April 2024). (90.2% as opposed to 90.9%).<\/strong><\/p>\n\n\n\n<p>Because of its better efficiency, AutoCoder is a great tool for developers looking for open-source, dependable solutions for their coding demands as well as sophisticated code interpretation skills.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>What makes AutoCoder special?<\/strong><\/h2>\n\n\n\n<p>Code Interpreter is one of the key parts of a coding assistant nowadays. The Code Interpreter helps the model to debug and run code, which is necessary for completely automating operations connected to scientific computations, sophisticated coding, and related duties.<\/p>\n\n\n\n<p>The model must correctly identify the code blocks that it needs to execute in order to build a code interpreter. But currently, only a few AI models such as GPT-4o and InternLM-Chat support such interpreters.<\/p>\n\n\n\n<p>These interpreters&#8217; inability to communicate with external systems and their closed operating environment, however, severely restricts their ability to run code that calls for the installation of external packages.<\/p>\n\n\n\n<p><strong>This is where AutoCoder\u2019s ability to install dependencies without requiring any sorts of instructions comes into play. This is accomplished by training the model to execute commands with the bash script when necessary.<\/strong><\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"1454\" height=\"561\" src=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-721.png\" alt=\"AutoCoder Using Code Interpreter\" class=\"wp-image-5352\" srcset=\"https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-721.png 1454w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-721-768x296.png 768w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-721-750x289.png 750w, https:\/\/favtutor.com\/articles\/wp-content\/uploads\/2024\/06\/Screenshot-721-1140x440.png 1140w\" sizes=\"(max-width: 1454px) 100vw, 1454px\" \/><\/figure>\n<\/div>\n\n\n<p>For a simple single execution feedback example, the original data entry contains three parts: natural language from the <strong>User<\/strong>; natural language + bash command + natural language + code block + natural language from the <strong>Assistant<\/strong>; and execution result from the <strong>code interpreter<\/strong>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Conclusion<\/strong><\/h2>\n\n\n\n<p>AI coding assistants will never be the same after AutoCoder. With its innovative capabilities, distinct training approach, and exceptional benchmark performance, it&#8217;s an excellent option for developers and enterprises searching for a potent open-source solution. AutoCoder could change the way we approach coding tasks with its capacity to install external packages, generate precise code, and learn continually through feedback and interaction.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>AutoCoder, a brand-new open-source LLM for coding, has shocked the world by defeating OpenAI\u2019s GPT-4o, across benchmarks related to coding.<\/p>\n","protected":false},"author":15,"featured_media":5361,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jnews-multi-image_gallery":[],"jnews_single_post":null,"jnews_primary_category":{"id":"","hide":""},"footnotes":""},"categories":[57],"tags":[56,61],"class_list":["post-5348","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","tag-ai","tag-chatgpt"],"_links":{"self":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/5348","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/comments?post=5348"}],"version-history":[{"count":6,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/5348\/revisions"}],"predecessor-version":[{"id":5362,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/posts\/5348\/revisions\/5362"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/media\/5361"}],"wp:attachment":[{"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/media?parent=5348"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/categories?post=5348"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/favtutor.com\/articles\/wp-json\/wp\/v2\/tags?post=5348"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}