Here’s What Developers Found After Testing Devin AI

It’s been almost three days since Devin AI’s announcement and developers worldwide have eagerly awaited early access to the software engineering tool.

However, a few users have the early first-hand Devin AI experience. Some have highly praised the mind-blowing coding and debugging capabilities that come with the tool, however, a few have had second thoughts about the tool potentially replacing them as software developers.

In this article, we will take an in-depth look at the reactions of a few users who explored Devin AI’s features and the vast use cases that the tool offered. So, let’s find out!

Devin’s Performance in Early Access

For now, Devin has been rolled out only for early access. Developers can request early access by filling out a Google form stating their tool needs and preferences. Depending on their need types, they will be queued on the waitlist.

However, we found some users who got early access to Devin and have provided an in-depth analysis of their experience with this ‘AI Software Engineer’.

1. Devin Setting up a Chatbot UI Project

Mckay Wrigley, a passionate and popular AI enthusiast took to X, to give a detailed video of his experience with Devin AI. Take a look at the video below in the tweet to see Devin in action.

He asked Devin to set up his locally running Chatbot UI app. As shown in the video above, it does an excellent job in first cloning the GitHub Repository entirely. It also continues further to analyze the README file from the repository to understand the project needs and requirements.

It then installs the required project dependencies using npm. This approach is excellent from Devin as the repository had a lot of .json files. In the end, it successfully recreates the UI Chatbot app in the interface, just as the user was running on his local machine.

Another interesting aspect of this cloning was that Devin even logged in to the Chatbot app when the user gave it his User ID and Password. This just goes to show that its capabilities go beyond just coding, debugging, and cloning repositories.

2. Creating a US Airport Map

Ethan Mollick, an AI professional who also got early access, asked Devin to create a map stating the US airports along with the travel times associated with them.

Devin did an excellent job first in gathering data sources on the US Airports. It then set up a local development environment designed for a React application. It took to Chakra UI components for this functionality.

It did an excellent job of developing a mapping library for the frontend interface using Leaflet.js. This shows Devin can also set up interactive web apps or mobile apps using various APIs or available software packages.

Lastly, Devin deployed the generated map interface using Netlify. It didn’t just stop with deployment, as it continued to test the code thoroughly. It ensures perfect performance in the production environment.

An interesting moment in the whole interaction was when the user found a bug during the process, not only did it autonomously fix the error, but also continued to assure the user that it was working on fixing the error and it was an important step of the development process.

The output was an excellent map-like interface with all US airports stated for optimal travel times. The only complaint the user had was that the tool was slow and sometimes broke down during the process, but still, the positives made up for this.

3. A website to play chess against an LLM (GPT-4 API)

Andrew Kean Gao, an AI enthusiast studying at Stanford University and working at LangChain AI, also got early access to Devin. The task he gave to the tool was to create a website to play chess against an LLM-based opponent, specifically GPT-4 API.

Andrew further stated in his tweet that this is an extremely complex task as most LLMs have no idea how to use APIs, especially the GPT-4 API. He also mentioned further whether Devin would securely handle the API keys and deal with any associated package errors.

To his surprise, Devin not only asked for the API key but also handled it securely. However, this user also complained about Devin being slow as the tool took almost 19 minutes to process before it asked for the API key. Further, it was working on making the game plans using chess.js as per the npm documentation and related errors. This is a crucial step in deciding the game’s movements and errors.

Lastly, the user shared an image in which Devin was seen debugging a rendering issue and reviewing an export method to fix the import statement of the Chess playing environment.

4. Developing a Chrome Extension

Andrew even continued to test Devin’s capabilities. He asked it to develop a Chrome extension that turns GitHub Repositories into Claude Prompts.

Depending on the page you are now viewing, the extension automatically fills in the Repo2Prompt GitHub URL. You then need to provide a GitHub token. Additionally, it generates the user’s download link for the TXT file.

This is excellent work from Devin! Who knew that a day would come when AI tools would generate Chrome extensions for us? We are on the verge of having everything in the palm of our hands!

You can even download the extension and try it for yourself through here. open the developer settings for Chrome extensions, unzip the file, and load it.

How is Devin being perceived?

All the reactions and tests by developers mentioned above have highly enjoyed and experienced Devin’s capabilities first-hand. The early access phase has mostly been positive. Users have hardly had any issues with the tool, except for the one fact that the tool is slow when it comes to code processing and repository cloning.

But we can assume that Cognition will address this issue in the days to come. The team behind Devin AI has still not offered any updates on when it will be made public.

For now, we can’t help but appreciate how marvellous the tool already is in its beginning phase. Developers worldwide can’t help but wait before they finally have the full access and script their projects and personal deployments.

However, some users have considered the possibility of Devin potentially replacing humans in the field of software development. It can highly steal the jobs of software developers and debuggers, as major firms and enterprises are already considering the usage of Devin for scripting source codes.

Do we as humans need and should promote an AI tool that can potentially replace us entirely in the long run? Maybe AI is just getting started.

Conclusion

Devin is an excellent advancement in the field of Generative AI as it narrows the gap between AI and Software Development. The early access phase has been mind-blowing as users worldwide have been in both awe and tension about its potential impacts on society. Only time will tell how far it can go, as more users across vast sectors and departments get their hands on the tool.