Companies hire developers for entry-level positions based on how well they solve DSA problems and by looking at their Coding Platform performances. Leetcode is considered to be one of the best platforms for solving DSA-related problems. But what if you can automate the entire process? Concerning but interesting right?
Claude 3.5 Sonnet Solved 633 Leetcode Problems
A Reddit user named “TimS2024” has automated Leetcode using Claude 3.5 Sonnet API and a small Python script to solve 633 problems in 24 hours with a success rate of 86%.
People are getting frustrated with building and maintaining their coding platform profiles. Tim Shelton was one such person. So, he decided to build a project and showcase his skills by building an automated leetcode problem solver.
Here is the automation code in progress.
— AshutoshShrivastava (@ai_for_success) August 21, 2024
📹 via @rohanpaul_aipic.twitter.com/HIRIk1MrNK
The way the model works is very interesting. It starts looking for a leetcode problem on its problem solver page, reads the given instructions and writes code accordingly.
In case, it fails to answer any test cases – it takes note of them and rewrites the entire code. If it succeeds in the problem, it goes again to the leetcode problem-solving page and picks a random question that is not Leetcode premium. This entire process goes on a loop.
Using this, the developer has solved over 633 problems ( 217 easy, 359 medium, 57 hard) in 24 hours. How much do you think this costs? Just $9 Yes, you read it right. The entire program cost just 9 dollars in API credits for 24 hours.
Some interesting observations that the user has made while building this project are :
- Anthropic’s Claude was very disciplined in following the instructions given. Google’s Gemini and OpenAI’s GPT models did not perform well according to the expectations.
- LeetCode has almost 20 layers of HTML/CSS/Java elements which makes it very difficult to dig through. Finding problem URLs or problems which are not premium was a huge task.
- The results submitted were always in the top 30% in terms of runtime leaderboards which is unexpected as the code generated by AI models would always be a bit low.
Some users on Twitter responded to a post on this, saying how crazy it is to evaluate engineering skills based on leetcode. Some also remarked that this is going to bring a change in how software engineer interviews are going to be conducted in the future.
AI can get that $500k FAANG job for <$10. pic.twitter.com/mzyfiJMmDY
— Deedy (@deedydas) August 21, 2024
Here are some reactions on X: “This just shows how stupid leetcode is for evaluating engineering skill“, or “This will change the interview process for sure“.
A user also wrote: “Software engineering will become leveraging AI to solve business problems. There will be a lot more competition from other technically apt people though, that weren’t previously software engineers“.
This also sparked the debate on whether AI will replace Software Engineers or not. See some user responses on this issue, but all hope is not lost:
The only people AI will replace are the ones that are not upskilling. If you’re just making react frontends and aren’t interested in doing more than that, your job will be replaced in no time
— Anthony Hagidimitriou (@anthony_hagi) August 21, 2024
Those that continue to upskill will always have a job because AI, even though it’s fast…
Conclusion
While this opens up new avenues for efficiency and automation, it also invites a broader discussion on the role of such tools in the learning process. Evaluating people based on only DSA skills during interviews is not going to be enough because recent developments like this are happening using AI.
