In this article, we explore AI’s impact on tech assessments and what companies need to do to adapt their processes and ensure they’re still providing a great candidate experience.
Coding challenges are declining as AI tools rise in popularity
AI has become an integral part of software development for South African developers, and shows no signs of going away: The majority of local developers now use code-writing assistants and ChatGPT is their most used AI tool.
This has led many companies to worry about developers using these tools to gain an edge in technical assessments. We can see this playing out in the data:
Developers assessed with an online coding challenge
The number of companies using online coding challenges to assess developers has fallen 21.6% over the past 12 months, while whiteboard interviews have also seen their popularity decline.
It’s no secret that AI tools like ChatGPT can solve problems commonly found in online coding challenges. Developers are also aware of the the advantages of AI tools in assessments: We asked the dev community about this and the majority said that using AI tools in assessments can misrepresent their skills:
At the same time, it’s not enough to ban AI from your assessment process, as it’s becoming an integral part of developers’ day-to-day workflow. How you conduct your hiring process is one of the top factors developers use to assess your company culture. On top of that, an irrelevant assessment or asking too many technical questions that are unrelated to the role are significant contributors to a negative experience.
This made us ask: What can companies do to adapt their hiring processes in response to the rise of AI?
Work with AI to improve your technical assessments
One approach companies can take to adapt their hiring processes is working with AI tools to improve their assessments. Karin Bothma, Engineering Manager at OfferZen, suggests running your technical tests through ChatGPT to see what answers it gives you. You can use this information to refine your questions or the problems you’re asking candidates to solve, as anything that relies on straightforward solutions wouldn’t make for a good test.
Instead, focus on questions that test candidates’ creativity, ability to give thoughtful answers, familiarity with specific technologies, and application of technical concepts beyond what ChatGPT can provide.
Test how potential team members leverage AI
A second option is to work with AI rather than trying to circumvent it. Getting great answers from ChatGPT requires good domain knowledge and creativity — these are skills you’d want in a good candidate. So, you can let candidates use ChatGPT or another LLM-based tool and make it part of your assessment.
Ilya Sakharov, CTO of Codility, suggests testing for AI-specific skills as part of your process. This can include things like proofreading and reviewing of AI-generated code, or how they prompt AI tools to achieve a desired outcome.
Alternatively, you can ask how developers are already using AI to determine if they have the fundamentals they need to use these tools to improve their productivity or if they’re masking shortcomings.
For more data and insights to make your hiring process as efficient as possible, get our 2024 Developer hiring & retention report.
Further reading:
- 2024 developer hiring and retention support
- 2024 Software Developer Salary Benchmarking Report
- Decoding the 2024 tech job market
- Developer salaries are stagnating: Here’s how to prepare for tougher salary conversations
- Senior developers are losing their ability to negotiate high increases, but can bonuses sweeten the deal?
- Not all seniors are equally affected by slowing salary growth — your tech stack matters too
- Week in Review: Full stack skills are in demand but it still pays more to specialise on the backend
- More developers are looking to go full stack to drive more product impact
- Management red flags that kill retention
- Enabling and not replacing: How AI is freeing up devs to do more impactful work