It's likely ChatGPT has already broken your traditional technical assessment. With the advent of AI tools like ChatGPT, you need to rethink your approach to assessing developers' skills and making solid hiring decisions. As an Engineering Hiring Manager at OfferZen, here are my key takeaways on how to do this.
How ChatGPT Has Broken the Technical Assessment
ChatGPT is an AI tool that has skyrocketed in popularity. Developers can use it to write and debug code and even get explanations for code changes. While you might see some errors and low-quality answers, for the most part, it’s a powerful tool that I expect will continue to see widespread adoption as it evolves.
As I was experimenting with its capabilities, I wondered what impact ChatGPT has on the technical hiring process. It turns out the implications are profound.
ChatGPT can solve many forms of technical assessment
Some forms of technical assessment rely on providing candidates with code ahead of time to analyse as part of a technical discussion or code review.
Using ChatGPT, candidates can get accurate answers and suggestions for code changes. The only safe territory seems to be questions requiring problem-solving abilities and creativity.
What happened when I fed OfferZen’s technical test through ChatGPT? It provided many of the answers I was looking for using our assessment rubric, especially those aimed at more junior and intermediate candidates. This means that candidates can do the same before the interview and do fairly well without producing these answers themselves.
See this example of ChatGPT providing the answer to a question from a technical test:
Asking a candidate to write code ahead of time poses similar challenges: Many take-home assignments can be solved by ChatGPT, especially if it relies on less complex or self-contained problems.
In these cases, ChatGPT can give good explanations for code changes or provide code suggestions that could pass a technical assessment.
Ways To Adapt Your Technical Interview Process
Here are some ways to mitigate the effects of AI tools like ChatGPT on various forms of technical assessment:
1. Use ChatGPT to refine your test
Before you use a technical test in your interview process, run it through ChatGPT to see what answers it gives you. You can use this to refine your questions or the problems you’re asking candidates to solve — anything that relies on straightforward solutions wouldn’t make for a good test.
Focus on questions that test candidates’ creativity, ability to give thoughtful answers, familiarity with specific technologies, and application of technical concepts beyond what ChatGPT can provide.
The downside is that junior developers might struggle to do well in some of these tests, simply from a lack of experience. A different approach might be needed when assessing junior developers. This brings me to my next point:
2. Use methods that test candidates in a live setting
Interviewing techniques that don’t provide candidates with code ahead of time, like pair programming and whiteboard interviews, will be less affected by ChatGPT.
You could also consider only revealing code to the candidate during the interview. The drawback of this is that candidates do not have time to familiarise themselves with the code, which will raise candidates' anxiety. You will also have to adjust the interview length to give the candidate enough time to read through and understand the code.
If you’re conducting a remote interview, this doesn’t entirely rule out candidates using ChatGPT in the background, but it would be easier to pick up on from interactions with the candidate.
3. Make ChatGPT part of your interview
Lastly, you can also work with AI rather than trying to circumvent it. Getting great answers from ChatGPT requires good domain knowledge and creativity — these are skills we’re looking for in a good candidate. You can allow the use of ChatGPT and make it part of your assessment.
Asking candidates to solve a problem while allowing for the use of ChatGPT can reveal their level of expertise and creative thinking abilities. For example, let candidates share their methods for solving the problem with the help of ChatGPT.
The benefit is that this is more authentic to how developers do their job in the real world or will likely work in future.
Eventually, all developers will use ChatGPT, or some form of AI, in their day-to-day work. When the internet came along, we tried to curb its use when assessing candidates. Today, we accept it as part of the assessment process.
We need to figure out how to work alongside AI: How can you gauge a person's technical ability, even when AI augments it? The solutions we come up with could raise the bar for technical interviews and, in the process, get rid of technical assessments that do not test for real-world ability.
Karin Bothma is a senior software engineering manager at OfferZen, with extensive experience in software development and leading engineering teams. She currently leads an engineering team at OfferZen and is helping other engineering managers drive performance improvements in their teams. When she’s not coaching teams to reach their highest potential, she’s out on the trails with her Jack Russels or playing board games with friends.
Read more
- How to Conduct an Effective Tech Assessment. This chapter from our Ultimate Developer Hiring Guide provides a breakdown of common types of tech assessments, including online assessments, technical interviews and take-home assessments, to help you choose the most effective method for your hiring process.
- Exploring Artificial Intelligence: Understanding the Basics and Its Applications
- How to Run a Technical Interview to Hire Developers Efficiently