With so many artificial intelligence (AI) tools available, it’s unsurprising that the majority of coders are using AI to get things done. Given how these tools can boost devs’ abilities, tech team leads and hiring managers need to adjust their hiring process to work with AI.
OfferZen Senior Engineering Manager Karin Bothma chatted to Harley Ferguson, Co-founder and CEO at Origen Software about how tech businesses can revamp their hiring processes to accurately and effectively assess developers’ skills.
The realities of interviewing developers in the age of AI
The surge in coding assistants has given devs the ability to increase productivity, but they can make it challenging to get a full view of candidates’ abilities during the interview process.
“More and more developers are leveraging these tools to get lots done, but we have to figure out how to assess that. How do you know when someone knows something or when someone is using AI? And what’s the difference? And, should you really care?” said Harley.
As far as coding tools go, the Origen Software co-founder and CEO believes that AI is going to become the “Swiss Army Knife of our lives”. He noted that this is something that tech businesses should be aware of and prepared to evaluate when hiring talent.
“It’s pretty much a guarantee that any developer is using or knows about AI coding systems and how they can implement them in their day-to-day work to be more productive or just have a higher quality of output. Understanding how and when to use it is going to be really crucial, and I think that if companies don’t adapt how they hire in this environment, they’ll start to struggle to identify top candidates,” he said.
How to leverage AI to identify and assess top talent
Tech leaders need to understand which tools are out there, what they’re used for and how they can help to enhance the hiring process.
“It’s the responsibility of business owners, technical interviewers and developer managers to catch up and understand that the jack is out of the box with AI. Then we need to figure out how we can use these tools to still do justice to the individual, not just assuming that everyone is the same and definitely not discrediting them for using AI,” he said.
The natural question that flows from this is: How do you effectively assess developers’ ability when they’re using AI during interviews?
Harley outlined a few strategies that the team at Origen Software has used to identify devs’ specific skills and better understand these coding proficiencies, as well as how adept they are at using AI to augment their abilities.
1. Integrate AI tools into your hiring process
A great place to start with using AI to identify top talent is to streamline your recruitment cycle. “The hiring process is being impacted by AI just like every other industry, so there’s opportunity to leverage it to find candidates and interview them,” said Harley.
“We use Notion, its AI is really powerful and there are a lot of cool integration tools like Zapier to keep things organised. We also use Otter.ai for all of our interviews. That records everything and creates pretty accurate transcripts with action points that we can use to share the highlights from our conversations with candidates.”
Harley also suggests training your own local LLM on your codebase and feeding a candidate’s coding assessment into it. “It provides a summary of what they've done well, according to our standards, what they've missed, what they've done poorly. And obviously, we still go through the code. But it’s a great way to quickly measure where this potential person is or what they've missed.”
2. Assess the candidate’s CV
It may seem like an obvious place to start, but the best way to tell if you’re speaking to a developer who uses AI effectively is by looking at their CV.
First off, decide whether potential hires using AI to help them generate their CVs is a dealbreaker. “Do you want a candidate to tell you about their experience in their own words or are you fine with them using AI to spruce it up?” said Harley.
Then, use AI yourself to pinpoint the standout points on the CV. “We take candidate CVs and put them through our LLM to quickly see what the highlights are and what the talking points could be during the interview,” he added.
3. Understand how candidates are using AI
Accepting that AI is everywhere and developers are going to have access to these tools – whether they’re interviewing or working with you – can open up a host of opportunities for better assessing candidates’ abilities.
“In our minds, AI is a tool just like anything else. So how you leverage that tool is more important than if you use it or not. What we’re looking for is: how are they using it? Why are they using it? What are they doing with the information that they get?” said Harley.
“Here, the team at Origen uses a variety of strategies that help them better understand how a candidate has used AI to complete an assessment – the most important being the prompts they’ve input into their chosen tool.”
“We have a spreadsheet that we provide with a bunch of columns where candidates can input the prompts they’ve used and all the parts of the assessment that they used AI for.
“We’ve found that going through the prompts and how they use them is most beneficial because it’s somewhat like having them speak through a problem themselves. Kind of like the approach of being in a room with a whiteboard, the prompts give us some dots that we can draw a line through to understand their thinking,” said Harley.
There are also a couple of questions that you can ask when considering a potential hire’s outputs:
- Is there anything they did that could help to improve the codebase?
- Were they researching or did they understand standards and best practices?
- Were they able to use the tool to implement standards or make their file consistent with others in the repository?
Harley pointed out that you’ll also want to look out for AI comments in the outputs. “If someone is leaving AI comments in their pull requests, that’s a problem. Those definitely shouldn’t be left in there,” he said.
Hire to complement AI
Although AI is a “jetpack” for developers, it’s important for hiring managers to keep in mind that there are plenty of functions that it can’t fulfil. Harley noted that this means that assessing soft skills during the interview process will become more important than ever.
“The ability for someone to problem solve and their ability to work well on the team – understand what you need, to be a fun person to work with and align with your core values – is inherently more important now because the technical aspects are getting easier to achieve,” Harley said.
Tech team leads can simultaneously assess developers' technical skills and collaboration abilities by incorporating elements like simulation days into the hiring process. This helps to give you a holistic view of their skills and how they’ll add to your team’s culture.