It’s no secret that artificial intelligence (AI) has changed how developers work over the past few years. So it shouldn’t come as much of a surprise that it’s also affecting how technical assessments play out during the hiring process.
According to OfferZen’s 2024 AI Skills and Impact Data Report, the rise of AI is making complex problem-solving and soft skills more important than ever. This means that hiring managers need to ensure that their technical assessments identify whether a candidate has these competencies.
Nick Botes, Head of Engineering at OfferZen, recently chatted with Old Mutual’s Head of Quality Engineering, Dawie Greyling, and Codility CTO, Ilya Sakharov, to figure out some technical assessment best practices for finding top talent.
Build expert-led assessments
With 53% of developers reporting that they dislike technical assessments, it’s essential to align this part of your recruitment process with the role you’re hiring for. Here, Dawie said, it’s useful to bring experts on board to help you build and review your assessments.
“I would urge people to get specialists to come in and design the assessment for a particular role using their specific knowledge. You also want to look at who is doing the interview and review the results and make sure that they’re capable of understanding what’s involved and what’s important,” he explained.
Ilya added that bringing subject-matter experts in can result in a fit-for-purpose assessment that adds value to the interview process on both sides – rather than being something that’s done just because it’s standard practice:
“It’s extremely important to tie the assessment to the role that you’re hiring for. If you don’t, you’re going to get something that’s not aligned with your development plans and not aligned with what the developer is going to do. Then everyone loses.”
Separate skills and tools
It’s important to understand what gaps you have in your team in terms of both technical and soft skills before building a technical assessment. According to Dawie and Ilya, this is the best way to develop standards that align with your organisational needs.
“You need to analyse the landscape of existing capabilities in your team and your organisation to see what the gaps are and where you want to take your team in the future. You need to understand what is missing and how you can enable your team. From there, you can identify what skills you need,” said Ilya.
One major pitfall to avoid here is conflating the tools you use with the skills that you need. Although it can be far easier to describe what you’re looking for by mentioning a programming language, this can lead to missing out on the best talent:
“If you're looking for someone working in Java, it doesn't mean you need a person who knows Java. You need a person who understands the OOP concepts of the third generation languages and they will be able to adapt within a relatively short period of time, if they are senior enough. So you’re looking for that understanding – the skill – not knowledge of the job,” added Ilya.
Dawie suggested a practical approach to pinpoint these core skills to test for: “What we did was analyse each technical role within our teams to see what thought processes and skills individuals who perform those roles have. We were looking at, for example, whether they need analytical skills or to have good attention to detail. Then we could create our assessments according to those skills.”
“You need to look at what is specific to the role that you need to fill and then figure out how you can test for the skills it requires. You also might have to keep reminding yourself that tools aren’t skills and they don’t necessarily matter too much; those are things that you can teach in a matter of weeks if you hire someone with the right skills.”
Establish your evaluation configuration
There are a variety of ways that a technical assessment can play out. But whether it’s a whiteboard session, coding interview or simulation day, Dawie noted that you’ll want to be clear on what you want to evaluate before you start building your process out.
Dawie and his team focus on three specific skills: soft skills, theoretical knowledge and technical skills.
“So, we use a psychological test that helps us to assess growth capability and analytical skills, like logical solutioning or out-of-the-box thinking. Then we look at the theoretical with a multiple-choice test and then on to a full practical where we give short, practical scenarios from test demo sites in the world.”
Once you know what you’re looking for and how you’d like to assess it, Ilya pointed out two major types of technical assessments that you can choose from – take-home tests and practical, collaborative sessions:
- Take-home tests: “The take-home test we know, you give a person a task and they take that task and do it in their spare time. That tends to be losing relevance now because of AI and, also, because more senior developers are reluctant to spend the time doing these things when they have their own GitHub reports or contributions to open source code that you can go and look at,” he said.
- Collaborative sessions: “Now we’re seeing another assessment that I believe is starting to take prevalence in hiring senior developers, which is collaborative creation. This is a situation in which you have a collaborative IDE [integrated development environment] that multiple people can connect to and can emulate a programming session and use it to create a solution for a problem.
A collaborative session not only allows you to see how people write code, but also how they interact with others. This type of interview allows you to answer three key questions:
- How do they lobby for their ideas?
- How do they defend their principles?
- How do they receive and give feedback?
“So, it gives them the opportunity to see what it’s like to work in your team and you can also identify whether they’d be a match for your culture,” Ilya said.
Tips for improving your technical assessment process
In addition to knowing what skills gaps you’re trying to fill and figuring out which type of technical assessment would be most appropriate for identifying these, Ilya and Dawie gave some quick tips for making the most of your assessment process:
- Leverage AI: “Ask AI to assess candidates based on the wording that they use, how they talk and the language that they use to determine their skill level,” advised Dawie.
- Be flexible: “Offer the candidate a variety of ways to implement the solution to the problem rather than telling them to use a specific tool, because you’re assessing their soft skills not how well they use a tool,” said Ilya.
- Regularly review your assessments: “Every time there’s a new role going out, try to look at your assessments again. When candidates all start getting high scores or we find that we have enough of a certain personality type, we change our structure,” said Dawie.