3 pro-AI strategies to evaluate dev skills in your tech assessment

If you want reliable community data on AI’s impact, check out the 2024 AI Skills and Impact Report. Get access to the report now.

Using AI to complete assessments took off as rapidly as AI developments over the past two years. Now, most developers say they’ve used an AI tool to complete an assessment but most companies still don’t have policies around AI use in place. The impact? No standard approach for gauging if candidates can use AI tools effectively.

Here, we unpack how AI impacts dev skills and three ways companies can respond to ensure they get the best holistic understanding of candidates’ skills.

OfferZen_How-to-stand-out-in-technical-assessments-with-the-rise-of-AI-tools_inner-article

AI is shaking up how companies think about dev skills, but not always how best to test them

AI tools have made it easier to generate code at speed, shifting the focus away from developers’ technical skills to their soft and problem-solving skills. At the same time, the ability to critically assess AI outputs and how they contribute to the broader business is more important than ever.

However, many companies have not responded fast enough:

Online coding challenges remain the third most used assessment tool, even though their use has dropped by 21.6% in the past year due to developers using AI tools to pass them.

On top of that, more than half of South African developers say they've used AI tools to complete a technical assessment.

Have you used an AI tool during a technical assessment?

Get more reliable data on AI’s impact

The current challenge faced by companies is understanding how to assess skills when AI use is widespread and becoming an integral part of the software development process.

The icing on the cake? Most companies don’t have policies in place around AI use in assessments. This is causing two main issues:

The lack of a standardised approach for using AI in assessments leads to unreliable, uncomparable test results
Candidates may use AI tools without necessarily showcasing their proficiency due to the absence of standardised testing methods.

The current challenge faced by companies face is understanding how to assess skills when AI use is widespread and becoming an integral part of the software development process. The best place to start is by embracing AI and its growing role in the process.

What’s your policy for the use of AI in tech assessments?

AI requires embracing new tools and rethinking what success in assessments looks like

AI is not going anywhere. Making sure AI’s contributing to, and not hindering, your assessment process starts by embracing AI's new role within it.

By doing this, you're in a better position to choose assessments that determine if someone has the right AI skills and if they’re using these tools appropriately.

From there, you can explore using assessments that measure how candidates use AI or ones that test their approach to problems more generally. Here are three strategies for doing that in practice.

1. Let candidates document how they use AI during the tech assessment

The first option is letting candidates use AI to complete assignments and assessing how well they utilise those tools. Harley Furgeson, CEO and Co-founder of Origen Software suggests asking candidates how they could improve their code with AI.

He also recommends requiring developers to keep track of any prompts they used, their reasoning for using those prompts and how they'll use the outputs. This way someone can’t use an AI tool to cheat an assessment because the goal is now measuring how well they understand the tools they use.

“There's efficiency with AI, and that's what we're looking for. We want to know: how are they using it? Why are they using it? And what are they doing with that information? We provide candidates with a spreadsheet with a bunch of columns to make it as easy as possible for them to fill in. Because it's a tool and how you leverage it is more important than whether you use it or not.”

It’s worth pointing out that the assignment you provide must be relevant to the role. Otherwise, you risk hurting your candidate experience and overall company reputation through two of devs’ biggest pain points: irrelevant assessments and technical questions unrelated to the role.

2. Choose an assessment that directly measures role proficiency

An alternative approach is choosing an assessment that measures how well someone tackles problems directly related to the role they’re interviewing for, like a take-home assessment. Here’s how software engineer, Minenhle Dlamini puts it:

“Online coding assessments have always been a joke they have never simulated the real job. Personally, I prefer take-home assessments to being told to reverse a linked list in 60 seconds. Unless I'm applying for a job to build frameworks and OS. This allows companies to test the intangible qualities key to succeeding in modern development: Problem-solving and soft skills. “

In these assessments, you’re able to see, for example, how well does someone have a solid grasp of the problems they’d likely work on? Do they know how to identify and solve the right problems? And, can they communicate and motivate their approach to other team members and leaders?

3. Ask a good set of questions to understand their problem-solving skills

Assessing problem-solving skills directly is a third AI-proof strategy you can use in an assessment. Both developers and engineering managers now rate complex problem-solving as the most important skill in an AI age. Harley Ferguson, Co-founder and CEO of Origen Software, explains the importance of problem solving as follows:

“It's about how developers tackle the problems. Developers see themselves as problem solvers. Code and all of the things associated with code is our toolbox. It's about knowing what to use and when. Having a good set of questions that are maybe somewhat open-ended, and asking those questions and seeing how they walk through and approach the problem can help “

As a result, assessing how someone tackles a problem can give you insight into how they’ll meet the challenges in a role, as Chris Gillam explains:

“If you want to be sure the candidate understands what he is doing, ask him to modify his code in a live environment. A good way to assess developer skill level is to call them in for a panel interview and ask them questions like: How would you do this? What tools or frameworks would you use to do task? Please explain how you would accomplish such and such.”

No matter what approach you take, establish and communicate clear policies to candidates early in the hiring process around AI tool use and what you’re looking for with an assessment.

If you’re looking for reliable data on AI’s impacts on the SA tech ecosystem, download the 2024 AI Skills and Impact Report. With data from the South African tech community, you'll get exclusive insights and data into:

How developers are freeing up capacity with AI tools
The AI skills developers need to drive impact
The growing importance of soft skills