4 Lessons Wonga Learned About Rewriting Their System with New Tech

Keeping your company’s systems up-to-date is important, but can be especially hard when there’s inherited “Frankenstein” and legacy code. Brandon Pearman, Principal Software Engineer at Wonga, and his team had to overhaul their system to new tech, which had built up a lot of this “Frankenstein” code. It’s been a long process, and a huge undertaking, but they’ve learned some valuable lessons about moving a system over to entirely new tech. In this article, Brandon shares those lessons, and some considerations that might help anyone going through, or thinking about going through, something similar.

In 2019, Wonga South Africa broke away from the Wonga Group and became its own entity in South Africa. Brandon’s tech team had inherited a lot of “Frankenstein” code from the Wonga Group which was headquartered out of the UK. Although they’re running that code, they don’t have all the source code to change it. To become more independent, and therefore more efficient, they realised that they would need to migrate their systems over to new tech.

Their goal was to build a new system infrastructure that was less rigid, less fragile, less viscous, and less immobile, so that they could be more agile, particularly when it came to working with the Cloud and containerisation. Brandon explains, “We wanted to see… code that we wrote a month ago - how easy is it to change now? We should be able to do it very rapidly.”

This was a huge undertaking for the Wonga team, and they had to figure out how to integrate the system overhaul into normal operations: “Obviously the business can't just stop everything. We need to add new features, fix bugs, change things... We can't go to the business and say, ‘Look, we want to move to new tech and for the next year or two there's no new features, the system is dormant."

Over the course of this project, they have learned some valuable lessons about being able to achieve such a mammoth task without derailing business objectives. The lessons he shares below are as follows:

• Break it down, and just start somewhere
• Get the entire dev team involved
• Be proactive about your workflow changing
• Test every single assumption you have

Break it down, and just start somewhere

Brandon’s team realised quickly that such a huge task was not something they could map out from start to finish. “During our first attempt, we tried to plan every single little detail, and that was an absolute crash. We spent a whole month planning, and we actually got nowhere. We were like, ‘Wow, we got through 1% of the system in a month, just mapping it out.’”

When attempting to tackle a task this big, the team found that you risk spending most of your time trying to digest it before getting started – and often, the details are going to change anyway, in which case you have to go back to the drawing board.

For Brandon, the number one thing to do was just start working. "Even if what you're coding isn't going to be the final product, start coding something," he says. By taking this approach, the team found that either they had something that was production ready, or they had an example from which to learn from. "Just diving in was the best thing to get over that overwhelming feeling," says Brandon.

Get the entire dev team involved

In trying to maintain normal business operations, Brandon says his team could have isolated the disruption by dedicating a team to work on the project. In his experience, however, it was important to get his entire dev team involved so that everyone could invest time in understanding the ins and outs of the new system they were building: “If you have a completely separate team that does this”, Brandon explains, “what happens when you go live? Then you've got all these developers that need to now maintain this new system that they weren't involved in building, and they don't know what's going on.”

He also found the diversity in thinking really valuable. Because no one had any real experience with what they were faced with, he found that having many different suggestions on how to tackle it made their solutions stronger. By reducing the amount of decisions made by one person, Brandon says their solutions to the challenges they faced became stronger: “If you have one person making the decision, you're stuck with a single thought pattern. The goal is not to take the exact code structure and just move it across. We need to make it better, more flexible, better to read, all those things.”

This collaborative approach also helped his team feel less overwhelmed, because everyone could take ownership of the project in their own way: “Some people will take leadership roles, some people will be more technical, some people will wake up in the middle of the night and fix everything. But people have the choice to do that, and that's how we're finding that overwhelming feeling is going away. It's up to each developer to go and innovate and come up with new ideas.”

By getting the whole tech team involved, with roles from frontend to backend to team lead, everyone feels like there is something they can work on, and something they can influence the outcome of. “There’s a mindset shift when you create an environment where they can choose to take ownership and choose to be accountable. You’re actually developing your team at the same time.”

Pro-tip: In order to make team contributions effective, Brandon says just get all ideas out into the discussion first. “Add as many ideas as you can - bad ideas, good ideas, I don't care, just get it all out.” From there, he says that it’s easier to become less attached to your own idea, and teams start developing a more robust collective idea. “When you only have one idea, you're like, ‘This is THE idea’, and your ego gets attached to that idea.”

He also suggests changing the language you use in these kinds of discussions: “We stay away from words like ‘I feel like this is a better idea’ and ‘I just feel like there's something wrong here’. If we think it's good or bad, we lay out logical reasons for why that’s the case as opposed to saying what we feel is right or wrong.”

Be proactive about your workflow changing

With this project, Brandon says he hadn't expected how the workflow and business processes would change. Not only did they have to juggle working on restructuring a new system in tandem with keeping the business running on the old system, but they also had to change the way they worked with their business: Whereas the business would normally ask for certain features to be built by his tech team, Brandon says that developers now needed to take ownership of their own work, and get buy in from the business.

“Typically, business creates a ticket that says, ‘We need this’, they feed it down to the dev team, and we do what gets fed down. But that's not happening anymore.” The system move was really technical, so it sat entirely with the dev team. This means Brandon’s team had to generate their own tickets, and keep the business updated with progress. “Business couldn’t give us tickets on what to change from a tech side, which meant that the dev team had to create their own work.”

His team’s strategy has been to use each sprint to work on both ‘old system’ changes and ‘new system’ changes. Brandon says that as any new features the business needs are added to their old system, his team needs to build them into the new system as well: “In a single sprint, we do both ‘old’ and ‘new’ changes, but we might not do the same feature at the same time. While adding the new feature to the old system, we are still writing the new system, but on something unrelated. Then, the next sprint, we add that new feature to the new system, and do new, unrelated work on the old system..”

This not only means they get new tickets done for refactoring their old system, but they also get more data on how much time the system restructuring is saving his team: “On the old system, it takes us two weeks to add that new feature. On the new system, it'll take us a day or two. That shows that we've designed the new system well. We're finding a 10 to 20 times faster development rate on the new system.”

For Brandon, building this workflow between his team and his business has helped not only manage a mammoth task, but has helped build trust within the business. His team can compare the time they’re saving on features, and make a case for how valuable the system rewrite is, and that it will set them up better in the long run.

Test every single assumption you have

On their old system, Brandon says they knew and could anticipate the kinds of bugs they’d encounter. On the new system, however, they were facing new unknowns. “There's a brand new structure, so we're almost definitely going to encounter a new type of bug – a brand new strain of bug that we don't understand, and don't know how it works.” In order to reduce the risks involved in this, Brandon says that debugging became paramount – not for something in particular, but for everything.

Although Brandon says ‘bugs’ are something every software maker expects and knows happens, he found that shortening their feedback loops wherever they could helped mitigate some of the oversight that could happen in a project as big as a system restructure.

“The shorter I can make my feedback loops, the better. So, I look at every single possibility that can happen. Basically: If you have an assumption, test for it, prove that it's real.”

Writing tests for every assumption could be an expensive thing to add to your workflow, but the trade off made, in Brandon’s experience, is worth the time you save if new bugs appear all over the place. If you don’t catch new bugs early, the effects that that bug can have compound: “When someone does a surface-level, manual test through your system, they cover the common cases. It's those edge cases that are really difficult. Those are the ones that are normally these unforeseen bugs.” Testing rigorously, and catching them early, is an easy way to save you that pain later on.

Although building their system from scratch is a long process and a huge undertaking, Brandon says that he’s seen the time it saves. In his opinion, saving development time like that translates into being a more robust and adaptable business as a whole: “If you want to be agile, and respond to changes in your environment, you need to be able to respond very rapidly as a business. And you can't do that if your dev team takes forever to get something done.”

Brandon’s book recommendations:

• “Pragmatic Programmer” by Dave Thomas and Andrew Hunt: This is an old-ish book, but one that Brandon recommends for anyone interested in building their programming foundations.
• “Domain-Driven Design” by Eric Evans: This book talks a lot about collaboration with business, and - to quote Kent Beck - “heal the divide between development and business.” Brandon particularly enjoys how this book helps make that quote a reality, and its focus on language. He says, “Brevity. When you write a variable name, take time and think about it. It's one of the hardest things to do in programming and it's so important. We're writing code for other developers to read, so language is important.”