Every week, we run a meetup for the Investec Programmable Banking community. Developers demo cool projects they've been working on and everyone has the chance to ask questions. If you've been wondering where programmable banking is at, here's your sneak peek!
In this demo, Renen Watermeyer shows how he built a system that checks an account via the Investec OpenAPI to find new transactions every minute. He also demos how he created a webhook mechanism to push new notifications to another service.
Click here to download Renen’s presentation slide deck.
Check out his repo here.
Transcript of the demo
Renen [00:04]
Where did this all start? I was quite excited when I discovered this programmable banking thing, and I immediately looked at it from the perspective of what the business I work for does and noticed that I couldn't do what I wanted.
Renen [00:17]
I chirped Ben and Ben chirped me back, and one thing led to another, and he said, build something that checks the API every few minutes for a new transaction and then do something useful with it.
Renen [00:30]
With that thought, this is where that came from, and we'll look at why in a second. We'll also look at how we do it and a couple of other things around the edges.
Renen [00:45]
Firstly, my name is Renen. I run a business called SASA Solutions. I come from KnowledgeTree, AFB and JUMO more recently, which some of you may have heard of. I'm functionally a software developer, mostly databases and Ruby. Most of our work is in East Africa.
Renen [01:13]
We help organisations, not so much a punt for SASA – just considering the context of why this whole functionality matters to us. SASA enables organisations to build or to digitise their financial processes, usually in a mobile money ecosystem context. Most of our work is done in Tanzania, based on USSD, mobile money and fun things like that. We do a bit of work here, but there's less mobile money here, so there's less work for us here.
Renen [01:48]
A lot of what we do deals with getting cash in and getting cash out again. You can immediately see why this whole Investec programmable banking thing is exciting as it allows us to manage bank accounts in ways that are similar to the way you can manage a mobile money wallet.
Renen [02:14]
In this country, for example, we get money in via Pay@ now. In Tanzania and Kenya, we get money in and out using M-Pesa, Airtel and things like that – and some of the banks – but what we're all about, at our core, is managing cash flow in and out.
Renen [02:34]
With this Investec programmable banking thing, I spotted an opportunity to deal with cash in. Of course, the programmable API doesn't allow you to do that. What I set out to do here was to create a little POLAR that smacks Investec every minute, downloads new transactions, and then triggers some activity based on what it finds. That's effectively the model that I would like to follow.
Renen [03:15]
At the outset, I should say that SASA is completely – in fact, all the projects that I work on – are entirely built on Amazon. We're thoroughly committed to that ecosystem already. For us, building a Lambda2 is quite a natural extension of what we do. Most of what we do is still old school, built and driven on, both in Ruby and Rails running on easy to scalable instances but we're starting to experiment increasingly with Lambda functions and the like.
Renen [03:50]
The model that we're going to use is an AWS cron job mechanism to trigger this functionality every minute. It's going to run a Lambda function. A Lambda function, for those of you who don't know, is a stored procedure or a chunk of code that runs in Amazon's managed code environment. That little function is going to dwell on the function part of that name, not the Lambda part.
Renen [04:34]
It's a function, and it's going to pull all the transactions – or recent transactions – down from Investec and then it's going to push them into DynamoDB. Why is that useful for us? DynamoDB is much like a database, and it has triggers. You can use triggers on DynamoDB to kick off other Lambda functions. Once we're inserting new transactions consistently into DynamoDB, we can then hook Lambda functions onto that to do whatever we want.
Renen [05:04]
The problem is once you push data into DynamoDB, which is just a database, you've effectively solved the problem. It can trigger whatever you want. Launch space rockets, take over the world based on the actions that fire from that database table.
Renen [05:17]
Typically, what we do is push that event that comes out of DynamoDB into Amazon, Amazon SNS queue. That's a simple notification service. It's a PubSub mechanism – we're not going to go there. In fact, we're going to stop with the DynamoDB thing. I'll show you how you can activate those other kinds of last steps, but frankly, if we get those first three blocks right, we've cracked this problem.
Renen [05:49]
Any questions? No. Okay, good.
Renen [05:54]
Usually, with an audience, you can see people and eyeball them, and you can understand whether they're watching or not. Now, I've got two of you and one space rocket.
The methodology, just to go a little bit more here to cut to the chase, if we drill into that Lambda function in slightly more detail to interrogate, how do we make this happen?
Renen [06:16]
The first thing we're going to do is, at a very high level, we're going to get all the code, get all the transactions from Investec. Once we've got that list of transactions, for each transaction, we're going to calculate an MD5 checksum effectively, just a little hash, and then we're going to use that to uniquely identify these transactions to DynamoDB. We'll use that as our key and DynamoDB as a conditional push mechanism so that if the key is there already, it doesn't update, which is, in effect solving the problem for us.
Renen [06:20]
The GET was probably the trickier part of this for me because I didn't know the Investec programmable API particularly well. It's a case of getting an access token, getting a list of accounts, finding the account we care about and then downloading all the transactions.
Renen [07:08]
That's the methodology. Again, if there are any questions or any uncertainty, just wave at me. I have shared the repo that this code is sitting in, in the message chat. If you want to open that up in the interim, go for it. I'll keep talking, and we'll come to the code in a second. Feel free to open that up so that you can see it more easily. Casually saying, let's just build an AWS Lambda function and push it to the cloud and Ruby is a satanic undertaking. Adam. Sorry, did I see you gesticulating?
Adam [07:50]
Yes. Sorry. Did you say that you put it in the group chat?
Renen [07:53]
Yes. I tried to.
Adam [07:56]
Maybe I joined a bit too late for it?
Huggs [08:00]
The Zoom chat. It should be in the Zoom chat.
Adam [08:05]
If somebody could repeat that it would be great. I want to look.
Huggs [08:09]
Next level.
Adam [08:11]
Alright. Thanks, appreciate that.
Renen [08:13]
Huggs, how do I pronounce your name? Is it Huggs?
Hugs [08:16]
Yes, that works.
Renen [08:17]
Yes. Easy enough.
Huggs [08:16]
It doesn't get easier than that.
Renen [08:21]
Good. Back to this undertaking. To build a Lambda function, you could do it reasonably easily through the Amazon console. But as soon as you try and deviate from…
You haven't got it yet?
Adam [08:39]
No, I have it. I was just responding involuntarily to your last comment about doing it through the Amazon console.
Renen [08:47]
You can certainly do that. We did a little program and it's quite a convenient, quick way of doing things. It's a contained problem, but it channels you into a node which I have some reservations about. And it's limited, as Adam indicated, there are clear problems and it's not sustainable.
Renen [09:07]
The way we're doing it now, which leaves a lot to be desired, is, we use a thing called AWS SAM. Getting it installed is a bastard. If you're running a Mac, you're in business. If you're running Windows, you're probably out of the game. If you are running Windows, let's ignore you. If you're running Ubuntu, it's not trivial. There's quite a lot of uncertainty around it, but once you get it installed, it does become a lot easier.
Renen [09:39]
You literally type SAM, and it creates a Hello_World application for you which you can then bastardise into whatever sort of shape you want. You can then deploy that by typing SAM deploy. Once I got to that point, I then added the other functionality around the edges. First, I added the cron to make sure that it ran regularly. Then I added the HTTP code and finally, I pushed it to DynamoDB. This was the process that I followed. I must be a bit careful if you build Lambdas, but Lambdas are great once you get in there. SAM is probably the way to go. Adam, do you have another way of doing it? No?
Adam [10:25]
Yes, I do. SAM is the devil.
Adam [10:30]
I'm posting in the chat now for everyone. I was fortunate to get a heads up from one of my co-workers, who is our cloud expert.
Adam [10:41]
I looked at SAM and CDK at the same time. CDK is phenomenal. There is no reason to do Amazon any other way at this point.
Renen [10:50]
Is that the new one? If I understand correctly.
Adam [10:54]
The link that I've sent you is to my repo, which is a quick start guide to using CDK because the documentation for Amazon's everything is all over the place. CDK has now come up with an extension, where it synthesises to terraform. You can use it not just for AWS, but any of the platforms. And it's a completely different way of doing it. I warmly recommend checking that out before continuing with your development.
Renen [11:22]
It's right. I almost cried when I googled something yesterday and found CDK. It's a new way to do something else. With Amazon, that just means more pain. Anyway, there's the repo. Whatever CDK does, the core of this will not change. What's going to change is the first and second steps, and how you provision the cron job, but frankly, the guts of this problem will not.
Renen [11:52]
The actual code, this stuff won't change, particularly because that still sits inside your Lambda. What will change is the pain that you have to go through to get the Lambda up and running. Hopefully, you've all found your way into the repo. There are two files in there that matter. There's a template.YAML, which we'll look at in a second, or you can look at it already, and there's app.rb.
Renen [12:23]
Having done that, let's quickly look, I'm going to try and share. No. I was going to try and share another screen, but that's too advanced. How do I get out of this thing? I'm going to click it to the repo.
Renen [12:39]
Make it bigger. Can you still see my screen?
Renen [12:44]
Somebody can nod. Adam?
Adam [12:46]
That's great.
Renen [12:48]
Thanks, Adam. If I go into here, most of the stuff is generated by our friends at Amazon. If you look at the template.YAML, this is the bit of code that generates the auto provisions infrastructure or provisions the resources that we're going to use. I've kept it quite simple.
Renen [13:09]
It's not perfect. There are a couple of issues with it, which we'll touch on in a second but hopefully, it's simple enough that it can be digested by you in a reasonably short amount of time. These things can get nasty. The first thing is you'll notice – Adam might notice – is that the timeout is 15 seconds. The Investec APIs are not particularly speedy. These things typically run for 3, 4 seconds at a shot, hence the need to up that. We created a database DynamoDB table – that's the resources that we're provisioning. We create a function, which is a Ruby function. Here's the function and here's the stuff that defines how it's going to work.
Renen [14:09]
It tells us where to find it, what code language it is, and I created some environment variables here. This is sub-optimal because, in a public repo, you don't want your details floating around. What you need to do is move these into Amazon's configuration mechanisms. I've included a snippet of code in the deck which I will show you roughly how to do that, but for the demo, for the purpose to get my proof of concept up and running, I just slammed the credentials right into the file there like that.
Renen [14:41]
And the last thing is to create a cron schedule. It's going to fire every minute. Look through this file, it should help you understand roughly how the resources are provisioned. What to do with that – I mentioned already that you type SAM and it creates the initial Hello_World function. Once you've created your code, you then say, SAM deploy, and that pushes the stuff up, we'll do that in a moment as well. That's the one bit of code.
Renen [15:11]
The other bit of code, which is worth touching on for a second – the real guts of this is our app.rb file. For those of you who don't speak Ruby, there's not that much to it, it's a series of downloads, a series of eight GET requests to make the various API calls, followed by each transaction being pushed to DynamoDB.
Renen [15:45]
You can see here I'm reading my environment variables that get set by the resource creation scripts. That's how you get the code in here. I'm cheating. I don't know how big a cheat it is, but I'm taking the table name that was provisioned and stuck it in here. The handler effectively gets all the transactions for this Investec account.
Renen [16:15]
We're using those environment variables, and then push them to DynamoDB. To start at the end, all we do is we turn the transaction into JSON, we create an MD5 checksum for it, and we push that to DynamoDB with a conditional expression saying, the attribute does not exist. That prevents it from re-adding so that then means if we get a new one, we can then fire transactions on top of that.
Renen [16:48]
The rest of the code is mundane, pulling record-making requests to the Investec API, sucking down the data and parsing it out. It is not rocket science and in fact, all I did was I took the postman library/collection that Ben shared with me, and I had postman generate that code for me. It's relatively mindless, but ultimately gets a list of transactions which you can then process through the system. I think that's the key stuff there.
Renen [17:31]
Are there any questions? There's plenty bad about that as well. There's plenty of scope to criticise it. It is a proof of concept for us. It's not production-ready, although it's not too far from it either.
Renen [17:47]
Okay, since Adam has remained silent. Thanks, Adam.
Renen [17:53]
Let me do two more things before I wrap up. One, I'll show you how it's deployed because I think it is quite fascinating for those of you who haven't done it before to see how you can deploy a whole crapload of infrastructure in one line. Then we'll have a look at how it runs in AWS. In fact, there are some problems there and we might look at those as well.
Renen [18:24]
How do I share a different screen? I want to share that one like that.
Renen [18:30]
Hopefully, you can all see an Ubuntu monitor.
Renen [18:36]
Just kill that. Good. The two commands that matter. The first one is SAM BUILD, which takes everything in our template, takes everything in our code and bundles it up so that it can get deployed. There are three commands, SAM INIT, the new modifier code. SAM BUILD, it does its build thing. Once it's done that, if it's successful, you can tell it to deploy it to the cloud. It asks you a few questions, where you want to put it? It asks you to confirm things. And off it goes.
Renen [19:21]
That is quite awesome. The fact that it can be up and running and live now is – with no changes to deploy – for me, that's compelling as it gets the development cycle happening nice and quickly, which matters a lot for my workflow anyway.
Renen [19:44]
Let's have a look at this thing running in AWS. The first thing is I deployed it earlier and I hope you can see that. Let me make it a little bit bigger. It promptly succeeded, and I thought it was done then it failed subsequently. It was working when I did this demo two weeks ago. The whole thing did work and I don't think it's too complicated. I think it's a permissions problem or something along those lines. We'll look at that in a second. You can see that it's running relatively slowly, about three seconds a cycle and most of that time it's interacting with the Investec API.
Renen [20:27]
The codes, this is the part of the equation that you can review the code. Despite Adams' comments, you can get useful stuff in here. It does break your workflow because this is not quite as easy to get into the GET repo, but the code is there. That's the Lambda. There's the code and as I said, you can see it's failing.
Adam [20:58]
Just to be clear, I am all in favour of trial and error and fiddling around with the Lambda code live, until you get it working, but it has to be checked in with the codebase.
Renen [21:10]
Yes, otherwise, the world ends when things go wrong.
Renen [21:14]
If we can click across to the Dynamo database and DynamoDB database. If I go look at tables, I can find in this list of play tables, the Investec POLAR table and if we're lucky, there are all our transactions sitting there.
Renen [21:34]
This is neatly pulling the stuff down. I had it running every minute for a while and stopped it. Started again about half an hour ago but it does what it's supposed to do. The next step in the journey is to fire off subsequent Lambda functions it's simply a case of going into more triggers and adding either an existing or you can create a new Lambda
Renen [22:13]
You can link it to an existing Lambda function, and that allows you to take over the world. Once you're getting events out of the DynamoDB system, you can do whatever you want and it's cost-effective.
Renen [22:32]
From where I stand, I'm happy that this is something we can build on. It's certainly something we could work with. I suspect that if we are polling it too hard, the Investec guys will become a little uptight with us polling every minute so, we probably will dial it back a little and do it every 10 minutes or every 15 minutes.
Renen [22:55]
The downside of that is if somebody makes payments or purchases insurance policy or something like that, you want to message them quickly after the transaction has happened to confirm that it's happened. The downside of slowing it down is that you make your customers less happy.
Renen [23:15]
I hope that the Investec guys notice and provide these APIs and these call-backs more intelligently in due course because they are running on Amazon. They should be able to create a call back mechanism reasonably straightforwardly. It's worth touching on that Lambda a little bit more.
Renen [23:37]
For those of you not familiar with it, it's worth going to…. There underneath me.
Renen [23:48]
In terms of diagnosing what's going on, Amazon provides you with print. It's failing. Why? If we go and look at our logs, there, just one of a lot of Amazon's clear cut unambiguous simple solutions. Not. We can go see what's going on. We can see that it is complaining that the conditional request failed, and where it failed on line 17. I'm not sure why it's failing. I'll have to go debug that – a bit surprised anyway.
Renen [24:32]
That's the way our code goes we will have to get into our code, and we can go figure that out. I don't know how much we will benefit from doing that right now, and it might take a while. That's me. I don't have anything else to say.
Renen [24:48]
We've looked at most of those. What remains to do? When I did this talk initially, I needed to publish a repo. I was quite uncomfortable with having my Investec identity strapped into a public repo. I wanted to get the code into secrets.
Renen [25:08]
This little snippet here is the clue, you need to be able to do that. It drops into the resource templates, creating a policy to allow you to access and store a client ID. A piece of configuration data inside of Amazon's configuration store.
Renen [25:35]
Other than jumping out of my head, but that works easily then you can read that in your code. Finally, the fudge that I did, I had to create the table permissions manually. I've got some work to do on that. It's not right. I should be able to provision the table correctly through the code. It's not doing that currently.
Renen [26:09]
Just to be very clear, this code here in our template on YAML, this provisioning of the DynamoDB table, there are two problems. One is I'm not getting the name to use in the code. You can see that hardcoded embedded in there.
Renen [26:23]
And secondly, this policy that's going to be DynamoDB crud policy, which should give me access to that table, is this reference. I think it is not working correctly. I need to fix that. Those are the key things that remain to be done here. but that can be hacked if one needs to get something up and running for one account. Suppose you try to do generically or sustainably, well in that case, you need to solve those, but it's in a tractable space, which means two or three days of hard struggle with Amazon. That's me done. I hope you found it interesting. Are there any questions?
Huggs [27:05]
Thanks. That was very informative and nice and detailed. Thanks for the effort.
Renen [27:16]
Let's do Adam first, and then we'll go there.
Adam [27:19]
Mine's not much of a question. Hats off for persevering with SAM. The fact that you managed to get this up and running with it is quite an achievement. I'm excited to see how you interact with CDK, in which you can use multiple languages, not just JavaScript because what you're describing right now, the last thing that you said about getting the permissions right, all that stuff is taken care of, for free by Amazon. They've really done a good job.
Renen [27:50]
SAM's spec says that they don't do that. Not so much to take care of permissions, I'm sure we can solve that but getting the name of the table into your code? I don't know.
Adam [28:00]
Everything. SAM by design is incompatible with CDK. But as you said, you can take your Lambdas, shove them in somewhere else, and they'll work fine. You wouldn't have lost any of your functional code. You wouldn't have lost any time as far as that's concerned.
Renen [28:20]
If you worked with Amazon, you'd know that the functional code is never the problem. It's trying to work out which bloody arcane piece of incantation you need to try to win to get the stuff to work.
Adam [28:29]
Exactly. I'm so excited for you because you're going to be jumping from SAM to CDK. It's just a whole other experience.
Renen [28:40]
I may never go there because I've got this to work. That's all balancing neatly at this point.
Renen [28:42]
I will check it out. I will take that on board.
Renen [28:46]
Vic Provinsi did you want to interject? I hope you're an Amazon employee listening loudly.
Vic [28:55]
Dear Lordy, I'll kill myself. I was going to ask how quickly and effectively does this build if you include more Gems or similar? How scalable is this with libraries? Does that matter?
Renen [29:15]
Yes. Let's put this in perspective. With a mediocre degree of understanding of the whole SAM service space, this took me three hours to put together end-to-end. That's the first part of that answer. We're not asked directly, but it is a tractable solvable problem once you got the SAM stuff done. If you're happy to deal with a few compromises.
Renen [29:39]
Your question, though. The other half of the question is, what if you start loading this thing up with Gems and stuff like that, what the consequences are. I don't know if it gives it to me here. No, let's look over here. It should give it to me here. If I look here, you can see it.
Renen [30:33]
The build duration, that's how long it takes to run. In fact, this took 1.2 seconds, and they billed us for 1300 seconds. They round it up to the next hundred milliseconds. Your execution time matters. Your memory size matters. It matters in two ways.
Renen [30:50]
One. It matters in terms of start-up time, the bigger the slug, the bigger amount of code you got to get loaded. Things just slow down, it's not ideal so adding loads and loads of Gems, really comes down to….
Huggs [31:11]
They charge you more.
Renen [31:14]
They charge you more and your code, everything is slower. You want to keep things as tight and tidy as possible. They do provide you with some tools to manage that.
Renen [31:22]
They've got things called layers, which you can use to simplify your code, not necessarily the size. You can keep things quite tidy. We use the stuff as sort of salt. We use it as connectors, and we have a Rails application running, which we use for the user interfaces.
Renen [31:40]
We have a lot of ingestion of data and in our case GETs are done through Lambda stuff, which doesn't have user interfaces. There, our need for libraries is relatively small. You need JSON, HTTP and a couple of other things. This has JSON and HTTP. That's a large part of what we need to do anyway.
Renen [32:05]
You can see that its max memory uses 81 MB. I think the slug side of the size for this is 4.5 MB, or somewhere thereabout. Not too dramatic. Yes, you don't want to go completely bananas, but frankly, in any code, you want to keep it quite tight. You don't want to have too much stuff scuttling around.
Renen [32:28]
The other problem with dependencies is that, for example, not in this case but if you use, for example, the MySQL Gem, it uses libraries that are not installed in Amazon's environment. You must then create a Docker function or Docker image and build this thing in a Docker image, then deploy it using some weird incantations which relate to the Docker image. That just adds complexity to your world and more pain.
Renen [32:58]
Once you figure these things out, it becomes less painful. The problem with Gems is much more than the configuration, and that's what you must watch for. I don't know if that answers your question?
Vic [33:13]
Yes, that completely answers it. Sorry, I asked because I'm from a very firmly traditional Rails background. These super lean environments are fascinating to me.
Renen [33:24]
What I can't answer, what I can't speak to, is what happens when you have 50 Lambdas drifting around, how you structure your code. We haven't answered that yet. We've five applications with 20 or 30 Lambdas drifting around. Now it's still tractable. When we get to thousands, it obviously will be a different story. I don't know if anybody else has answers to this. Maybe that's beyond the scope of this though. Any other calls, questions?
Huggs [33:59]
What are your next steps? Where are you taking this next or are you taking it anywhere?
Renen [34:04]
It's nice to have it in our back pocket. We have, at this point, three clients in South Africa. To have this here means that we've got another trick up our sleeves.
Renen [34:18]
As I said at the beginning, a lot of what we do deals with getting cash in and getting cash out. There's more to it than that. It's about managing customers in that context. Frankly, being able to get cash in this way, it is helpful. Pay@, which is the solution we're using now, is antiquated, it leaves a lot to be desired.
Renen [34:42]
It works well for taking deposits in supermarkets etc., Pep Stores and the like. But if you have fought for other parts of the market, this is compelling. Our next step is to find a customer who will leverage it and to build some real production functionality around it.
Huggs [35:06]
This touched on the interesting stuff, which is the SAM stuff.
Renen [35:15]
The audience will do well to listen to Adam's comments on CDK because there has to be a better way.
Huggs [35:23]
Yes, CDK is a much better way.
Renen [35:40]
So, where it's useful beyond now the APIs deal with credit cards and this allows you to get beyond credit cards. Yes. Which is extremely useful. I haven't tested how fast the transactions arrive there. Seems to be reasonably quick.
Vic [36:00]
The other thing I could see this being very useful for is certain types of small businesses that have a flow system that relies on EFTs, and money being paid into an account to verify your order.
Renen [36:18]
That's exactly what I'll do. That sort of thing.
Vic [36:20]
We have a couple of clients who rely on similar things. It's a bloody nightmare to handle that by hand. Where you have some business developer or some per sad admin clerk, sitting there with a page open and checking at once, every hour, to see whose orders are now confirmed, so that they can send them an email and proceed with the rest of the order. This potentially allows you to automate that entire process.
Renen [36:44]
What you described doesn't scale very well. It doesn't normally scale much beyond Tuesday.
Vic [36:49]
No.
Renen [36:52]
Exactly. The reason for this code is to be able to take that admin clerk out of the equation. That's precisely why it matters to us.
Renen [37:06]
Thank you all for listening. Appreciate it.
Huggs [37:08]
Thank you. Thank you for a great demo. Very informative.
Renen [37:13]
Thank you. Pleasure. I will let you know how it goes with the CDK.
For more than twenty years Renen has worked in financial services in Africa. He was involved in developing the first tech platforms that enabled a mobile network operators to provide loans to their customers in Tanzania and across East Africa.
He has a strong technical background with experience spanning the development of mobile money platforms, trading systems and insurance platforms. He understands first-hand the market and the technical challenges facing rural Africans and uses this knowledge, and his passion for change, to continue to develop and improve SASA Connect.
Renen spends his spare time driving his wife to distraction with a range of ill-conceived home automation projects that are seldom concluded!
Get involved in the Programmable Banking Community
If you’re not part of our community yet, sign up and join the fun. You can also see more of the demos from our meetups on our YouTube channel here.
For those of you in the community, check out our GitLab to see more of the awesome projects members of our community are working on. You can also sign up for challenges, where you can help find solutions for real life problems.
For more information, get in touch!