Software developers prefer to not solve the same problems over and over. This mindset coupled with a business requirement of being agile and efficient is part of what led my team at Mezzanine to develop a rapid application development platform called Helium. Building our own platform came with a lot of challenges, so I’d like to share how we approached it and what we learned.
When I joined Mezzanine, most of the problems we were solving were focused in the domain of mobile health. The applications we were developing were mostly crud-based web and mobile applications with some complex back-end logic for notifications and other asynchronous processing.
We realised that there were many similar components from one application to the next. We suspected that we could massively improve development velocity and efficiency by reusing some of those components.
At the time, there were many application development frameworks available that we could potentially use to develop apps. However, none of them provided us with the simplicity and flexibility to work with that we required, nor did any of the available frameworks have the domain specific elements that we preferred.
Developing our own platform seemed like the logical answer to this problem: It would allow us all the flexibility we required and would allow us to adapt the platform as our requirements, and the requirements of clients, changed over time.
As development of the platform started, it became clear that there were three main challenges that would need to be solved in order to be successful. We had to:
- Determine which components of our applications were similar enough so that they could be abstracted away from the application business logic, and then implementing these as components of the platform. We also needed to anticipate future components that might be useful in similar ways.
- Find ways to expose these components to application developers so that they could easily use them, while still having flexibility in the way they implement business logic in their applications.
- Provide a set of tools to help both developers and non-developer stakeholders deploy, manage, monitor and configure applications for their clients.
Here’s how we approached solving these challenges, and a few of the main lessons we learned along the way.
Challenge 1: Determining which components were similar
The first challenge we faced was determining what functionality was common to many of our applications, and what core features the platform should provide. These features needed to be implemented in a more general way and eventually exposed to developers somehow.
Since we had already developed various applications, we had an idea of the features that were common to our applications and therefore what needed to be implemented as core features in the platform. We also needed to anticipate what might be needed in the near future.
Getting this part of the process right would allow us to hit the ground running with application development once the first version of Helium was ready.
Some of the key requirements that we identified were as follows:
- Since a large part of our applications was focused on data capturing, we needed to provide persistence functionality by integrating with a relational database management system.
- Since there were potentially many different actors that would interact in the same application – and given that access to the data needed to be strictly controlled – we needed an easy and secure way to manage user authentication and role based authorisation.
- Applications also had to be cloud-based and easily accessible from a web browser or mobile devices.
- Hosting concerns needed to be hidden from developers and handled by the platform.
- Developers needed to be able to create user interfaces and backing logic that could satisfy client requirements without spending days of development time on “look and feel”.
- We needed to integrate with external services, such as SMS aggregators and mobile payment providers.
- We also needed to provide partners with ways to interact with our applications by means of their own integrations.
Once the requirements were identified, the core functionality was implemented using Java EE and encapsulated as separate components that would be exposed to application developers.
The result was a feature-rich back-end that could leverage the functionality and architecture provided by Java EE, with the aim being to provide developers with a complete and simple to use platform. With the core back-end functionality implemented – but not yet exposed to developers in an easy to use package – there was still work to do.
Generalisation does not mean less complexity
Because of the fact that the reusable components in Helium need to be generalised for potentially many different (but similar) use cases, it requires a considerably more complex implementation to ensure that all edge cases are covered. This means that adding new features to the platform is complex and potentially time consuming.
The so-called black box approach of the platform also has a disadvantage. In cases where the system does not behave as expected or an issue is encountered developers might not be empowered to solve problems themselves. This potentially results in high support costs.
Challenge 2: Exposing these components to developers
With many of the back-end features implemented and abstracted away, developers needed to be able to use these implementations in their applications.
At the core of this challenge, the goal was to have a platform that could provide a way to rapidly develop applications. As such, we needed a way to expose all of the back-end features to developers in a way that would help developers build apps faster.
We specifically wanted to avoid a situation where developers needed to know the intricacies of the Java EE ecosystem in order to be productive. We wanted the platform to take care of those aspects so that developers could focus on the business logic of their applications. In addition – and because our applications were domain specific – we decided that a domain specific language (DSL) would be appropriate. The result was the Helium DSL.
The DSL is a programming language with related tools developed in-house specifically for developing the types of applications that Mezzanine develops and expects to develop in the future. It is implemented as a model view presenter (MVP) framework using XML for specifying views and a custom C-like language for specifying data model objects and presenter logic. This is helpful because the DSL allows developers all the functionality previously mentioned, without a need to know the intricacies of the underlying implementations.
With the DSL in place, the Helium server could be used as an execution/hosting environment supporting a multi-tenant architecture. In other words, it could be used as a single server instance that could house many different applications.
The advantages were easy to see: Any improvements to the platform implementation were immediately available to apps running on the platform. Solving a problem for one application would allow all applications running on the server to benefit from the solution. The result was a platform that satisfied many of our initial requirements.
Here’s what we achieved:
Database integration
The DSL provides syntax for specifying model objects along with internal ORM functionality. It's as easy as specifying model objects and then using presenter logic and calls to built-in methods to save or query from the database.
To provide developers with even more control, native SQL can also be executed using built-in functions. The code snippet below shows a simple example of persisting and querying data using these methods.
void persistInlandFarmer(string name) {
Farmer farmer = Farmer:new();
farmer.name = name;
farmer.location = FARMER_LOCATION.Inland;
farmer.save();
}
Farmer[] getFarmersWithSelector(FARMER_LOCATION locationParam) {
Farmer[] farmers = Farmer:equals(location, locationParam);
return farmers;
}
Farmer[] getFarmersWithQuery(FARMER_LOCATION locationParam) {
Farmer[] farmers = sql:query(
"select * from farmer where location = '?'",
locationParam
);
return farmers;
}
User management
Objects specified in an application's data model can be marked as a user type of specific role using a provided annotation. The DSL provides built-in functions that can then be used to invite users for that role. A single user can be invited as different types of roles and easily use built-in UI components to switch between roles.
Model objects can further be annotated to only allow access to that object for certain roles using custom criteria.
Cloud-based
Since the Helium back-end is implemented in Java EE, this requirement was easily satisfied as part of the tech stack and execution environment.
Hosting concerns
The Helium platform was implemented using a multi-tenant architecture that would allow many apps to be deployed and run on the same server. Developers therefore do not have to concern themselves with the underlying architecture and implementation details of the hosting and execution environment.
Simple UIs
The DSL facilitates UI development by providing a variety of components out of the box. These include simple components for data capturing and complex components such as data tables, maps, an image gallery, and more.
The MVP architecture along with these UI components, a fixed layout scheme, and fixed uniform styling means that developers have flexibility in what UI components they want to display and how these should interact with application logic without having to spend hours on other aspects of the look and feel.
Integration with external services
Many external services can be called from within DSL applications by simply making calls to provide built-in functions. Initially these included functions to send SMS and e-mail messages but later enhancements also included calls to Mpesa and general outbound APIs.
Developing a programming language is not easy
A lesson that we learnt was that developing a programming language from scratch is not easy. In part, this was expected. What was slightly more of a surprise was the difficulty in expanding and maintaining a programming language. The result was a somewhat slower turnaround time in developing new language features for the platform than expected.
We also realised that there is a constant balance that needs to be maintained to avoid expanding the DSL with too many general programming language (GPL) features that would essentially change its nature from a DSL to that of a GPL.
Allowing for very quick development will always be a big advantage of the Helium platform and the Helium DSL; but, with any framework or platform, there is some compromise of flexibility. This means that it’s possible that the platform approach adds complexity for certain use cases instead of reducing it.
With regards to hosting apps and utilising a multi-tenant architecture, additional challenges were faced. Since apps are sharing resources and any misbehaving apps have the potential to bring down the hosting and execution environment, a considerable dedicated effort needed to be made to ring-fence apps in such a way that this did not occur.
Challenge 3: Providing the set of tools for stakeholders to work with
With the Helium DSL and back-end in place, the final piece of the initial puzzle was providing the tools for both developers and non-developers to deploy, manage, monitor and configure running instances of their applications.
A key requirement here was a core web application that could provide authenticated users the ability to, amongst other things, create and configure apps and to upgrade apps with previously released source code. Developers also needed a simple way to build and release source code to a Helium server.
These requirements were met by building two stand-alone tools that would integrate with the web services provided by the Helium back-end:
- Helium Core web application: Implemented as a Java EE based web application and providing all the management features required to manage applications and their configurations.
- Helium Dev command line tool: Checking code for compilation issues and releasing source code to a Helium server is facilitated by a stand-alone command line tool that developers can download and run locally. Later additions allowed the Helium Dev client to be fully utilised from within a CI/CD tool such as Jenkins.
Collectively, the Helium DSL, execution/hosting environment, all backing services and tools are now referred to as the Helium platform. The tools essentially provide many different users the ability to interact with the platform in different ways.
They also allow developers a short turnaround time when developing and deploying applications that further satisfies the original requirement of being agile and able to react quickly.
Tools like this are key for accessibility and development
In general we realised the importance of these types of tools and how, despite not being part of the core back-end and DSL, they provide essential features required to deploy and develop applications. The tools also demonstrated the extensibility of the whole platform, made possible through integration resources in the Helium back-end.
As with any complex software project, the first iteration is rarely a success that satisfies initial expectations. This was the case for us. There were two predecessors to Helium that were focused on many of the same ideas, but eventually did not provide us with all that we needed even though each iteration represented a step in the right direction. This highlighted the complexity of what we were attempting to achieve.
Given everything that we've learned in developing and using the Helium platform up to this point, the future for Helium is very exciting. Although we've had to re-evaluate some of the details, the general idea of providing reusable components is still very much relevant and something that the future strategy will rely heavily on.
With the complexity of a monolithic multi-tenant platform such as Helium and the potential limitations of a DSL, we are looking forward to an approach that provides a framework for development instead of a single platform.
Jacques Marais is the Team Lead and Architect of the Helium Platform Development Team at Mezzanine. Jacques is passionate about software development and using technology to solve real world problems. In his spare time he enjoys photography, hiking and growing his own vegetables.