Recently we got engaged in a short project which involves around centralising sensor data from various sources. A small project with big ambitions. However, the small nature of this project regarding budget and time to deliver did not at all reflect in the non-functional requirements. So how to deliver a fully web based project integrating six external API’s to deliver a monitoring and reporting platform that by nature should always be available, while keeping operations costs at a minimum. In this article I will tell about our design considerations for choosing Serverless.
Getting feeling for size
It started out with your average napkin/whiteboard boxed architecture diagram. Just to see what functional components (not services!) we could distinguish.
In the first half hour we immediately recognised two types of storage, lots of independent external API’s with formats ranging from REST, FTP, to scraping email, and some infrastructural services (in pink). I have seen projects for which all of these independent components would take up a similar size project.
Managing the non-functional requirements
So in itself a challenging feat. I already mentioned the non-functional side. A second challenge in an environment that processes a continuous flow of sensor data. Although such a datastream is quite ‘monotonous’ for most of the time, you don’t want to miss the interesting bit when some sensor should alert. Delivering reliability has traditionally put pressure on the operations side of software.
However we have come to learn that this model is flawed in different ways:
- It shifts expenditure from development (increasing value), to operations (delivering value).
- It does not scale well. This model has decreasing returns to scale. Meaning, that for every 10% more reliable, you need to invest increasingly more to achieve it.
- It promotes a focus on making components robust, while at the start of a project things should not be set in stone.
- It requires a specialised skill to operate a resilient and scalable system.
It’s obvious that these days there are more ways to achieve reliability. Cloud technology has provided us with a comparatively cheap runtime environment/platform. This enabled us to design for reliability. In a way to shift reliability from operations to architecture/systems design. To make it an integral part of the platform, Netflix, Spotify, Google, and other companies with a strong focus on cloud native IT departments, have showed us it is possible to shift reliability to development.
How to keep complex software small?
The nature of this small project does not offer much leeway for setting up a platform with a microservice architecture, completely containerised, with multi-availability zones, service discovery, messaging. During this project, the time to market for an MVP was 10 weeks!!! I have seen teams failing in bootstrapping even one of the before mentioned aspects within such a timeframe. Also, this would drag us into a technical implementation frenzy away from delivering business value.
The premise of Serverless is to shift operations concerns to a 3rd party. To quote Martin Fowler:
Serverless, or else
This quote sounds like the perfect recipe for this project. We have experience using Serverless (databases, authentication, lambda’s) mainly as infrastructural or platform components. It proved to be a nice tool for building utility or ‘platform services’ that should ‘always be there’. Or for components that are not differentiating (like authorization, or running a database). In this project we decided to use Serverless components as the main way of delivering business value. So one of the fundamental architecture principles are: ‘Serverless, or else’. So whenever we make a design decision, we try to look at a Serverless way of dealing with it. It has become the default way of delivering business value in this project.
In the past weeks we have bootstrapped the project and delivered the first features delivering business value. We have intensely used Lambda, DynamoDB, Cognito, S3 and Cloudfront. We chose AWS because of their comprehensive service offering. Also knowledge about Serverless development is more widely dispersed, making transfer between teams and organizations possible. Choosing AWS really enabled our ‘serverless, or else’ attitude.
We will continue to share our experiences and considerations with building this platform in the months to come.
Serverless is part of every major cloud vendor. If you want to know more about how Serverless can help you on the Google Cloud, join us on October 25th at our meetup Firebase | Google Cloud Platform Serverless 101. Read more!