12 Factor-App: Modern Application Development Patterns
Post co-written by Mohamed Ben Slimen (Cellenza) and Mahdi Ghandi (Squadra)
The relationship or dependency between the Build and Run environments is widely recognized now. The changes in our web application development habits, so-called cloud-native applications with the normalization of “You build it, you run it,” or the adoption of so-called “DevOps” methods, means we are increasingly in need of a methodology that puts the Build environment at the service of the Run environment.
Based on hundreds of feedback comments from their development and operations teams, IT teams from the US firm Heroku published a manifesto in 2012. This manifesto established the 12 rules to follow when designing and developing modern apps. These are called Twelve-Factor Apps.
These principles are universal. They can be applied regardless of language and structure, no matter which tools or frameworks you use.
Let’s take a closer look at each of these 12 factors.
“One codebase tracked in revision control, many deploys”
The codebase principle states that an application cannot be archived and versioned in multiple source code repositories and vice versa.
This means that a source code repository can only contain one application at a time. Each commit to this single repository can produce artifacts that will be deployed to one or more application execution environments (test, acceptance, or production). This single repository can also contain resources like build or release scripts (e.g., YAML files) and resource provisioning scripts (e.g., Terraform files).
If an anomaly or bug occurs, rolling back the commit will restore the application to a stable state.
If multiple applications share the same codebase, it’s best to factor the part of the code to be published from another repository as an external dependency.
“Explicitly declare and isolate dependencies”
A dependency is an external yet vital component of an application. Dependencies are the one thing we have no control over. That is why you need to pay extra attention to them!
A 12-factor app should:
- Explicitly identify these dependencies (package name + version);
- Isolate the dependencies from the code. The code is all that matters;
- Dependencies must be fixed: there must be no assumptions about whether a dependency exists in the runtime environments;
- List the dependencies. This list will be used in the Build and deployment phase;
- Verify the integrity of the dependencies.
Here are various tools that, depending on the technology, can help you track your app’s dependencies:
- NUGET => .Net
- Maven => JAVA
- NPM => NodeJs
- Bundler => RUBY
“Store config in the environment”
An application’s configuration refers to the variables that depend on the environment: connection strings to storage spaces, URLs to external APIs or internal services like namespaces of BUS services, or even a basic SMTP server for sending emails.
Configuration data also includes more sensitive data such as database passwords or login credentials, so these must be protected by separating them from the rest of the application code.
For example, a vault such as Azure Key Vault can protect sensitive configuration settings and serve them to client applications through URLs.
This approach also allows you to change the values of variables based on the target environment (test or production).
⚠️Make sure you don’t confuse this with the application settings, which do not change with the environment but rather according to how the application is used. These include the theme for the interface, the duration of user sessions, etc.
The runtime environment must inject configuration information as environment variables in 12-factor applications for these reasons. This keeps it strictly separate from the code.
IV. Backing Services
“Treat backing services as attached resources”
The fourth principle states that bus services, databases, mail services, storage space, etc., should be considered external resources and decoupled from the application.
These resources will be consumed and used via URLs or similar access points specified in environment variables (see principle 3: Config).
In the case of a failure, the objective is to be able to swap (detach/attach) these resources without affecting the operation or requiring changes to the application code.
V. Build, Release, Run
“Strictly separate build and run stages”
12-factor apps divide deployment into three steps:
- Build: this is the generation of artifacts from the source code. In this step, the code is extracted from the source and checked with the compiler (syntax check). The binaries (.exe, dll, jar, etc.) are then created, and the code is tested using unit tests and analyzed for quality control. The output artifacts are stored in storage spaces that the “release” service can access.
- Release: once the artifacts have been generated, they are deployed to the target environments by applying the necessary configurations at the release stage. The release must be immutable, versioned, and unchangeable.
- Run: this step includes the following tasks:
- environment provisioning, such as running Terraform scripts or database migration scripts
- releasing binaries, dependencies, etc.
- running the application and starting one or more processes
The deployment process is, therefore, completely ephemeral. All the artifacts and environments can be recreated using the source code repository assets.
A Continuous Integration/Continuous Delivery (CI/CD) approach is strongly recommended when using this principle.
“Execute the app as one or more stateless processes”
The processes must be independent in a 12-factor application. They do not keep track of or make assumptions about the existence of information produced by other processes such as session variables.
A stateless architecture facilitates horizontal scaling by adding multiple run processes to handle a huge workload.
It’s better to use resources like a storage space or cache server that can be considered and treated as external resources when you need to save the state or run results for a process.
VII. Port binding
“Export services via port binding”
The Port Binding principle allows an application to be self-sufficient and identifiable on the network by giving it a port number (e.g., 80 for HTTP services or 443 for HTTPS).
For example, a web application that uses HTTP must have its own web server. The latter is an external dependency that can be injected at runtime.
This internal port will be linked to an external port usually assigned by the cloud provider, thus avoiding port collision problems with other services.
“Scale out using the process model”
When a web application reached its capacity limits, the solution used to be to increase the size and power of the server running it. So, if you only need to support a high number of requests per minute for a given period, making your application more elastic and scalable may be a reasonable solution. There are two approaches for scaling your applications or infrastructures: scale out and scale up.
Scale out is one of the modern approaches for 12-factor applications. You can manage the number of cloud instances to grow your application and make it scalable and elastic. A single large process can be turned into a series of smaller processes by scaling out an application. You can then spread your native cloud application load across these small processes.
Scale up is not the recommended first choice for scaling 12-factor applications. However, it’s an option for scaling virtual machines, for example. Scale up allows you to increase RAM, the number of processors, disk space, etc. Remember that scaling up to increase machine power will inevitably involve additional costs, so you need to select your machine attributes based on your requirements.
Finally, scale out is supported by various cloud providers. If you are designing disposable, stateless applications and processes with a shared-nothing architecture, you are in an excellent position to benefit from a scale out approach. This allows you to run several instances of your application simultaneously and successfully integrate it into the cloud as a 12-factor application.
“Maximize robustness with fast startup and graceful shutdown”
An application and its infrastructure have an ephemeral life cycle in a cloud instance. It’s worth noting that a 12-factor application is disposable, which means it can be started or stopped at any time.
For instance, an application that cannot rapidly start up or shut down cannot be scaled out, deployed, put into production, or recovered efficiently. Many applications are written so that they run several lengthy processes on startup: for example, fetching data to fill a cache or setting up other runtime dependencies. Handling this type of activity separately is critical when implementing a cloud-native architecture.
For example, you might outsource the cache to a backing service so your application can start up and shut down quickly without performing front-loaded operations.
X. Dev/Prod parity
“Keep development, staging, and production as similar as possible”
The Dev and Prod parity factor recommends keeping our environments as similar as possible.
Today, many organizations are working hard to evolve and improve innovation in their information systems quickly and efficiently. As IT consultants, we are often faced with the following scenarios, which can sometimes be frustrating:
- A development environment that differs from a test QA or acceptance environment in terms of reliability and scaling.
- The database drivers used in the development and QA environments are not the same as those used in the production environment.
- The environments have different security rules, firewalls, and configuration settings.
These real examples of problems we frequently encounter in the business world can be terrifying, and they can sometimes make Build and Run teams less confident during deployments and releases.
The goal of applying consistency and discipline to Dev and Prod parity is to give your team and the rest of your organization confidence that the software will work everywhere. This means in all approved target environments.
A whole host of things can cause mismatches between environments, but the three most common factors that can have a negative impact are typically:
In many businesses, it might take weeks—or even months—from the time a developer tests their code to the time it’s deployed. In this scenario, developers often forget about the changes in the release, even though they have release notes. Or worse still, they might forget what their source code looked like if it was implemented a long time ago.
Organizations should aim to shorten this time between archiving the code commit and releasing it into production in a modern approach. We can reduce code deployment time from months/weeks to hours/minutes using this type of modern method. However, in a true continuous deployment pipeline, the final step should be to run automated tests to ensure an application’s continuous reliability.
For many companies, the thought of continuously automating everything from Build to release can be a bit unnerving. As a result, once developers get used to coding knowing that their source code will be deployed on the same day as a commit archive, their discipline and code quality often improve drastically. This can only improve the company’s quality and productivity.
It used to be that the organization’s size dictated the type of people who deployed the apps. This is because developers are typically involved in everything in smaller companies, from coding to deployment, whereas in larger firms, there are more people, roles, and teams involved.
The Dev/Prod parity factor says that developers and deployers should be the same individuals. This makes sense if your goal is to deploy to a public cloud. However, this is not necessarily true when the environment is a private cloud in a large corporation. There is a school of thought that Devs should never deploy applications, at least not outside their workstation or lab environments.
An application can be automatically deployed to all relevant environments with a CI/CD strategy and a suitable Build and Release pipeline. The application can still be deployed manually to other environments based, for example, on security restrictions within the Continuous Integration CI tool and the target cloud instance.
You are on the right track if you can deploy your apps instantly at the touch of a button or if they are deployed automatically following a series of events.
We are all guilty of taking shortcuts when pressed for time. These shortcuts can result in technical debt or put us at risk of catastrophic failure.
Today’s developers have a virtually limitless toolbox. There are very few “valid excuses” left for not using the same resource types in all environments. Developers can be given their own database instances (particularly straightforward if the database is a service on a PaaS instance). If that is not an option, container tools like Docker can help make Prod-like environments more accessible to developer workstations.
To summarize, remember when developing your 12-factor apps that whenever you validate a change (to the source code, for example, or a configuration), it should be put into production after a short period: this is the time needed to run all the tests (e.g., unit tests, functional tests, non-regression tests, etc.) and check the changes against all the integration suites.
If your development, test, and production environments differ (even in ways you might think don’t matter), you lose the ability to reliably predict how your code change will behave in production.
This trust in the source code from development to production is critical for the kind of “CI/CD” continuous delivery and deployment that allows 12-factor apps and their teams to thrive in the cloud. This is also the point of the Dev and Prod parity factor.
“Treat logs as event streams”
Application logs, also known as event logs, are a set of data collected during the execution of an application. They are the only reliable data source for monitoring and supervising the application. They are also used for analysis and troubleshooting in a production incident.
In the past, the application was often responsible for collecting the logs, filtering them by level (ERROR, WARN, DEBUG, etc.), contextualizing them, and saving them in files. This method makes it difficult and time-consuming to use the logs for operational or monitoring purposes.
This is why 12-factor applications favor a more dynamic approach in which the logs are considered a stream of events that describe the application’s behavior without having to store the output stream.
This way, the routing and processing of logs can be done separately, allowing multiple consumers of logging events for different purposes. For example, Azure Application Insights allows you to use this approach.
Source: Microsoft documentation
XII. Admin processes
“Run admin/management tasks as one-off processes”
According to the admin processes principle, scripts or processes such as database migration scripts or data populating jobs must comply with the following rules:
- These scripts should share the same deployment and execution path as the base application and must not be separated from the development life cycle. This means associating the same Build, Release, and Run sequence for the same codebase.
- The administration processes must be executed separately and isolated from the other production processes to avoid any risk of interfering with the production processes in the event of a problem (failure or potential performance impact). Starting new execution instances is preferable.
- Complete parity between the dev, test, and production environments means you can confirm that these scripts are working correctly before deploying them in production.
12-Factor App: A Basis for Reflection
Developers can use the 12-factor app methodology to build and deploy modern cloud-native or on-premises applications.
These 12 principles are designed to help developers think about how to build robust, scalable, and easy-to-use applications. Most of these principles are inspired by Martin Flower’s work: “Patterns of Enterprise Application Architecture” and “Refactoring.”
At first glance, the 12-factor app methodology appears to be all about making applications more resilient and powerful, but it also allows developers to take part in improving business processes and the Run.