UPDATE September 2022: An updated version of this article has been published here: Deployment and infrastructure for a bootstrapped webapp with 150k monthly visits.
This is an article outlining how we deploy our scoreboard webapp. By following our setup you can deploy your app so that it also reaps the benefits of this method.
Let’s get 2 caveats out of the way first:
- Our CRUD application runs on one application server. If you’re using a more complex setup, then what we describe here will need to be adjusted.
- Like many others out there, we’re learning this stuff as we go along. Please be gentle.
By the way, We run our servers on DigitalOcean but this is incidental. It should work equally well for other providers too.
Why we chose blue-green deployment
Before we get into the details, let’s quickly look at our situation before we switched to a blue-green deployment:
- We had one application server running on DigitalOcean, plus a hosted Postgres database.
- To deploy, we used a script that SSHed into that server and did a
This was fine to begin with however there were several issues:
Bad Gatewayerrors 💥
- If there was a bug in production this could be fixed by checking out the previous commit. However, this invariably took too long and always involved frenzied googling of the correct git commands.
- There was no way of testing the production setup, other than in production.
Switching to blue-green deployments fixed all of these issues.
What is blue-green deployment?
Here’s our definition of a blue-green environment:
- There are two identical and independent servers hosting the application. One is called green, the other blue.
- There is a shared production database that both servers can access.
- There is a quick and painless way of routing traffic to the green or the blue server.
One of the 2 servers is always serving production traffic, the other is idle. Let’s say green is serving production traffic, and blue is idle. When a new release is ready, it gets deployed to the idle blue server. Here it can be tested and issues fixed. Remember, the blue server is accessing the production database, so the application can be tested with real data.
Once you’re satisfied that you’re ready to go you switch traffic from the green (live) server to the blue server. If any problems occur, you can simply switch back to the green server within seconds, effectively doing a roll-back.
Basic components of our setup
For our blue-green setup we did the following things:
- We cloned our application server. On DigitalOcean this is super simple: you can create a snapshot (even of a running machine) and create a new machine from that snapshot. An even more elegant way to do this would be to use Docker… but we haven’t watched enough YouTube tutorials to do that yet.
- Setup a way to switch traffic from one server to the other. We use a floating IP from DigitalOcean. Basically they are publicly-accessible static IP addresses that you can assign to servers and instantly remap between other servers in the same datacenter. Our domain (keepthescore.co) resolves to this static IP address.
- Setup a way to determine whether the blue or the green server is currently live.
- Created a deployment script that always deploys to the idle server.
Let’s dive in a little more:
Setting up the servers
Once we’d cloned the application server, we gave them 2 different hostnames:
green-production. To do this on Ubuntu you have to do 2 things on the actual servers (in these examples for the green server):
- Carry out this command:
sudo hostnamectl set-hostname green-production
- Edit the hosts file with
sudo vim /etc/hostsand add
Then we ensured that our app can expose the hostname of the server it’s currently running on. On Flask you can create a route like this:
import socket @app.route('/hostname') def server_info(): host_name = socket.gethostname() return host_name + '\n'
Now it’s possible for a human or a machine (using
curl) to discover which the current production server is. We simply call https://keepthescore.co/hostname/. Give it a try by clicking on the link!
One final thing we needed to do is to add the public IP addresses for
green-production to the local
hosts file of our development machine(s).
The deployment script can now use this information to deploy the new version of the software to the idle server. Here’s our deployment script:
#!/usr/bin/env bash # Get the current production server and # set TARGET to the other server CURRENT=$(curl -s https://keepthescore.co/hostname) if [ "$CURRENT" = "blue-production" ]; then TARGET="green-production" elif [ "$CURRENT" = "green-production" ]; then TARGET="blue-production" else echo "Something is not right! 😬" exit -1 fi echo "Current deployment is " $CURRENT echo "Deploying to " $TARGET # Do deployment ssh -q [email protected]$TARGET "cd keepthescore && git pull" echo "Deploy to " $TARGET " complete"
We are now repeating ourselves but the beauty of this script is that it will always deploy to the idle server and not to the live production server. We can test the deployment on our development machine by simply entering
green-production into our browser – because we’ve added these IP addresses to our local
Once we’re sure that everything’s working we route traffic to the newly deployed idle server using DigitalOceans’s web interface for the floating IP addresses.
Our users get routed to the newly deployed software without noticing (hopefully).
What about the database?
The database is a sore point, because we don’t have 2 instances of the database. Martin Fowler, who wrote a great article about blue-green deployments wrote the following:
“Databases can often be a challenge with this technique, particularly when you need to change the schema to support a new version of the software. The trick is to separate the deployment of schema changes from application upgrades. So first apply a database refactoring to change the schema to support both the new and old version of the application, deploy that, check everything is working fine so you have a rollback point, then deploy the new version of the application. (And when the upgrade has bedded down remove the database support for the old version.)”
We’d love to get some feedback on our deployment strategy. Do you have questions? Are we over-engineering? Should we learn Docker? Let us know in the comments below.