Our tech stack includes multiple databases and data-stores such as MongoDB, Redis and Neo4j, and our production environment consists of multiple backend processes, async worker processes and web servers. All of our services are running inside Docker containers and are built and deployed using Bamboo.
Sometimes there’s an actual need to take a “snapshot” of the production environment and clone it into a temporary environment for development, debugging, customer support and QA needs.
Diving a little bit into our Bamboo build process, a Docker image is built and tagged with the Bamboo build number. E.g.: build #123 of the backend project will result in an image tagged as augury/backend:123.
Our mission is to build an automated method for launching a cloned environment, returning a DNS of a temporary EC2 instance that we can use.
Fig is a tool for quickly launching isolated development environments using Docker and we can use it for launching our entire stack pretty easily. Let’s start with an example of defining our stack in a file named fig.yml. It’ll look like:
image: augury/backend:123
ports:
– “3000:3000”
links:
– mongodb
– redis
– neo4j
– rabbitmq
environment:
– MONGODB_URL=mongodb://mongodb:27017
– MONGODB_DB=production_db
– REDIS_URL=redis:6379
– NEO4J_URL=https://neo4j:7474/db/data/
– AMQP_URL=amqp://rabbitmqmongodb:
image: dockerfile/mongodb
command: mongod –smallfiles
ports:
– “27017:27017″redis:
image: dockerfile/redis
command: redis-server /etc/redis/redis.conf
ports:
– “6379:6379″rabbitmq:
image: dockerfile/rabbitmq
ports:
– “5672:5672″neo4j:
image: tpires/neo4j
ports:
– “7474:7474”
– “1337:1337”
By running fig up we can launch our entire stack.
- All Docker image versions in
fig.ymlare supposed to be a “snapshot” of the currently running production environment. For that, we can use Bamboo’s (or any other deployment system) REST API to get the currently deployed build numbers for each one of the services. - We use Docker’s linking functionality and host aliases to configure the services which must follow the twelve-factor app concept and accept all settings from environment variables.
Next, we can launch a new EC2 instance using the AWS command line interface. It’s preferable to use a minimalistic AMI such as Debian with ephemeral instance store (it’s a temporary instance, after all).
To make things even simpler, we should do all instance setup through cloud-init (passing user-data to the command line tool), such as:
- Install Docker
- Install Fig
- Create
fig.ymlfile as before, we the relevant image versions - Copy a snapshot of the databases and other data-stores. For simplicity, the Docker images in
fig.ymlcan be run with shared folders mounted as volumes. - Launch the stack in by running
fig up -d
Once the instance is up and installed, we can use the public DNS returned by the AWS command line tool to access the cloned environment. Just remember to terminate it when you’re finished using it.
In conclusion, running everything inside Docker containers can simplify complicated dev ops tasks (such as the one described in this post), as long as the twelve-factor app methodology is followed. Fig is an awesome tool for quickly launching development environments but can also be used for running an entire stack with one command. Finally, in order to reduce Docker pull times, it’s possible to mount an EBS volume and set Docker to store its images on it – when running a new instance, most of the images will already be present.
Another thing that may be considered is that if there’s compiled or minimized code running in production and its source should be present for debugging purposes, the relevant version control commit can be grabbed from the build/deployment system. The source code (or unminimized version) can then be pulled from version control and mounted as a volume.
