Product, Use Cases, Partners

Demo: Migrating MongoDB data with Mesos and Flocker

May 21, 2015

D2iQ

D2iQ

This is a guest post from Kai Davenport at ClusterHQ.

 

NOTE: ClusterHQ has published a more up-to-date version of this demo that uses the official Docker plugin instead of Powerstrip.

 

The amazing app you just published is gaining traction and the post to Hacker News last week hit home -- you've seen a 400 percent increase in average load!

 

Clearly, your MongoDB server needs more resources because it's becoming a bottleneck in your stack. Because you wanted to keep costs down, your Mesos cluster is made up from low-power machines, but you decide to add some much more-powerful hardware and faster disks. Because you deployed your database inside a Docker container using Marathon, it is simple to reschedule the container onto a machine with more-powerful hardware.

 

The problem

 

When we move the database container, we need a way to migrate the data also. This is essential, lest users become unhappy because their data was left on the old machine. We also need to minimize the downtime required to make this migration happen.

 

The solution

 

Using Powerstrip, we can use the Flocker local-storage migration feature to move the data to a new server. This means we are treating the container and the data as an atomic unit -- when Marathon reschedules the container to another machine, Flocker moves the data alongside it.

 

We could use MongoDB sharding to eliminate some of this problem in the future, but because we started out with m1.small machines on Amazon Web Services and need to upgrade to c3.xlarge machines with SSDs, we need a way to migrate the data whatever we do.

 

Demonstration

 

Our demo is a Backbone version of the classic TodoMVC application. It is plugged into a node.js TodoMVCBackend, which saves its data inside a MongoDB container.

 

We have added attributes to the two Mesos slaves -- disk=spinning and disk=ssd-- to represent the types of disk they have. The Mongo container is first allocated onto the node with the spinning disk and then migrated (along with its data) onto the node with an SSD drive.

 

Before migration

 

[caption id="attachment_1766" align="aligncenter" width="532"][Figure 1: Node.js container accessing Mongo container on node 1]10 Figure 1: Node.js container accessing Mongo container on node 1[/caption]

 

After migration

 

[caption id="attachment_1767" align="aligncenter" width="530"][Figure 2: MongoDB container and data volume migrated to node 2]11 Figure 2: MongoDB container and data volume migrated to node 2[/caption]

 

The roles of Mesos and Marathon

 

Mesos plays the role of a kernel for accessing hardware resources in a cluster of machines. It can offer resources to registered frameworks, which will schedule jobs onto machines and invoke the framework's executor on the chosen machine to run the job.

 

Marathon plays the role of a cluster-wide init and control system. It runs as a Mesos framework and presents a REST API that can be used to deploy long-running Docker containers across the cluster.

 

 

Installation

 

First you need to install:

 

 

We'll use Virtualbox to supply the virtual machines that our Mesosphere cluster will run on. We'll use Vagrant to simulate our application stack locally. You could also run this demo on AWS or Rackspace with minimal modifications.

 

Step 1: Start VMs

 

The first step is to clone this repo and start the three VMs:

 

$ git clone https://github.com/binocarlos/powerstrip-mesosphere-demo

 

$ cd powerstrip-mesosphere-demo

 

$ vagrant up

 

Step 2: Open Marathon/Mesos GUI

 

Now, we open the Marathon GUI so we can monitor our deployment. Open a web browser and point it to this URL:

 

http://172.16.255.250:8080

 

[caption id="attachment_1771" align="aligncenter" width="600"][Figure 3: The Marathon web GUI before deployment]15 Figure 3: The Marathon web GUI before deployment[/caption]

 

We can also open the Mesos GUI to monitor the underlying resource usage. Open another web browser and point it to this URL:

 

http://172.16.255.250:5050

 

[caption id="attachment_1776" align="aligncenter" width="600"][Figure 4:  The Mesos web GUI before deployment]15 Figure 4: The Mesos web GUI before deployment[/caption]

 

Step 3: Deploy the MongoDB container

 

First, we deploy our MongoDB container to Marathon using the example/todomvc/db.json config. This will schedule the container onto the host with the spinning disk:

 

$ cat example/todomvc/db.json \  | curl -i -H 'Content-type: application/json' -d @- http://172.16.255.250:8080/v2/apps

 

Step 4: Deploy the node.js container

 

Then, we deploy our app container to Marathon using the example/todomvc/app.json config:

 

$ cat example/todomvc/app.json \  | curl -i -H 'Content-type: application/json' -d @- http://172.16.255.250:8080/v2/apps

 

Step 5: Check deployment

 

The Marathon GUI should display the 2 deployments.

 

[caption id="attachment_1777" align="aligncenter" width="600"][Figure 5:  The Marathon web GUI after deployment]16 Figure 5: The Marathon web GUI after deployment[/caption]

 

We can also check the status by using the REST API:

 

$ curl http://172.16.255.250:8080/v2/tasks

 

Step 6: Add some data

 

Next, open the application in a browser and add some todo entries. Once you have typed an entry press Enter to submit it to the database:

 

http://172.16.255.251:8000/

 

[caption id="attachment_1778" align="aligncenter" width="600"][Figure 6: The todoMVC application]17 Figure 6: The todoMVC application[/caption]

 

Step 7: Stop the MongoDB container

 

Now, we stop the MongoDB container in preparation for moving it to node2:

 

$ curl -X "DELETE" http://172.16.255.250:8080/v2/apps/mongo

 

Step 8: Re-deploy the MongoDB container

 

Next, we use the same deployment file but replace the constraint so the Mongo container is scheduled onto the node with the SSD disk:

 

$ cat example/todomvc/db.json \  | sed 's/spinning/ssd/' \  | curl -i -H 'Content-type: application/json' -d @- http://172.16.255.250:8080/v2/apps

 

Step 9: Check deployment

 

Now we have moved the Mongo container; lets check the Marathon GUI for the deployment status.

 

[caption id="attachment_1779" align="aligncenter" width="600"][Figure 7: The Marathon web GUI after the 2nd deployment]18 Figure 7: The Marathon web GUI after the 2nd deployment[/caption]

 

Step 10: Check data

 

Reload the application in a browser and check that the todo entries we added before are still there (meaning we have migrated the data successfully):

 

http://172.16.255.251:8000/

 

Note: It sometimes take 10 seconds for the MongoDB container to deploy and for the node.js container to connect to it. If the data does not appear, press refresh after 10 seconds.

 

How it works

 

The key part of this demonstration is the usage of Flocker to migrate data from one server to another. To make Flocker work natively with Mesos and Marathon, we've used Powerstrip. Powerstrip is an open-source project we started to prototype Docker extensions.

 

This demo uses the Flocker extension prototype (powerstrip-flocker). Once the official Docker extensions mechanism is released, Powerstrip will go away and you'll be able to use Flocker directly with Mesos and Marathon (or Docker Swarm or Kubernetes) to perform database migrations.

 

We have installed Powerstrip and powerstrip-flocker on each host. This means that when Marathon starts a container with volumes, powerstrip-flocker is able to prepare/migrate the required data volumes before docker starts the container.

 

Please note: Because this demo uses Powerstrip, which is only meant for prototyping Docker extensions, we do not recommend this configuration for anything approaching production usage. When Docker extensions become official, Flocker and Weave will support them. Until then, this is just a proof-of-concept.

 

The Mesos cluster

 

The master node (which controls the cluster) is running the following components:

 

  • mesos-master: the master node for the mesos cluster
  • marathon: a mesos framework that runs long running processes
  • zookeeper: a distributed key/value store
  • flocker-control-service: the control service for the Flocker cluster

 

The two slave nodes each run:

 

  • mesos-slave: the slave process that communicates with the mesos-master
  • flocker-zfs-agent: the flocker slave process that communicates with the flocker-control-service
  • powerstrip: the prototyping tool for Docker extensions
  • powerstrip-flocker: a powerstrip adapter that creates ZFS volumes for containers
  • powerstrip-weave: a powerstrip adapter that networks containers together across hosts

 

[caption id="attachment_1781" align="aligncenter" width="656"][Figure 8: Overview of the Mesosphere cluster]21 Figure 8: Overview of the Mesosphere cluster[/caption]

 

Conclusion

 

Mesos and Marathon are powerful tools to manage a cluster of machines as though they are one large computer. We have shown in this demo that you can extend the behavior of Mesos slaves using Powerstrip adapters (and soon official Docker extensions).

 

This demo made use of local storage for your data volumes. Local storage is fast and cheap and with Flocker, it's also portable between servers and even clouds.

 

We are also working on adding support for shared storage to enable failover support when your hosts have access to network (e.g., EBS or SAN) storage.

Ready to get started?