Jul 17, 2015
Gabriel Hartmann
D2iQ
2 min read
Apache Mesos is software system that pools datacenter resources and facilitates their allocation and use by distributed systems. It's often claimed that Mesos is analogous to the kernel in a single-machine operating system. Just as the kernel in a conventional OS mediates allocation of CPU time and memory resources, so does Mesos provide a mechanism for managing CPU, memory and other resources in a datacenter.
To take advantage of this mechanism to perform interesting work, it is necessary to write frameworks. Frameworks are distributed applications running as services in the datacenter. They are analogous to the applications that software developers are used to writing for single machines.
In brief, these datacenter services interface with Mesos by providing a scheduler, which accepts or rejects resource offers provided by Mesos, and an executor, which launches tasks that do the bulk of the desired work. The Mesos documentation provides a more detailed explanation.
The best way to get a good feel for how frameworks achieve their goals and interact with Mesos is to write one yourself. In order to demonstrate how this works, we developed a tutorial in which we develop a datacenter service from scratch in Go. These steps are largely generic and can be used to write your own arbitrary datacenter service.
The tutorial -- you can view the the whole thing here -- progresses through three main stages:
- Empty template: In this stage, we'll deploy an empty template and observe that our scheduler has indeed received a resource offer from Mesos.
- Empty tasks: We'll then flesh out the scheduler and executor templates so that they complete tasks that have no work content.
- Batch image-processing: Finally, we'll further extend the scheduler and executor implementations to produce a framework that performs batch image-processing (in this case, inverting the images, as shown below).
After completing the tutorial, you should have a good idea how Mesos makes it easy to write a distributed application.
As written, the framework above is only as stable and available as the scheduler process. In order to write more complex and highly available frameworks, one possible path is to leverage the meta-framework Marathon. Documentation and further tutorials about Marathon and its use in the context of the DCOS are available here.
For even more DCOS information and tooling, sign up for early access to the Mesosphere SDK, which provides documentation, a reference project and a local development environment for building datacenter-scale services.
