Jun 09, 2015
D2iQ
D2iQ
4 min read
There are big days for a startup, and then there are big days for a startup. Today qualifies as the latter, because after two years of building it, our Datacenter Operating System (DCOS) is now generally available. Best of all, it's free to get started.
When Mesosphere started, its founders had a vision of making the datacenter as easy to use as a laptop. They asked themselves questions that seemed crazy at the time. Why should 40,000 cores in a datacenter or cloud be any different than the 4 cores in a laptop? Why can't we use a command line or even a graphical UI to navigate a datacenter like it was a desktop machine? Why can't we install datacenter-scale services with the same ease that we install apps on our iPhones?
From that vision and those questions emerged the Datacenter Operating System (DCOS), an OS as simple to use as the one in your laptop, but that controls entire datacenters or clouds. The DCOS is a new kind of operating system that can scale across tens of thousands of servers. It is a complete OS built around the open-source distributed-systems kernel Apache Mesos. It runs on all of the machines in your datacenter and pools their resources into what looks like one big computer.
The DCOS becomes the platform for deploying and managing all of the workloads in the datacenter, including cloud-native apps built with Kubernetes, big data apps built on YARN or Spark, and traditional workloads like Java apps and MySQL databases.
"Cisco knows firsthand the complexities in the datacenter, and we've seen how the Mesosphere DCOS changes the game for enterprises," said Ken Owens, CTO of cloud cervices at Cisco. "With Mesosphere DCOS, enterprises can very quickly build, deploy and scale modern applications with unprecedented operational simplicity, both on-premises and in hybrid clouds."
Two new editions to cover all your needs
Today, we're making the DCOS available in two editions:
- Community Edition (free, cloud only).
- Enterprise Edition (paid license, all platforms and configurations)
With the DCOS Community Edition, anyone can launch the Mesosphere DCOS on Amazon Web Services and have access to the same core technology that automates infrastructure for popular applications such as Twitter, Hubspot and Apple's Siri. The Community Edition is free and comes without any limitations on scalability -- so users can see for themselves just how powerful the software is and how far it can take them.
The DCOS Community Edition is also available in beta on Microsoft Azure and Google Compute Engine.
When users are ready to deploy DCOS into production, the paid Enterprise Edition offers additional features such as 24/7/365 technical support, and increased security and policy management. Enterprise Edition may be installed in a private datacenter, in a public cloud, or both, in a hybrid scenario.
Why a Datacenter Operating System. Why now.
At the same time Mesosphere was building the DCOS, two big transformations happened in the world of application development and datacenter operations. Namely, application containers (such as Docker) took off and Apache Mesos transformed into an application platform at some significant companies. The two transformations are at the same time very related and very distinct.
They also highlight why we think the Mesosphere DCOS is poised to become the next great computing platform. Let me explain.
Containers and Docker
Container technology has been around for more than a decade as a tool for isolating the resources needed by the processes running on a host OS, but it wasn't until the past couple years that containers really caught on in the mainstream. This was, in large part, thanks to the efforts of an open source project (and eponymous commercial backer) called Docker.
What makes Docker so special is how easy it is to use. While containers always made it possible to maximize server utilization by running all sorts of isolated tasks on it, Docker has made the process something developers can do themselves with minimal effort. A Docker container is essentially a special file that dictates what services an application connects to and what resource levels it needs, and that can be created on a desktop and easily launched -- unmodified -- onto a server.
Developers don't to worry about things like OS versions or server specs. They can create once and deploy anywhere. This creates a virtuous cycle in which easier tooling leads to happier developers, which in turn leads to better applications and architectures.
Mesos catches on
Mesos is also concerned with resource efficiency, developer productivity, and abstracting away concerns over things such as OS types and server specs. The big difference it and something like Docker is where they fit into the datacenter: while Docker is very much at the application level, Mesos manages the actual servers on which those applications run.
Here's how it works: When a workload requests resources to run, Mesos will assess what it needs and what's available in the cluster, and then use a type of container technology called Linux cgroups in order to provision the necessary memory, CPU and storage. Mesos will locate the job on whichever server(s) happen to have the appropriate resources available and automate failover should a machine die.
Mesos also supports jobs deployed as Docker containers. And while Docker containers are traditionally deployed one at a time on servers, Mesos can launch thousands of them in seconds.
Now that it's deployed at scale inside a growing number of companies -- including at Apple and Twitter -- we're seeing some interesting trends pop up:
- Platform as a service, redefined: So-called platform-as-a-service projects that have been created by the Mesos community, such as Marathon (Mesosphere), Aurora (Twitter), Singularity (Hubspot), J.A.R.V.I.S. (Apple) and PaaSta (Yelp), are helping users rethink their strategies around application platforms. Developers can easily launch new services (via Docker container or otherwise) into production or test-dev environments without waiting for machines to be provisioned or worrying what the underlying infrastructure looks like.
- Hybrid cloud realized: Some Mesos users, including Yelp and Orbitz, have used it to create hybrid environments that span private datacenters and public clouds. This means jobs can run wherever it's more economical, secure or whatever else matters, without forcing developers or operations staff to follow different protocol for each environment. Differences between the specs of local machines and cloud instances are abstracted away when they join a cluster. Launching jobs onto Mesos via Docker, a homegrown PaaS or directly from a framework like Spark is the same regardless where it's running.
- Same servers, more innovation: Aside from improvements at the application and architectural levels, Mesos is also letting users do more business on the same of resources. Groupon, for example, used to provision physical servers for each new workload, but now many small services share the same server. Among other benefits, the move freed up capacity for data analysts to test more real-time advertising models using Apache Storm. Mattermark, a startup running on Amazon Web Services, is able to save on monthly cloud bills by operating its Mesos-powered cloud instances at 70-75 percent capacity.
Operating like Twitter, without a team of systems engineers
However, as impressive as some of these Mesos deployments are, it's important to remember that using open source -- and sometimes homemade -- components often requires teams of experienced distributed systems engineers. Mesosphere wants bring the benefits of Mesos, and then some, to companies of all shapes and sizes. We want to let users build and deploy applications whose components span entire datacenters, but with the simplicity of programming for a single machine.
The Mesosphere DCOS is built atop Mesos, but we take it much farther by giving the datacenter a simple, yet powerful command line and dashboard. DCOS ships with Marathon, our open source PaaS-like service, pre-installed, but additional datacenter services can be installed in seconds.
Want to deploy HDFS as a shared file system? Deploy it in seconds using DCOS. Want to deploy thousands of Docker containers as a cluster in Kubernetes?
$ dcos package install kubernetes
Or the Cassandra NoSQL database to power a high-transaction web application? Deploy those in seconds, too, using the same command line.
$ dcos package install cassandra
The DCOS dashboard is the icing on the cake, letting ops teams and even developers and architects, if they're so inclined, monitor what's happening with their resource pool and all the services running on it. It's visual, intuitive and easy to use.
Furthermore, DCOS users will receive a consistent user experience and the same levels of automated orchestration and maximized resource utilization wherever they choose to install it -- in the public cloud, in a private cloud or in a hybrid environment.
What this means to companies is that developers spend less time worrying about server configuration; IT spends less time worrying about server provisioning and failure; CIOs spend less time worrying about hardware budgets and server utilization; and everyone spends less time worrying about important applications going down.
Growing more powerful with more services
Mesosphere is also building out an ecosystem of partners both commercial and open source so that it's easier to launch and run even more services on the DCOS. One early and excited partner is MemSQL, the startup whose scale-out in-memory database is already deployed by some very large companies.
Speaking of databases, Mesosphere DCOS also gains functionality as the open source community around Apache Mesos continues to grow. Case in point: Twitter recently formed an Apache Incubator project called Mysos, which centers around its code for running MySQL at scale on Mesos. Two Mesosphere engineers are among the project's early committers.
There are many other open source projects underway at other Mesos user-companies, many of which will be supported or integrated Mesosphere DCOS.
If all of this sounds like something you're interested in, get started today with the GA version on AWS (or beta versions on Azure and GCE), or give us a call.
