Jul 16, 2015
D2iQ
D2iQ
5 min read
Autodesk, a multi-national software company with revenue surpassing $2.5 billion, is a global leader in 3-D design, engineering and entertainment software that helps customers imagine, design and create a better world.
As Autodesk gears up for continued growth in its third decade, it's looking at Apache Mesos and Mesosphere to help bolster its existing infrastructure. A major goal is to accelerate the release of new types of bleeding-edge cloud technology specializing in visualization, analysis, simulation and collaboration.
Autodesk thinks a Mesos-based platform that abstracts the complexities of the underlying infrastructure will be a big help in making that happen. Autodesk Cloud Services are hosted in a hybrid cloud infrastructure in both Amazon Web Services and private datacenters -- but that's information that only select people should really care about.
"It shouldn't matter to our developers where our stuff is hosted," explains Stephen Voorhees, the company's director of engineering for cloud platforms. "Our focus should be on making deploying applications as simple as telling the system what resources they need to get the job done. I want no more provisioning of virtual machines or writing against infrastructure-level APIs."
The desire to leverage Mesos is because it provides that abstraction layer from Autodesk infrastructure and the datacenter. Mesos enables efficient allocation of resources on underlying infrastructure to control utilization as well as control costs.
"We really don't want to have to deal with knowing all that details of what types of servers are running where," says Vorhees. "We should just be able to press a button and deploy containers through our CI/CD pipeline onto Mesos without having to worry."
However, even with growing awareness of the intrinsic value that Mesos could provide for the type of resource abstraction that Autodesk Cloud needs, the company still had to prove it would work.
Putting Mesos through its paces
This is where Olivier Paugam, senior software architect for the Autodesk Cloud Platform and resident technology cowboy, enters the picture. He wanted to see if Mesos could do all the things it claimed it could, so he built a Mesos environment to power the company's new Event Streaming Service. The idea was to demonstrate how well Mesos might work cross-organization, cross-division and cross-company. (Paugam describes much of his experience with Mesos across several posts on the Autodesk Cloud Engineering blog, which is worth checking out.)
"The Event Streaming Service is already a critical part of the core backbone for our cloud platform," Paugam explains. "It handles communications between numerous backend systems, including rendering, data translation, analytics and identity management. As you can imagine, the service needs to be able to handle the scale and volume of data that is being published and subscribed to with the highest degree of reliability."
The Event Streaming Service is running in a production environment, and so far the results have been impressive. The system spans four separate Amazon Web Services clusters in different global regions, with about 12 machines per cluster. It's built on Mesos, Marathon and Ochopod, and runs ZooKeeper (for Ochopod data synchronization), Kafka, RabbitMQ, Redis, HAProxy and some Play Framework web stacks.
[caption id="attachment_2598" align="alignnone" width="800"][
]4Credit: Olivier Paugam / http://cloudengineering.autodesk.com/blog/2015/05/cicd-mesos-ochothon.html[/caption]
Ochopod is an open source, Python-based container orchestration layer developed at Autodesk that is meant to integrate with Kubernetes and Marathon. Autodesk has also open sourced a project called Ochothon, which is a drop-in component for Mesos/Marathon that provides simple tools to help deploy, manage and maintain Ochopod containers -- a mini-PaaS, of sorts.
With the Event Streaming Service now running in production, Autodesk expects the infrastructure to grow as large as 30 machines per cluster. "Using Mesos, we can actually very effectively scale the whole thing to accommodate the growing load," Paugam says.
Additionally, Paugam, like Voorhees, also wants as much abstraction as possible between developers and the underlying resources. Mesos, along with Docker as the container format, hasn't disappointed.
Paugam enthusiastically explains: "As soon as we were able to use Mesos as a general resource platform, it was like day and night in terms of overall improvement. It was fantastic! Mesos provides that strong decoupling we were looking for. That alone is a huge improvement."
Paugam shared that previously, deploying new services like the Event Streaming Service would have taken weeks. It would have required asking operations for new Amazon EC2 images, waiting for them, and then figuring out how to deploy and connect to Kafka, ZooKeeper and the rest of the stack. If machines died or their specs changed, it often meant troubleshooting configuration problems.
With the new setup, Paugam just asks operations for a cluster of images, makes sure they're hardened, and he deploys his containers. Three easy steps to deployment.
[caption id="attachment_2597" align="alignnone" width="800"][
]6 Credit: Olivier Paugam / http://cloudengineering.autodesk.com/blog/2015/05/scalable-eventing-over-mesos.html[/caption]
Making Mesos mainstream inside of Autodesk
Autodesk was so impressed with the Event Streaming Service project results that it considers the Mesosphere's Datacenter Operating System (DCOS) a very viable option for rolling out Mesos and Marathon company-wide. The strong drive to abstract away the complexities of deploying applications for developers extends to Autodesk wanting to eliminate the operational complexity of setting up and scaling its Mesos environment and of keeping all the components up to date.
"We don't want to worry about all those little details," says Vorhees. "We just want to know that Mesos is there, doing it's thing."
The simplicity, efficiency, and utility of the Mesos features are key factors to broader adoption of the technology within Autodesk. Among other things, various big data frameworks, including Hadoop and Spark could eventually be running on a Mesos cluster.
Ultimately, providing a uniform, and scalable, internal platform for managing all of these various systems would be a huge win for Autodesk after years of trying to find one. "What we struggled with for a long time, was coming up with a standard approach for how we deploy our software," Vorhees explaines. "We feel that the combination of Mesos, Marathon and Docker solves a lot of the problems that we struggled with over the last couple of years. This could get us over that hump."
Laying the foundation for standard infrastructure and deployment policies internally will carry Autodesk toward its next big venture. Autodesk is looking to provide an external-facing PaaS for Autodesk third-party developers, which would be a prime candidate for running on Mesos.
There are definitely exciting times ahead for strategic technology partnerships, with Mesos helping Autodesk Cloud do it right.
