Testing DC/OS frameworks with Shakedown

For more than five years, DC/OS has enabled some of the largest, most sophisticated enterprises in the world to achieve unparalleled levels of efficiency, reliability, and scalability from their IT infrastructure. But now it is time to pass the torch to a new generation of technology: the D2iQ Kubernetes Platform (DKP). Why? Kubernetes has now achieved a level of capability that only DC/OS could formerly provide and is now evolving and improving far faster (as is true of its supporting ecosystem). That’s why we have chosen to sunset DC/OS, with an end-of-life date of October 31, 2021. With DKP, our customers get the same benefits provided by DC/OS and more, as well as access to the most impressive pace of innovation the technology world has ever seen. This was not an easy decision to make, but we are dedicated to enabling our customers to accelerate their digital transformations, so they can increase the velocity and responsiveness of their organizations to an ever-more challenging future. And the best way to do that right now is with DKP.

Jun 15, 2016

Scott Schneider


Quality assurance is an essential part of authoring great software. To help our community write great services that run on top of DC/OS, we are releasing Shakedown, a lightweight Python library that can be imported into your tests to aid and streamline your software testing workflow.




According to Wikipedia, a shakedown is a period of testing or a trial journey undergone by a ship, aircraft or other craft and its crew before being declared operational.


In the DC/OS world, Shakedown consists of both a testing harness as well as a Python library which abstracts many common Mesos and DC/OS-related tasks such as installing packages from the Universe, querying a framework service for associated tasks, running commands on individual DC/OS agents, and more.


Why we developed Shakedown


We had a few goals as we developed Shakedown:


  • Develop and achieve widespread adoption of an extensible, scalable testing harness that will serve automated testing need within the DC/OS community.
  • Deliver a well-understood, well-documented, user-friendly solution to software engineers and quality personnel, both internally and externally.
  • Optimize for test stability, usability, and speed, in that order.


Using the Shakedown tool to run tests




$ git clone github.com:dcos/shakedown.git && cd shakedown && pip3 install -e .



Shakedown requires a Python 3.4+ environment. Installation is as straightfoward as cloning the repository from GitHub and using pip to install the required libraries, as shown above.


Configuring Shakedown


Shakedown can be configured either via arguments or via a .shakedown configuration file in the user's $HOME directory.


See below for the arguments that you can pass to Shakedown on the command line (as described with shakedown --help):


--fail [fast|never]          Specify whether to continue testing when


                             encountering failures. (default: fast) 


--ssh-key-file PATH          Path to the SSH keyfile to use for




--stdout [pass|fail|skip|all|none]


                             Print the standard output of tests with the


                             specified result. (default: fail) 


--stdout-inline              Display output inline rather than after test


                             phase completion.



Alternatively, you can create a .shakedown configuration file in the user's $HOME directory to set your options:


ssh-key-file = '~/.ssh/id_rsa-mesosphere-shared' fail = 'never' stdout = 'all' stdout-inline = false



Running tests


The Shakedown testing tool is an opinionated wrapper around the popular pytest testing tool. Some default behavioral assumptions Shakedown makes are:


  • Fail fast. If any test fails, immediately abort the test run and report on what failed, and why.
  • Only verbosely report on what failed. Details and output of passing tests is not necessary.

We are testing DC/OS. First and foremost, make sure we can talk to the specified cluster. Secondly, if it is a cluster requiring authentication, all communications with the cluster should be authenticated.


$ shakedown --dcos-url http://dcos.example.com [options] [path_to_tests]


As shown above, --dcos-url is the only required Shakedown argument; all other configuration options can be specified on the command line or via a .shakedown file as detailed above under Configuring Shakedown.


Shakedown tests are pytest tests, and Shakedown will automatically discover and run any test files (filenames beginning with test_) contained within the recursive search of the current working directory.


Writing tests using the Shakedown Python library


Shakedown, aside from being a testing tool, includes a Python library which abstracts many common Mesos and DC/OS-related tasks. Full documentation on available methods can be found in the Shakedown API documentation.


Example acceptance tests


The jenkins-mesos project includes several tests which use the Shakedown library in order to test Jenkins functionality.


The (first test performed)[https://github.com/mesosphere/jenkins-mesos/blob/master/tests/acceptance/test_jenkins.py#L11-L25] by the jenkins-mesos suite uses Shakedown to install the jenkins package from the Universe and wait for it to register a "healthy" state within Mesos:


def test_install_jenkins():


    """Install the Jenkins package for DC/OS.






    assert package_installed(PACKAGE_NAME), 'Package failed to install'


    end_time = time.time() + WAIT_TIME_IN_SECS


    found = False


    while time.time() < end_time:


       found = get_service(PACKAGE_NAME) is not None


       if found and service_healthy(PACKAGE_NAME):






    assert found, 'Service did not register with DCOS'



In this example, install_package_and_wait, package_installed, get_service and service_healthy are all Python methods provided by Shakedown.




As shown above, Shakedown aims to help speed up and improve framework authoring and testing. As with the rest of DC/OS, Shakedown is an open-source project and contributions and enhancements are welcome!



Ready to get started?