Testing DC/OS frameworks with Shakedown
For more than five years, DC/OS has enabled some of the largest, most sophisticated enterprises in the world to achieve unparalleled levels of efficiency, reliability, and scalability from their IT infrastructure. But now it is time to pass the torch to a new generation of technology: the D2iQ Kubernetes Platform (DKP). Why? Kubernetes has now achieved a level of capability that only DC/OS could formerly provide and is now evolving and improving far faster (as is true of its supporting ecosystem). That’s why we have chosen to sunset DC/OS, with an end-of-life date of October 31, 2021. With DKP, our customers get the same benefits provided by DC/OS and more, as well as access to the most impressive pace of innovation the technology world has ever seen. This was not an easy decision to make, but we are dedicated to enabling our customers to accelerate their digital transformations, so they can increase the velocity and responsiveness of their organizations to an ever-more challenging future. And the best way to do that right now is with DKP.
Quality assurance is an essential part of authoring great software. To help our community write great services that run on top of DC/OS, we are releasing Shakedown, a lightweight Python library that can be imported into your tests to aid and streamline your software testing workflow.
According to Wikipedia, a shakedown is a period of testing or a trial journey undergone by a ship, aircraft or other craft and its crew before being declared operational.
In the DC/OS world, Shakedown consists of both a testing harness as well as a Python library which abstracts many common Mesos and DC/OS-related tasks such as installing packages from the Universe, querying a framework service for associated tasks, running commands on individual DC/OS agents, and more.
Why we developed Shakedown
We had a few goals as we developed Shakedown:
- Develop and achieve widespread adoption of an extensible, scalable testing harness that will serve automated testing need within the DC/OS community.
- Deliver a well-understood, well-documented, user-friendly solution to software engineers and quality personnel, both internally and externally.
- Optimize for test stability, usability, and speed, in that order.
Using the Shakedown tool to run tests
$ git clone github.com:dcos/shakedown.git && cd shakedown && pip3 install -e .
Shakedown can be configured either via arguments or via a .shakedown configuration file in the user's $HOME directory.
See below for the arguments that you can pass to Shakedown on the command line (as described with shakedown --help):
--fail [fast|never] Specify whether to continue testing when
encountering failures. (default: fast)
--ssh-key-file PATH Path to the SSH keyfile to use for
Print the standard output of tests with the
specified result. (default: fail)
--stdout-inline Display output inline rather than after test
Alternatively, you can create a .shakedown configuration file in the user's $HOME directory to set your options:
ssh-key-file = '~/.ssh/id_rsa-mesosphere-shared' fail = 'never' stdout = 'all' stdout-inline = false
The Shakedown testing tool is an opinionated wrapper around the popular pytest testing tool. Some default behavioral assumptions Shakedown makes are:
- Fail fast. If any test fails, immediately abort the test run and report on what failed, and why.
- Only verbosely report on what failed. Details and output of passing tests is not necessary.
We are testing DC/OS. First and foremost, make sure we can talk to the specified cluster. Secondly, if it is a cluster requiring authentication, all communications with the cluster should be authenticated.
$ shakedown --dcos-url http://dcos.example.com [options] [path_to_tests]
As shown above, --dcos-url is the only required Shakedown argument; all other configuration options can be specified on the command line or via a .shakedown file as detailed above under Configuring Shakedown.
Shakedown tests are pytest tests, and Shakedown will automatically discover and run any test files (filenames beginning with test_) contained within the recursive search of the current working directory.
Writing tests using the Shakedown Python library
Shakedown, aside from being a testing tool, includes a Python library which abstracts many common Mesos and DC/OS-related tasks. Full documentation on available methods can be found in the Shakedown API documentation.
Example acceptance tests
The (first test performed)[https://github.com/mesosphere/jenkins-mesos/blob/master/tests/acceptance/test_jenkins.py#L11-L25] by the jenkins-mesos suite uses Shakedown to install the jenkins package from the Universe and wait for it to register a "healthy" state within Mesos:
"""Install the Jenkins package for DC/OS.
assert package_installed(PACKAGE_NAME), 'Package failed to install'
end_time = time.time() + WAIT_TIME_IN_SECS
found = False
while time.time() < end_time:
found = get_service(PACKAGE_NAME) is not None
if found and service_healthy(PACKAGE_NAME):
assert found, 'Service did not register with DCOS'
In this example, install_package_and_wait, package_installed, get_service and service_healthy are all Python methods provided by Shakedown.
As shown above, Shakedown aims to help speed up and improve framework authoring and testing. As with the rest of DC/OS, Shakedown is an open-source project and contributions and enhancements are welcome!