Product, Use Cases

Sharing secret data in Kubernetes

For more than five years, DC/OS has enabled some of the largest, most sophisticated enterprises in the world to achieve unparalleled levels of efficiency, reliability, and scalability from their IT infrastructure. But now it is time to pass the torch to a new generation of technology: the D2iQ Kubernetes Platform (DKP). Why? Kubernetes has now achieved a level of capability that only DC/OS could formerly provide and is now evolving and improving far faster (as is true of its supporting ecosystem). That’s why we have chosen to sunset DC/OS, with an end-of-life date of October 31, 2021. With DKP, our customers get the same benefits provided by DC/OS and more, as well as access to the most impressive pace of innovation the technology world has ever seen. This was not an easy decision to make, but we are dedicated to enabling our customers to accelerate their digital transformations, so they can increase the velocity and responsiveness of their organizations to an ever-more challenging future. And the best way to do that right now is with DKP.

Nov 12, 2015

Michael Hausenblas


In this post, we will have a look into a topic you'll sooner or later bump into when using Kubernetes to build your microservices in a real-world setting. Whenever you need to pass keys for APIs, such as with Twitter apps or Amazon S3 access, or you need to provide credentials for databases, you will want to do this in a secure and safe way. This is a general access control and authorization pattern, not specific to Kubernetes but important to understand as you start working with any new technology.


The basics




A simple way to pass secret data to a pod is to use environment variables. This is conceptually better than baking them into a Docker image, which is a big anti-pattern. Don't even think about doing this, really bad things can happen, such as your cluster getting kidnapped by a botnet. While you could use environment variables in Kubernetes pod specs, this is not much better than doing so directly in the Dockerfile. Conceptually, the pod spec is seen as part of the app blueprint as well, in the same way the Dockerfile is.


What we want is (1) to separate the secrets from the app blueprint, and (2) to pass the secrets securely into the containers.


Luckily, there is a native solution for this: Kubernetes Secrets. These Secrets are first-class citizens in the Kubernetes ecosystem. They effectively are directories and files that you can mount into your containers (well, actually into a pod). Kubernetes does not support passing secrets as environment variables (see also ISSUE-4710), among other reasons due to security considerations.


There are some small gotchas with using Secrets, more information on which can be found in the docs:


  • the key must be DNS subdomain as per RFC 1035 with a max length of 253 characters
  • you need to base64-encode the value yourself
  • a secret's value is limited to 1MB in size


But enough talk, let's see Secrets in action.


In action


Suppose you want to share the value some-base64-encoded-payload under the key my-super-secret-key as a Kubernetes Secret for a pod. This means first you'd base64-encode it like so:


$ echo -n some-base64-encoded-payload | base64





Note the -n parameter with echo; this is necessary to suppress the trailing newline character. If you don't do this, your value is not correctly encoded.


Then, you can use it in a YAML doc, like this:


$ cat example-secret.yaml


apiVersion: v1


kind: Secret




  name: mysecret


type: Opaque




  my-super-secret-key: c29tZS1iYXNlNjQtZW5jb2RlZC1wYXlsb2Fk



Now it's time to let Kubernetes know about it:


$ kubectl create -f example-secret.yaml





Let's check it:


$ kubectl get secrets


NAME        TYPE      DATA


mysecret    Opaque    1



So we're ready now to use it in a pod like so:


$ cat example-secret-pod.yaml


apiVersion: v1


kind: Pod






    name: secretstest


  name: secretstest






    - name: "secretstest"




        secretName: mysecret




    - image: nginx:1.9.6


      name: awebserver




        - mountPath: "/tmp/mysec"


          name: "secretstest"



And submit it:


$ kubectl create -f example-secret-pod.yaml





Now, to check if it worked, do the following:


$ kubectl describe pods secretstest


Name:               secretstest


Namespace:          default


Image(s):           nginx:1.9.6




Labels:             name=secretstest


Status:             Running.





From this above command you can learn where Kubernetes decided to launch the pod secretstest, in our case on a node with the IP So, we ssh into this node and go like:


core@ip-10-0-1-193 ~ $ docker ps


CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS               NAMES


fe02326de9e7        nginx:1.9.6                            "nginx -g"             48 seconds ago      Up 47 seconds                           k8s_awebserver.7c974a55_secretstest_default_87f0cb42-8861-11e5-8ebb-0a9e3fc51b85_3cb2020e


dbbf535a482b   "/pause"               58 seconds ago      Up 57 seconds                           k8s_POD.e4cc795_secretstest_default_87f0cb42-8861-11e5-8ebb-0a9e3fc51b85_fcadcb9a


core@ip-10-0-1-193 ~ $ docker exec -it fe02326de9e7 /bin/sh


# cat /tmp/mysec/my-super-secret-key





And there we have it: Kubernetes has mounted the Secret on the pod and made it available under /tmp/mysec/my-super-secret-key, which unsurprisingly contains the value some-base64-encoded-payload as we would have expected and indeed hoped for.


Note that rather than doing kubectl describe pods secretstest, then sshing into the node and using docker exec, we could have used kubectl exec -i --tty secretstest /bin/sh mount directly. But I thought showing the more traditional, portable way in the first place made more sense.


One word of warning here, in case it's not obvious: example-secret.yaml should never ever be committed to a source control system such as Git. If you do that, you're exposing the secret and the whole exercise would have been for nothing.


For a complete, real-world example see also our time series DCOS demo. In this demo we use Kubernetes for the so called offline part, a aggregated view of crimes in a certain region, visualized using a overlay heatmap. There, a helper process (one container) fetches JSON data from a pre-defined S3 bucket and makes it available to another process (a second container in the same pod) that has a simple Webserver serving the map.


I'd like to thank my colleague Stefan Schimanski from the Mesosphere Kubernetes team for reviewing this post and providing very valid feedback. If you want to try out the steps above, I suggest you ramp up a Mesosphere DCOS CE cluster and you'll be done in less than 20 minutes end-to-end.


Ready to get started?