Product, Use Cases

Sharing secret data in Kubernetes

Nov 12, 2015

Michael Hausenblas

D2iQ

In this post, we will have a look into a topic you'll sooner or later bump into when using Kubernetes to build your microservices in a real-world setting. Whenever you need to pass keys for APIs, such as with Twitter apps or Amazon S3 access, or you need to provide credentials for databases, you will want to do this in a secure and safe way. This is a general access control and authorization pattern, not specific to Kubernetes but important to understand as you start working with any new technology.


The basics




A simple way to pass secret data to a pod is to use environment variables. This is conceptually better than baking them into a Docker image, which is a big anti-pattern. Don't even think about doing this, really bad things can happen, such as your cluster getting kidnapped by a botnet. While you could use environment variables in Kubernetes pod specs, this is not much better than doing so directly in the Dockerfile. Conceptually, the pod spec is seen as part of the app blueprint as well, in the same way the Dockerfile is.


What we want is (1) to separate the secrets from the app blueprint, and (2) to pass the secrets securely into the containers.


Luckily, there is a native solution for this: Kubernetes Secrets. These Secrets are first-class citizens in the Kubernetes ecosystem. They effectively are directories and files that you can mount into your containers (well, actually into a pod). Kubernetes does not support passing secrets as environment variables (see also ISSUE-4710), among other reasons due to security considerations.


There are some small gotchas with using Secrets, more information on which can be found in the docs:



  • the key must be DNS subdomain as per RFC 1035 with a max length of 253 characters

  • you need to base64-encode the value yourself

  • a secret's value is limited to 1MB in size


  • But enough talk, let's see Secrets in action.


    In action


    Suppose you want to share the value some-base64-encoded-payload under the key my-super-secret-key as a Kubernetes Secret for a pod. This means first you'd base64-encode it like so:


    $ echo -n some-base64-encoded-payload | base64

    c29tZS1iYXNlNjQtZW5jb2RlZC1wYXlsb2Fk


    Note the -n parameter with echo; this is necessary to suppress the trailing newline character. If you don't do this, your value is not correctly encoded.


    Then, you can use it in a YAML doc, like this:


    $ cat example-secret.yaml

    apiVersion: v1

    kind: Secret

    metadata:

    name: mysecret

    type: Opaque

    data:

    my-super-secret-key: c29tZS1iYXNlNjQtZW5jb2RlZC1wYXlsb2Fk


    Now it's time to let Kubernetes know about it:


    $ kubectl create -f example-secret.yaml

    secrets/mysecret


    Let's check it:


    $ kubectl get secrets

    NAME TYPE DATA

    mysecret Opaque 1


    So we're ready now to use it in a pod like so:


    $ cat example-secret-pod.yaml

    apiVersion: v1

    kind: Pod

    metadata:

    labels:

    name: secretstest

    name: secretstest

    spec:

    volumes:

    - name: "secretstest"

    secret:

    secretName: mysecret

    containers:

    - image: nginx:1.9.6

    name: awebserver

    volumeMounts:

    - mountPath: "/tmp/mysec"

    name: "secretstest"


    And submit it:


    $ kubectl create -f example-secret-pod.yaml

    pods/secretstest


    Now, to check if it worked, do the following:


    $ kubectl describe pods secretstest

    Name: secretstest

    Namespace: default

    Image(s): nginx:1.9.6

    Node: 10.0.1.193/10.0.1.193

    Labels: name=secretstest

    Status: Running.

    ..


    From this above command you can learn where Kubernetes decided to launch the pod secretstest, in our case on a node with the IP 10.0.1.193. So, we ssh into this node and go like:


    core@ip-10-0-1-193 ~ $ docker ps

    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

    fe02326de9e7 nginx:1.9.6 "nginx -g" 48 seconds ago Up 47 seconds k8s_awebserver.7c974a55_secretstest_default_87f0cb42-8861-11e5-8ebb-0a9e3fc51b85_3cb2020e

    dbbf535a482b gcr.io/google_containers/pause:0.8.0 "/pause" 58 seconds ago Up 57 seconds k8s_POD.e4cc795_secretstest_default_87f0cb42-8861-11e5-8ebb-0a9e3fc51b85_fcadcb9a

    core@ip-10-0-1-193 ~ $ docker exec -it fe02326de9e7 /bin/sh

    # cat /tmp/mysec/my-super-secret-key

    some-base64-encoded-payload#


    And there we have it: Kubernetes has mounted the Secret on the pod and made it available under /tmp/mysec/my-super-secret-key, which unsurprisingly contains the value some-base64-encoded-payload as we would have expected and indeed hoped for.


    Note that rather than doing kubectl describe pods secretstest, then sshing into the node and using docker exec, we could have used kubectl exec -i --tty secretstest /bin/sh mount directly. But I thought showing the more traditional, portable way in the first place made more sense.


    One word of warning here, in case it's not obvious: example-secret.yaml should never ever be committed to a source control system such as Git. If you do that, you're exposing the secret and the whole exercise would have been for nothing.


    For a complete, real-world example see also our time series DCOS demo. In this demo we use Kubernetes for the so called offline part, a aggregated view of crimes in a certain region, visualized using a overlay heatmap. There, a helper process (one container) fetches JSON data from a pre-defined S3 bucket and makes it available to another process (a second container in the same pod) that has a simple Webserver serving the map.


    I'd like to thank my colleague Stefan Schimanski from the Mesosphere Kubernetes team for reviewing this post and providing very valid feedback. If you want to try out the steps above, I suggest you ramp up a Mesosphere DCOS CE cluster and you'll be done in less than 20 minutes end-to-end.


    Ready to get started?