Role-based access control is a key to properly secure Kubernetes cluster. The roles can be namespace or cluster based.
Role is set of resources and operations allowed on them. It is a set of rules for resources (grouped in apiGroups) and allowed actions (verbs) associated with a specific namespace.
Role binding connects role to the user/group.
ClusterRole is a version of the role which is applied to the whole cluster.
Cluster role binding connects cluter role to the user/group.
- cluster-admin - allows to perform any access on any resource
- admin - allows admin access, does not allow to modify resource quota or namespace for which it is granted.
- system-masters - full admin for the cluster
- system:basic-user, system:unauthenticated - unauthenticated, no operations allowed
By default, pods are non-isolated; they accept traffic from any source.
Pods become isolated by having a NetworkPolicy that selects them. Once there is any NetworkPolicy in a namespace selecting a particular pod, that pod will reject any connections that are not allowed by any NetworkPolicy. (Other pods in the namespace that are not selected by any NetworkPolicy will continue to accept all traffic.)
In staging/test environment and production cluster all pods should be isolated using network policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Production cluster/namespace may have no network policies for pods for easier prototyping and development.
Certificates and user accounts¶
1) SSH env jumpbox
ssh firstname.lastname@example.org -I jumpbox.pem
2) Login to credhub and grab private key for CA and CA certs
credhub_login credhub get -n /bosh/k8s/kubo_ca
3) Save certs to files
Private key (--tls-private-key-file, /etc/kubernetes/pki/private/server.key): kubernetes-key.pem CA (--client-ca-file, /etc/kubernetes/pki/ca.crt): kubernetes-ca.pem
Certificate (--tls-cert-file, /etc/kubernetes/pki/issued/server.crt): kubernetes-crt.pem
4) Create private key and CSR using openssl
openssl genrsa -out adko.key 2048 openssl req -new -key adko.key -out adko.csr -subj "/CN=adko/O=dev/O=test"
CN - common name (login, username) O - organisation unit (group)
openssl x509 -req -in adko.csr -CA kubernetes-ca.pem -CAkey kubernetes-key.pem -CAcreateserial -out adko.crt -days 365
5) Configure kubectl
kubectl config set-cluster adko-aws --certificate-authority kubernetes.pem --embed-certs=true --server=https://cloudboostr-k8s-api.us-west-1.elb.amazonaws.com:8443 kubectl config set-credentials adko --client-certificate=adko.crt --client-key=adko.key --embed-certs=true kubectl config set-context adko-aws-test --cluster=adko-aws --user=adko
For easier context switching use: https://github.com/ahmetb/kubectx
Yaml samples for roles¶
Instead of specific resources or verbs it is possible to use wildcards: ["*"] To see existing roles and allocation in specific apiGroups check: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml
View-only role for test namespace:¶
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: test name: test-view-role rules: - apiGroups: [""] resources: ["pods", "deployments", "services", "replicasets"] verbs: ["get", "watch", "list"]
Binding role to the user:
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: dev-view-rolebinding namespace: test subjects: - kind: User name: adko apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: test-view-role apiGroup: rbac.authorization.k8s.io
View access for all developers to prod¶
kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: prod name: prod-view-role rules: - apiGroups: [""] resources: ["pods", "deployments", "services", "replicasets", "replicationcontrollers"] verbs: ["get", "list", "watch"] - apiGroups: ["extensions"] resources: ["deployments"] verbs: ["get", "list", "watch"]
Binding role to the users group:
kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: prod-view-rolebinding namespace: prod subjects: - kind: Group name: dev apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: prod-view-role apiGroup: rbac.authorization.k8s.io
If you have an error below kubectl update is required:
error: SchemaError(io.k8s.api.rbac.v1.RoleBindingList): invalid object doesn't have additional properties
Helm, security and colocating production with test and production¶
Helm runs inside the cluster. This means effectively anyone can run anything from the inside if the Helm installation is using default, cluster-wide, serivce account. Bear in mind that in general helm charts itself are also not safe and may create service accounts with cluster-wide permissions.
The best solution for that will be to have multiple tillers - e.g. tiller per dev team with the same privileges the team has set with a service account. Another way would be to have tiller per namespace depending on the security layout.
Port-forward in tiller namespace
Without network policies in place, all pods can access each other. This is not a problem in dev environment, but very much is for production namespace. Having prod colocated in the same cluster with all other envs means that network policies are required.
To avoid resources starvation by dev environment node affinity/anti-affinity or nodeSelectors should be applied, so the production pods are physically isolated from the test and dev. Additionally in general dev/test pods should never have higher priority than prod ones to avoid eviction.
As mentioned in kubernetes docs:
Setting resource limits properly, and testing that they are working takes a lot of time and effort so unless you can measurably save money by running production in the same cluster as staging or test, you may not really want to do that.
Using namespaces to partition the cluster¶
The article below explains with examples how namespaces can be used to partition the cluster: