Install the Gremlin Daemon¶
Gremlin must be installed on each host you wish to attack, and every installed gremlin must be registered with the Gremlin service. If you would prefer to install Gremlin with Docker instead of running it directly on the host, read our guide on How to Install and Use Gremlin in a Docker Container.
How to install Gremlin with Debian¶
# Add the Gremlin repo echo "deb https://deb.gremlin.com/ release non-free" | sudo tee /etc/apt/sources.list.d/gremlin.list # Import the GPG key sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C81FC2F43A48B25808F9583BDFF170F324D41134 9CDB294B29A5B1E2E00C24C022E8EF3461A50EF6 # Install Gremlin client and daemon sudo apt-get update && sudo apt-get install -y gremlin gremlind
Note that you might also need to install the
apt-transport-https package to be able to install Gremlin from our repo via HTTPS.
How to install Gremlin with RPM¶
# Add the Gremlin repo sudo curl https://rpm.gremlin.com/gremlin.repo -o /etc/yum.repos.d/gremlin.repo # Install Gremlin client and daemon sudo yum install -y gremlin gremlind
How to setup Docker Permissions for Gremlin Attacks¶
gremlind to attack Docker containers, you need to add the
gremlin user to the
docker group after installing Gremlin and Docker.
sudo adduser gremlin docker
How to install Gremlin with Kubernetes¶
Gremlin has been tested to work on Kubernetes versions 1.6 and up.
Create a Kubernetes secret¶
If you do not already have your certificates locally, you can download them by going the teams page and selecting the team for which you’d like to install the client. From there you can select ‘Download’ to download the current certificate, or ‘Create New’ if you have not yet created your client certificates.
When you download your certificate files, they will have a name like
YOUR_TEAM_NAME-client.pub_cert.pem. Rename these files to
gremlin.cert respectively. Then create your secret as follows:
kubectl create secret generic gremlin-team-cert --from-file=./gremlin.cert --from-file=./gremlin.key
Installation with Helm¶
Before installing with Helm, be sure to configure your team secret as described in the section above.
The simplest way to install the Gremlin client on your Kubernetes cluster is to use helm. If you do not already have Helm installed, go here to get started. Once helm is installed and configured, add the gremlin repo and install the client:
helm repo add gremlin https://helm.gremlin.com helm install --set gremlin.teamID=YOUR-TEAM-ID gremlin/gremlin
For more information on the Gremlin Helm chart, including more configuration options, check out the chart on Github.
Installation with kubectl¶
Here is a sample DaemonSet configuration template for installing Gremlin into your nodes.
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: gremlin namespace: <namespace where you want to run an attack> labels: k8s-app: gremlin version: v1 spec: template: metadata: labels: k8s-app: gremlin version: v1 spec: # If you want to enable host-level process-killing, add this flag: #hostPID: true # If you want to enable host-level network attacks, add this flag: #hostNetwork: true containers: - name: gremlin image: gremlin/gremlin args: [ "daemon" ] imagePullPolicy: Always securityContext: capabilities: add: - NET_ADMIN - SYS_BOOT - SYS_TIME - KILL env: - name: GREMLIN_TEAM_ID value: <YOUR TEAM ID GOES HERE> - name: GREMLIN_TEAM_PRIVATE_KEY_OR_FILE value: file:///var/lib/gremlin/cert/gremlin.key - name: GREMLIN_TEAM_CERTIFICATE_OR_FILE value: file:///var/lib/gremlin/cert/gremlin.cert - name: GREMLIN_IDENTIFIER valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts: - name: docker-sock mountPath: /var/run/docker.sock - name: gremlin-state mountPath: /var/lib/gremlin - name: gremlin-logs mountPath: /var/log/gremlin - name: shutdown-trigger mountPath: /sysrq - name: gremlin-cert mountPath: /var/lib/gremlin/cert readOnly: true volumes: # Gremlin uses the Docker socket to discover eligible containers to attack, # and to launch Gremlin sidecar containers - name: docker-sock hostPath: path: /var/run/docker.sock # The Gremlin daemon communicates with Gremlin sidecars via its state directory. # This should be shared with the Kubernetes host - name: gremlin-state hostPath: path: /var/lib/gremlin # The Gremlin daemon forwards logs from the Gremlin sidecars to the Gremlin control plane # These logs should be shared with the host - name: gremlin-logs hostPath: path: /var/log/gremlin # If you want to run shutdown attacks on the host, the Gremlin Daemon requires a /proc/sysrq-trigger:/sysrq mount - name: shutdown-trigger hostPath: path: /proc/sysrq-trigger - name: gremlin-cert secret: secretName: gremlin-team-cert
Considerations when Attacking the Network of a Kubernetes Pod¶
By definition, containers of a Kubernetes Pod all share a network interface. This means when Gremlin applies a network impact to one container within a Kubernetes pod, the impact will be observed for all containers in the Pod. Note that this does not apply to containers in Pod replicas. If you attack a specific Pod replica, the effect applies to containers within that replica only, and does not apply to the rest of the replicas.
It is always recommended to target only a single container of a Pod. If you wish to exclude some containers from the network impact, reduce your blast radius by specifying ports relevant to the containers you wish to see impact.
Once Gremlin is installed, you want to make sure it will run properly on your system.
How to use Gremlin Syscheck¶
Note: DO NOT run this command on production hosts
syscheck command is a quick way to verify that all or a set of desired gremlins will work as intended. When you run
gremlin syscheck without any additional arguments, the Gremlin client will run some prepared attacks for each of the gremlin attack types. These attacks are short in length (10 to 15 seconds each) and designed to test the efficacy of Gremlin on the system in which it is running.
Syscheck Test Types¶
Each Type can be supplied as the argument to
syscheck to run that test only.
gremlin syscheck blackhole
|Type||Assert Gremlin can…|
||consume up to 1 cpu core on the system|
||occupy up to 50% of the block device that
||consume up to 512Mb on the system|
||drop all egress traffic from the system|
||introduce 100ms of latency for all egress traffic from the system|
||introduce up to 100% packet loss of egress traffic from the system|
||drop all DNS requests made from the system|
||alter system time|
||spin up and kill processes on the system|
Run Gremlin Syscheck in Docker¶
Gremlin provides a special Docker tag for running
syscheck tests in Docker:
docker run -it \ --cap-add=NET_ADMIN \ --cap-add=NET_RAW \ --cap-add=SYS_TIME \ --cap-add=KILL \ gremlin/gremlin:syscheck
How to Configure Gremlin¶
You’ve installed Gremlin and validated that Gremlin can run on your system by running the
gremlin syscheck command. The next step will be to configure your Gremlin clients using our Gremlin Client Configuration guide.
Gremlin’s Developer Guide is a great resource and reference for using Gremlin to do Chaos Engineering. You can also explore the Gremlin Blog for more information on how to use Chaos Engineering with your application infrastructure.