How to Setup and Use the Gremlin Datadog Integration


Datadog is a monitoring service that provides monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. If you use Datadog, it’s useful to enable the Gremlin Datadog integration for your Engineering team to see the Gremlins in action while they are running attacks. This tutorial will explain how to setup and use the Gremlin Datadog integration.


Before you begin this tutorial, you’ll need the following:

Step 1 - Setting up the Gremlin Datadog Integration

In this step, you’ll setup the Gremlin Datadog integration.

In order to activate this integration, you will need to pass your Datadog API key to Gremlin. First retrieve your API key from Datadog, you can find it in Datadog Settings.

API Key Datadog

Next add the Datadog API key to Gremlin Integrations by clicking the Add button on the row for Datadog.

Add API Key Datadog Gremlin

You will be prompted for your Datadog API key. Paste your Datadog API key in the box and click save. The Gremlin Datadog integration will now be initialized.

API Key Datadog Gremlin

Step 2 - Validating Installation

After enabling the Gremlin Slackbot you will now be able to see Gremlin attack events in your Event Stream.

Gremlin Events Datadog

Step 3 - Receive Datadog events for Gremlin attacks

Your Gremlin Datadog integration is now ready for you to use. You will now be able to overlay attacks on top of your dashboards to pinpoint exactly how and when Gremlin is impacting your metrics.

You can also click from within your Datadog Event Stream to:

  • Rerun Gremlin attacks
  • Show logs for Gremlin attacks
  • Halt Gremlin attacks

Example: Create a Gremlin Process Killer Attack

Next we are going to create a Gremlin Process Killer Attack that will be visible in Datadog using the Gremlin Datadog integration. Though this example uses an infrastructure-level attack to tigger Datadog events, this integration works the same way with ALFI Experiments.

To create a Process Killer Attack first select the target you will run it on, this can be a host or a container. Next you will choose a Gremlin, for the Process Killer Attack select “State” and then “Process Killer”.

The Process Killer Attack will kill a process of your choice. Enter the process name or PID of your choice to continue.

Finally, click Unleash Gremlin and your Process Killer Attack will begin to run.

View and interact with the attack using the Gremlin Datadog integration

Open your Datadog Event Stream, you will be able to view the Gremlin Process Killer Attack you started to run. From the Datadog Event Stream you can now click to show logs of the attack, halt the attack or rerun the attack.

When the attack finishes successfully you will see an Ended Process Killer attack event in the Datadog Event Stream.

Gremlin Events Datadog Running


You’ve setup the Gremlin Datadog integration and validated that it works by creating a Process Killer Attack. You now possess tools that make it possible for your entire engineering team to view the Gremlin attacks that are running on your infrastructure.

Gremlin’s Developer Guide is a great resource and reference for using Gremlin to do Chaos Engineering. Another useful Gremlin integration you can setup for your engineering team is the Gremlin Slack integration. You can view the guide to setup the Gremlin Slack integration here: How to Setup and Use the Gremlin Slackbot.