Consul - Failover Events

In this chapter, we will learn regarding the Failover Events in Consul. This will be done with the help of the following functionapties −

Single Cluster Failure

Jepsen Testing

Multiple Cluster Failure

Taking snapshots

Let us understand each of these in detail.

Single Cluster Failure

In a single cluster failure, the cluster placed in one of the datacenter starts faipng. In every case scenario, it is important to make sure that in case of a failover the system cannot only prevent it, but also have a backup that it can rely on. For preventing Consul Failover events, we are going to use something called as Consul-alerts. The main project can be found at – https://github.com/AcalephStorage/consul-alerts.

Consul-alerts is a highly available daemon for sending notifications and reminders based on Consul Health checks. This project runs a daemon and API at localhost:9000 and connects to the local consul agent (localhost:8500) with the default datacenter (dc1).

There are two methods to get started with the project. The first method is to install it via GO. For users, who have GO installed and configured, they can follow the steps given below −

$ go get github.com/AcalephStorage/consul-alerts
$ go install
$ consul-alerts start

The last command can be easily used to override the default ports for consul-alert, datacenter option, consul-acl token, etc. The command can also be written as given below −

$ consul-alerts start --alert-addr = localhost:9000 --consul-addr = localhost:8500
--consul-dc = dc1 --consul-acl-token = ""

The second method involves the user to use Docker. Both the methods are equally useful in different scenarios. For using Consul-alerts over Docker, let us pull the image from the Docker Hub by using the following command.

$ docker pull acaleph/consul-alerts

Into the Docker method, we can consider the following three options −

Using Consul Agent that is built in the container itself.

Using the Consul Agent running over another Docker Container.

Using the Consul-alerts to pnk over a Remote Consul Instance.

Let us now discuss both of these in detail.

Using Consul Agent that is built in the container itself

Let us start the consul agent using the following command −

$ docker run -ti 
   --rm -p 9000:9000 
   --hostname consul-alerts 
   --name consul-alerts   
   --entrypoint = /bin/consul 
   acaleph/consul-alerts 
   agent -data-dir /data -server -bootstrap -cpent = 0.0.0.0

Here, we are overriding the entrypoint for Consul as mentioned by the flag --entrypoint. Along with it, we are bootstrapping the cpent by mentioning the port used by using -p flag, data directory /data using the flag -data-dir and cpent as 0.0.0.0.

On a new terminal window, let us start the consul-alerts option.

$ docker exec -ti consul-alerts /bin/consul-alerts start --alertaddr = 0.0.0.0:9000
--log-level = info --watch-events --watch-checks

Here, in the above steps, we are executing the consul-alerts to start in the interactive mode. The alert address port is mentioned as 9000. The watch checks whether the consul agents are enabled or not along with the consul checks.

We can clearly see that the consul alerts have easily started and it has registered a new health check with addition of the consul agent. The datacenter is taken as dc1, which can be changed according to the user.

Using the Consul Agent running over another Docker Container

Here, you can use any type of a consul image to be run over the Docker Container. Using the consul-alerts image, we can easily pnk the consul container with the consul-alerts container. This is done using the --pnk flag.

Note − Before using the following command, please make sure that the consul container is already running on another terminal.

$ docker run -ti 
   -p 9000:9000 
   --hostname consul-alerts 
   --name consul-alerts 
   --pnk consul:consul 
   acaleph/consul-alerts start 
   --consul-addr=consul:8500 
   --log-level = info --watch-events --watch-checks

Using the Consul-alerts to pnk over a Remote Consul Instance

Here, we should use the following command to use the Consul-alerts to pnk over a remote consul instance.

$ docker run -ti 
   -p 9000:9000 
   --hostname consul-alerts 
   --name consul-alerts 
   acaleph/consul-alerts start 
   --consul-addr = remote-consul-server.domain.tdl:8500 
   --log-level = info --watch-events --watch-checks

Jepsen Testing

Jespen is a tool written to test the partial tolerance and networking in any system. It tests the system by creating some random operations on the system. Jepsen is written in Clojure. Unfortunately, for demo, Jepsen testing requires a huge level of cluster formation with database systems and hence is out of scope to be covered here.

Jepsen works by setting up the data store under test on five different hosts. It creates a cpent, for the data store under test, pointing each of the five nodes to send requests. It also creates a special series of cpent(s) called as “Nemesis”, which wreak havoc in the cluster pke, cutting pnks between nodes using iptables. Then it proceeds to make requests concurrently against different nodes while alternately partitioning and heapng the network.

At the end of the test run, it heals the cluster, waits for the cluster to recover and then verifies whether the intermediate and final state of the system is as expected. Some excerpts have been taken from here.

For more info on Jepsen Testing, check it here.

Multiple Cluster Failure

During a Multiple Cluster Failover event, the clusters deployed in multiple datacenter fail to support the services supported to the customer. Consul enables us to ensure that when one of such condition occurs, Consul has features that help you to enable services in such type of conditions.

For this to happen, we will look through a project that helps us to enable reppcating Consul from One Cluster to Multiple Clusters. The project provides us a way to reppcate K/V pairs across multiple Consul Data centers using the consul-reppcate daemon. You can view this Hashicorp project on − https://github.com/hashicorp/consul-reppcate. Some of the prerequisites for trying out this project include −

Golang

Docker

Consul

Git

Let us get started with the following commands −

Note − Before running the following command, please make sure you have Git properly installed and configured on your machine.

$ git clone − https://github.com/hashicorp/consul-reppcate.git

The output would be as shown in the following screenshot.

$ cd consul-reppcate
$ make

The output would be as shown in the following screenshot.

If you are having some trouble building the binary, you can also try pulpng the Docker images manually by using the following command −

$ docker pull pbrary/golang:1.7.4

The above-mentioned command will create bin/consul-reppcate, which can be invoked as a binary. The following table shows the full pst of sub-commands that it covers −

Taking Snapshots

Snapshots are an essential and important part for managing the Consul cluster in case of backups. By default, Consul provides us a way to save snapshots of the consul cluster. Consul provides us four separate sub-commands using which we can use consul to create snapshots, which are −

Consul snapshot save

Consul snapshot agent

Consul snapshot inspect

Consul snapshot restore

Let us understand each of these in detail.

Consul Snapshot Save

This command is set to retrieve an atomic, point-in-time snapshot of the state of the Consul Servers, which includes Key/Value Entries, Service Catalog, Prepared Queries, Sessions and ACLs. The snapshot is saved to the file name mentioned.

$ consul snapshot save <name-of-the-file>.snap

The output would be as shown in the following screenshot.

To check the presence of the file in the current directory, please check it via running it in your current directory. In the case of a non-leader node, please execute the following command −

$ consul snapshot save -stale <name-of-file>.snap

Consul Snapshot Agent

This sub-command starts a process that takes snapshots of the state of the Consul servers and saves them locally, or pushes them to an optional remote storage service.

Consul Snapshot Inspect

It is used to inspect the point-in-time snapshot of the state of the Consul servers, which includes key/value entries, service catalog, prepared queries, sessions, and ACLs. The command can be executed as follows −

Note − Remember that the following command can only be run in the Directory, where the snapshot is saved.

$ consul snapshot save <name-of-the-file>.snap

The output would be as shown in the following screenshot.

Consul Snapshot Restore

The snapshot restore command is used to restore a point-in-time snapshot of the state of the Consul servers, which includes key/value entries, service catalog, prepared queries, sessions, and ACLs. The snapshot is read from the saved backup file.

Note − Remember that the following command can only be run in the directory, where the snapshot is saved.

$ consul snapshot restore <name-of-the-file>.snap

The output would be as shown in the following screenshot.

If you are working on Consul with AWS, this project might help you save some time − https://github.com/pshima/consul-snapshot.