Leader Election

This project implements Leader Election to ensure only one worker is the leader and handles tasks from the message queue. The leader election is managed using etcd, and tasks are processed by workers that listen to messages from Kafka.

Features

• Leader Election with etcd: Guarantees that only one worker is elected as the leader, and it is responsible for managing task scheduling.
• Message Queue with Kafka: The system uses Kafka as a message queue, where tasks are sent to the queue, and workers listen and process them.
• Fault Tolerance: If the current leader fails, etcd automatically triggers a new leader election to ensure the system remains available and tasks are handled without interruption.

Setup

Prerequisites

Before you can run the project, ensure that you have the following tools installed:

• Docker 27.5.0
• Golang go1.23.4 linux/amd64
• Tmux 3.4

Steps to start on local

1. Run the following command to bring up the necessary services using Docker Compose:

   docker compose up -d

2. Open a new terminal window and run the provided script to start the workers and scheduler: ./start-all

Others

1. Kafka UI: localhost:7777

2. ETCD Keeper UI: localhost:8080

   Please use the etcd:2379 connection string when connecting to the etcd cluster.