StreamNest

🚀 A minimal, Kafka-inspired clustered message streaming system in Go. Launch a broker cluster with any number of brokers, create topics with any number of partitions, and interactively produce and consume messages—partition ownership is distributed across brokers just like in real Kafka!

---

✨ Features

• Multi-Broker Clustering: Start N brokers at once, each automatically aware of peers.
• Any Partition Count: Create topics with any number of partitions, independently of broker count.
• Round-Robin Assignment: Partitions are spread evenly across brokers (round-robin).
• HTTP APIs: Create topics, list topics, produce to and consume from any partition over HTTP.
• CLI Producer & Consumer: Simple interactive clients for message publishing and consumption.
• Persistent Logs: Each partition's data is stored on disk and survives restarts.

---

🚀 Quickstart

Prerequisites

• Go 1.19 or higher

Clone & Build

git clone https://github.com/rohankumardubey/StreamNest.git
cd StreamNest
go mod tidy
go build -o stream-nest-cluster ./cmd/stream-nest-cluster

1. Start Three Brokers

Run this in one terminal (spawns all brokers as subprocesses):

./stream-nest-cluster broker --count=3

Each will print:

Starting broker 1 on port 8080 with peers: localhost:8081,localhost:8082
Broker 1 running on :8080
Starting broker 2 on port 8081 with peers: localhost:8080,localhost:8082
Broker 2 running on :8081
Starting broker 3 on port 8082 with peers: localhost:8080,localhost:8081
Broker 3 running on :8082

2. Create a Topic (with Any Partition Count!)

In a second terminal, run:

curl -X POST -H "Content-Type: application/json" \
  -d '{"topic":"demo","partitions":7}' \
  http://localhost:8080/create-topic

Response:

{"status":"created"}

3. List Topics

curl http://localhost:8080/list-topics

Response:

{"topics":["test"]}

4. View Cluster Metadata

curl http://localhost:8080/metadata

Response Example:

{
  "topic_partitions": {
    "demo": {
      "partitions": [
        {"partition":0,"broker":"localhost:8080"},
        {"partition":1,"broker":"localhost:8081"},
        {"partition":2,"broker":"localhost:8082"},
        {"partition":3,"broker":"localhost:8080"},
        {"partition":4,"broker":"localhost:8081"},
        {"partition":5,"broker":"localhost:8082"},
        {"partition":6,"broker":"localhost:8080"}
      ]
    }
  }
}

5. Register Schema

Register a schema for a topic:

curl -X POST -H "Content-Type: application/json" \
  -d '{
    "topic":"demo",
    "schema":{
      "type":"object",
      "properties":{
        "name":{"type":"string"},
        "age":{"type":"number"}
      },
      "required":["name","age"]
    }
  }' \
  http://localhost:8080/register-schema

Note: If you are registering a schema in Schema Registry, you must send the messages in the schema defined in schema registry format; otherwise the messages will be ignored by the consumer due to strict Schema Registry validations.If you are unaware of the schema just ignore this step so no validations happen and broker & consumer accepts all messages.

6. Produce Messages

Producing a valid message with the schema from /produce endpoint

(if you plan to specify the partition value manually without key then it will send the message to specified partition)

curl -X POST -H "Content-Type: application/json" \
  -d '{
    "topic":"demo",
    "partition":0,
    "message":"{\"name\":\"Alice\",\"age\":30}"
  }' \
  http://localhost:8080/produce

(if no partition value is specified and key is given then it will calculate partition based on hash function)

curl -X POST -H "Content-Type: application/json" \
  -d '{
    "topic":"demo",
    "key": "id1",
    "message":"{\"name\":\"Alice\",\"age\":30}"
  }' \
  http://localhost:8080/produce

(if no partition value and no key is given then it will follow the round robin strategy)

curl -X POST -H "Content-Type: application/json" \
  -d '{
    "topic":"demo",
    "message":"{\"name\":\"Alice\",\"age\":30}"
  }' \
  http://localhost:8080/produce

Producing a valid message without schema from CLI

./stream-nest-cluster producer --meta=localhost:8080

Response

Enter topic: demo
Partitions:
  0 on localhost:8080
  1 on localhost:8081
  2 on localhost:8082
  3 on localhost:8080
  4 on localhost:8081
  5 on localhost:8082
  6 on localhost:8080
Partition?> 4
Type messages (or 'exit'):
> Hello
offset: 0
> Another message
offset: 1
> exit

7. Consume Messages

./stream-nest-cluster consumer --meta=localhost:8080

Response

Enter topic: demo
Partitions:
  0 on localhost:8080
  1 on localhost:8081
  2 on localhost:8082
  3 on localhost:8080
  4 on localhost:8081
  5 on localhost:8082
  6 on localhost:8080
Partition?> 4
[Offset 0] Hello
[Offset 1] Another message

8. Monitor Metrics

Each broker exposes Prometheus metrics on /metrics. Metrics are registered automatically when a broker starts.

Query a broker's metrics with curl:

curl http://localhost:8080/metrics

You should see counters such as streamnest_messages_produced_total and streamnest_messages_consumed_total that increment as you produce and consume messages.

---

📁 Project Layout

StreamNest/
├── cmd/
│   └── stream-nest-cluster/
│       └── main.go
├── internal/
│   ├── broker/
│   │   ├── types.go
│   │   ├── storage.go
│   │   └── broker.go
│   └── client/
│       └── client.go
├── data/          # runtime logs: <topic>_<partition>.log
├── go.mod
└── README.md

---

🏗️ Roadmap

• Replication & Failover – mirror partitions across brokers
• Consumer Groups – manage offsets per group
• Docker Compose – launch cluster with a single command
• Metrics & Monitoring – Prometheus endpoints

---

🤝 Contributing

1. Fork the repo
2. Create a branch:

   git checkout -b my-feature

3. Commit your changes:

   git commit -am 'Add cool feature'

4. Push to your branch:

   git push origin my-feature

5. Open a Pull Request

---

📄 License

This project is licensed under the MIT License. See LICENSE for details.