🛰️ Kafnus

Kafnus is a smart city data persistence system, designed as a modern replacement for Cygnus, built on top of Kafka.

It processes NGSI notifications from the Context Broker (CB) and stores them in PostGIS and MongoDB, leveraging technologies like Faust, Kafka Connect, and custom connectors.

---

📦 Main Components

• 🧭 Mosquitto + MQTT Kafka Connector
  Temporary entry point for CB notifications (to be removed once CB supports native Kafka output).

• ⚙️ Kafnus NGSI
  Using Faust library, transforms raw notifications into structured events. Each data flow (historic, lastdata, mutable, etc.) is handled by an independent agent.

• 🔄 Kafnus Connect
  Custom image of Kafka Connect with plugins integrated. Persists processed messages to:

  • PostGIS, via a modified JDBC connector and custom SMT.
  • MongoDB, via the official MongoDB connector.

• 📊 Monitoring
  Integrated with Prometheus and Grafana to expose metrics from Kafka, Kafnus Connect, and Kafnus NGSI.

• 🧪 End-to-End Testing
  Functional tests implemented in Python using Pytest and Testcontainers.

---

🚀 Purpose

• Replace Cygnus in FIWARE smart city stacks.
• Provide robust, extensible ingestion with real-time stream processing.
• Offer a Kafka-based architecture ready for future scalability.

---

📁 Documentation

Complete documentation is available in the doc/ directory:

• 00_overview.md – Project overview
• 01_installation.md – How to install & build
• 02_architecture.md – System architecture
• 03_operational_guide.md – Operational guide
• 04_docker.md – Docker details
• 05_kafnus_ngsi.md – Kafnus NGSI stream processor
• 06_kafnus_connect.md – Kafnus Connect and sinks details
• 07_monitoring.md – Metrics & observability
• 08_testing.md – Test structure

---

🛠️ Requirements

• Docker + docker compose
• Java 11+
• Python 3.11+ (for tests)
• Maven