A high performance, brokered messaging queue
It is a high performance, brokered messaging queue which supports transient (1M msg/sec with microseconds latency) and durable (~300K msg/sec with milliseconds latency) queues. Durable queues are similar to Kafka where data are written to the file and consumers consume from the file.
It is mostly header only project with main.cpp as an example for broker, producer and consumer.
NOTE: This is an initial version and may not be ready for production use.
##Protocol:
(Admin userid and password must be passed to create a topic. Also we need to define userid/password per topic which consumer/produder need to pass for authentication)
Send a request to the broker
{
"admin_password": "T0p$3cr31",
"admin_user_id": "myq_admin",
"broker_type": "queue",
"cmd": "create_topic",
"password": "T0p$3cr31",
"topic": "test",
"user_id": "test_admin"
}
Response:
{
"cmd": "create_topic",
"description": "topic created successfully",
"status": "ok"
}
###Join Topic (Consumer): (Need to pass userid/password for topic 'test')
Request:
{
"cmd": "join",
"connection_type": "zmq",
"password": "T0p$3cr31",
"topic": "test",
"type": "pull",
"user_id": "test_admin"
}
Response:
{
"bind_uri": "tcp://127.0.0.1:5002",
"cmd": "join",
"status": "ok",
"topic": "test"
}
(Need to pass userid/password for topic 'test')
Request:
{
"cmd": "join",
"connection_type": "zmq",
"password": "T0p$3cr31",
"topic": "test",
"type": "pub",
"user_id": "test_admin"
}
Response:
{
"bind_uri": "tcp://127.0.0.1:5003",
"cmd": "join",
"status": "ok",
"topic": "test"
}
Request:
{
"cmd": "stats",
"password": "T0p$3cr31",
"topic": "test",
"user_id": "test_admin"
}
Response:
{
"cmd": "stats",
"messages_received": 9499570,
"messages_sent": 9491554,
"publishers_count": 1,
"queue_size": 8016,
"status": "ok",
"subscribers_count": 1,
"total_bytes_read": 0,
"total_bytes_written": 0
}
#Performance:
Laptop hardware:
MacBook Pro (Retina, 15-inch, Late 2013)
Processor 2.3 GHz Intel Core i7
Memory 16 GB 1600 MHz DDR3
Broker Type: Transient ##100 bytes, 10M messages
Producer:
./dist/Release/GNU-MacOSX/myq producer 10000000 100 event
Total Messages:10000000, Time Taken:8.46577 seconds.
Start Time: 1427658489112, End Time:1427658497577
1181227 messages per seconds.
1000000000 bytes sent
112.6507 MB per second
Consumer:
./dist/Release/GNU-MacOSX/myq consumer queue zmq pull event
Total Messages:10000001, Time Taken:8.47781 seconds.
Start Time: 1427658489122, End Time:1427658497600
1179550 messages per seconds.
1000000004 bytes received
112.4907 MB per second.
##256 bytes 10M Messages
Producer:
./dist/Release/GNU-MacOSX/myq producer 10000000 256 event
Total Messages:10000000, Time Taken:9.2752 seconds.
Start Time: 1427658738559, End Time:1427658747834
1078143 messages per seconds.
2560000000 bytes sent
263.2186 MB per second.
Consumer:
./dist/Release/GNU-MacOSX/myq consumer queue zmq pull event
Total Messages:10000001, Time Taken:9.30292 seconds.
Start Time: 1427658738562, End Time:1427658747865
1074931 messages per seconds.
2560000004 bytes received
262.4345 MB per second.
Producer:
./dist/Release/GNU-MacOSX/myq producer 10000000 512 event
Total Messages:10000000, Time Taken:10.5182 seconds.
Start Time: 1427658940094, End Time:1427658950612
950734 messages per seconds.
5120000000 bytes sent
464.2258 MB per second.
Consumer:
./dist/Release/GNU-MacOSX/myq consumer queue zmq pull event
Total Messages:10000001, Time Taken:10.5296 seconds.
Start Time: 1427658940097, End Time:1427658950627
949706 messages per seconds.
5120000004 bytes received
463.7239 MB per second.
Producer:
./dist/Release/GNU-MacOSX/myq producer 10000000 1024 event
Total Messages:10000000, Time Taken:19.8285 seconds.
Start Time: 1427659063592, End Time:1427659083420
504324 messages per seconds.
10240000000 bytes sent
492.5049 MB per second.
Consumer:
./dist/Release/GNU-MacOSX/myq consumer queue zmq pull event
Total Messages:10000001, Time Taken:19.8222 seconds.
Start Time: 1427659063603, End Time:1427659083425
504485 messages per seconds.
10240000004 bytes received
492.6617 MB per second.
#Performance: (Durable broker: file) ##100 bytes, 10M messages
Producer:
./dist/Release/GNU-MacOSX/myq producer 10000000 100 event
Total Messages:10000000, Time Taken:25.0786 seconds.
Start Time: 1427659575158, End Time:1427659600236
398746 messages per seconds.
1000000000 bytes sent
38.0275 MB per second.
Consumer:
./dist/Release/GNU-MacOSX/myq consumer file socket pull event
Total Messages:10000001, Time Taken:25.0945 seconds.
Start Time: 1427659575170, End Time:1427659600264
398493 messages per seconds.
1000000004 bytes received
38.0033 MB per second.
##256 bytes 10M Messages
Producer:
./dist/Release/GNU-MacOSX/myq producer 10000000 256 event
Total Messages:10000000, Time Taken:28.3802 seconds.
Start Time: 1427659339399, End Time:1427659367779
352358 messages per seconds.
2560000000 bytes sent
86.0250 MB per second.
Consumer:
./dist/Release/GNU-MacOSX/myq consumer file socket pull event
Total Messages:10000001, Time Taken:28.3939 seconds.
Start Time: 1427659339410, End Time:1427659367804
352188 messages per seconds.
2560000004 bytes received
85.9834 MB per second.
Producer:
./dist/Release/GNU-MacOSX/myq producer 10000000 512 event
Total Messages:10000000, Time Taken:31.0832 seconds.
Start Time: 1427890737326, End Time:1427890768409
321716 messages per seconds.
5120000000 bytes sent
157.0884 MB per second.
Consumer:
./dist/Release/GNU-MacOSX/myq consumer file socket pull event
Total Messages:10000001, Time Taken:31.0935 seconds.
Start Time: 1427890737329, End Time:1427890768423
321610 messages per seconds.
5120000004 bytes received
157.0363 MB per second.
Producer:
./dist/Release/GNU-MacOSX/myq producer 10000000 1024 event
Total Messages:10000000, Time Taken:37.4027 seconds.
Start Time: 1427890878446, End Time:1427890915848
267360 messages per seconds.
10240000000 bytes sent
261.0942 MB per second.
Consumer:
./dist/Release/GNU-MacOSX/myq consumer file socket pull event
Total Messages:10000001, Time Taken:37.4166 seconds.
Start Time: 1427890878455, End Time:1427890915871
267260 messages per seconds.
10240000004 bytes received
260.9970 MB per second.
Start Broker: (broker type: queue, logging level: event)
./dist/Release/GNU-MacOSX/myq queue event
Start Consumer: (client: consumer, broker type: queue, client socket: zmq, logging level: event)
./dist/Release/GNU-MacOSX/myq consumer queue zmq pull event
Start Producer: (client: producer, number of messages 10M, payload size: 100 bytes, logging level: event)
./dist/Release/GNU-MacOSX/myq producer 10000000 100 event
##License :
Dependecies:
ZeroMQ LGPL:
