Node.js vs vert.x

WORK IN PROGRESS

Simple and naive Node.js vs vert.x benchmark. I've written it to check if there's any significant performance difference between Node.js (which is getting more popular than even before and undoubtedly more stable and mature) and new kid on the block -- vert.x (which shares the asynchronous and message-oriented philosophy with Node.js). I'm playing with Node since v0.2, this benchmark took me 2 hours of work mostly due to trial-and-error approach to vert.x, so the code may not be the most optimized and sophisticated.

Node.js server uses Restify framework and Mongoose ORM. Vert.x is a complete stack so there's no need for external modules. The database is MongoDB.

Testing environment

• Processor: Intel(R) Core(TM) i5-2410M CPU @ 2.30GHz
• Memory: 4GB DDR3 SDRAM @ 1333MHz (1725MB used)
• Hard drive: KINGSTON SSD SVP100S
• Operating System: Ubuntu 11.04

Yes. It's a laptop.

• Node.js v0.6.6

• vert.x v1.0.1.final

• ulimit = unlimited

Default V8/JVM and Node.js/vert.x settings unless stated otherwise.

Fairness considerations

Restify adds it's own headers such as X-Response-Time, Content-MD5, X-Request-Id, Date and so on. To be fair I removed unnecessary calculations from Restify and added some of them to vert.x code so that both servers return the same set of headers and perform (more or less) the same calculations. I got rid of MD5 response hash because it took about 40ms to calculate it using JavaScript function (Restify uses OpenSSL for that).

As far as I know, vert.x uses one 'vertex' instance by default (I'm certain about that in Node's case). Multiple instances were tested for vert.x, I will add node-cluster benchmarks in the future.

Benchmark results

Benchmarks were performed using Siege. All test data is generated on the fly.

Posting and selecting from database

Request adds a random JSON document to database:

author: String(16)
date: Date
content: String(160)

and gets 100 already stored documents in response (no sorting).

Benchmarking command: siege -c100 -d1 -r100 http://localhost:1337/post

	[1.1] Node.js (default)	[1.2] vert.x (default)
Transactions	10000 hits	10000 hits
Availability	100.00 %	100.00 %
Elapsed time	203.10 secs	64.04 secs
Data transferred	257.50 MB	283.23 MB
Response time	1.45 secs	0.04 secs
Transaction rate	49.24 trans/sec	156.15 trans/sec
Throughput	1.27 MB/sec	4.42 MB/sec
Concurrency	71.27	5.66
Successful transactions	10000	10000
Failed transactions	0	0
Longest transaction	2.32	3.03
Shortest transaction	0.01	0.00
Memory consumed	35 MB	50 MB -> 1 GB

• Couldn't make multiple vert.x instances work:

  Jun 18, 2012 5:05:45 PM org.vertx.java.core.logging.impl.JULLogDelegate error SEVERE: Exception in JavaScript verticle: Wrapped java.lang.IllegalStateException: Response has already been written at core/http.js:142 (anonymous) at file:/home/adamus/nodejs-vs-vertx/vert.js:69 (anonymous) at core/event_bus.js:56 (anonymous)

Returning 'hello world' response

Short JSON was returned in response to every request, no calculations performed.

Benchmarking command: siege -c100 -b -r1000 http://localhost:1337/hello

	[2.1] Node.js (default)	[2.2] vert.x (default)	[2.3] vert.x (4 instances)
Transactions	39515 hits	100000 hits	100000 hits
Availability	97.37 %	100.00 %	100.00 %
Elapsed time	39.72 secs	22.41 secs	13.84 secs
Data transferred	1.21 MB	1.81 MB	1.81 MB
Response time	0.09 secs	0.02 secs	0.01 secs
Transaction rate	994.84 trans/sec	4462.29 trans/sec	7225.43 trans/sec
Throughput	0.03 MB/sec	0.08 MB/sec	0.13 MB/sec
Concurrency	93.96	87.33	38.29
Successful transactions	39515	100000	100000
Failed transactions	1066	0	0
Longest transaction	0.64	3.01	3.01
Shortest transaction	0.00	0.00	0.00
Memory consumed	35 MB	60 MB	83 MB

• 2.1 was aborted by Siege due to 'excessive socket failure'.

String concatenation

Random 10000 characters long strings were returned to this requests. No buffers were used, just plain old strings and concatenation (one character at a time).

Benchmarking command: siege -c100 -b -r100 http://localhost:1337/concat

	[3.1] Node.js (default)	[3.2] vert.x (default)	[3.3] vert.x (4 instances)
Transactions	10000 hits	10000 hits	10000 hits
Availability	100.00 %	100.00 %	100.00 %
Elapsed time	20.17 secs	61.27 secs	35.83 secs
Data transferred	95.49 MB	95.49 MB	95.49 MB
Response time	0.20 secs	0.61 secs	0.34 secs
Transaction rate	495.79 trans/sec	163.21 trans/sec	279.10 trans/sec
Throughput	4.73 MB/sec	1.56 MB/sec	2.67 MB/sec
Concurrency	99.47	99.53	94.48
Successful transactions	10000	10000	10000
Failed transactions	0	0	0
Longest transaction	0.48	1.34	2.24
Shortest transaction	0.00	0.01	0.00
Memory consumed	30 MB	320 MB	360 MB

Fibonacci

The most famous benchmark out there. Calculating fib(30) recursively.

Benchmarking command: siege -c100 -b -r10 http://localhost:1337/fibonacci

	[4.1] Node.js (default)	[4.2] vert.x (default)	[4.3] vert.x (4 instances)
Transactions	1000 hits	1000 hits	1000 hits
Availability	100.00 %	100.00 %	100.00 %
Elapsed time	26.88 secs	101.46 secs	64.01 secs
Data transferred	0.04 MB	0.02 MB	0.02 MB
Response time	2.54 secs	9.63 secs	5.72 secs
Transaction rate	37.20 trans/sec	9.86 trans/sec	15.62 trans/sec
Throughput	0.00 MB/sec	0.00 MB/sec	0.00 MB/sec
Concurrency	94.39	94.96	89.38
Successful transactions	1000	1000	1000
Failed transactions	0	0	0
Longest transaction	5.42	18.51	13.78
Shortest transaction	0.02	0.11	0.20
Memory consumed	15 MB -> 30 MB	300 MB	335 MB

Webpage render

[ETA: end of June]

Conclusions

• Dummy database constrained benchmarks favors 3x vert.x over Node.js.
• As for 'requests per second' it's 3.5x faster (7x using both processor cores, but this is unfair).
• Vert.x sent more data down the tubes, I suppose it's caused by different 'Date' header formatting (it uses Rhino's NativeDate instead of JavaScript's Date) . It also prints slightly longer 'X-Response-Timeout' (e.g. 13.0 instead of 13). I might look into it in the future. I tried to even out basic benchmarks by adding extra data to headers. The problem is, vert.x's database persistor uses different/longer '_id' format (looks like some kind of UUID) than Mongoose, the 'Date' string returned is longer as well.
• Native string manipulation (simple concat) in V8 seems faster than JVM's.
• It looks like recursive Fibonacci's fans should go with Node.js.

Fun facts

• Naming your vert.x application file 'vertx.js' is a bad idea.
• Vert.x crashed with 'java.lang.OutOfMemoryError: GC overhead limit exceeded' exception because of the MongoDB socket timeouts.