SurrealDB Performance Benchmark

[theseusyang]

Anybody can show a surrealdb's benchmark, to describe the performance?

[tobiemh]

Hi @theseusyang thanks for starting the discussion!

We'll definitely be performing benchmarking for SurrealDB in due course.

Currently, we are focussed on functionality and stability, and want to make the following changes for performance: #26, #24, and #22.

Once those are implemented, performance should be far better than it is now!

[bencz]

Benchmarking is interesting...
Test, in addition to speed, the limits... for example, maximum number of simultaneous connections and etc

[MrBns]

@tobiemh peoples are making new comment everyday but its really disappointing that you guys are not interested in this discussion. no reply from your team. no hope .

[JABirchall]

I'd also be interested in seeing some benchmarks.

Not a describe command, but for example a benchmark of the database loaded with data. Often times databases can start to lose performance with millions of records.

[Developeranees]

I'd also be interested in seeing some benchmarks.

[AlexFrid]

The benchmarks are currently being worked on, we've used some initial results to improve performance and will release benchmarks publically once we have improved on a few more things.

[FI-Mihej]

Hello!

I'm currently pondering whether I have the capability to utilize SurrealDB effectively. Is its performance significantly slow, or is it somewhat tolerable at this present moment, without considering future improvements?

[AlexFrid]

Hey @FI-Mihej,
As for the current state v1.1.0-beta very large aggregation queries can be annoyingly slow, as well as bulk inserts of millions of records might take much longer than expected if each record is also large or is a graph relation. Inserts of small records however perform well.

Apart from that, you'll find that most things perform similarly to comparable DBs with other things being tolerable.

Therefore I think you will currently be able to utilize SurrealDB effectively, as long as you can accept a few rough edges here and there.

[AlexFrid]

Now talking about future improvements, we are working towards ironing out these rough edges in time for our cloud offering.

[carueda]

I continue to be super interested in incorporating SurrealDB in some of my professional projects. However, I still interpret your answer to the Is SurrealDB ready for production use? FAQ basically as, "No, it's not." I'd be more that happy to be corrected, or simply learn about a more concrete timeline for significant performance improvements to land in a version (even if beta). Please keep up the great work.

[AlexFrid]

Hey @carueda,
It's basically ready for people who can accept the risks of being an early adopter 😅
You can see an example of an early adopter using it in production here: https://www.youtube.com/watch?v=kHnn1XTnZCk

What we are working on currently is making it ready for the more risk-averse early majority with 2.0, which will include some necessary breaking changes, but all the improvements that don't need a breaking change are being released in a 1.x release each month now.

[nottisa]

Replying to this as it seems it has not been addressed in quite some time. It does not appear that benchmarks are currently publicly available, but some pre-made benchmarking tools are available here: https://github.com/surrealdb/surrealdb/tree/main/lib/benches

[nickchomey]

They have stated various times that benchmarks will be made available when v2 is fully released. The first alpha was just released, so if folks can contribute with testing and feedback, the full v2 and benchmarks will come sooner.

[nottisa]

I only said what I said because the current benchmarking tools were not yet mentioned. I understand what they've said previously and am not discounting that.

[mahbubabbas]

I have used SurrealDB in 2 applications recently. Honestly speaking I feel Postgres is outperforming Surreal. I am not so happy with Surreal.

[AlexFrid]

Thanks for the feedback, could you say a bit more about what kind of application you were using it for and what postgres extensions you were using (if any)?

[MrBns]

Now SurrealDB 2.0 is out. is there any performance benchmark yet ?

[MrBns]

@AlexFrid thanks 😊

[MrBns]

is there any benchmark out yet ?

[JABirchall]

From a few user made benchmarks ive seen, its not great. Once you load up the database it degrades in performance.

[MrBns]

@JABirchall dude, can you share anyone please ?

[ghost]

Would also love to see some data :)

[nickchomey]

I just did a relatively simple query on a single table with about 40,000 rows imported from Mysql, returning 1000 results. It took 280ms while it takes 7ms seconds in Mysql (with query cache turned off). That's almost 40x slower.

However, if I use a record ID to select a specific record, it takes 0.15ms whereas mysql takes 3ms seconds to select a specific record.

If I remove the indexes from the mysql table columns, it makes no difference for this query. If I add indexes to those columns in SurrealDB, it actually gets SLOWER - 400ms seconds for 1000 records, though it is the same 0.15ms for 1 record.

Also, interestingly, I got similar numbers with both Memory and RocksDB storage. So, the bottleneck here seems to be SurrealDB itself.

One factor could be that im using Surrealist over websockets whereas I use the mysql cli client. Perhaps network latency is a factor when transferring data.

So, I have to conclude based off these crude tests that Surreal is extremely slow when doing large queries returning many records in a table. That's somewhat what I expected given how there's such an emphasis on having, linking and using specific record IDs. I think that probably means we should architect our applications in such a way that we have specific record IDs in hand when we run a query - and then do "joins" via embedded records or graph relations, rather than always starting from scratch with a big full table query. We may very well see better performance for complicated joins, graph traversals etc, as well as if we have sort of pregenerated "materialized views"

I have to figure the team's delays in releasing benchmarks are because they've found similarly disappointing performance and are trying to improve things.

HOWEVER, I have to assume that Surreal will keep getting faster as they keep working on it. Moreover, performance isn't necessarily the most important factor for a DB. Im using Surreal (or at least something other than the legacy Mysql) in order to implement a different application architecture which should improve my legacy application's performance and scalability by many orders of magnitude. Also, there's significant value in having full text and vector search, graph db, good tooling and syntax, live websocket queries, etc.. all in one package.

So, I remain optimistic and eager for the future of SurrealDB! Still, I hope that benchmarks will come with native performance improvements as well as guidance on how to analyze and improve performance of our DBs

[JABirchall]

TBH, i'd also be interested in a comparison with Cassandra, so SurrealDB is simular to cassandra with the graph database.

[nickchomey]

I see nothing stopping anyone from giving it a try! Though the official tools will use YCSB, among other things. And that tests Cassandra and many other dbs

https://github.com/pingcap/go-ycsb?tab=readme-ov-file#supported-database

[ShanonJackson]

@nickchomey

It's a good question. I'd say to make sure both have network + parse time.
Get 40_000 rows as an array of data in your backend language of choice.

Yes, this does mean slower parse times from poor driver implementation may effect results, but I believe it's a fair test because people 99% of the time will be querying the database from an application layer so that total performance should include parse/transfer time. If a database protocol uses an inefficient binary format to send the data that should be accounted for when evaluating it's performance.

Getting the data into a backend of choice also means no-one can cheat the network or parse time in any way. It's simply when the array has a length of 40_000 the test concludes.

I've written a simple MySQL driver for NodeJS before and for 40_000 rows network transfer time can be several seconds depending on how many columns you're selecting.

[nickchomey]

@ShanonJackson

Yeah, fair enough - interacting with the db from a "real" application makes most sense.

But you continue to misunderstand what I tested - the total dataset was 40k rows (or 3.5k in the surrealdb test dataset) and was only returning a small amount of rows (dozens or less).

The queries certainly interact with thousands of rows while processing it, but only return a small portion. Mysql is mature and have good indexes, optimizers etc. Surreal seemingly has none of that (I didn't even notice indexes helping much, if at all, when doing simple relational select * from x where y=z

[nickchomey]

I'll add a few more notes here.

I previously mentioned that I tried using pre computed views to mitigate the poor performance, but that it wasn't particularly useful. As it turns out, that's because they're broken altogether. But the issue hasn't received any acknowledgement yet

#4881

I've also proposed an improvement to the mechanism, such that it can essentially automatically track and pre-compute all queries. It's inspired by Readyset, which does something like that for Sql dbs.

#5148

Finally, I had mentioned above that I experienced poor performance in surrealist, but that was due to me using debug logging when starting the db. If you set it to info, it works as expected.

surrealdb/surrealist#532

[nickchomey]

I just asked Tobie in his Discord AMA about benchmarks. He said that they have built out and are using their own custom benchmarking tool that allows you to test every aspect of Surrealdb, different types of queries etc... It also tests against various industry-standard DB benchmarks.

They'll be releasing it all "in November", so hopefully sometime in the next month!

Also, he separately mentioned that there's many enterprises already using Surreal in production with millions of users, queries etc... Given the basic performance observations that I and others have shared, perhaps they only use it in very particular (or perhaps background/async) ways.

[nickchomey]

Yeah, didn't happen. Just another own goal by surreal...

My question was simply "any estimate of when it might be available?" and they could have replied with "it's something we're working on but don't have a timeline" but instead said a couple times that the November (2.1)release would bring the tooling.

I get the impression that the tooling is already ready and in use internally, in at least some form. So all I can assume is that they don't want to release it because it confirms the terrible perf that everyone has reported via their adhoc benchmarking.

If that's the case, I again urge the team to just release it anyway. We already know that's the case so it doesn't do anything to try to hide it. In fact, it just erodes trust. Conversely, being transparent builds trust, allows for us to put surreal in proper context, and make appropriate plans to for integrating it.

If it's not the case, well you still need to release it because we're going to jump to conclusions anyway in the absence of communication about the elephant in the room.

[carueda]

I have put great confidence on Surreal (with significant time spent experimenting, learning, evaluating, providing feedback, etc.), while being aware of its many bugs(*) and not-ready-for-production status, despite all the fanfare, most recently with v2. But, broken promises (and the fact that they even removed "performance" from the FAQ page!) are very disappointing.

(*) Some of the "index" related bugs are rather perplexing to me, here they are (not including the ones waiting triage). Seems like "indexes" are not first-class citizens in Surrealdb.

[JABirchall]

I have put great confidence on Surreal (with significant time spent experimenting, learning, evaluating, providing feedback, etc.), while being aware of its many bugs(*) and not-ready-for-production status, despite all the fanfare, most recently with v2. But, broken promises (and the fact that they even removed "performance" from the FAQ page!) are very disappointing.

(*) Some of the "index" related bugs are rather perplexing to me, here they are (not including the ones waiting triage). Seems like "indexes" are not first-class citizens in Surrealdb.

indeed, i dont know if i can confidently propose the use of surreal at my company for any products. i looks like it has no use at all. The features always looked great, but thats where the cliff starts, if the performance isnt there, it cant be used for a database.

if 1 user takes 7 seconds to queries 40k rows, that marks a desaster. What about more than 1 user? What about more rows (millions infact)? What about more users queries more rows? it seems performance and scalability isnt there.

[nickchomey]

Indeed.

The only reasonable way forward is for the benchmarking tooling and datasets to be released, so that everyone can start testing and comparing on the same baseline.

Even if it confirms all of the terrible results reported by adhoc tests, that's fine! At least everyone will be on solid ground and can more easily find and report issues. Likewise, it could be run on every commit and PR to identify regressions (of which there seem to have been many in v2.1).

More importantly, it'll signal that the team is being transparent and accountable, which itself is a sign of maturity and humility. Which are pre-requisites for successfully developing any project and business, and especially something as complicated as the "one database to rule them all"...

Until these benchmarking tools are released, I don't think I'll be giving Surreal any more of my attention. I've wasted far too much time already.

[joakim]

disappointing.

Disappointing is a good description of SurrealDB. That's what hype gets you. Shoot for the stars and you might end up in a treetop.

I really hope they will land this SpaceX rocket, because I do like the ideas. I just think they spend too much money and energy on hype and marketing rather than focusing on building a robust, performant database. I don't care what your marketing says, marketing people are professional liars. The only way to convince engineers that a product is good is to deliver a good product.

Until I hear good things about SurrealDB from people I trust, I'm not going to invest any more of my time and energy in it.

[mahbubabbas]

Similar query in Postgres took only 1s. Hence Surreal is horribly slow.

[nickchomey]

Which storage backend are you using?

Also, I wonder what difference it would make to do this via bulk insert?

Also, I'm more curious/concerned about select query performance - writes can generally happen async/invisible to the users

[mahbubabbas]

@nickchomey I tested in my local machine (laptop i5, intel CPU 13th gen, 16gb ram, 512g SSD).

[nickchomey]

But did you use memory, rocksdb, tikv, etc?

[mahbubabbas]

By the way.. I tried with 1 million records too which took about 50s in Surreal but about 7.5s in Postgres. Forgot to mention in my last message, I used SurrealKV.

[arriqaaq]

@mahbubabbas SurrealKV is a versioned in-memory key-value store that stores the entire index in memory. It is currently in beta. We are continuously working on improving its performance, but for production or real-time use, please consider using RocksDB (if on a single node) or TiKV/FDB, depending on your requirements.

We recommend using SurrealKV in production after the 1.0 launch.

[arriqaaq]

Also, I'm curious about which OS you're running this on, or if this is a debug build. When I run the latest release with SurrealKV, I get the following results. The time for SurrealKV is under 1 second.

[arriqaaq]

The debug build will be slower of course, same run on debug build. Are you running this in debug mode?

[mahbubabbas]

@tobiemh I strongly suggest that you and your team prioritize addressing performance issues as your topmost priority. Without immediate attention, I fear this otherwise awesome product could face an unrecoverable and disastrous situation. While you’ve made great progress on features, it’s now crucial to shift focus to performance. Frankly speaking, I am very much skeptical now in using this db even though I still love it. But because of performance issue I am planning to switch back to Postgres.

[nickchomey]

Performance is surely already a priority. What needs to be top priority is getting the benchmarking tools released so we can all operate on the same baseline.

Ps you should reply in the same comment thread, rather than creating a new thread for each comment.

[mahbubabbas]

@arriqaaq Assalamu alaikum, I am using SDB 2.1.2 on Windows 11. Machine conf: i5 13th Gen, 16g RAM, 512 g SSD. Please let me know yours.

[arriqaaq]

Waalaikumassalam warehmatullahi wabarakatuhu @mahbubabbas

This is my config

Hardware:

    Hardware Overview:

      Model Name: MacBook Air
      Model Identifier: Mac14,2
      Chip: Apple M2
      Total Number of Cores: 8 (4 performance and 4 efficiency)
      Memory: 16 GB

It seems the in-memory index behavior of SurrealKV (because of the in-memory index) and RocksDB differs on Windows compared to other platforms. I recommend testing them in a production-like setup, like on Linux. If you encounter any issues, please let us know.

[nickchomey]

@arriqaaq But the screenshot you showed above showed it also being very slow

I just tried this on my laptop in Ubuntu with rocksdb and info logging and it took 40+ seconds.

I then did the following and it shows a bulk insert being a bit faster, but still slow.

LET $data = CREATE |test:1000000| SET name = id.id(), created_at=time::now();

INSERT INTO test2 $data return none;



-------- Query 1 (44.689133312s) --------

NONE

-------- Query 2 (40.305937017s) --------

[]

[arriqaaq]

@nickchomey That is the debug build on the sdb repo, which is always going to be slow (for any storage engine). I just wanted to check if the user was running a debug build or release build. The release binary perf is <1s. You can install v2.1.2 from here to check

[arriqaaq]

Also @nickchomey the above test by @mahbubabbas was 100k records, yours is 1mil, so there will be a diff

[nickchomey]

Ah, my mistake. I also just re-ran it a few times and its going faster now - 2 seconds or less (for 100k records). That seems tolerable

[mahbubabbas]

@arriqaaq, Farhan bhai, Jazakallaah. I will test Sdb in my cloud m/c running Ubuntu in sha Allaah. As you say I think Sdb is not fully optimized for Windows OS. I have already started another LMS project using Sdb latest edition, Alhamdulillaah.

[nickchomey]

Friend, PLEASE reply to your own comments with threaded/nested comments - you're creating tremendous clutter here