Multi datacentre Cassandra storage backend

[naphta]

I'm looking at implementing Jaeger and have a bias towards using Cassandra as we already have an existing setup in place going across our 9 Kubernetes clusters and 3 datacentres.

How well it does it work when replicating data back using Cassandra in a multi datacentre way? We only intend to run the UI in a single datacentre, just get each collector to talk to the cluster-local cassandra.

[yurishkuro]

I think this is more of a question of your Cassandra setup, why would Jaeger itself be affected?

When I was at Uber we ran Jaeger backend with Cassandra cluster whose nodes were distributed across 3 availability zones.

[naphta]

It's more I can't find a lot of information around schemas used in production, nor could I see anyone using Cassandra in a multi datacentre way with Jaeger specifically (everyone seems to have one per cluster). I could use the operator to create my keyspace of course but it looks like I don't have a huge amount of control over the schema and it's replication etc then.

The way we have it with our other applications is they use a local quorum, and then we let the replication between datacentres happen by itself. I'm essentially wanting to run the Jaeger UI in one datacentre, and then have collectors and agents running in every other cluster to collect data.

I'm just struggling to get a setup down on paper which makes sense and there's very little documentation around running Jaeger across multiple datacentres.

[yurishkuro]

This might be related #2618

[naphta]

Possibly a litttle related at least in terms of configuring the automatically created schema but there doesn't seem to be any especially authoritative documentation / advice on running Jaeger across datacentres or in a multi cluster manner. I'm sure plenty of people are doing it but there's just not a lot to tune into. Kafka seems to be the only half way documented way of doing it and that's quite a burden to configure and use solely for Jaeger.

I suppose what I'm asking here ultimately is if I point several Jaeger's at a multi datacentre cassandra setup what's going to happen?

Or maybe, is a better strategy to just have one centralised collector and run agents on the other clusters which talk back?

Maybe cassandra is the wrong thing to do and we should work towards an elasticsearch backend with kafka to make it replicate messages from the other datacentres?