Unable to connect to cosmos db cassandra storage: unable to connect to initial hosts: EOF

[Steve887]

I'm trying to setup a connection to an Azure Cosmos DB account via the Cassandra API. I think I have the config setup correctly, although it's difficult to tell, but when my Jaeger app starts up I get the following error in the console:

failed to start extensions: failed to initialize storage 'cassandra_storage': gocql: unable to create session: unable to connect to initial hosts: EOF

My storage config is as follows:

  jaeger_storage:
    backends:
      cassandra_storage:
        cassandra:
          connection:
            servers: '<accountname>.cassandra.cosmos.azure.com'
            port: 10350
            tls:
              insecure: true
            auth: 
              basic:
                username: 'username'
                password: 'password'
          schema:
            create: true
            keyspace: jaeger_dc1
            datacenter: dc1
            trace_ttl: 48h
            #dependencies_ttl: 48d
            compaction_window: 2h
            replication_factor: 1

Do I have something wrong in my config, or anything else I can try>

[dosubot[bot]]

Hey @Steve887! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together!

The error message "unable to connect to initial hosts: EOF" suggests that there might be an issue with the connection settings to your Azure Cosmos DB using the Cassandra API. Here are a few things you can check or try:

1. Disable Snappy Compression: Azure Cosmos DB with the Cassandra API does not support Snappy Compression. Ensure that you have disabled it by setting cassandra-archive.disable-compression to true ^[1].

2. TLS Configuration: Since you're using TLS, make sure that the tls.insecure setting is appropriate for your environment. If possible, try setting up a proper TLS configuration with certificates instead of using insecure: true.

3. Port and Server Address: Verify that the server address and port are correct. The port 10350 is typically used for secure connections, so ensure that your Cosmos DB account is configured to accept connections on this port.

4. Authentication: Double-check the username and password for accuracy. Ensure that the credentials have the necessary permissions to access the keyspace.

5. Firewall and Network: Ensure that there are no network or firewall rules blocking the connection to Azure Cosmos DB. You might need to whitelist the IP address of your Jaeger instance in the Azure Cosmos DB firewall settings.

6. Consistency Level: Consider setting the consistency level to a value supported by Azure Cosmos DB, such as LOCAL_ONE or QUORUM, if not already set.

If these steps do not resolve the issue, you might want to enable more detailed logging to get additional insights into the connection process.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Steve887]

None of these worked.

[Steve887]

I am making progress. I changed my TLS settings to:

        tls:
          insecure_skip_verify: true

And this gave me a different error:

failed to start extensions: failed to initialize storage 'cassandra_storage': gc_grace_seconds value must be zero.

From some research it seems to be related to a flag being passed during schema creation that Azure Cosmos doesn't support. However there doesn't seem to be any way to override this except for manually creating the schema. Is that the only solution?

[dosubot[bot]]

The gc_grace_seconds setting is not directly in the script but is part of the schema definition templates used by the script. In the Jaeger codebase, gc_grace_seconds is set to 10800 seconds (3 hours) in the schema definition files for various tables. These files are located in the internal/storage/v1/cassandra/schema/ directory, such as v003.cql.tmpl and others ^[1][2][3].

To address the issue with Azure Cosmos DB, you can manually edit these .cql.tmpl files to set gc_grace_seconds to zero before running the schema creation script. This adjustment should help you comply with Azure Cosmos DB's requirements.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Steve887]

@dosu I am trying to run the create command and cqlsh. I cannot turn off certificate verification, and when I remove the SSL option I get the following error:

Connection error: ('Unable to connect to any servers', {':10350': ConnectionShutdown('Connection to :10350 was closed')})

[dosubot[bot]]

To address the connection issue with Azure Cosmos DB using cqlsh, you need to ensure that SSL is properly configured, as Azure Cosmos DB requires SSL for connections. Here are some steps you can take:

1. Use SSL with Certificate Verification:

   • Ensure that you have the correct SSL certificate for Azure Cosmos DB. You can download the certificate from the Azure portal or use a trusted CA certificate that matches the server's certificate.

2. Specify the SSL Certificate in cqlsh:

   • Use the --ssl option along with the --ssl-certificate option to specify the path to the SSL certificate file. Here's an example command:

     cqlsh <accountname>.cassandra.cosmos.azure.com 10350 -u <username> -p <password> --ssl --ssl-certificate=/path/to/certificate.pem

   • Replace /path/to/certificate.pem with the actual path to your SSL certificate file.

3. Verify Network and Firewall Settings:

   • Ensure that there are no network or firewall rules blocking the connection to Azure Cosmos DB. The IP address of your machine should be whitelisted in the Azure Cosmos DB firewall settings.

4. Check for Connection Issues:

   • If you continue to experience connection issues, consider enabling verbose logging in cqlsh to get more detailed error messages. You can do this by adding the --debug flag to your cqlsh command.

These steps should help you establish a secure connection to Azure Cosmos DB using cqlsh with SSL. Adjust the command and settings as necessary to fit your specific environment.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Steve887]

@dosu i do not want to use a certificate, I shouldn't have to. Plus, there is no --ssl-certificate option. Perhaps a human can chime in?

[dosubot[bot]]

This is a tough nut to crack, could really use your help on this, @yurishkuro.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[jkowall]

We do not support this type of setup, there is probably something missing in the API or the way table creation works on this emulation layer. Jaeger creates the schema and setup so I assume there is some kind of issue there.

Your setup looks fine to me, did you try to test with another real Cassandra database versus the emulation layer?

I would suggest using a different datastore option.

[Steve887]

@jkowall So CosmosDB doesn't work as a data source? This post seems to suggest it does: #1667

[yurishkuro]

It is not an officially supported backend and we're not testing against it. The issue you linked suggests people found a way to make it work. You may try contacting them, and you're welcome to submit a PR with a brief doc describing which configuration is needed to make it work with Cosmos. For example, someone contributed a working example for Scylla

$ l ./docker-compose/scylladb
total 16
-rw-r--r--  1 2.0K Feb  3 18:39 README.md
-rw-r--r--  1 2.3K Feb  3 18:39 docker-compose.yml