Stopping and then starting the node fails for Cassandra

[muzarski]

Branch: master

I believe this might be due to #645 - when C* node is down, and we call ccm node start, the hostid method is called on a node that is down. I saw that ScyllaNode::hostid logic was adjusted, so it's possible to obtain a host id from Scylla node that is down.

Reproducer from cpp-rust-driver CI:

ccm> ccm create -n 3:0 -i 127.0.0. -v 3.11.19 -b cpp-driver_3-11-19_3-0
ccm> ccm updateconf --rt=10000 read_request_timeout_in_ms:10000 write_request_timeout_in_ms:10000 request_timeout_in_ms:10000 phi_convict_threshold:16 hinted_handoff_enabled:false enable_materialized_views:true dynamic_snitch_update_interval_in_ms:1000 native_transport_max_threads:1 concurrent_reads:2 concurrent_writes:2 concurrent_compactors:1 compaction_throughput_mb_per_sec:0 key_cache_size_in_mb:0 key_cache_save_period:0 memtable_flush_writers:1 max_hints_delivery_threads:1 cas_contention_timeout_in_ms:10000 file_cache_size_in_mb:0 rpc_min_threads:1 rpc_max_threads:1
ccm> ccm start --wait-other-notice --wait-for-binary-proto
ccm> ccm status
ccm> ccm node2 stop
ccm> ccm node3 stop
ccm> ccm node2 start --wait-other-notice --wait-for-binary-proto
unknown file: Failure
C++ exception with description "Traceback (most recent call last):
  File "/home/runner/.local/bin/ccm", line 74, in <module>
    cmd.run()
  File "/home/runner/.local/lib/python3.11/site-packages/ccmlib/cmds/node_cmds.py", line 196, in run
    self.node.start(not self.options.no_join_ring,
  File "/home/runner/.local/lib/python3.11/site-packages/ccmlib/node.py", line 675, in start
    node.watch_log_for_alive(self, from_mark=mark)
  File "/home/runner/.local/lib/python3.11/site-packages/ccmlib/node.py", line 534, in watch_log_for_alive
    tofind = [f"({node.address()}|{node.hostid()}).* now UP" for node in tofind]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.11/site-packages/ccmlib/node.py", line 534, in <listcomp>
    tofind = [f"({node.address()}|{node.hostid()}).* now UP" for node in tofind]
                                   ^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.11/site-packages/ccmlib/node.py", line 1442, in hostid
    info = self.nodetool('info', capture_output=True, timeout=timeout)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.11/site-packages/ccmlib/node.py", line 880, in nodetool
    return self._do_run_nodetool(nodetool, capture_output, wait, timeout, verbose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/lib/python3.11/site-packages/ccmlib/node.py", line 832, in _do_run_nodetool
    raise NodetoolError(" ".join(nodetool), exit_status, stdout, stderr)
ccmlib.node.ToolError: Subprocess /home/runner/.ccm/repository/3.11.19/bin/nodetool -h localhost -p 7200 -Dcom.sun.jndi.rmiURLParsing=legacy info exited with non-zero status; exit status: 1; 
stderr: nodetool: Failed to connect to 'localhost:7200' - ConnectException: 'Connection refused (Connection refused)'.

The reproducer does not work for me locally - it means that I might be lucky and C* node already listens on JMX port when ccm asks for hostid(). However, it reliably reproduces in cpp-rust-driver CI (GH action). See for example: https://github.com/scylladb/cpp-rust-driver/actions/runs/14170661751/job/39693671740?pr=231

[fruch]

@muzarski should be a simple fix to try/expect, and return None or "" as fallback.

or even node host_id for Cassandra should return empty string

[gleb-cloudius]

I do not understand what is going on here. Are we using our ccm with cassandra as well?! Why not use cassandra's ccm?

[fruch]

I do not understand what is going on here. Are we using our ccm with cassandra as well?! Why not use cassandra's ccm?

Our driver forks are using this CCM, for testing both Scylla and Casandra

[gleb-cloudius]

I do not understand what is going on here. Are we using our ccm with cassandra as well?! Why not use cassandra's ccm?

Our driver forks are using this CCM, for testing both Scylla and Casandra

This is very sad since it is not tested in CI with cassandra and hence may break any time. Casandra itself may change something and it will break.

Here I have no idea what can be done. Of course we can return zero, and for this particular problem it will be enough, but then we will have an API that sometimes returns garbage.

[Lorak-mmk]

This is very sad since it is not tested in CI with cassandra and hence may break any time.

Then we should start testing with Cassandra in the CI. It is not reasonable to expect developers to install a different tool each time we want to change if we run Scylla or Cassandra.

[gleb-cloudius]

This is very sad since it is not tested in CI with cassandra and hence may break any time.

Then we should start testing with Cassandra in the CI. It is not reasonable to expect developers to install a different tool each time we want to change if we run Scylla or Cassandra.

You do install different tool already. You install Cassandra instead of Scylla. Scylla and Cassandra evolve in different directions and it will become harder and harder to preserver exactly same functionality as you can see here.

[Lorak-mmk]

This is very sad since it is not tested in CI with cassandra and hence may break any time.

Then we should start testing with Cassandra in the CI. It is not reasonable to expect developers to install a different tool each time we want to change if we run Scylla or Cassandra.

You do install different tool already. You install Cassandra instead of Scylla. Scylla and Cassandra evolve in different directions and it will become harder and harder to preserver exactly same functionality as you can see here.

And yet Scylla wants to be compatible with Cassandra, and we want to test both in CI of the drivers / tools.
If our CCM does not support cassandra, then we would need to implement a second "backend" in ccm wrappers present in testing suites that will abstract away differences between our ccm and upstream ccm.
That sound like a lot of work because we want to do it in X projects, right?
If only there was a project that could do it once and everyone could utilize this... Oh wait that is scylla-ccm.

[gleb-cloudius]

This is very sad since it is not tested in CI with cassandra and hence may break any time.

Then we should start testing with Cassandra in the CI. It is not reasonable to expect developers to install a different tool each time we want to change if we run Scylla or Cassandra.

You do install different tool already. You install Cassandra instead of Scylla. Scylla and Cassandra evolve in different directions and it will become harder and harder to preserver exactly same functionality as you can see here.

And yet Scylla wants to be compatible with Cassandra, and we want to test both in CI of the drivers / tools.

Only on CQL level. But ccm relies on internals.

If our CCM does not support cassandra, then we would need to implement a second "backend" in ccm wrappers present in testing suites that will abstract away differences between our ccm and upstream ccm. That sound like a lot of work because we want to do it in X projects, right? If only there was a project that could do it once and everyone could utilize this... Oh wait that is scylla-ccm.

By all means if you perceive it like that and think that it is easier for you to fix scylla-ccm to work with cassandra than writing those "backends" then keep maintaining scylla-ccm to work with cassandra. You are in the best position to do so since you are the one who actually run cassandra with it.

[fruch]

the philosophical debate is interesting, but...

1. we can't revert the change @gleb-cloudius did, since dtest is now counting on it on master
2. we probably don't want over complicate all of the code base that need to run Cassandra to use anything else.

so we'll need to fix it, question is who, and how we can cover it with test in this repo, to avoid it happening again.

I wouldn't be able to attend to it, in the next 2 weeks.

so it's between you too..

[muzarski]

I'll fix it.