Discussion:
multiple node bootstrapping
Osman YOZGATLIOĞLU
6 years ago
Permalink
Hello,

I have 2 dc cassandra 3.0.14 setup. I need to add 2 new nodes to each dc.

I started one node in dc1 and its already joining. 3TB of 50TB finished in 2 weeks. One year ttl time series data with twcs.

I know, its not best practise..

I want to start one node in dc2 and cassandra refused to start with mentioning already one node in joining state.

I find some workaround with jmx directives, but i'm not sure if I broke something on the way.

Is it wise to bootstrap in both dc at the same time?


Regards,

Osman
Vitali Dyachuk
6 years ago
Permalink
You can use auto_bootstrap set to false to add a new node to the ring, it
will calculate the token range for the new node, but will not start
streaming the data.
In this case you can add several nodes into the ring quickly. After that
you can start nodetool rebuild -dc <> to start streaming data.
In your case 50Tb of data per node is quite a large amount of data i would
recommend, based on own experience keeping 1Tb per node, since when
streaming can be interrupted for some reason and it cannot be resumed so
you'll have to restart streaming. Also there will be compaction problems.

Vitali.
On Wed, Nov 28, 2018 at 12:03 PM Osman YOZGATLIOĞLU <
...
Jeff Jirsa
6 years ago
Permalink
This violates any consistency guarantees you have and isn’t the right approach unless you know what you’re giving up (correctness, typically)
--
Jeff Jirsa
...
Jonathan Haddad
6 years ago
Permalink
Agree with Jeff here, using auto_bootstrap:false is probably not what you
want.

Have you increased your streaming throughput?

Upgrading to 3.11 might reduce the time by quite a bit:
https://issues.apache.org/jira/browse/CASSANDRA-9766

You'd be doing committers a huge favor if you grabbed some histograms and
flame graphs on both the sending an receiving nodes:
http://thelastpickle.com/blog/2018/01/16/cassandra-flame-graphs.html and
sent them to the dev mailing list.
...
--
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade
Continue reading on narkive:
Loading...