Cassandra 2.1 bootstrap - No streaming progress from one node

[Steinmaurer, Thomas]

Hello,

while bootstrapping a new node into an existing cluster, a node which is acting as source for streaming got restarted unfortunately. Since then, from nodetool netstats I don't see any progress for this particular node anymore.

E.g.:

/X.X.X.X
Receiving 94 files, 260.09 GB total. Already received 26 files, 69.33 GB total

Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our case) not doing any resume here, in case there have been e.g. connectivity troubles or in our case, Cassandra on the node acting as stream source got restarted?

Can I force the joining node to recover connection to X.X.X.X or do I need to restart the bootstrap via restart on the new node from scratch?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist?dterstra?e 313

[Durity, Sean R]

I would wipe the new node and bootstrap again. I do not know of any way to resume the streaming that was previously in progress.


Sean Durity
From: Steinmaurer, Thomas <***@dynatrace.com>
Sent: Wednesday, November 07, 2018 5:13 AM
To: ***@cassandra.apache.org
Subject: [EXTERNAL] Cassandra 2.1 bootstrap - No streaming progress from one node

Hello,

while bootstrapping a new node into an existing cluster, a node which is acting as source for streaming got restarted unfortunately. Since then, from nodetool netstats I don't see any progress for this particular node anymore.

E.g.:

/X.X.X.X
Receiving 94 files, 260.09 GB total. Already received 26 files, 69.33 GB total

Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our case) not doing any resume here, in case there have been e.g. connectivity troubles or in our case, Cassandra on the node acting as stream source got restarted?

Can I force the joining node to recover connection to X.X.X.X or do I need to restart the bootstrap via restart on the new node from scratch?

Thanks,
Thomas

The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.

[Alain RODRIGUEZ]

Hello,

+1 with Sean above.
In Cassandra 2.2 you got the new 'nodetool resume' command to resume a
bootstrap. For C*2.1, nothing equivalent sadly. Thus old school technics
apply and the possible alternatives are

Option 1 - safe and slow- Stop Cassandra from the stuck joining node.
Remove everything (comitlog/data), restart bootstrap, mentioned by Sean
above.
Option 2 - More wild / hopefully quicker - Stop the joining node and start
it with 'auto_bootstrap: false'. The node joins with missing data, that you
can repair after. Yet this can be done without inconsistencies only using a
strong consistency (CL.R+CL.W > RF). This node will be read from but is
then not enough to induce a stale read.

Unless you're confident or have a huge amount of data and most of it made
it to the new node already, I would stick with the option 1, safe and slow
(and upgrade soon ;-))

C*heers,
-----------------------
Alain Rodriguez - ***@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

Post by Durity, Sean R
I would wipe the new node and bootstrap again. I do not know of any way to
resume the streaming that was previously in progress.
Sean Durity
*Sent:* Wednesday, November 07, 2018 5:13 AM
*Subject:* [EXTERNAL] Cassandra 2.1 bootstrap - No streaming progress
from one node
Hello,
while bootstrapping a new node into an existing cluster, a node which is
acting as source for streaming got restarted unfortunately. Since then,
from nodetool netstats I don’t see any progress for this particular node
anymore.
/X.X.X.X
Receiving 94 files, 260.09 GB total. Already received 26 files, 69.33 GB total
Basically, it is stuck at 69.33GB for hours. Is Cassandra (2.1 in our
case) not doing any resume here, in case there have been e.g. connectivity
troubles or in our case, Cassandra on the node acting as stream source got
restarted?
Can I force the joining node to recover connection to X.X.X.X or do I need
to restart the bootstrap via restart on the new node from scratch?
Thanks,
Thomas
The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or
disclose it to anyone else. If you received it in error please notify us
immediately and then destroy it. Dynatrace Austria GmbH (registration
number FN 91482h) is a company registered in Linz whose registered office
is at 4040 Linz, Austria, FreistÀdterstraße 313
------------------------------
The information in this Internet Email is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this Email
by anyone else is unauthorized. If you are not the intended recipient, any
disclosure, copying, distribution or any action taken or omitted to be
taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject
to the terms and conditions expressed in any applicable governing The Home
Depot terms of business or client engagement letter. The Home Depot
disclaims all responsibility and liability for the accuracy and content of
this attachment and for any damages or losses arising from any
inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
items of a destructive nature, which may be contained in this attachment
and shall not be liable for direct, indirect, consequential or special
damages in connection with this e-mail message or its attachment.