Thanks Sean. I have automation in place that can put the new binary and
restart the node to a newer version as quickly as possible. upgradesstables
node. Given these constraints, is there a risk due to prolonged
Post by Durity, Sean RWe have had great success with Cassandra upgrades with applications
staying on-line. It is one of the strongest benefits of Cassandra. A couple
- The main task is getting the new binaries loaded, then
restarting the node â in a rolling fashion. Get this done as quickly as
possible
- Streaming between versions is usually problematic. So, I never
do any node additions or decommissions during an upgrade
- With applications running, there is not an acceptable back-out
plan (either lose data or take a long outage or both), so we are always
going forward. So, lower life cycle testing is important before hitting
production
- Upgrading is a more frequent activity, so get the
process/automation in place. The upgrade process should not be a reason to
delay, especially for minor version upgrades that might be quickly
necessary (security issue or bug fix).
Sean Durity
*Sent:* Tuesday, December 04, 2018 12:22 PM
*Subject:* [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded
cluster (2.1.9 + 3.x) and I was *not* able to do so. The schema never
settled.
How does the below approach sound like?
1. Update the software binary on all nodes to use cassandra-3.x upon a
restart.
2. Restart all nodes in a rolling fashion
3. Run nodetool upgradesstables in a rolling fashion
Is there a risk on pending nodetool upgradesstables?
Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the
version part that you suggested. The reason to go direct to 3.x is to take
a bi leap and reduce overall effort to upgrade a large cluster (development
included).
I have these questions from my original post. Appreciate if you could shed
some light and point me in the right direction.
1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
If any of the replicas have already upgraded, which is almost guaranteed
if youâre using vnodes, Itâs hard / you donât. Youâd basically upgrade
everything else and then deal with it. If a host fails mid upgrade youâll
likely have some period of unavailables while you bounce the replicas to
finish, then you can decom
2) How to bootstrap a 3.x node to a partially upgraded cluster?
This may work fine, but test it because Iâm not certain. It should be able
to read the 2.1 and 3.0 sstables thatâll stream so itâll just work
3) Is there an alternative approach to the upgrade large clusters. i.e
instead of going through nodetool upgradesstables on each node in rolling
fashion
Bounce them all as quickly as is practical, do the upgradesstables after
the bounces complete
Dmitry is right. Generally speaking always go with the latest bug fix release.
See more here
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_plugins_servlet_mobile-23issue_CASSANDRA-2D13004&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=8csmPWgUEWao6E4wthrG_-BX5a2OQJKXpkKtFLjSPlI&e=>
Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug
corrupting sstables on alter.
Hi Shravan,
Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before
doing the major upgrade? It's generally favorable to go to the latest patch
release as often times they include fixes that smooth over the upgrade
process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current
version)
Best,
Marc
Hello,
I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9.
I came up with the version based on [1]. I followed upgrade steps as in
[2]. I was testing the same in the lab and encountered issues (streaming
just fails and hangs for ever) with bootstrapping a 3.0.9 node on a
partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The
production cluster that I am supporting is pretty large and I am
anticipating to end up in a situation like this (Hope not) and would like
to be prepared.
1) How do deal with decommissioning a 2.1.9 node in a partially upgraded cluster?
2) How to bootstrap a 3.x node to a partially upgraded cluster?
3) Is there an alternative approach to the upgrade large clusters. i.e
instead of going through nodetool upgradesstables on each node in rolling
fashion
As per [1] the general restriction is to avoid decommissioning or adding
nodes but in reality there can be failures or maintenance that warrants us
to do so.
Please point me in the right direction.
Thanks,
Shravan
[1]
https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
<https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_upgrade_doc_upgrade_datastax-5Fenterprise_upgdDSE50.html-23upgdDSE50-5F-5Fcstar-2Dversion-2Dchange&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=IV10uiJPS2fyCHGYw7yR90A8cxcpjy9Is40YUWgPvF0&e=>
[2]
https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
<https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=>
--
Marc Selwan | DataStax | Product Management | (925) 413-7079
--
Jon Haddad
http://www.rustyrazorblade.com
<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=HPfRIl5knAn9tpUyT06P6YaktRTDwyFSMO7r4GHXBzY&e=>
twitter: rustyrazorblade
------------------------------
The information in this Internet Email is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this Email
by anyone else is unauthorized. If you are not the intended recipient, any
disclosure, copying, distribution or any action taken or omitted to be
taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject
to the terms and conditions expressed in any applicable governing The Home
Depot terms of business or client engagement letter. The Home Depot
disclaims all responsibility and liability for the accuracy and content of
this attachment and for any damages or losses arising from any
inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
items of a destructive nature, which may be contained in this attachment
and shall not be liable for direct, indirect, consequential or special
damages in connection with this e-mail message or its attachment.