Upgrade from 1.0.9 to 1.2.8

[Mike Neir]

Greetings folks,

I'm faced with the need to update a 36 node cluster with roughly 25T of data on
disk to a version of cassandra in the 1.2.x series. While it seems that 1.2.8
will play nicely in the 1.0.9 cluster long enough to do a rolling upgrade, I'd
still like to have a roll-back plan in case the rolling upgrade goes sideways.

I've tried to upgrade a single node in my dev cluster, then roll back using a
snapshot taken previously, but things don't appear to be going smoothly. The
node will rejoin the ring eventually, but not after spending some time in the
"Joining" state as shown by "nodetool ring", and spewing a ton of error messages
similar to the following:

ERROR [MutationStage:31] 2013-08-29 14:07:20,530 RowMutationVerbHandler.java
(line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1178

My test procedure is as follows:
1) nodetool -h localhost snapshot
2) nodetool -h localhost drain
3) service cassandra stop
4) back up cassandra configs
5) remove cassandra 1.0.9
6) install cassandra 1.2.8
7) restore cassandra configs, alter them to remove configuration entries no
longer used
8) start cassandra 1.2.8, let it run for a bit, then drain/stop it
9) remove cassandra 1.2.8
10) reinstall cassandra 1.0.9
11) restore original cassandra configs
12) remove any commit logs present
13) remove folders for system_auth and system_traces Keyspaces (since they don't
seem to be present in 1.0.9)
14) Move snapshots back to where they should be for 1.0.9 and remove cass 1.2.8 data
# cd /var/lib/cassandra/data/$KEYSPACE/
# mv */snapshots/$TIMESTAMP/* .
# find . -mindepth 1 -type d -exec rm -rf {} \;
# cd /var/lib/cassandra/data/system
# mv */snapshots/$TIMESTAMP/* .
# find . -mindepth 1 -type d -exec rm -rf {} \;
15) start cassandra 1.0.9
16) observe cassandra system.log

Does anyone have any insight on things I may be doing wrong, or whether this is
just an unavoidable pain point caused by rolling back? It seems that since there
are no schema changes going on, the node should be able to just hop back into
the cluster without error and without transitioning through the "Joining" state.

[Jon Haddad]

Does your previous snapshot include the system keyspace? I haven't tried upgrading from 1.0.x then rolling back, but it's possible there's some backwards incompatible changes. Other than that, make sure you also rolled back your config files?

Post by Mike Neir
Greetings folks,
I'm faced with the need to update a 36 node cluster with roughly 25T of data on disk to a version of cassandra in the 1.2.x series. While it seems that 1.2.8 will play nicely in the 1.0.9 cluster long enough to do a rolling upgrade, I'd still like to have a roll-back plan in case the rolling upgrade goes sideways.
ERROR [MutationStage:31] 2013-08-29 14:07:20,530 RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1178
1) nodetool -h localhost snapshot
2) nodetool -h localhost drain
3) service cassandra stop
4) back up cassandra configs
5) remove cassandra 1.0.9
6) install cassandra 1.2.8
7) restore cassandra configs, alter them to remove configuration entries no longer used
8) start cassandra 1.2.8, let it run for a bit, then drain/stop it
9) remove cassandra 1.2.8
10) reinstall cassandra 1.0.9
11) restore original cassandra configs
12) remove any commit logs present
13) remove folders for system_auth and system_traces Keyspaces (since they don't seem to be present in 1.0.9)
14) Move snapshots back to where they should be for 1.0.9 and remove cass 1.2.8 data
# cd /var/lib/cassandra/data/$KEYSPACE/
# mv */snapshots/$TIMESTAMP/* .
# find . -mindepth 1 -type d -exec rm -rf {} \;
# cd /var/lib/cassandra/data/system
# mv */snapshots/$TIMESTAMP/* .
# find . -mindepth 1 -type d -exec rm -rf {} \;
15) start cassandra 1.0.9
16) observe cassandra system.log
Does anyone have any insight on things I may be doing wrong, or whether this is just an unavoidable pain point caused by rolling back? It seems that since there are no schema changes going on, the node should be able to just hop back into the cluster without error and without transitioning through the "Joining" state.
--
Mike Neir
Liquid Web, Inc.
Infrastructure Administrator

[Jon Haddad]

Sorry, I didn't see the test procedure, it's still early.

Post by Mike Neir
Greetings folks,
I'm faced with the need to update a 36 node cluster with roughly 25T of data on disk to a version of cassandra in the 1.2.x series. While it seems that 1.2.8 will play nicely in the 1.0.9 cluster long enough to do a rolling upgrade, I'd still like to have a roll-back plan in case the rolling upgrade goes sideways.
ERROR [MutationStage:31] 2013-08-29 14:07:20,530 RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1178
1) nodetool -h localhost snapshot
2) nodetool -h localhost drain
3) service cassandra stop
4) back up cassandra configs
5) remove cassandra 1.0.9
6) install cassandra 1.2.8
7) restore cassandra configs, alter them to remove configuration entries no longer used
8) start cassandra 1.2.8, let it run for a bit, then drain/stop it
9) remove cassandra 1.2.8
10) reinstall cassandra 1.0.9
11) restore original cassandra configs
12) remove any commit logs present
13) remove folders for system_auth and system_traces Keyspaces (since they don't seem to be present in 1.0.9)
14) Move snapshots back to where they should be for 1.0.9 and remove cass 1.2.8 data
# cd /var/lib/cassandra/data/$KEYSPACE/
# mv */snapshots/$TIMESTAMP/* .
# find . -mindepth 1 -type d -exec rm -rf {} \;
# cd /var/lib/cassandra/data/system
# mv */snapshots/$TIMESTAMP/* .
# find . -mindepth 1 -type d -exec rm -rf {} \;
15) start cassandra 1.0.9
16) observe cassandra system.log
Does anyone have any insight on things I may be doing wrong, or whether this is just an unavoidable pain point caused by rolling back? It seems that since there are no schema changes going on, the node should be able to just hop back into the cluster without error and without transitioning through the "Joining" state.
--
Mike Neir
Liquid Web, Inc.
Infrastructure Administrator

[Robert Coli]

Post by Mike Neir
I'm faced with the need to update a 36 node cluster with roughly 25T of
data on disk to a version of cassandra in the 1.2.x series. While it seems
that 1.2.8 will play nicely in the 1.0.9 cluster long enough to do a
rolling upgrade, I'd still like to have a roll-back plan in case the
rolling upgrade goes sideways.

Upgrading two major versions online is an unsupported operation. I would
not expect it to work. Is there a detailed reason you believe it should
work between these versions? Also, instead of 1.2.8 you should upgrade to
1.2.9, released yesterday. Everyone headed to 2.0 has to pass through 1.2.9.

=Rob

[Mike Neir]

In my testing, mixing 1.0.9 and 1.2.8 seems to work fine as long as there is no
need to do streaming operations (move/repair/bootstrap/etc). The reading I've
done confirms that 1.2.x should be network-compatible with 1.0.x, sans streaming
operations. Datastax seems to indicate here that doing a rolling upgrade from
1.0.x to 1.2.x is viable:

http://www.datastax.com/documentation/cassandra/1.2/webhelp/#upgrade/upgradeC_c.html#concept_ds_nht_czr_ck

See the second bullet point in the Prerequisites section.

I'll look into 1.2.9. It wasn't available when I started my testing.

MN

Post by Mike Neir
I'm faced with the need to update a 36 node cluster with roughly 25T of data
on disk to a version of cassandra in the 1.2.x series. While it seems that
1.2.8 will play nicely in the 1.0.9 cluster long enough to do a rolling
upgrade, I'd still like to have a roll-back plan in case the rolling upgrade
goes sideways.
Upgrading two major versions online is an unsupported operation. I would not
expect it to work. Is there a detailed reason you believe it should work between
these versions? Also, instead of 1.2.8 you should upgrade to 1.2.9, released
yesterday. Everyone headed to 2.0 has to pass through 1.2.9.
=Rob

[Mohit Anchlia]

If you have multiple DCs you at least want to upgrade to 1.0.11. There is
an issue where you might get errors during cross DC replication.

Post by Mike Neir
In my testing, mixing 1.0.9 and 1.2.8 seems to work fine as long as there
is no need to do streaming operations (move/repair/bootstrap/etc). The
reading I've done confirms that 1.2.x should be network-compatible with
1.0.x, sans streaming operations. Datastax seems to indicate here that
http://www.datastax.com/**documentation/cassandra/1.2/**
webhelp/#upgrade/upgradeC_c.**html#concept_ds_nht_czr_ck<http://www.datastax.com/documentation/cassandra/1.2/webhelp/#upgrade/upgradeC_c.html%23concept_ds_nht_czr_ck>
See the second bullet point in the Prerequisites section.
I'll look into 1.2.9. It wasn't available when I started my testing.
MN

Post by Mike Neir
I'm faced with the need to update a 36 node cluster with roughly 25T of data
on disk to a version of cassandra in the 1.2.x series. While it seems that
1.2.8 will play nicely in the 1.0.9 cluster long enough to do a rolling
upgrade, I'd still like to have a roll-back plan in case the rolling upgrade
goes sideways.
Upgrading two major versions online is an unsupported operation. I would not
expect it to work. Is there a detailed reason you believe it should work between
these versions? Also, instead of 1.2.8 you should upgrade to 1.2.9, released
yesterday. Everyone headed to 2.0 has to pass through 1.2.9.
=Rob

--
Mike Neir
Liquid Web, Inc.
Infrastructure Administrator

[Jeremiah D Jordan]

You probably want to go to 1.0.11/12 first no matter what. If you want the least chance of issue you should then go to 1.1.12. While there is a high probability that going from 1.0.X->1.2 will work. You have the best chance at no failures if you go through 1.1.12. There are some edge cases that can cause errors if you don't do that.

-Jeremiah

Post by Mike Neir
http://www.datastax.com/documentation/cassandra/1.2/webhelp/#upgrade/upgradeC_c.html#concept_ds_nht_czr_ck
See the second bullet point in the Prerequisites section.
I'll look into 1.2.9. It wasn't available when I started my testing.
MN

Post by Mike Neir
I'm faced with the need to update a 36 node cluster with roughly 25T of data
on disk to a version of cassandra in the 1.2.x series. While it seems that
1.2.8 will play nicely in the 1.0.9 cluster long enough to do a rolling
upgrade, I'd still like to have a roll-back plan in case the rolling upgrade
goes sideways.
Upgrading two major versions online is an unsupported operation. I would not
expect it to work. Is there a detailed reason you believe it should work between
these versions? Also, instead of 1.2.8 you should upgrade to 1.2.9, released
yesterday. Everyone headed to 2.0 has to pass through 1.2.9.
=Rob

--
Mike Neir
Liquid Web, Inc.
Infrastructure Administrator

[Mike Neir]

Is there anything that you can link that describes the pitfalls you mention? I'd
like a bit more information. Just for clarity's sake, are you recommending 1.0.9
-> 1.0.12 -> 1.1.12 -> 1.2.x? Or would 1.0.9 -> 1.1.12 -> 1.2.x suffice?

Regarding the placement strategy mentioned in a different post, I'm using the
Simple placement strategy, with the RackInferringSnitch. How does that play into
the bugs mentioned previously about cross-DC replication?

MN

Post by Jeremiah D Jordan
You probably want to go to 1.0.11/12 first no matter what. If you want the least chance of issue you should then go to 1.1.12. While there is a high probability that going from 1.0.X->1.2 will work. You have the best chance at no failures if you go through 1.1.12. There are some edge cases that can cause errors if you don't do that.
-Jeremiah

[Jeremiah D Jordan]

1.0.9 -> 1.0.12 -> 1.1.12 -> 1.2.x?

Because this fix in 1.0.11:
* fix 1.0.x node join to mixed version cluster, other nodes >= 1.1 (CASSANDRA-4195)

-Jeremiah

Is there anything that you can link that describes the pitfalls you mention? I'd like a bit more information. Just for clarity's sake, are you recommending 1.0.9 -> 1.0.12 -> 1.1.12 -> 1.2.x? Or would 1.0.9 -> 1.1.12 -> 1.2.x suffice?
Regarding the placement strategy mentioned in a different post, I'm using the Simple placement strategy, with the RackInferringSnitch. How does that play into the bugs mentioned previously about cross-DC replication?
MN

Post by Jeremiah D Jordan
You probably want to go to 1.0.11/12 first no matter what. If you want the least chance of issue you should then go to 1.1.12. While there is a high probability that going from 1.0.X->1.2 will work. You have the best chance at no failures if you go through 1.1.12. There are some edge cases that can cause errors if you don't do that.
-Jeremiah

[Mike Neir]

Ah. I was going by the upgrade recommendations in the NEWS.txt file in the
cassandra source tree, which didn't make mention of that version (1.0.11)
whatsoever. I didn't see any show-stoppers that would have prevented me from
going straight from 1.0.9 to 1.2.x.

https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-1.2.4

Looks like a multi-step upgrade is the way I'll be proceeding. Thanks for the
insight, everyone.

MN

Post by Jeremiah D Jordan

1.0.9 -> 1.0.12 -> 1.1.12 -> 1.2.x?

* fix 1.0.x node join to mixed version cluster, other nodes >= 1.1 (CASSANDRA-4195)
-Jeremiah