Discussion:
Adjusting Token Spaces and Rebalancing Data
Jon Graham
2010-02-25 00:17:48 UTC
Permalink
Hello,

I have 6 node Cassandra 0.5.0 cluster
using org.apache.cassandra.dht.OrderPreservingPartitioner with replication
factor 3.

I mistakenly set my tokens to the wrong values, and have all the data being
stored on the first node (with replicas on the seconds and third nodes)

Does Cassandra have any tools to reset the token values and re-distribute
the data?

Thanks for your help,
Jon
Jonathan Ellis
2010-02-25 04:17:45 UTC
Permalink
nodeprobe loadbalance and/or nodeprobe move

http://wiki.apache.org/cassandra/Operations
Post by Jon Graham
Hello,
I have 6 node Cassandra 0.5.0 cluster
using org.apache.cassandra.dht.OrderPreservingPartitioner with replication
factor 3.
I mistakenly set my tokens to the wrong values, and have all the data being
stored on the first node (with replicas on the seconds and third nodes)
Does Cassandra have any tools to reset the token values and re-distribute
the data?
Thanks for your help,
Jon
Jon Graham
2010-03-01 18:40:56 UTC
Permalink
Hello Everyone,

Jonathan, Thanks for your advice :-)

I have started a loadbalance operation on a busy cassandra node.

The http://wiki.apache.org/cassandra/Operations web page indicates that
nodetool streams can
be used to monitor the status of the load balancing operation.

I can't seem to find the nodetool command for cassandra 0.5.0.

Is this a separate package/tool?

Thanks,
Jon
Post by Jonathan Ellis
nodeprobe loadbalance and/or nodeprobe move
http://wiki.apache.org/cassandra/Operations
Post by Jon Graham
Hello,
I have 6 node Cassandra 0.5.0 cluster
using org.apache.cassandra.dht.OrderPreservingPartitioner with
replication
Post by Jon Graham
factor 3.
I mistakenly set my tokens to the wrong values, and have all the data
being
Post by Jon Graham
stored on the first node (with replicas on the seconds and third nodes)
Does Cassandra have any tools to reset the token values and re-distribute
the data?
Thanks for your help,
Jon
Jonathan Ellis
2010-03-01 18:47:01 UTC
Permalink
nodetool is the 0.6 replacement for nodeprobe. the stream info is new
in that version.

(0.6 beta release is linked from
http://wiki.apache.org/cassandra/GettingStarted)

-Jonathan
Hello Everyone,
Jonathan,  Thanks for your advice :-)
I have started a loadbalance operation on a busy cassandra node.
The http://wiki.apache.org/cassandra/Operations web page indicates that
nodetool streams can
be used to monitor the status of the load balancing operation.
I can't seem to find the nodetool command for cassandra 0.5.0.
Is this a separate package/tool?
Thanks,
Jon
Post by Jonathan Ellis
nodeprobe loadbalance and/or nodeprobe move
http://wiki.apache.org/cassandra/Operations
Post by Jon Graham
Hello,
I have 6 node Cassandra 0.5.0 cluster
using org.apache.cassandra.dht.OrderPreservingPartitioner with replication
factor 3.
I mistakenly set my tokens to the wrong values, and have all the data being
stored on the first node (with replicas on the seconds and third nodes)
Does Cassandra have any tools to reset the token values and re-distribute
the data?
Thanks for your help,
Jon
Jon Graham
2010-03-01 19:44:24 UTC
Permalink
Jonathan, Thanks for the quick reply.

After starting a loadbalance operation for about 30 minutes, I can see 3
ColumnFamily-tmp-Data, Filter and Index files
on a lightly loaded node. The Data file has a size of 2,147,483,647 (max
signed int) on the node being loaded. I hope I didn't run out of Int space
during the load balance operation.

The file sizes don't seem to be changing in the column family folder of the
newly loaded node. The system/LocationInfo files haven't changed in 20
minutes. The 3 ColumnFamily files still have the -tmp- name.

Can I tell if the load balancing operaion is still running ok or if it
has terminated?

Is there a rough computation to determine how long the process should take?

If the load balancing is successful, will the cluster ring
information reflect the load balanacing changes?

Thanks,
Jon
Post by Jonathan Ellis
nodetool is the 0.6 replacement for nodeprobe. the stream info is new
in that version.
(0.6 beta release is linked from
http://wiki.apache.org/cassandra/GettingStarted)
-Jonathan
Post by Jon Graham
Hello Everyone,
Jonathan, Thanks for your advice :-)
I have started a loadbalance operation on a busy cassandra node.
The http://wiki.apache.org/cassandra/Operations web page indicates that
nodetool streams can
be used to monitor the status of the load balancing operation.
I can't seem to find the nodetool command for cassandra 0.5.0.
Is this a separate package/tool?
Thanks,
Jon
Post by Jonathan Ellis
nodeprobe loadbalance and/or nodeprobe move
http://wiki.apache.org/cassandra/Operations
Post by Jon Graham
Hello,
I have 6 node Cassandra 0.5.0 cluster
using org.apache.cassandra.dht.OrderPreservingPartitioner with replication
factor 3.
I mistakenly set my tokens to the wrong values, and have all the data being
stored on the first node (with replicas on the seconds and third
nodes)
Post by Jon Graham
Post by Jonathan Ellis
Post by Jon Graham
Does Cassandra have any tools to reset the token values and re-distribute
the data?
Thanks for your help,
Jon
Jonathan Ellis
2010-03-01 19:55:07 UTC
Permalink
Can I tell if the load balancing operaion is still running ok or if it
has terminated?
Is there a rough computation to determine how long the process should take?
Not really, although you can guess from cpu/io usage. This is much
improved in 0.6.
If the load balancing is successful, will the cluster ring
information reflect the load balanacing changes?
Yes.
Jon Graham
2010-03-01 21:18:52 UTC
Permalink
Thanks Jonathan.

It seems like the load balance operation isn't moving. I haven't seen any
data file time changes in 2 hours and no location file time
changes in over an hour.

I can see a tcp port # 7000 opened on the node where I ran the loadbalance
command. It is connected to
port 39033 on the node receiving the data. The CPU usage on both systems is
very low. There are about 10
million records on the node where the load balance command was issued.

My six node Cassandra ring consists of tokens for nodes 1-6 of: 0
(ascii 0x30) 6 B H O (the letter O) T

The load balance target node initially had a token of 'H' (using ordered
partitioning). The source node has a key of 0 (ascii 0x30). Most of the data
on the source node has keys starting with '/'. Slash falls between tokens T
and 0 in my ring so most of the data landed on the node with token 0 with
replicas on the next 2 nodes. My token space is badly divided for the data I
have already inserted.

Does the initial token value of the load balance target node selected by
Cassandra need to be cleared or set to a specific value before hand to
accomodate the load balance data transfer?

Would I have better luck decommissioning nodes 4,5,6 and trying to
bootstrapping these nodes one at a time
with better initial token values?

Would the existing data on nodes 1,2,3 be moved to the boot strapped nodes?

I am looking for a good way to move/split/re-balance data from nodes 1,2,3
to nodes 4, 5, 6 while achiving a better token space distribution.

Thanks for your help,
Jon
Post by Jonathan Ellis
Post by Jon Graham
Can I tell if the load balancing operaion is still running ok or if it
has terminated?
Is there a rough computation to determine how long the process should
take?
Not really, although you can guess from cpu/io usage. This is much
improved in 0.6.
Post by Jon Graham
If the load balancing is successful, will the cluster ring
information reflect the load balanacing changes?
Yes.
Jonathan Ellis
2010-03-01 22:54:59 UTC
Permalink
Post by Jon Graham
Thanks Jonathan.
It seems like the load balance operation isn't moving. I haven't seen any
data file time changes in 2 hours and no location file time
changes in over an hour.
I can see a tcp port # 7000 opened on the node where I ran the loadbalance
command. It is connected to
port 39033 on the node receiving the data. The CPU usage on both systems is
very low. There are about 10
million records on the node where the load balance command was issued.
Did you check logs for exceptions?
Post by Jon Graham
My six node Cassandra ring consists of tokens for nodes 1-6 of:  0
(ascii 0x30)  6  B  H  O (the letter O)  T
The load balance target node initially had a token of 'H' (using ordered
partitioning). The source node has a key of 0 (ascii 0x30). Most of the data
on the source node has keys starting with '/'. Slash falls between tokens T
and  0 in my ring so most of the data landed on the node with token 0 with
replicas on the next 2 nodes. My token space is badly divided for the data I
have already inserted.
Does the initial token value of the load balance target node selected by
Cassandra need to be cleared or set to a specific value before hand to
accomodate the load balance data transfer?
No.
Post by Jon Graham
Would I have better luck decommissioning nodes 4,5,6 and trying to
bootstrapping these nodes one at a time
with better initial token values?
LoadBalance is basically sugar for decommission + bootstrap, so no.
Post by Jon Graham
I am looking for a good way to move/split/re-balance data from nodes 1,2,3
to nodes 4, 5, 6 while achiving a better token space distribution.
I would upgrade to the 0.6 beta and try loadbalance again.

-Jonathan
Jon Graham
2010-03-01 23:39:26 UTC
Permalink
Hello,

I did find these exceptions. I issued the loadbalance command on node
192.168.2.10.

INFO [MESSAGING-SERVICE-POOL:3] 2010-03-01 10:34:40,764 TcpConnection.java
(line 315) Closing errored connection
java.nio.channels.SocketChannel[connected local=/192.168.2.10:55973 remote=/
192.168.2.13:7000]
WARN [MESSAGE-DESERIALIZER-POOL:1] 2010-03-01 10:34:40,964
MessagingService.java (line 555) Running on default stage - beware
WARN [MESSAGING-SERVICE-POOL:1] 2010-03-01 10:34:40,964 TcpConnection.java
(line 484) Problem reading from socket connected to :
java.nio.channels.SocketChannel[connected local=/192.168.2.10:40758 remote=/
192.168.2.13:7000]
WARN [MESSAGING-SERVICE-POOL:1] 2010-03-01 10:34:40,964 TcpConnection.java
(line 485) Exception was generated at : 03/01/2010 10:34:40 on thread
MESSAGING-SERVICE-POOL:1
Reached an EOL or something bizzare occured. Reading from:
/192.168.2.13BufferSizeRemaining: 16
java.io.IOException: Reached an EOL or something bizzare occured. Reading
from: /192.168.2.13 BufferSizeRemaining: 16
at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44)
at org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39)
at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
at
org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
INFO [MESSAGING-SERVICE-POOL:1] 2010-03-01 10:34:40,964 TcpConnection.java
(line 315) Closing errored connection
java.nio.channels.SocketChannel[connected local=/192.168.2.10:40758 remote=/
192.168.2.13:7000]
INFO [MESSAGE-STREAMING-POOL:1] 2010-03-01 10:35:23,171 TcpConnection.java
(line 315) Closing errored connection
java.nio.channels.SocketChannel[connected local=/192.168.2.10:56728 remote=/
192.168.2.13:7000]
INFO [MESSAGE-STREAMING-POOL:1] 2010-03-01 10:35:23,221 FileStreamTask.java
(line 79) Exception was generated at : 03/01/2010 10:35:23 on thread
MESSAGE-STREAMING-POOL:1
Value too large for defined data type
java.io.IOException: Value too large for defined data type
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at org.apache.cassandra.net.TcpConnection.stream(TcpConnection.java:226)
at org.apache.cassandra.net.FileStreamTask.run(FileStreamTask.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

I can certainly upgrade to 0.6 and try a loadbalance there, do you
still think it is advisable?

All of my key/value entries are well under 1024 bytes but I have millions of
them.

Do you think I have a data corruption problem?

Thanks,
Jon
Post by Jonathan Ellis
Post by Jon Graham
Thanks Jonathan.
It seems like the load balance operation isn't moving. I haven't seen any
data file time changes in 2 hours and no location file time
changes in over an hour.
I can see a tcp port # 7000 opened on the node where I ran the
loadbalance
Post by Jon Graham
command. It is connected to
port 39033 on the node receiving the data. The CPU usage on both systems
is
Post by Jon Graham
very low. There are about 10
million records on the node where the load balance command was issued.
Did you check logs for exceptions?
Post by Jon Graham
My six node Cassandra ring consists of tokens for nodes 1-6 of: 0
(ascii 0x30) 6 B H O (the letter O) T
The load balance target node initially had a token of 'H' (using ordered
partitioning). The source node has a key of 0 (ascii 0x30). Most of the
data
Post by Jon Graham
on the source node has keys starting with '/'. Slash falls between tokens
T
Post by Jon Graham
and 0 in my ring so most of the data landed on the node with token 0
with
Post by Jon Graham
replicas on the next 2 nodes. My token space is badly divided for the
data I
Post by Jon Graham
have already inserted.
Does the initial token value of the load balance target node selected by
Cassandra need to be cleared or set to a specific value before hand to
accomodate the load balance data transfer?
No.
Post by Jon Graham
Would I have better luck decommissioning nodes 4,5,6 and trying to
bootstrapping these nodes one at a time
with better initial token values?
LoadBalance is basically sugar for decommission + bootstrap, so no.
Post by Jon Graham
I am looking for a good way to move/split/re-balance data from nodes
1,2,3
Post by Jon Graham
to nodes 4, 5, 6 while achiving a better token space distribution.
I would upgrade to the 0.6 beta and try loadbalance again.
-Jonathan
Jonathan Ellis
2010-03-02 00:55:48 UTC
Permalink
Reached an EOL or something bizzare occured. Reading from: /192.168.2.13
BufferSizeRemaining: 16
This one is harmless
java.io.IOException: Value too large for defined data type
    at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
    at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
    at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
    at org.apache.cassandra.net.TcpConnection.stream(TcpConnection.java:226)
    at org.apache.cassandra.net.FileStreamTask.run(FileStreamTask.java:55)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
This one is killing you.

Are you on windows? If so
https://issues.apache.org/jira/browse/CASSANDRA-795 should fix it.
That's in both 0.5.1 and 0.6 beta.

-Jonathan
Jon Graham
2010-03-02 16:46:00 UTC
Permalink
Hello,

I am running a 32-bit linux version 2.6.27.24. My original data set was
copied from a 64-bit cassandra cluster to a 32-bit cassandra cluster. I am
trying to load balance the data on a 32-bit cluster.

Is the cassandra-795 issue applicable for 32-linux too for the 0.5.0
release?

Thanks,
Jon
Post by Jonathan Ellis
Reached an EOL or something bizzare occured. Reading from: /192.168.2.13
BufferSizeRemaining: 16
This one is harmless
java.io.IOException: Value too large for defined data type
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at
org.apache.cassandra.net.TcpConnection.stream(TcpConnection.java:226)
at
org.apache.cassandra.net.FileStreamTask.run(FileStreamTask.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
This one is killing you.
Are you on windows? If so
https://issues.apache.org/jira/browse/CASSANDRA-795 should fix it.
That's in both 0.5.1 and 0.6 beta.
-Jonathan
Jonathan Ellis
2010-03-02 17:15:33 UTC
Permalink
Doing some googling, this is a different JRE bug than the on addressed
by 795: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6253145.
It is marked fixed in JDK 6u18, so try upgrading to that.

-Jonathan
Post by Jon Graham
Hello,
I am running a 32-bit linux version 2.6.27.24. My original data set was
copied from a 64-bit cassandra cluster to a 32-bit cassandra cluster. I am
trying to load balance the data on a 32-bit cluster.
Is the cassandra-795 issue applicable for 32-linux too for the 0.5.0
release?
Thanks,
Jon
Post by Jonathan Ellis
Reached an EOL or something bizzare occured. Reading from: /192.168.2.13
BufferSizeRemaining: 16
This one is harmless
java.io.IOException: Value too large for defined data type
    at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
    at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
    at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
    at
org.apache.cassandra.net.TcpConnection.stream(TcpConnection.java:226)
    at
org.apache.cassandra.net.FileStreamTask.run(FileStreamTask.java:55)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
This one is killing you.
Are you on windows?  If so
https://issues.apache.org/jira/browse/CASSANDRA-795 should fix it.
That's in both 0.5.1 and 0.6 beta.
-Jonathan
Jon Graham
2010-03-02 17:24:56 UTC
Permalink
Thanks Jonathan,

My 32-bit java version is at: 1.6.0_13-b03. I'll try a java upgrade.
This tracks well with the exact MaxInt -tmp- Data file size

Jon
Post by Jonathan Ellis
Doing some googling, this is a different JRE bug than the on addressed
by 795: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6253145.
It is marked fixed in JDK 6u18, so try upgrading to that.
-Jonathan
Post by Jon Graham
Hello,
I am running a 32-bit linux version 2.6.27.24. My original data set was
copied from a 64-bit cassandra cluster to a 32-bit cassandra cluster. I
am
Post by Jon Graham
trying to load balance the data on a 32-bit cluster.
Is the cassandra-795 issue applicable for 32-linux too for the 0.5.0
release?
Thanks,
Jon
Post by Jonathan Ellis
Reached an EOL or something bizzare occured. Reading from: /
192.168.2.13
Post by Jon Graham
Post by Jonathan Ellis
BufferSizeRemaining: 16
This one is harmless
java.io.IOException: Value too large for defined data type
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at
org.apache.cassandra.net.TcpConnection.stream(TcpConnection.java:226)
at
org.apache.cassandra.net.FileStreamTask.run(FileStreamTask.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
This one is killing you.
Are you on windows? If so
https://issues.apache.org/jira/browse/CASSANDRA-795 should fix it.
That's in both 0.5.1 and 0.6 beta.
-Jonathan
Jon Graham
2010-03-02 20:51:09 UTC
Permalink
Thanks!

Switching to java 1.6.0_18 seems to have gotten past the 2GB file boundary.
I now have a new ring token for the first node in my cluster.

Can I run a "loadbalance" on nodes 2-6 to achive more data and token
balancing?

Should I perform a cleanup operation on node 1?

During the loadbalance operation, the following changes occurred:

node 1: changed token value ; data size change from 5.7 GB to 8.2 GB
(loadbalance was performed on this node)
node 2: no token ring changes ; data size remained at 5.7 GB ; again found
the following warning(s) in log file

java.io.IOException: Reached an EOL or something bizzare occured. Reading
from: /node1 BufferSizeRemaining: 16
at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44)
at
org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39)
at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
at
org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)

node 3: no token ring changes ; data size remained at 5.7 GB
node 4: no token ring changes ; data size change from 3KB to 5.7 GB
(cassandra selected target of load balance)
node 5: no token ring changes ; data size remained at 3 KB
node 6: no token ring changes ; data size remained at 3 KB

Thanks again, I really appreciate your help on this.
Jon
--------------------------------------------------------------------------------------------------------------------
Post by Jon Graham
Thanks Jonathan,
My 32-bit java version is at: 1.6.0_13-b03. I'll try a java upgrade.
This tracks well with the exact MaxInt -tmp- Data file size
Jon
Post by Jonathan Ellis
Doing some googling, this is a different JRE bug than the on addressed
by 795: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6253145.
It is marked fixed in JDK 6u18, so try upgrading to that.
-Jonathan
Post by Jon Graham
Hello,
I am running a 32-bit linux version 2.6.27.24. My original data set was
copied from a 64-bit cassandra cluster to a 32-bit cassandra cluster. I
am
Post by Jon Graham
trying to load balance the data on a 32-bit cluster.
Is the cassandra-795 issue applicable for 32-linux too for the 0.5.0
release?
Thanks,
Jon
Post by Jonathan Ellis
Reached an EOL or something bizzare occured. Reading from: /
192.168.2.13
Post by Jon Graham
Post by Jonathan Ellis
BufferSizeRemaining: 16
This one is harmless
java.io.IOException: Value too large for defined data type
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(Unknown Source)
at sun.nio.ch.FileChannelImpl.transferTo(Unknown Source)
at
org.apache.cassandra.net.TcpConnection.stream(TcpConnection.java:226)
at
org.apache.cassandra.net.FileStreamTask.run(FileStreamTask.java:55)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
This one is killing you.
Are you on windows? If so
https://issues.apache.org/jira/browse/CASSANDRA-795 should fix it.
That's in both 0.5.1 and 0.6 beta.
-Jonathan
Jonathan Ellis
2010-03-02 20:58:00 UTC
Permalink
Post by Jon Graham
Thanks!
Switching to java 1.6.0_18 seems to have gotten past the 2GB file boundary.
I now have a new ring token for the first node in my cluster.
Can I run a "loadbalance" on nodes 2-6 to achive more data and token
balancing?
You should run it on the least-loaded half. After that there will be
diminishing returns.
Post by Jon Graham
Should I perform a cleanup operation on node 1?
Yes. And others if your replication factor > 1.

You should really upgrade to 0.5.1 at the very least, if not 0.6 beta,
there are some bugs fixed related to load calculation that will affect
you here.

-Jonathan

Loading...