What version of Cassandra did you upgrade to 2.2.0 *from*?
This would help with looking at config differences, changelogs, etc.
each other. I'm not terribly familiar with using SSL with Cassandra, but
cipher_suites: configuration.
Post by Carlos ScheideckerThanks for the reply, Michael!
Yes, I did followed the upgrade nodes.
I am running Ubuntu Ubuntu 14.04.2 LTS on all and
kernel 3.13.0-57-generic on all.
I have 4 machines: .31, .32, .33 and .34. If I run nodetool status from
.34 I now see all the others as DN the same happens if I log in from the
DN 192.168.1.31 ? 256 ?
1f8000f5-026c-42c7-8189-cf19fbede566 RAC1
DN 192.168.1.32 ? 256 ?
12478d45-3d5e-418b-a0dc-dba6d4307af3 RAC1
DN 192.168.1.33 ? 256 ?
994172b3-cd36-4558-a4b8-054cfac027f3 RAC1
UN 192.168.1.34 1.7 MB 256 ?
b66be1f3-bb4a-49bd-9835-5c8ee2a71e5c RAC1
tcp 0 0 127.0.1.1:53 <http://127.0.1.1:53>
0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 <http://0.0.0.0:22>
0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 <http://127.0.0.1:631>
0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:7001 <http://192.168.1.34:7001>
0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:7199 <http://127.0.0.1:7199>
0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:9160 <http://192.168.1.34:9160>
0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:59441 <http://127.0.0.1:59441>
0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:52951 <http://192.168.1.34:52951>
192.168.1.31:7001 <http://192.168.1.31:7001> ESTABLISHED
WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:48,764 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:48,764 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:48,764 OutboundTcpConnection.java:316 - error writing
to /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.33 <http://192.168.1.33>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:50,763 OutboundTcpConnection.java:316 - error writing
to /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:51,766 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:51,767 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing
to /192.168.1.33 <http://192.168.1.33>
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing
to /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:53,767 SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
2015-07-22 17:29:53,767 OutboundTcpConnection.java:229 - error
processing a message intended for /192.168.1.31 <http://192.168.1.31>
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
endpoint_snitch: GossipingPropertyFileSnitch
On the cassandra-rackdc.properties
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# These properties are used with GossipingPropertyFileSnitch and will
# indicate the rack and dc for this node
dc=DC1
rack=RAC1
# Add a suffix to a datacenter name. Used by the Ec2Snitch and
Ec2MultiRegionSnitch
# to append a string to the EC2 region name.
#dc_suffix=
# Uncomment the following line to make this snitch prefer the internal
ip when possible, as the Ec2MultiRegionSnitch does.
# prefer_local=true
On my cassandra-topology.properties
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Cassandra Node IP=Data Center:Rack
192.168.1.31=DC1:RAC1
192.168.1.32=DC1:RAC1
192.168.1.33=DC1:RAC1
192.168.1.34=DC1:RAC1
# default for unknown nodes
default=DC1:RAC1
I have a 4 node Cassandra system running on 4 Ubuntu boxes. After
updating to Cassandra 2.2.0 and keeping the same cassandra.yaml file,
the nodes cannot see each other.
What version did you upgrade from?
Usually, when upgrading, it is probably a good idea to start with
the default cassandra.yaml from the new version (2.2.0 in your case)
and edit the necessary items from your old version; i.e. num_tokens,
initial_token, listen_address, broadcast_address, etc. You are
perhaps missing some sort of default setting that 2.2.0 is looking for?
When I do a nodetool status it only reports as being up the machine
where I had issue the command.
In other words, all the machines cannot communicate to each other any
longer. Nodetool status behave the same on each machine.
I am trying to debug that, hopefully only something on the configuration
that has changed.
Any ideas?
Anything helpful in the system.log on each of your nodes?
Did you follow all the upgrade notes from your previous release to 2.2.0?
https://github.com/apache/cassandra/blob/cassandra-2.2.0/NEWS.txt
--
Kind regards,
Michael