Discussion:
Unable to create table
(too old to reply)
Hari Sreekumar
2011-03-28 13:29:27 UTC
Permalink
Hi,

I am trying to create table in hbase v0.90.1 and I get the following error:

11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket connection to
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection established
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment complete
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x12efc946d6600
0b
11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: 0x12efc946d66000b
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException: No
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while creating
table: Table1

This is the code I am using:
.....
HTableDescriptor desc = CreateTableByXML.convertSchemaToDescriptor(schema);
try {
hbaseAdmin.createTable(desc);
} catch (IOException e) {
CreateTableByXML.LOG.error("Caught IOException: " + e.getMessage()
+ " while creating table: " + tableName);
....
.....


It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it started
giving this error. Any ideas?

Hari
Stack
2011-03-28 15:29:32 UTC
Permalink
Can I see more of the stack track please Hari and is AcContact the
table you are creating? Is the schema you've saved aside one you
created with 0.20 hbase? I don't think it matters but asking just in
case.
St.Ack

On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket connection to
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection established
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment complete
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
Closed zookeeper sessionid=0x12efc946d6600
0b
11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: 0x12efc946d66000b
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException: No
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while creating
table: Table1
.....
HTableDescriptor desc = CreateTableByXML.convertSchemaToDescriptor(schema);
   try {
     hbaseAdmin.createTable(desc);
   } catch (IOException e) {
     CreateTableByXML.LOG.error("Caught IOException: " + e.getMessage()
             + " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it started
giving this error. Any ideas?
Hari
Hari Sreekumar
2011-03-28 18:55:06 UTC
Permalink
Hi Stack

yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just something
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do with
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack trace
asap

hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact the
table you are creating?  Is the schema you've saved aside one you
created with 0.20 hbase?  I don't think it matters but asking just in
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket connection to
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection established
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment complete
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
Closed zookeeper sessionid=0x12efc946d6600
0b
11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: 0x12efc946d66000b
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException: No
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while creating
table: Table1
.....
HTableDescriptor desc = CreateTableByXML.convertSchemaToDescriptor(schema);
   try {
     hbaseAdmin.createTable(desc);
   } catch (IOException e) {
     CreateTableByXML.LOG.error("Caught IOException: " + e.getMessage()
             + " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it started
giving this error. Any ideas?
Hari
Hari Sreekumar
2011-03-29 07:17:43 UTC
Permalink
Here is the stack trace:

11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection to
server hadoop2/192.168.1.111:2181
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection established
to hadoop2/192.168.1.111:2181, initiating session
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment complete
on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c,
negotiated timeout = 180000
11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries from
META. Retrying to create table
11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create table even
after Cleaning Meta entries
11/03/28 18:48:02 FATAL create.CreateTableByXML:
*************************************************************
11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
192.168.1.57:60000 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
11/03/28 18:48:02 FATAL create.CreateTableByXML:
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy4.createTable(Unknown Source)
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
Source)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)

The "CleanFromMeta" function catches IOException and deletes all rows from
.META. We had added this in the exception catch block because we used to
face the "Table taking too long to be disabled" exception often. It seems
the rows in META already get created when the IOException is thrown.
CleanFromMeta cleans .META. and then I try again to create the table, after
which I get the socket timeout exception.

I am using Hadoop v0.20.2, r911707. Can this be a reason for this error? I
get the message "You are currently running the HMaster without HDFS append
support enabled. This may result in data loss. Please see the HBase
wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
details." on the HBase Master UI.

On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi Stack
yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just something
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do with
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack trace
asap
hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact the
table you are creating? Is the schema you've saved aside one you
created with 0.20 hbase? I don't think it matters but asking just in
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
I am trying to create table in hbase v0.90.1 and I get the following
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket connection
to
Post by Stack
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Stack
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Stack
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO
Closed zookeeper sessionid=0x12efc946d6600
0b
11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: 0x12efc946d66000b
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException: No
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while
creating
Post by Stack
Post by Hari Sreekumar
table: Table1
.....
HTableDescriptor desc =
CreateTableByXML.convertSchemaToDescriptor(schema);
Post by Stack
Post by Hari Sreekumar
try {
hbaseAdmin.createTable(desc);
} catch (IOException e) {
CreateTableByXML.LOG.error("Caught IOException: " + e.getMessage()
+ " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it
started
Post by Stack
Post by Hari Sreekumar
giving this error. Any ideas?
Hari
Jean-Daniel Cryans
2011-03-29 18:00:20 UTC
Permalink
The 60 secs timeout means that the client was waiting on the master
for some operation but the master took longer than 60 secs to do it,
so its log should be the next place too look for something whack.

BTW deleting the rows from .META. directly is probably the worst thing
you can do.

J-D

On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection to
server hadoop2/192.168.1.111:2181
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection established
to hadoop2/192.168.1.111:2181, initiating session
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment complete
on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c,
negotiated timeout = 180000
11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries from
META. Retrying to create table
11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create table even
after Cleaning Meta entries
*************************************************************
11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy4.createTable(Unknown Source)
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
Source)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown Source)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
The "CleanFromMeta" function catches IOException and deletes all rows from
.META. We had added this in the exception catch block because we used to
face the "Table taking too long to be disabled" exception often. It seems
the rows in META already get created when the IOException is thrown.
CleanFromMeta cleans .META. and then I try again to create the table, after
which I get the socket timeout exception.
I am using Hadoop v0.20.2, r911707. Can this be a reason for this error? I
get the message "You are currently running the HMaster without HDFS append
support enabled. This may result in data loss. Please see the HBase
wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
details." on the HBase Master UI.
On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi Stack
yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just something
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do with
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack trace
asap
hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact the
table you are creating?  Is the schema you've saved aside one you
created with 0.20 hbase?  I don't think it matters but asking just in
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
I am trying to create table in hbase v0.90.1 and I get the following
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket connection
to
Post by Stack
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Stack
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Stack
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO
Closed zookeeper sessionid=0x12efc946d6600
0b
11/03/28 18:39:52 INFO zookeeper.ZooKeeper: Session: 0x12efc946d66000b
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught IOException: No
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while
creating
Post by Stack
Post by Hari Sreekumar
table: Table1
.....
HTableDescriptor desc =
CreateTableByXML.convertSchemaToDescriptor(schema);
Post by Stack
Post by Hari Sreekumar
   try {
     hbaseAdmin.createTable(desc);
   } catch (IOException e) {
     CreateTableByXML.LOG.error("Caught IOException: " + e.getMessage()
             + " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it
started
Post by Stack
Post by Hari Sreekumar
giving this error. Any ideas?
Hari
Hari Sreekumar
2011-03-29 18:29:18 UTC
Permalink
Hi J-D,

Here is the tail of HMaster log:

2011-03-29 23:48:51,155 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:52,158 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:53,161 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:54,163 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:55,164 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:56,166 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:57,168 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:58,170 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:59,172 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:00,174 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:01,176 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 ERROR
org.apache.hadoop.hbase.master.handler.TableEventHandler: Error manipulating
table AcContact
java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for
region to leave region
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions
at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

What is "clearing regions in transition"? What could be the issue?

I was told before too that deleting from META isn't recommended. But we were
facing the disable problem way too often. So we thought we'd make it
in-built. What other alternatives do I have?

Thx,
Hari
Post by Jean-Daniel Cryans
The 60 secs timeout means that the client was waiting on the master
for some operation but the master took longer than 60 secs to do it,
so its log should be the next place too look for something whack.
BTW deleting the rows from .META. directly is probably the worst thing
you can do.
J-D
On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection to
server hadoop2/192.168.1.111:2181
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating session
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c,
negotiated timeout = 180000
11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries
from
Post by Hari Sreekumar
META. Retrying to create table
11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create table
even
Post by Hari Sreekumar
after Cleaning Meta entries
*************************************************************
11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
java.nio.channels.SocketChannel[connected
Post by Hari Sreekumar
local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
Post by Hari Sreekumar
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy4.createTable(Unknown Source)
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
Post by Hari Sreekumar
Source)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
Source)
Post by Hari Sreekumar
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Post by Hari Sreekumar
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
The "CleanFromMeta" function catches IOException and deletes all rows
from
Post by Hari Sreekumar
.META. We had added this in the exception catch block because we used to
face the "Table taking too long to be disabled" exception often. It seems
the rows in META already get created when the IOException is thrown.
CleanFromMeta cleans .META. and then I try again to create the table,
after
Post by Hari Sreekumar
which I get the socket timeout exception.
I am using Hadoop v0.20.2, r911707. Can this be a reason for this error?
I
Post by Hari Sreekumar
get the message "You are currently running the HMaster without HDFS
append
Post by Hari Sreekumar
support enabled. This may result in data loss. Please see the HBase
wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
details." on the HBase Master UI.
On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi Stack
yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just something
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do with
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack trace
asap
hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact the
table you are creating? Is the schema you've saved aside one you
created with 0.20 hbase? I don't think it matters but asking just in
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
I am trying to create table in hbase v0.90.1 and I get the following
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
connection
Post by Hari Sreekumar
Post by Hari Sreekumar
to
Post by Stack
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Stack
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Stack
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO
Closed zookeeper sessionid=0x12efc946d6600
0b
0x12efc946d66000b
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
No
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while
creating
Post by Stack
Post by Hari Sreekumar
table: Table1
.....
HTableDescriptor desc =
CreateTableByXML.convertSchemaToDescriptor(schema);
Post by Stack
Post by Hari Sreekumar
try {
hbaseAdmin.createTable(desc);
} catch (IOException e) {
CreateTableByXML.LOG.error("Caught IOException: " +
e.getMessage()
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
+ " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it
started
Post by Stack
Post by Hari Sreekumar
giving this error. Any ideas?
Hari
Jean-Daniel Cryans
2011-03-29 18:51:48 UTC
Permalink
There's a reason why disabling takes time, if you delete rows from
.META. you might end up in an inconsistent situation and we'll have a
hard time helping you :)

So HBASE-3557 is what you want.

Regarding your current issue, RIT is "region in transition" meaning
that the region is in a state recognized by the master as either
moving from one region server to another, or just closing. Normally
after disabling a table its regions all left their in transition
state, but by deleting the rows directly in .META. who knows exactly
what's the state of your table?

J-D

On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi J-D,
2011-03-29 23:48:51,155 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:52,158 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:53,161 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:54,163 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:55,164 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:56,166 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:57,168 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:58,170 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:59,172 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:00,174 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:01,176 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 ERROR
org.apache.hadoop.hbase.master.handler.TableEventHandler: Error manipulating
table AcContact
java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for
region to leave region
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions
       at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
       at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
       at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:662)
What is "clearing regions in transition"? What could be the issue?
I was told before too that deleting from META isn't recommended. But we were
facing the disable problem way too often. So we thought we'd make it
in-built. What other alternatives do I have?
Thx,
Hari
Post by Jean-Daniel Cryans
The 60 secs timeout means that the client was waiting on the master
for some operation but the master took longer than 60 secs to do it,
so its log should be the next place too look for something whack.
BTW deleting the rows from .META. directly is probably the worst thing
you can do.
J-D
On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection to
server hadoop2/192.168.1.111:2181
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating session
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c,
negotiated timeout = 180000
11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries
from
Post by Hari Sreekumar
META. Retrying to create table
11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create table
even
Post by Hari Sreekumar
after Cleaning Meta entries
*************************************************************
11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
java.nio.channels.SocketChannel[connected
Post by Hari Sreekumar
local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
Post by Hari Sreekumar
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy4.createTable(Unknown Source)
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
Post by Hari Sreekumar
Source)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
Source)
Post by Hari Sreekumar
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Post by Hari Sreekumar
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
The "CleanFromMeta" function catches IOException and deletes all rows
from
Post by Hari Sreekumar
.META. We had added this in the exception catch block because we used to
face the "Table taking too long to be disabled" exception often. It seems
the rows in META already get created when the IOException is thrown.
CleanFromMeta cleans .META. and then I try again to create the table,
after
Post by Hari Sreekumar
which I get the socket timeout exception.
I am using Hadoop v0.20.2, r911707. Can this be a reason for this error?
I
Post by Hari Sreekumar
get the message "You are currently running the HMaster without HDFS
append
Post by Hari Sreekumar
support enabled. This may result in data loss. Please see the HBase
wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
details." on the HBase Master UI.
On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi Stack
yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just something
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do with
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack trace
asap
hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact the
table you are creating?  Is the schema you've saved aside one you
created with 0.20 hbase?  I don't think it matters but asking just in
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
I am trying to create table in hbase v0.90.1 and I get the following
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
connection
Post by Hari Sreekumar
Post by Hari Sreekumar
to
Post by Stack
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Stack
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Stack
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO
Closed zookeeper sessionid=0x12efc946d6600
0b
0x12efc946d66000b
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
No
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while
creating
Post by Stack
Post by Hari Sreekumar
table: Table1
.....
HTableDescriptor desc =
CreateTableByXML.convertSchemaToDescriptor(schema);
Post by Stack
Post by Hari Sreekumar
   try {
     hbaseAdmin.createTable(desc);
   } catch (IOException e) {
     CreateTableByXML.LOG.error("Caught IOException: " +
e.getMessage()
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
             + " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1 and it
started
Post by Stack
Post by Hari Sreekumar
giving this error. Any ideas?
Hari
Hari Sreekumar
2011-03-29 19:12:41 UTC
Permalink
Yep I know, I think I was also on the mailing list thread that inspired
HBASE-3557 :) But what can I do in the current version better than that?

In any case, I do that only when I catch IOException. So why am I getting
this IOException. I deleted the hbase folder in HDFS and tried doing
everything from start. Here is the HMaster log before the IOException was
thrown: http://pastebin.com/x1BUuPpQ

thanks,
Hari
Post by Jean-Daniel Cryans
There's a reason why disabling takes time, if you delete rows from
.META. you might end up in an inconsistent situation and we'll have a
hard time helping you :)
So HBASE-3557 is what you want.
Regarding your current issue, RIT is "region in transition" meaning
that the region is in a state recognized by the master as either
moving from one region server to another, or just closing. Normally
after disabling a table its regions all left their in transition
state, but by deleting the rows directly in .META. who knows exactly
what's the state of your table?
J-D
On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi J-D,
2011-03-29 23:48:51,155 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:52,158 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:53,161 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:54,163 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:55,164 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:56,166 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:57,168 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:58,170 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:59,172 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:00,174 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:01,176 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 ERROR
org.apache.hadoop.hbase.master.handler.TableEventHandler: Error
manipulating
Post by Hari Sreekumar
table AcContact
java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for
region to leave region
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions
at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
Post by Hari Sreekumar
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
Post by Hari Sreekumar
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
Post by Hari Sreekumar
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
Post by Hari Sreekumar
at java.lang.Thread.run(Thread.java:662)
What is "clearing regions in transition"? What could be the issue?
I was told before too that deleting from META isn't recommended. But we
were
Post by Hari Sreekumar
facing the disable problem way too often. So we thought we'd make it
in-built. What other alternatives do I have?
Thx,
Hari
On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans <
Post by Jean-Daniel Cryans
The 60 secs timeout means that the client was waiting on the master
for some operation but the master took longer than 60 secs to do it,
so its log should be the next place too look for something whack.
BTW deleting the rows from .META. directly is probably the worst thing
you can do.
J-D
On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client
connection,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection
to
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating session
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c,
negotiated timeout = 180000
11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries
from
Post by Hari Sreekumar
META. Retrying to create table
11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create
table
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
even
Post by Hari Sreekumar
after Cleaning Meta entries
*************************************************************
11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
java.net.SocketTimeoutException: 60000 millis timeout while waiting
for
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
java.nio.channels.SocketChannel[connected
Post by Hari Sreekumar
local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy4.createTable(Unknown Source)
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Source)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
Source)
Post by Hari Sreekumar
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
The "CleanFromMeta" function catches IOException and deletes all rows
from
Post by Hari Sreekumar
.META. We had added this in the exception catch block because we used
to
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
face the "Table taking too long to be disabled" exception often. It
seems
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
the rows in META already get created when the IOException is thrown.
CleanFromMeta cleans .META. and then I try again to create the table,
after
Post by Hari Sreekumar
which I get the socket timeout exception.
I am using Hadoop v0.20.2, r911707. Can this be a reason for this
error?
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
I
Post by Hari Sreekumar
get the message "You are currently running the HMaster without HDFS
append
Post by Hari Sreekumar
support enabled. This may result in data loss. Please see the HBase
wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
details." on the HBase Master UI.
On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi Stack
yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just something
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do with
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack trace
asap
hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact the
table you are creating? Is the schema you've saved aside one you
created with 0.20 hbase? I don't think it matters but asking just
in
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
I am trying to create table in hbase v0.90.1 and I get the
following
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
connection
Post by Hari Sreekumar
Post by Hari Sreekumar
to
Post by Stack
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Stack
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Stack
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO
Closed zookeeper sessionid=0x12efc946d6600
0b
0x12efc946d66000b
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught
No
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while
creating
Post by Stack
Post by Hari Sreekumar
table: Table1
.....
HTableDescriptor desc =
CreateTableByXML.convertSchemaToDescriptor(schema);
Post by Stack
Post by Hari Sreekumar
try {
hbaseAdmin.createTable(desc);
} catch (IOException e) {
CreateTableByXML.LOG.error("Caught IOException: " +
e.getMessage()
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
+ " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1 and
it
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
started
Post by Stack
Post by Hari Sreekumar
giving this error. Any ideas?
Hari
Jean-Daniel Cryans
2011-03-29 19:54:41 UTC
Permalink
hadoop3 was trying to open it, but it seems like it's not able to. It
really looks like: https://issues.apache.org/jira/browse/HBASE-3669

You should also check what's going on on that slave.

J-D

On Tue, Mar 29, 2011 at 12:12 PM, Hari Sreekumar
Post by Hari Sreekumar
Yep I know, I think I was also on the mailing list thread that inspired
HBASE-3557 :) But what can I do in the current version better than that?
In any case, I do that only when I catch IOException. So why am I getting
this IOException. I deleted the hbase folder in HDFS and tried doing
everything from start. Here is the HMaster log before the IOException was
thrown: http://pastebin.com/x1BUuPpQ
thanks,
Hari
Post by Jean-Daniel Cryans
There's a reason why disabling takes time, if you delete rows from
.META. you might end up in an inconsistent situation and we'll have a
hard time helping you :)
So HBASE-3557 is what you want.
Regarding your current issue, RIT is "region in transition" meaning
that the region is in a state recognized by the master as either
moving from one region server to another, or just closing. Normally
after disabling a table its regions all left their in transition
state, but by deleting the rows directly in .META. who knows exactly
what's the state of your table?
J-D
On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi J-D,
2011-03-29 23:48:51,155 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:52,158 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:53,161 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:54,163 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:55,164 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:56,166 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:57,168 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:58,170 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:48:59,172 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:00,174 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:01,176 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
 region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. state=OPENING,
ts=1301422405271
2011-03-29 23:49:02,178 ERROR
org.apache.hadoop.hbase.master.handler.TableEventHandler: Error
manipulating
Post by Hari Sreekumar
table AcContact
java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for
region to leave region
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in transitions
       at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
Post by Hari Sreekumar
       at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
Post by Hari Sreekumar
       at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
       at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
Post by Hari Sreekumar
       at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
Post by Hari Sreekumar
       at java.lang.Thread.run(Thread.java:662)
What is "clearing regions in transition"? What could be the issue?
I was told before too that deleting from META isn't recommended. But we
were
Post by Hari Sreekumar
facing the disable problem way too often. So we thought we'd make it
in-built. What other alternatives do I have?
Thx,
Hari
On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans <
Post by Jean-Daniel Cryans
The 60 secs timeout means that the client was waiting on the master
for some operation but the master took longer than 60 secs to do it,
so its log should be the next place too look for something whack.
BTW deleting the rows from .META. directly is probably the worst thing
you can do.
J-D
On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client
connection,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
connectString=hadoop2:2181 sessionTimeout=180000 watcher=hconnection
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket connection
to
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating session
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sessionid = 0x12efc946d66000c,
negotiated timeout = 180000
11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table entries
from
Post by Hari Sreekumar
META. Retrying to create table
11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create
table
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
even
Post by Hari Sreekumar
after Cleaning Meta entries
*************************************************************
11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
java.net.SocketTimeoutException: 60000 millis timeout while waiting
for
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
java.nio.channels.SocketChannel[connected
Post by Hari Sreekumar
local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
$Proxy4.createTable(Unknown Source)
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Source)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
Source)
Post by Hari Sreekumar
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
The "CleanFromMeta" function catches IOException and deletes all rows
from
Post by Hari Sreekumar
.META. We had added this in the exception catch block because we used
to
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
face the "Table taking too long to be disabled" exception often. It
seems
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
the rows in META already get created when the IOException is thrown.
CleanFromMeta cleans .META. and then I try again to create the table,
after
Post by Hari Sreekumar
which I get the socket timeout exception.
I am using Hadoop v0.20.2, r911707. Can this be a reason for this
error?
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
I
Post by Hari Sreekumar
get the message "You are currently running the HMaster without HDFS
append
Post by Hari Sreekumar
support enabled. This may result in data loss. Please see the HBase
wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
details." on the HBase Master UI.
On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi Stack
yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just something
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do with
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack trace
asap
hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact the
table you are creating?  Is the schema you've saved aside one you
created with 0.20 hbase?  I don't think it matters but asking just
in
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
I am trying to create table in hbase v0.90.1 and I get the
following
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
connection
Post by Hari Sreekumar
Post by Hari Sreekumar
to
Post by Stack
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Stack
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Stack
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO
Closed zookeeper sessionid=0x12efc946d6600
0b
0x12efc946d66000b
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut down
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught
No
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5. while
creating
Post by Stack
Post by Hari Sreekumar
table: Table1
.....
HTableDescriptor desc =
CreateTableByXML.convertSchemaToDescriptor(schema);
Post by Stack
Post by Hari Sreekumar
   try {
     hbaseAdmin.createTable(desc);
   } catch (IOException e) {
     CreateTableByXML.LOG.error("Caught IOException: " +
e.getMessage()
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
             + " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1 and
it
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
started
Post by Stack
Post by Hari Sreekumar
giving this error. Any ideas?
Hari
Hari Sreekumar
2011-03-30 04:42:40 UTC
Permalink
Ah, yea. I have this error in the regionserver log:

2011-03-30 10:09:26,534 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.63 MB,
free=196.71 MB, max=198.34 MB, blocks=0, accesses=0, hits=0, hitRatio=ᅵ%,
cachingAccesses=0, cachingHits=0, cachingHitsRatio=ᅵ%, evictions=0,
evicted=0, evictedPerRun=NaN
2011-03-30 10:09:37,328 DEBUG
org.apache.hadoop.hbase.regionserver.LogRoller: Hlog roll period 3600000ms
elapsed
2011-03-30 10:09:46,369 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open
region: AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
2011-03-30 10:09:46,369 DEBUG
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing
open of AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
2011-03-30 10:09:46,370 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x12f02e9bdec0007 Attempting to transition node
95652cfd0dab6dcf3be0c837c539f8d3 from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2011-03-30 10:09:46,372 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:60020-0x12f02e9bdec0007 Successfully transitioned node
95652cfd0dab6dcf3be0c837c539f8d3 from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
2011-03-30 10:09:46,372 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Opening region: REGION => {NAME =>
'AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.', STARTKEY =>
'', ENDKEY => '', ENCODED => 95652cfd0dab6dcf3be0c837c539f8d3, TABLE =>
{{NAME => 'AcContact', FAMILIES => [{NAME => 'Data', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS => '1000', TTL =>
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}]}}
2011-03-30 10:09:46,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Instantiated AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
2011-03-30 10:09:46,373 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of region=AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
java.io.IOException: Compression algorithm 'lzo' previously failed test.
at
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:2555)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2544)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:262)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:94)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Will try to fix LZO here and let you know..
Post by Jean-Daniel Cryans
hadoop3 was trying to open it, but it seems like it's not able to. It
really looks like: https://issues.apache.org/jira/browse/HBASE-3669
You should also check what's going on on that slave.
J-D
On Tue, Mar 29, 2011 at 12:12 PM, Hari Sreekumar
Post by Hari Sreekumar
Yep I know, I think I was also on the mailing list thread that inspired
HBASE-3557 :) But what can I do in the current version better than that?
In any case, I do that only when I catch IOException. So why am I getting
this IOException. I deleted the hbase folder in HDFS and tried doing
everything from start. Here is the HMaster log before the IOException was
thrown: http://pastebin.com/x1BUuPpQ
thanks,
Hari
On Wed, Mar 30, 2011 at 12:21 AM, Jean-Daniel Cryans <
Post by Jean-Daniel Cryans
There's a reason why disabling takes time, if you delete rows from
.META. you might end up in an inconsistent situation and we'll have a
hard time helping you :)
So HBASE-3557 is what you want.
Regarding your current issue, RIT is "region in transition" meaning
that the region is in a state recognized by the master as either
moving from one region server to another, or just closing. Normally
after disabling a table its regions all left their in transition
state, but by deleting the rows directly in .META. who knows exactly
what's the state of your table?
J-D
On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi J-D,
2011-03-29 23:48:51,155 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:52,158 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:53,161 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:54,163 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:55,164 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:56,166 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:57,168 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:58,170 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:59,172 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:49:00,174 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:49:01,176 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:49:02,178 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:49:02,178 ERROR
org.apache.hadoop.hbase.master.handler.TableEventHandler: Error
manipulating
Post by Hari Sreekumar
table AcContact
java.io.IOException: Waited hbase.master.wait.on.region (300000ms) for
region to leave region
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in
transitions
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at java.lang.Thread.run(Thread.java:662)
What is "clearing regions in transition"? What could be the issue?
I was told before too that deleting from META isn't recommended. But
we
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
were
Post by Hari Sreekumar
facing the disable problem way too often. So we thought we'd make it
in-built. What other alternatives do I have?
Thx,
Hari
On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans <
Post by Jean-Daniel Cryans
The 60 secs timeout means that the client was waiting on the master
for some operation but the master took longer than 60 secs to do it,
so its log should be the next place too look for something whack.
BTW deleting the rows from .META. directly is probably the worst
thing
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
you can do.
J-D
On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client
connection,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
connectString=hadoop2:2181 sessionTimeout=180000
watcher=hconnection
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket
connection
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
to
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating session
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sessionid =
0x12efc946d66000c,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
negotiated timeout = 180000
11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table
entries
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
from
Post by Hari Sreekumar
META. Retrying to create table
11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create
table
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
even
Post by Hari Sreekumar
after Cleaning Meta entries
*************************************************************
11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
java.net.SocketTimeoutException: 60000 millis timeout while waiting
for
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
java.nio.channels.SocketChannel[connected
Post by Hari Sreekumar
local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
$Proxy4.createTable(Unknown Source)
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Source)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
Source)
Post by Hari Sreekumar
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
The "CleanFromMeta" function catches IOException and deletes all
rows
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
from
Post by Hari Sreekumar
.META. We had added this in the exception catch block because we
used
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
to
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
face the "Table taking too long to be disabled" exception often. It
seems
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
the rows in META already get created when the IOException is
thrown.
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
CleanFromMeta cleans .META. and then I try again to create the
table,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
after
Post by Hari Sreekumar
which I get the socket timeout exception.
I am using Hadoop v0.20.2, r911707. Can this be a reason for this
error?
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
I
Post by Hari Sreekumar
get the message "You are currently running the HMaster without HDFS
append
Post by Hari Sreekumar
support enabled. This may result in data loss. Please see the HBase
wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
details." on the HBase Master UI.
On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi Stack
yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just
something
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do
with
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack
trace
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
asap
hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact
the
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
table you are creating? Is the schema you've saved aside one
you
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
created with 0.20 hbase? I don't think it matters but asking
just
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
in
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
I am trying to create table in hbase v0.90.1 and I get the
following
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
connection
Post by Hari Sreekumar
Post by Hari Sreekumar
to
Post by Stack
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Stack
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session
establishment
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
complete
Post by Stack
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO
Closed zookeeper sessionid=0x12efc946d6600
0b
0x12efc946d66000b
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut
down
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught
No
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
while
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
creating
Post by Stack
Post by Hari Sreekumar
table: Table1
.....
HTableDescriptor desc =
CreateTableByXML.convertSchemaToDescriptor(schema);
Post by Stack
Post by Hari Sreekumar
try {
hbaseAdmin.createTable(desc);
} catch (IOException e) {
CreateTableByXML.LOG.error("Caught IOException: " +
e.getMessage()
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
+ " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1
and
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
it
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
started
Post by Stack
Post by Hari Sreekumar
giving this error. Any ideas?
Hari
Hari Sreekumar
2011-03-30 05:45:58 UTC
Permalink
The Lzo compression test works when I run it through hadoop but not when I
run it through hbase:

[***@hadoop1 opt]$ $HADOOP_HOME/bin/hadoop
org.apache.hadoop.hbase.util.CompressionTest hdfs://hadoop1:54310 lzo
11/03/30 10:18:26 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
11/03/30 10:18:26 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library
11/03/30 10:18:26 INFO compress.CodecPool: Got brand-new compressor
[***@hadoop1 opt]$ $HBASE_HOME/bin/hbase
org.apache.hadoop.hbase.util.CompressionTest hdfs://hadoop1:54310 lzo
11/03/30 10:19:48 ERROR lzo.GPLNativeCodeLoader: Could not load native gpl
library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1734)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1028)
at
com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31)
at
com.hadoop.compression.lzo.LzoCodec.isNativeLzoLoaded(LzoCodec.java:69)
at
com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:146)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:98)
at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:200)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.getCompressingStream(HFile.java:397)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.newBlock(HFile.java:383)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.checkBlockBoundary(HFile.java:354)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:536)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:515)
at
org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:126)
11/03/30 10:19:48 ERROR lzo.LzoCodec: Cannot load native-lzo without
native-hadoop
java.lang.RuntimeException: native-lzo library not available
at
com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:147)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:98)
at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:200)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.getCompressingStream(HFile.java:397)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.newBlock(HFile.java:383)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.checkBlockBoundary(HFile.java:354)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:536)
at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:515)
at
org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:126)
FAILED

I had the native libs added in $HADOOP_HOME/lib only and not in
$HBASE_HOME/lib because I had not created the directory
$HBASE_HOME/lib/native. Now I have them in HBase too. Also, earlier I was
building the native libs in the master node only and rsyncing the
$HADOOP_HOME/lib to all nodes. Now I am building individually on all nodes
(not sure if this made any difference). Works now.

Thanks a lot for the help,
Hari


On Wed, Mar 30, 2011 at 10:12 AM, Hari Sreekumar
Post by Hari Sreekumar
2011-03-30 10:09:26,534 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.63 MB,
free=196.71 MB, max=198.34 MB, blocks=0, accesses=0, hits=0, hitRatio=ᅵ%,
cachingAccesses=0, cachingHits=0, cachingHitsRatio=ᅵ%, evictions=0,
evicted=0, evictedPerRun=NaN
2011-03-30 10:09:37,328 DEBUG
org.apache.hadoop.hbase.regionserver.LogRoller: Hlog roll period 3600000ms
elapsed
2011-03-30 10:09:46,369 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open
region: AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
2011-03-30 10:09:46,369 DEBUG
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing
open of AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
regionserver:60020-0x12f02e9bdec0007 Attempting to transition node
95652cfd0dab6dcf3be0c837c539f8d3 from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
regionserver:60020-0x12f02e9bdec0007 Successfully transitioned node
95652cfd0dab6dcf3be0c837c539f8d3 from M_ZK_REGION_OFFLINE to
RS_ZK_REGION_OPENING
Opening region: REGION => {NAME =>
'AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.', STARTKEY =>
'', ENDKEY => '', ENCODED => 95652cfd0dab6dcf3be0c837c539f8d3, TABLE =>
{{NAME => 'AcContact', FAMILIES => [{NAME => 'Data', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS => '1000', TTL =>
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}]}}
Instantiated AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
2011-03-30 10:09:46,373 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of region=AcContact,,1301424009632.95652cfd0dab6dcf3be0c837c539f8d3.
java.io.IOException: Compression algorithm 'lzo' previously failed test.
at
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:77)
at
org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:2555)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2544)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:2532)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:262)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:94)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Will try to fix LZO here and let you know..
Post by Jean-Daniel Cryans
hadoop3 was trying to open it, but it seems like it's not able to. It
really looks like: https://issues.apache.org/jira/browse/HBASE-3669
You should also check what's going on on that slave.
J-D
On Tue, Mar 29, 2011 at 12:12 PM, Hari Sreekumar
Post by Hari Sreekumar
Yep I know, I think I was also on the mailing list thread that inspired
HBASE-3557 :) But what can I do in the current version better than that?
In any case, I do that only when I catch IOException. So why am I
getting
Post by Hari Sreekumar
this IOException. I deleted the hbase folder in HDFS and tried doing
everything from start. Here is the HMaster log before the IOException
was
Post by Hari Sreekumar
thrown: http://pastebin.com/x1BUuPpQ
thanks,
Hari
On Wed, Mar 30, 2011 at 12:21 AM, Jean-Daniel Cryans <
Post by Jean-Daniel Cryans
There's a reason why disabling takes time, if you delete rows from
.META. you might end up in an inconsistent situation and we'll have a
hard time helping you :)
So HBASE-3557 is what you want.
Regarding your current issue, RIT is "region in transition" meaning
that the region is in a state recognized by the master as either
moving from one region server to another, or just closing. Normally
after disabling a table its regions all left their in transition
state, but by deleting the rows directly in .META. who knows exactly
what's the state of your table?
J-D
On Tue, Mar 29, 2011 at 11:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi J-D,
2011-03-29 23:48:51,155 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:52,158 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:53,161 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:54,163 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:55,164 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:56,166 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:57,168 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:58,170 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:48:59,172 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:49:00,174 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:49:01,176 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:49:02,178 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Waiting on
region to clear regions in transition;
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215.
state=OPENING,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
ts=1301422405271
2011-03-29 23:49:02,178 ERROR
org.apache.hadoop.hbase.master.handler.TableEventHandler: Error
manipulating
Post by Hari Sreekumar
table AcContact
java.io.IOException: Waited hbase.master.wait.on.region (300000ms)
for
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
region to leave region
AcContact,,1301416789483.0cd6d132b2f367f21e88f00778349215. in
transitions
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
org.apache.hadoop.hbase.master.handler.DeleteTableHandler.handleTableOperation(DeleteTableHandler.java:60)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
org.apache.hadoop.hbase.master.handler.TableEventHandler.process(TableEventHandler.java:66)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
at java.lang.Thread.run(Thread.java:662)
What is "clearing regions in transition"? What could be the issue?
I was told before too that deleting from META isn't recommended. But
we
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
were
Post by Hari Sreekumar
facing the disable problem way too often. So we thought we'd make it
in-built. What other alternatives do I have?
Thx,
Hari
On Tue, Mar 29, 2011 at 11:30 PM, Jean-Daniel Cryans <
Post by Jean-Daniel Cryans
The 60 secs timeout means that the client was waiting on the master
for some operation but the master took longer than 60 secs to do it,
so its log should be the next place too look for something whack.
BTW deleting the rows from .META. directly is probably the worst
thing
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
you can do.
J-D
On Tue, Mar 29, 2011 at 12:17 AM, Hari Sreekumar
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ZooKeeper: Initiating client
connection,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
connectString=hadoop2:2181 sessionTimeout=180000
watcher=hconnection
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Opening socket
connection
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
to
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating session
11/03/28 18:47:02 INFO zookeeper.ClientCnxn: Session establishment
complete
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sessionid =
0x12efc946d66000c,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
negotiated timeout = 180000
11/03/28 18:47:02 INFO tools.CleanFromMeta: Deleting row
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
11/03/28 18:47:02 INFO create.CreateTableByXML: Cleaned table
entries
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
from
Post by Hari Sreekumar
META. Retrying to create table
11/03/28 18:48:02 FATAL create.CreateTableByXML: Could not create
table
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
even
Post by Hari Sreekumar
after Cleaning Meta entries
*************************************************************
11/03/28 18:48:02 FATAL create.CreateTableByXML: Call to hadoop3/
java.net.SocketTimeoutException: 60000 millis timeout while
waiting
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
for
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
java.nio.channels.SocketChannel[connected
Post by Hari Sreekumar
local=/192.168.0.233:51525 remote=hadoop3/192.168.1.57:60000]
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:802)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:775)
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
$Proxy4.createTable(Unknown Source)
org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:341)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:303)
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:227)
com.clickable.dataengine.hbase.create.CreateTableByXML.createTable(Unknown
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Source)
com.clickable.dataengine.hbase.create.CreateTableByXML.main(Unknown
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Source)
Post by Hari Sreekumar
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
The "CleanFromMeta" function catches IOException and deletes all
rows
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
from
Post by Hari Sreekumar
.META. We had added this in the exception catch block because we
used
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
to
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
face the "Table taking too long to be disabled" exception often.
It
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
seems
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
the rows in META already get created when the IOException is
thrown.
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
CleanFromMeta cleans .META. and then I try again to create the
table,
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
after
Post by Hari Sreekumar
which I get the socket timeout exception.
I am using Hadoop v0.20.2, r911707. Can this be a reason for this
error?
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
I
Post by Hari Sreekumar
get the message "You are currently running the HMaster without
HDFS
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
append
Post by Hari Sreekumar
support enabled. This may result in data loss. Please see the
HBase
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
wiki<http://wiki.apache.org/hadoop/Hbase/HdfsSyncSupport> for
details." on the HBase Master UI.
On Tue, Mar 29, 2011 at 12:25 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi Stack
yes the tablename is AcContact. The tableName variable was wrong.
Fixed it now but I still get the same error. Schema is just
something
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
created by parsing an XML file which has stuff like column family
name, compression type etc so I guess it doesn't have much to do
with
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
version. Except that I had to change the bloom filter variable to
String ( used to be boolean in 0.20.6). I will paste the stack
trace
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
asap
hari
Post by Stack
Can I see more of the stack track please Hari and is AcContact
the
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
table you are creating? Is the schema you've saved aside one
you
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
created with 0.20 hbase? I don't think it matters but asking
just
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
in
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
case.
St.Ack
On Mon, Mar 28, 2011 at 6:29 AM, Hari Sreekumar
Post by Hari Sreekumar
Hi,
I am trying to create table in hbase v0.90.1 and I get the
following
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Opening socket
connection
Post by Hari Sreekumar
Post by Hari Sreekumar
to
Post by Stack
Post by Hari Sreekumar
server hadoop2/192.168.1.111:2181
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Socket connection
established
Post by Stack
Post by Hari Sreekumar
to hadoop2/192.168.1.111:2181, initiating s
ession
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: Session
establishment
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
complete
Post by Stack
Post by Hari Sreekumar
on server hadoop2/192.168.1.111:2181, sess
ionid = 0x12efc946d66000b, negotiated timeout = 180000
11/03/28 18:39:52 INFO
Closed zookeeper sessionid=0x12efc946d6600
0b
0x12efc946d66000b
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
closed
11/03/28 18:39:52 INFO zookeeper.ClientCnxn: EventThread shut
down
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
11/03/28 18:47:02 ERROR create.CreateTableByXML: Caught
No
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
server address listed in .META. for region
AcContact,,1301317792604.16d1f5fd49478f79002e89ce02cf37b5.
while
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
creating
Post by Stack
Post by Hari Sreekumar
table: Table1
.....
HTableDescriptor desc =
CreateTableByXML.convertSchemaToDescriptor(schema);
Post by Stack
Post by Hari Sreekumar
try {
hbaseAdmin.createTable(desc);
} catch (IOException e) {
CreateTableByXML.LOG.error("Caught IOException: " +
e.getMessage()
Post by Hari Sreekumar
Post by Hari Sreekumar
Post by Stack
Post by Hari Sreekumar
+ " while creating table: " + tableName);
....
.....
It used to work fine in v0.20.6. I upgraded today to v0.90.1
and
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
it
Post by Hari Sreekumar
Post by Jean-Daniel Cryans
Post by Hari Sreekumar
Post by Hari Sreekumar
started
Post by Stack
Post by Hari Sreekumar
giving this error. Any ideas?
Hari
Continue reading on narkive:
Loading...