Discussion:
disable table
(too old to reply)
Mohit Anchlia
2012-09-26 20:54:31 UTC
Permalink
When I try to disable table I get:

hbase(main):011:0> disable 'SESSIONID_TIMELINE'
ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: SESSIONID_TIMELINE
Here is some help for this command:
Start disable of named table: e.g. "hbase> disable 't1'"

But then I try to enable I get:

hbase(main):012:0> enable 'SESSIONID_TIMELINE'
ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
org.apache.hadoop.hbase.TableNotDisabledException: SESSIONID_TIMELINE
Here is some help for this command:
Start enable of named table: e.g. "hbase> enable 't1'"

I've tried flush, major_compaction also. I tseems it's stuck in
inconsistent state. Could someone point me to correct direction? I am using
92.1
Mohammad Tariq
2012-09-26 21:19:51 UTC
Permalink
Hello Mohit,

Try hbck once and see if it shows any inconsistency. Also, you can try
restarting your cluster and deleting the table again. Having a look at the
logs could also be useful.

Regards,
Mohammad Tariq



On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org>wrote:

> When I try to disable table I get:
>
> hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> org.apache.hadoop.hbase.TableNotEnabledException: SESSIONID_TIMELINE
> Here is some help for this command:
> Start disable of named table: e.g. "hbase> disable 't1'"
>
> But then I try to enable I get:
>
> hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> org.apache.hadoop.hbase.TableNotDisabledException: SESSIONID_TIMELINE
> Here is some help for this command:
> Start enable of named table: e.g. "hbase> enable 't1'"
>
> I've tried flush, major_compaction also. I tseems it's stuck in
> inconsistent state. Could someone point me to correct direction? I am using
> 92.1
>
Mohit Anchlia
2012-09-26 21:25:39 UTC
Permalink
Which node should I look at for logs? Is this the master node? I'll try
hbck.

On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <dontariq-***@public.gmane.org> wrote:

> Hello Mohit,
>
> Try hbck once and see if it shows any inconsistency. Also, you can try
> restarting your cluster and deleting the table again. Having a look at the
> logs could also be useful.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org
> >wrote:
>
> > When I try to disable table I get:
> >
> > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> > org.apache.hadoop.hbase.TableNotEnabledException: SESSIONID_TIMELINE
> > Here is some help for this command:
> > Start disable of named table: e.g. "hbase> disable 't1'"
> >
> > But then I try to enable I get:
> >
> > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> > org.apache.hadoop.hbase.TableNotDisabledException: SESSIONID_TIMELINE
> > Here is some help for this command:
> > Start enable of named table: e.g. "hbase> enable 't1'"
> >
> > I've tried flush, major_compaction also. I tseems it's stuck in
> > inconsistent state. Could someone point me to correct direction? I am
> using
> > 92.1
> >
>
Mohammad Tariq
2012-09-26 21:29:13 UTC
Permalink
Yes. Also have a look at the logs of the problematic region if hbck shows
any inconsistency.

Regards,
Mohammad Tariq



On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org>wrote:

> Which node should I look at for logs? Is this the master node? I'll try
> hbck.
>
> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <dontariq-***@public.gmane.org>
> wrote:
>
> > Hello Mohit,
> >
> > Try hbck once and see if it shows any inconsistency. Also, you can
> try
> > restarting your cluster and deleting the table again. Having a look at
> the
> > logs could also be useful.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> >
> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org
> > >wrote:
> >
> > > When I try to disable table I get:
> > >
> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> > > org.apache.hadoop.hbase.TableNotEnabledException: SESSIONID_TIMELINE
> > > Here is some help for this command:
> > > Start disable of named table: e.g. "hbase> disable 't1'"
> > >
> > > But then I try to enable I get:
> > >
> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> > > org.apache.hadoop.hbase.TableNotDisabledException: SESSIONID_TIMELINE
> > > Here is some help for this command:
> > > Start enable of named table: e.g. "hbase> enable 't1'"
> > >
> > > I've tried flush, major_compaction also. I tseems it's stuck in
> > > inconsistent state. Could someone point me to correct direction? I am
> > using
> > > 92.1
> > >
> >
>
Mohammad Tariq
2012-09-26 21:36:00 UTC
Permalink
A possible reason could be that the znode associated with this particular
table is not behaving properly. In such case, you can try the following:

Stop Hbase
Stop ZK
Take a backup of ZK data
Restart ZK
Remove the znode
Start Hbase again

After this hopefully your table would be enabled.

Regards,
Mohammad Tariq



On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq <dontariq-***@public.gmane.org> wrote:

> Yes. Also have a look at the logs of the problematic region if hbck shows
> any inconsistency.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org>wrote:
>
>> Which node should I look at for logs? Is this the master node? I'll try
>> hbck.
>>
>> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <dontariq-***@public.gmane.org>
>> wrote:
>>
>> > Hello Mohit,
>> >
>> > Try hbck once and see if it shows any inconsistency. Also, you can
>> try
>> > restarting your cluster and deleting the table again. Having a look at
>> the
>> > logs could also be useful.
>> >
>> > Regards,
>> > Mohammad Tariq
>> >
>> >
>> >
>> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org
>> > >wrote:
>> >
>> > > When I try to disable table I get:
>> > >
>> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
>> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
>> > > org.apache.hadoop.hbase.TableNotEnabledException: SESSIONID_TIMELINE
>> > > Here is some help for this command:
>> > > Start disable of named table: e.g. "hbase> disable 't1'"
>> > >
>> > > But then I try to enable I get:
>> > >
>> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
>> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
>> > > org.apache.hadoop.hbase.TableNotDisabledException: SESSIONID_TIMELINE
>> > > Here is some help for this command:
>> > > Start enable of named table: e.g. "hbase> enable 't1'"
>> > >
>> > > I've tried flush, major_compaction also. I tseems it's stuck in
>> > > inconsistent state. Could someone point me to correct direction? I am
>> > using
>> > > 92.1
>> > >
>> >
>>
>
>
Mohit Anchlia
2012-09-26 22:09:20 UTC
Permalink
Thanks! I do see Inconsistency. How do I remove the znode. And also could
you please help me understand how this might have happened?


ERROR: Region
SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. not
deployed on any region server.


On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq <dontariq-***@public.gmane.org> wrote:

> A possible reason could be that the znode associated with this particular
> table is not behaving properly. In such case, you can try the following:
>
> Stop Hbase
> Stop ZK
> Take a backup of ZK data
> Restart ZK
> Remove the znode
> Start Hbase again
>
> After this hopefully your table would be enabled.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq <dontariq-***@public.gmane.org>
> wrote:
>
> > Yes. Also have a look at the logs of the problematic region if hbck shows
> > any inconsistency.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> >
> > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org
> >wrote:
> >
> >> Which node should I look at for logs? Is this the master node? I'll try
> >> hbck.
> >>
> >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <dontariq-***@public.gmane.org>
> >> wrote:
> >>
> >> > Hello Mohit,
> >> >
> >> > Try hbck once and see if it shows any inconsistency. Also, you can
> >> try
> >> > restarting your cluster and deleting the table again. Having a look at
> >> the
> >> > logs could also be useful.
> >> >
> >> > Regards,
> >> > Mohammad Tariq
> >> >
> >> >
> >> >
> >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> mohitanchlia-***@public.gmane.org
> >> > >wrote:
> >> >
> >> > > When I try to disable table I get:
> >> > >
> >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> >> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> >> > > org.apache.hadoop.hbase.TableNotEnabledException: SESSIONID_TIMELINE
> >> > > Here is some help for this command:
> >> > > Start disable of named table: e.g. "hbase> disable 't1'"
> >> > >
> >> > > But then I try to enable I get:
> >> > >
> >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> >> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> SESSIONID_TIMELINE
> >> > > Here is some help for this command:
> >> > > Start enable of named table: e.g. "hbase> enable 't1'"
> >> > >
> >> > > I've tried flush, major_compaction also. I tseems it's stuck in
> >> > > inconsistent state. Could someone point me to correct direction? I
> am
> >> > using
> >> > > 92.1
> >> > >
> >> >
> >>
> >
> >
>
Mohammad Tariq
2012-09-26 22:57:50 UTC
Permalink
In order to delete a znode you have to go to the ZK shell and issue the
"delete" command along with the required path. For example :
delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit the ZK
homepage at : zookeeper.apache.org

Actually when we try to fetch data from an Hbase table, the client or app
first contacts the ZK to get the location of server holding the
-ROOT- table. From this we come to know about the server hosting the .META.
table. This tells us the location of the server which actually holds the
rows of interest. Because of some reasons the znode which was holding this
info has either faced some catastrophe or lost the info associated with
this particular table. Or sometimes the znode remains unable to keep itself
updated with the latest changes. That could also be a probable reason. We
should always keep in mind that ZK is the centralized service that actually
coordinating everything behind the scene. As a result, any problem to the
ZK quorum means problem with Hbase custer.

Regards,
Mohammad Tariq



On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org>wrote:

> Thanks! I do see Inconsistency. How do I remove the znode. And also could
> you please help me understand how this might have happened?
>
>
> ERROR: Region
> SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. not
> deployed on any region server.
>
>
> On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq <dontariq-***@public.gmane.org>
> wrote:
>
> > A possible reason could be that the znode associated with this particular
> > table is not behaving properly. In such case, you can try the following:
> >
> > Stop Hbase
> > Stop ZK
> > Take a backup of ZK data
> > Restart ZK
> > Remove the znode
> > Start Hbase again
> >
> > After this hopefully your table would be enabled.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> >
> > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq <dontariq-***@public.gmane.org>
> > wrote:
> >
> > > Yes. Also have a look at the logs of the problematic region if hbck
> shows
> > > any inconsistency.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > >
> > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org
> > >wrote:
> > >
> > >> Which node should I look at for logs? Is this the master node? I'll
> try
> > >> hbck.
> > >>
> > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <dontariq-***@public.gmane.org>
> > >> wrote:
> > >>
> > >> > Hello Mohit,
> > >> >
> > >> > Try hbck once and see if it shows any inconsistency. Also, you
> can
> > >> try
> > >> > restarting your cluster and deleting the table again. Having a look
> at
> > >> the
> > >> > logs could also be useful.
> > >> >
> > >> > Regards,
> > >> > Mohammad Tariq
> > >> >
> > >> >
> > >> >
> > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> > mohitanchlia-***@public.gmane.org
> > >> > >wrote:
> > >> >
> > >> > > When I try to disable table I get:
> > >> > >
> > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> > >> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> SESSIONID_TIMELINE
> > >> > > Here is some help for this command:
> > >> > > Start disable of named table: e.g. "hbase> disable 't1'"
> > >> > >
> > >> > > But then I try to enable I get:
> > >> > >
> > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > >> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> > SESSIONID_TIMELINE
> > >> > > Here is some help for this command:
> > >> > > Start enable of named table: e.g. "hbase> enable 't1'"
> > >> > >
> > >> > > I've tried flush, major_compaction also. I tseems it's stuck in
> > >> > > inconsistent state. Could someone point me to correct direction? I
> > am
> > >> > using
> > >> > > 92.1
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Mohit Anchlia
2012-09-26 23:27:23 UTC
Permalink
I don't see path like /hbase/SESSIONID_TIMELINE
This is what I see

[zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
[SESSIONID_TIMELINE]
[zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table

cZxid = 0x1000000fe
ctime = Mon Sep 10 15:31:45 PDT 2012
mZxid = 0x1000000fe
mtime = Mon Sep 10 15:31:45 PDT 2012
pZxid = 0x5000008f1
cversion = 3
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 1

On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq <dontariq-***@public.gmane.org> wrote:

> In order to delete a znode you have to go to the ZK shell and issue the
> "delete" command along with the required path. For example :
> delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit the ZK
> homepage at : zookeeper.apache.org
>
> Actually when we try to fetch data from an Hbase table, the client or app
> first contacts the ZK to get the location of server holding the
> -ROOT- table. From this we come to know about the server hosting the .META.
> table. This tells us the location of the server which actually holds the
> rows of interest. Because of some reasons the znode which was holding this
> info has either faced some catastrophe or lost the info associated with
> this particular table. Or sometimes the znode remains unable to keep itself
> updated with the latest changes. That could also be a probable reason. We
> should always keep in mind that ZK is the centralized service that actually
> coordinating everything behind the scene. As a result, any problem to the
> ZK quorum means problem with Hbase custer.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org
> >wrote:
>
> > Thanks! I do see Inconsistency. How do I remove the znode. And also could
> > you please help me understand how this might have happened?
> >
> >
> > ERROR: Region
> > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. not
> > deployed on any region server.
> >
> >
> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq <dontariq-***@public.gmane.org>
> > wrote:
> >
> > > A possible reason could be that the znode associated with this
> particular
> > > table is not behaving properly. In such case, you can try the
> following:
> > >
> > > Stop Hbase
> > > Stop ZK
> > > Take a backup of ZK data
> > > Restart ZK
> > > Remove the znode
> > > Start Hbase again
> > >
> > > After this hopefully your table would be enabled.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > >
> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq <dontariq-***@public.gmane.org>
> > > wrote:
> > >
> > > > Yes. Also have a look at the logs of the problematic region if hbck
> > shows
> > > > any inconsistency.
> > > >
> > > > Regards,
> > > > Mohammad Tariq
> > > >
> > > >
> > > >
> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
> mohitanchlia-***@public.gmane.org
> > > >wrote:
> > > >
> > > >> Which node should I look at for logs? Is this the master node? I'll
> > try
> > > >> hbck.
> > > >>
> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <dontariq-***@public.gmane.org
> >
> > > >> wrote:
> > > >>
> > > >> > Hello Mohit,
> > > >> >
> > > >> > Try hbck once and see if it shows any inconsistency. Also, you
> > can
> > > >> try
> > > >> > restarting your cluster and deleting the table again. Having a
> look
> > at
> > > >> the
> > > >> > logs could also be useful.
> > > >> >
> > > >> > Regards,
> > > >> > Mohammad Tariq
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> > > mohitanchlia-***@public.gmane.org
> > > >> > >wrote:
> > > >> >
> > > >> > > When I try to disable table I get:
> > > >> > >
> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> > SESSIONID_TIMELINE
> > > >> > > Here is some help for this command:
> > > >> > > Start disable of named table: e.g. "hbase> disable 't1'"
> > > >> > >
> > > >> > > But then I try to enable I get:
> > > >> > >
> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> > > SESSIONID_TIMELINE
> > > >> > > Here is some help for this command:
> > > >> > > Start enable of named table: e.g. "hbase> enable 't1'"
> > > >> > >
> > > >> > > I've tried flush, major_compaction also. I tseems it's stuck in
> > > >> > > inconsistent state. Could someone point me to correct
> direction? I
> > > am
> > > >> > using
> > > >> > > 92.1
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Mohit Anchlia
2012-09-26 23:39:22 UTC
Permalink
I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll restart
hbase and see if it works.

One thing I don't understand is why is zookeeper holding information about
this table if it is enabled or disabled? Wouldn't this information be with
master?

On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia <mohitanchlia-***@public.gmane.org>wrote:

> I don't see path like /hbase/SESSIONID_TIMELINE
> This is what I see
>
> [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> [SESSIONID_TIMELINE]
> [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
>
> cZxid = 0x1000000fe
> ctime = Mon Sep 10 15:31:45 PDT 2012
> mZxid = 0x1000000fe
> mtime = Mon Sep 10 15:31:45 PDT 2012
> pZxid = 0x5000008f1
> cversion = 3
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 0
> numChildren = 1
>
> On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq <dontariq-***@public.gmane.org>wrote:
>
>> In order to delete a znode you have to go to the ZK shell and issue the
>> "delete" command along with the required path. For example :
>> delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit the ZK
>> homepage at : zookeeper.apache.org
>>
>> Actually when we try to fetch data from an Hbase table, the client or app
>> first contacts the ZK to get the location of server holding the
>> -ROOT- table. From this we come to know about the server hosting the
>> .META.
>> table. This tells us the location of the server which actually holds the
>> rows of interest. Because of some reasons the znode which was holding this
>> info has either faced some catastrophe or lost the info associated with
>> this particular table. Or sometimes the znode remains unable to keep
>> itself
>> updated with the latest changes. That could also be a probable reason. We
>> should always keep in mind that ZK is the centralized service that
>> actually
>> coordinating everything behind the scene. As a result, any problem to the
>> ZK quorum means problem with Hbase custer.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia <mohitanchlia-***@public.gmane.org
>> >wrote:
>>
>> > Thanks! I do see Inconsistency. How do I remove the znode. And also
>> could
>> > you please help me understand how this might have happened?
>> >
>> >
>> > ERROR: Region
>> > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. not
>> > deployed on any region server.
>> >
>> >
>> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq <dontariq-***@public.gmane.org>
>> > wrote:
>> >
>> > > A possible reason could be that the znode associated with this
>> particular
>> > > table is not behaving properly. In such case, you can try the
>> following:
>> > >
>> > > Stop Hbase
>> > > Stop ZK
>> > > Take a backup of ZK data
>> > > Restart ZK
>> > > Remove the znode
>> > > Start Hbase again
>> > >
>> > > After this hopefully your table would be enabled.
>> > >
>> > > Regards,
>> > > Mohammad Tariq
>> > >
>> > >
>> > >
>> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq <dontariq-***@public.gmane.org>
>> > > wrote:
>> > >
>> > > > Yes. Also have a look at the logs of the problematic region if hbck
>> > shows
>> > > > any inconsistency.
>> > > >
>> > > > Regards,
>> > > > Mohammad Tariq
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
>> mohitanchlia-***@public.gmane.org
>> > > >wrote:
>> > > >
>> > > >> Which node should I look at for logs? Is this the master node? I'll
>> > try
>> > > >> hbck.
>> > > >>
>> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <
>> dontariq-***@public.gmane.org>
>> > > >> wrote:
>> > > >>
>> > > >> > Hello Mohit,
>> > > >> >
>> > > >> > Try hbck once and see if it shows any inconsistency. Also,
>> you
>> > can
>> > > >> try
>> > > >> > restarting your cluster and deleting the table again. Having a
>> look
>> > at
>> > > >> the
>> > > >> > logs could also be useful.
>> > > >> >
>> > > >> > Regards,
>> > > >> > Mohammad Tariq
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
>> > > mohitanchlia-***@public.gmane.org
>> > > >> > >wrote:
>> > > >> >
>> > > >> > > When I try to disable table I get:
>> > > >> > >
>> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
>> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
>> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
>> > SESSIONID_TIMELINE
>> > > >> > > Here is some help for this command:
>> > > >> > > Start disable of named table: e.g. "hbase> disable 't1'"
>> > > >> > >
>> > > >> > > But then I try to enable I get:
>> > > >> > >
>> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
>> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
>> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
>> > > SESSIONID_TIMELINE
>> > > >> > > Here is some help for this command:
>> > > >> > > Start enable of named table: e.g. "hbase> enable 't1'"
>> > > >> > >
>> > > >> > > I've tried flush, major_compaction also. I tseems it's stuck in
>> > > >> > > inconsistent state. Could someone point me to correct
>> direction? I
>> > > am
>> > > >> > using
>> > > >> > > 92.1
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>
>
Ramkrishna.S.Vasudevan
2012-09-27 04:29:21 UTC
Permalink
Hi Mohith
First of all thanks to Tariq for his replies.

Just to add on,
Basically HBase uses the Zookeeper to know the status of the cluster like
the no of tables enabled, disabled and deleted.
Enabled and deleted states are handled bit different in the 0.94 version.

ZK is used for various region assignments.

Also the ZK is used to track the Active master and standby master.

As you understand correctly that the master is responsible for the overall
maintenance of the no of tables and their respective states, it seeks the
help of ZK to do it and that is where the states are persisted.

Also there are few cases where the enable and disable table are having some
issues due to some race conditions in the 0.92 versions, In the latest
version we are trying to resolve them.
You can attach the master and RS logs to identify exactly what caused this
problem in your case which will be really help ful so that I can be fixed in
the kernel.

Regards
Ram

> -----Original Message-----
> From: Mohit Anchlia [mailto:mohitanchlia-***@public.gmane.org]
> Sent: Thursday, September 27, 2012 5:09 AM
> To: user-50Pas4EWwPEyzMRdD/***@public.gmane.org
> Subject: Re: disable table
>
> I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll
> restart
> hbase and see if it works.
>
> One thing I don't understand is why is zookeeper holding information
> about
> this table if it is enabled or disabled? Wouldn't this information be
> with
> master?
>
> On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia
> <mohitanchlia-***@public.gmane.org>wrote:
>
> > I don't see path like /hbase/SESSIONID_TIMELINE
> > This is what I see
> >
> > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> > [SESSIONID_TIMELINE]
> > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
> >
> > cZxid = 0x1000000fe
> > ctime = Mon Sep 10 15:31:45 PDT 2012
> > mZxid = 0x1000000fe
> > mtime = Mon Sep 10 15:31:45 PDT 2012
> > pZxid = 0x5000008f1
> > cversion = 3
> > dataVersion = 0
> > aclVersion = 0
> > ephemeralOwner = 0x0
> > dataLength = 0
> > numChildren = 1
> >
> > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq
> <dontariq-***@public.gmane.org>wrote:
> >
> >> In order to delete a znode you have to go to the ZK shell and issue
> the
> >> "delete" command along with the required path. For example :
> >> delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit
> the ZK
> >> homepage at : zookeeper.apache.org
> >>
> >> Actually when we try to fetch data from an Hbase table, the client
> or app
> >> first contacts the ZK to get the location of server holding the
> >> -ROOT- table. From this we come to know about the server hosting the
> >> .META.
> >> table. This tells us the location of the server which actually holds
> the
> >> rows of interest. Because of some reasons the znode which was
> holding this
> >> info has either faced some catastrophe or lost the info associated
> with
> >> this particular table. Or sometimes the znode remains unable to keep
> >> itself
> >> updated with the latest changes. That could also be a probable
> reason. We
> >> should always keep in mind that ZK is the centralized service that
> >> actually
> >> coordinating everything behind the scene. As a result, any problem
> to the
> >> ZK quorum means problem with Hbase custer.
> >>
> >> Regards,
> >> Mohammad Tariq
> >>
> >>
> >>
> >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia
> <mohitanchlia-***@public.gmane.org
> >> >wrote:
> >>
> >> > Thanks! I do see Inconsistency. How do I remove the znode. And
> also
> >> could
> >> > you please help me understand how this might have happened?
> >> >
> >> >
> >> > ERROR: Region
> >> >
> SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. not
> >> > deployed on any region server.
> >> >
> >> >
> >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq
> <dontariq-***@public.gmane.org>
> >> > wrote:
> >> >
> >> > > A possible reason could be that the znode associated with this
> >> particular
> >> > > table is not behaving properly. In such case, you can try the
> >> following:
> >> > >
> >> > > Stop Hbase
> >> > > Stop ZK
> >> > > Take a backup of ZK data
> >> > > Restart ZK
> >> > > Remove the znode
> >> > > Start Hbase again
> >> > >
> >> > > After this hopefully your table would be enabled.
> >> > >
> >> > > Regards,
> >> > > Mohammad Tariq
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq
> <dontariq-***@public.gmane.org>
> >> > > wrote:
> >> > >
> >> > > > Yes. Also have a look at the logs of the problematic region if
> hbck
> >> > shows
> >> > > > any inconsistency.
> >> > > >
> >> > > > Regards,
> >> > > > Mohammad Tariq
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
> >> mohitanchlia-***@public.gmane.org
> >> > > >wrote:
> >> > > >
> >> > > >> Which node should I look at for logs? Is this the master
> node? I'll
> >> > try
> >> > > >> hbck.
> >> > > >>
> >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <
> >> dontariq-***@public.gmane.org>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > Hello Mohit,
> >> > > >> >
> >> > > >> > Try hbck once and see if it shows any inconsistency.
> Also,
> >> you
> >> > can
> >> > > >> try
> >> > > >> > restarting your cluster and deleting the table again.
> Having a
> >> look
> >> > at
> >> > > >> the
> >> > > >> > logs could also be useful.
> >> > > >> >
> >> > > >> > Regards,
> >> > > >> > Mohammad Tariq
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> >> > > mohitanchlia-***@public.gmane.org
> >> > > >> > >wrote:
> >> > > >> >
> >> > > >> > > When I try to disable table I get:
> >> > > >> > >
> >> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> >> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> >> > SESSIONID_TIMELINE
> >> > > >> > > Here is some help for this command:
> >> > > >> > > Start disable of named table: e.g. "hbase> disable 't1'"
> >> > > >> > >
> >> > > >> > > But then I try to enable I get:
> >> > > >> > >
> >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> >> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> >> > > SESSIONID_TIMELINE
> >> > > >> > > Here is some help for this command:
> >> > > >> > > Start enable of named table: e.g. "hbase> enable 't1'"
> >> > > >> > >
> >> > > >> > > I've tried flush, major_compaction also. I tseems it's
> stuck in
> >> > > >> > > inconsistent state. Could someone point me to correct
> >> direction? I
> >> > > am
> >> > > >> > using
> >> > > >> > > 92.1
> >> > > >> > >
> >> > > >> >
> >> > > >>
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
Mohammad Tariq
2012-09-27 08:10:33 UTC
Permalink
Hello Mohit,

It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies for the
typo. For rest of the things, I feel Ramkrishna sir has provided a good and
proper explanation. Please let us know if you still have any doubt or
question.

Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to share
space with you people.

Regards,
Mohammad Tariq



On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan <
ramkrishna.vasudevan-hv44wF8Li93QT0dZR+***@public.gmane.org> wrote:

> Hi Mohith
> First of all thanks to Tariq for his replies.
>
> Just to add on,
> Basically HBase uses the Zookeeper to know the status of the cluster like
> the no of tables enabled, disabled and deleted.
> Enabled and deleted states are handled bit different in the 0.94 version.
>
> ZK is used for various region assignments.
>
> Also the ZK is used to track the Active master and standby master.
>
> As you understand correctly that the master is responsible for the overall
> maintenance of the no of tables and their respective states, it seeks the
> help of ZK to do it and that is where the states are persisted.
>
> Also there are few cases where the enable and disable table are having some
> issues due to some race conditions in the 0.92 versions, In the latest
> version we are trying to resolve them.
> You can attach the master and RS logs to identify exactly what caused this
> problem in your case which will be really help ful so that I can be fixed
> in
> the kernel.
>
> Regards
> Ram
>
> > -----Original Message-----
> > From: Mohit Anchlia [mailto:mohitanchlia-***@public.gmane.org]
> > Sent: Thursday, September 27, 2012 5:09 AM
> > To: user-50Pas4EWwPEyzMRdD/***@public.gmane.org
> > Subject: Re: disable table
> >
> > I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll
> > restart
> > hbase and see if it works.
> >
> > One thing I don't understand is why is zookeeper holding information
> > about
> > this table if it is enabled or disabled? Wouldn't this information be
> > with
> > master?
> >
> > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia
> > <mohitanchlia-***@public.gmane.org>wrote:
> >
> > > I don't see path like /hbase/SESSIONID_TIMELINE
> > > This is what I see
> > >
> > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> > > [SESSIONID_TIMELINE]
> > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
> > >
> > > cZxid = 0x1000000fe
> > > ctime = Mon Sep 10 15:31:45 PDT 2012
> > > mZxid = 0x1000000fe
> > > mtime = Mon Sep 10 15:31:45 PDT 2012
> > > pZxid = 0x5000008f1
> > > cversion = 3
> > > dataVersion = 0
> > > aclVersion = 0
> > > ephemeralOwner = 0x0
> > > dataLength = 0
> > > numChildren = 1
> > >
> > > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq
> > <dontariq-***@public.gmane.org>wrote:
> > >
> > >> In order to delete a znode you have to go to the ZK shell and issue
> > the
> > >> "delete" command along with the required path. For example :
> > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit
> > the ZK
> > >> homepage at : zookeeper.apache.org
> > >>
> > >> Actually when we try to fetch data from an Hbase table, the client
> > or app
> > >> first contacts the ZK to get the location of server holding the
> > >> -ROOT- table. From this we come to know about the server hosting the
> > >> .META.
> > >> table. This tells us the location of the server which actually holds
> > the
> > >> rows of interest. Because of some reasons the znode which was
> > holding this
> > >> info has either faced some catastrophe or lost the info associated
> > with
> > >> this particular table. Or sometimes the znode remains unable to keep
> > >> itself
> > >> updated with the latest changes. That could also be a probable
> > reason. We
> > >> should always keep in mind that ZK is the centralized service that
> > >> actually
> > >> coordinating everything behind the scene. As a result, any problem
> > to the
> > >> ZK quorum means problem with Hbase custer.
> > >>
> > >> Regards,
> > >> Mohammad Tariq
> > >>
> > >>
> > >>
> > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia
> > <mohitanchlia-***@public.gmane.org
> > >> >wrote:
> > >>
> > >> > Thanks! I do see Inconsistency. How do I remove the znode. And
> > also
> > >> could
> > >> > you please help me understand how this might have happened?
> > >> >
> > >> >
> > >> > ERROR: Region
> > >> >
> > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. not
> > >> > deployed on any region server.
> > >> >
> > >> >
> > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq
> > <dontariq-***@public.gmane.org>
> > >> > wrote:
> > >> >
> > >> > > A possible reason could be that the znode associated with this
> > >> particular
> > >> > > table is not behaving properly. In such case, you can try the
> > >> following:
> > >> > >
> > >> > > Stop Hbase
> > >> > > Stop ZK
> > >> > > Take a backup of ZK data
> > >> > > Restart ZK
> > >> > > Remove the znode
> > >> > > Start Hbase again
> > >> > >
> > >> > > After this hopefully your table would be enabled.
> > >> > >
> > >> > > Regards,
> > >> > > Mohammad Tariq
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq
> > <dontariq-***@public.gmane.org>
> > >> > > wrote:
> > >> > >
> > >> > > > Yes. Also have a look at the logs of the problematic region if
> > hbck
> > >> > shows
> > >> > > > any inconsistency.
> > >> > > >
> > >> > > > Regards,
> > >> > > > Mohammad Tariq
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
> > >> mohitanchlia-***@public.gmane.org
> > >> > > >wrote:
> > >> > > >
> > >> > > >> Which node should I look at for logs? Is this the master
> > node? I'll
> > >> > try
> > >> > > >> hbck.
> > >> > > >>
> > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <
> > >> dontariq-***@public.gmane.org>
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > Hello Mohit,
> > >> > > >> >
> > >> > > >> > Try hbck once and see if it shows any inconsistency.
> > Also,
> > >> you
> > >> > can
> > >> > > >> try
> > >> > > >> > restarting your cluster and deleting the table again.
> > Having a
> > >> look
> > >> > at
> > >> > > >> the
> > >> > > >> > logs could also be useful.
> > >> > > >> >
> > >> > > >> > Regards,
> > >> > > >> > Mohammad Tariq
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> > >> > > mohitanchlia-***@public.gmane.org
> > >> > > >> > >wrote:
> > >> > > >> >
> > >> > > >> > > When I try to disable table I get:
> > >> > > >> > >
> > >> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> > >> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> > >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> > >> > SESSIONID_TIMELINE
> > >> > > >> > > Here is some help for this command:
> > >> > > >> > > Start disable of named table: e.g. "hbase> disable 't1'"
> > >> > > >> > >
> > >> > > >> > > But then I try to enable I get:
> > >> > > >> > >
> > >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > >> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> > >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> > >> > > SESSIONID_TIMELINE
> > >> > > >> > > Here is some help for this command:
> > >> > > >> > > Start enable of named table: e.g. "hbase> enable 't1'"
> > >> > > >> > >
> > >> > > >> > > I've tried flush, major_compaction also. I tseems it's
> > stuck in
> > >> > > >> > > inconsistent state. Could someone point me to correct
> > >> direction? I
> > >> > > am
> > >> > > >> > using
> > >> > > >> > > 92.1
> > >> > > >> > >
> > >> > > >> >
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
>
>
Mohit Anchlia
2012-09-27 14:41:32 UTC
Permalink
Thanks everyone for the input, it's helpful. I did remove the znode from
/hbase/table/SESSIONID_TIMELINE and after that I was able to list the
table. At that point I tried to do a "put" but when I did a put I got a
message NoRegionServer online. I looked in the logs and it says the Failed
to open region server at nodexxx. When I went to nodexxx it complains
something about unable to run testcompression.

I setup SNAPPY compression on my table and I also ran SNAPPY compression
test which was successful. Not sure what's going on in the cluster.
On Thu, Sep 27, 2012 at 1:10 AM, Mohammad Tariq <dontariq-***@public.gmane.org> wrote:

> Hello Mohit,
>
> It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies for the
> typo. For rest of the things, I feel Ramkrishna sir has provided a good and
> proper explanation. Please let us know if you still have any doubt or
> question.
>
> Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to share
> space with you people.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan <
> ramkrishna.vasudevan-hv44wF8Li93QT0dZR+***@public.gmane.org> wrote:
>
> > Hi Mohith
> > First of all thanks to Tariq for his replies.
> >
> > Just to add on,
> > Basically HBase uses the Zookeeper to know the status of the cluster
> like
> > the no of tables enabled, disabled and deleted.
> > Enabled and deleted states are handled bit different in the 0.94 version.
> >
> > ZK is used for various region assignments.
> >
> > Also the ZK is used to track the Active master and standby master.
> >
> > As you understand correctly that the master is responsible for the
> overall
> > maintenance of the no of tables and their respective states, it seeks the
> > help of ZK to do it and that is where the states are persisted.
> >
> > Also there are few cases where the enable and disable table are having
> some
> > issues due to some race conditions in the 0.92 versions, In the latest
> > version we are trying to resolve them.
> > You can attach the master and RS logs to identify exactly what caused
> this
> > problem in your case which will be really help ful so that I can be fixed
> > in
> > the kernel.
> >
> > Regards
> > Ram
> >
> > > -----Original Message-----
> > > From: Mohit Anchlia [mailto:mohitanchlia-***@public.gmane.org]
> > > Sent: Thursday, September 27, 2012 5:09 AM
> > > To: user-50Pas4EWwPEyzMRdD/***@public.gmane.org
> > > Subject: Re: disable table
> > >
> > > I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll
> > > restart
> > > hbase and see if it works.
> > >
> > > One thing I don't understand is why is zookeeper holding information
> > > about
> > > this table if it is enabled or disabled? Wouldn't this information be
> > > with
> > > master?
> > >
> > > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia
> > > <mohitanchlia-***@public.gmane.org>wrote:
> > >
> > > > I don't see path like /hbase/SESSIONID_TIMELINE
> > > > This is what I see
> > > >
> > > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> > > > [SESSIONID_TIMELINE]
> > > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
> > > >
> > > > cZxid = 0x1000000fe
> > > > ctime = Mon Sep 10 15:31:45 PDT 2012
> > > > mZxid = 0x1000000fe
> > > > mtime = Mon Sep 10 15:31:45 PDT 2012
> > > > pZxid = 0x5000008f1
> > > > cversion = 3
> > > > dataVersion = 0
> > > > aclVersion = 0
> > > > ephemeralOwner = 0x0
> > > > dataLength = 0
> > > > numChildren = 1
> > > >
> > > > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq
> > > <dontariq-***@public.gmane.org>wrote:
> > > >
> > > >> In order to delete a znode you have to go to the ZK shell and issue
> > > the
> > > >> "delete" command along with the required path. For example :
> > > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit
> > > the ZK
> > > >> homepage at : zookeeper.apache.org
> > > >>
> > > >> Actually when we try to fetch data from an Hbase table, the client
> > > or app
> > > >> first contacts the ZK to get the location of server holding the
> > > >> -ROOT- table. From this we come to know about the server hosting the
> > > >> .META.
> > > >> table. This tells us the location of the server which actually holds
> > > the
> > > >> rows of interest. Because of some reasons the znode which was
> > > holding this
> > > >> info has either faced some catastrophe or lost the info associated
> > > with
> > > >> this particular table. Or sometimes the znode remains unable to keep
> > > >> itself
> > > >> updated with the latest changes. That could also be a probable
> > > reason. We
> > > >> should always keep in mind that ZK is the centralized service that
> > > >> actually
> > > >> coordinating everything behind the scene. As a result, any problem
> > > to the
> > > >> ZK quorum means problem with Hbase custer.
> > > >>
> > > >> Regards,
> > > >> Mohammad Tariq
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia
> > > <mohitanchlia-***@public.gmane.org
> > > >> >wrote:
> > > >>
> > > >> > Thanks! I do see Inconsistency. How do I remove the znode. And
> > > also
> > > >> could
> > > >> > you please help me understand how this might have happened?
> > > >> >
> > > >> >
> > > >> > ERROR: Region
> > > >> >
> > > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050. not
> > > >> > deployed on any region server.
> > > >> >
> > > >> >
> > > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq
> > > <dontariq-***@public.gmane.org>
> > > >> > wrote:
> > > >> >
> > > >> > > A possible reason could be that the znode associated with this
> > > >> particular
> > > >> > > table is not behaving properly. In such case, you can try the
> > > >> following:
> > > >> > >
> > > >> > > Stop Hbase
> > > >> > > Stop ZK
> > > >> > > Take a backup of ZK data
> > > >> > > Restart ZK
> > > >> > > Remove the znode
> > > >> > > Start Hbase again
> > > >> > >
> > > >> > > After this hopefully your table would be enabled.
> > > >> > >
> > > >> > > Regards,
> > > >> > > Mohammad Tariq
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq
> > > <dontariq-***@public.gmane.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Yes. Also have a look at the logs of the problematic region if
> > > hbck
> > > >> > shows
> > > >> > > > any inconsistency.
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > > Mohammad Tariq
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
> > > >> mohitanchlia-***@public.gmane.org
> > > >> > > >wrote:
> > > >> > > >
> > > >> > > >> Which node should I look at for logs? Is this the master
> > > node? I'll
> > > >> > try
> > > >> > > >> hbck.
> > > >> > > >>
> > > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <
> > > >> dontariq-***@public.gmane.org>
> > > >> > > >> wrote:
> > > >> > > >>
> > > >> > > >> > Hello Mohit,
> > > >> > > >> >
> > > >> > > >> > Try hbck once and see if it shows any inconsistency.
> > > Also,
> > > >> you
> > > >> > can
> > > >> > > >> try
> > > >> > > >> > restarting your cluster and deleting the table again.
> > > Having a
> > > >> look
> > > >> > at
> > > >> > > >> the
> > > >> > > >> > logs could also be useful.
> > > >> > > >> >
> > > >> > > >> > Regards,
> > > >> > > >> > Mohammad Tariq
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> >
> > > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> > > >> > > mohitanchlia-***@public.gmane.org
> > > >> > > >> > >wrote:
> > > >> > > >> >
> > > >> > > >> > > When I try to disable table I get:
> > > >> > > >> > >
> > > >> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> > > >> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
> > > >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> > > >> > SESSIONID_TIMELINE
> > > >> > > >> > > Here is some help for this command:
> > > >> > > >> > > Start disable of named table: e.g. "hbase> disable 't1'"
> > > >> > > >> > >
> > > >> > > >> > > But then I try to enable I get:
> > > >> > > >> > >
> > > >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > > >> > > >> > > ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
> > > >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> > > >> > > SESSIONID_TIMELINE
> > > >> > > >> > > Here is some help for this command:
> > > >> > > >> > > Start enable of named table: e.g. "hbase> enable 't1'"
> > > >> > > >> > >
> > > >> > > >> > > I've tried flush, major_compaction also. I tseems it's
> > > stuck in
> > > >> > > >> > > inconsistent state. Could someone point me to correct
> > > >> direction? I
> > > >> > > am
> > > >> > > >> > using
> > > >> > > >> > > 92.1
> > > >> > > >> > >
> > > >> > > >> >
> > > >> > > >>
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> >
> >
>
rajesh babu chintaguntla
2012-09-27 15:00:37 UTC
Permalink
Hi Mohit,

We should not delete znode's manually which will cause inconsistencies like
region may be shown as online on master, but it wont be on region server.
That's put is failing in your case. Master restart will bring back your
cluster to normal state(recovery any failures in enable/disable). Even hbck
also wont solve this problem.

FYI,
Presently discussion is going on this issue. You can follow jira associated
with this issue at
https://issues.apache.org/jira/browse/HBASE-6469

On Thu, Sep 27, 2012 at 8:11 PM, Mohit Anchlia <mohitanchlia-***@public.gmane.org>wrote:

> Thanks everyone for the input, it's helpful. I did remove the znode from
> /hbase/table/SESSIONID_TIMELINE and after that I was able to list the
> table. At that point I tried to do a "put" but when I did a put I got a
> message NoRegionServer online. I looked in the logs and it says the Failed
> to open region server at nodexxx. When I went to nodexxx it complains
> something about unable to run testcompression.
>
> I setup SNAPPY compression on my table and I also ran SNAPPY compression
> test which was successful. Not sure what's going on in the cluster.
> On Thu, Sep 27, 2012 at 1:10 AM, Mohammad Tariq <dontariq-***@public.gmane.org>
> wrote:
>
> > Hello Mohit,
> >
> > It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies for the
> > typo. For rest of the things, I feel Ramkrishna sir has provided a good
> and
> > proper explanation. Please let us know if you still have any doubt or
> > question.
> >
> > Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to share
> > space with you people.
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> >
> > On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan <
> > ramkrishna.vasudevan-hv44wF8Li93QT0dZR+***@public.gmane.org> wrote:
> >
> > > Hi Mohith
> > > First of all thanks to Tariq for his replies.
> > >
> > > Just to add on,
> > > Basically HBase uses the Zookeeper to know the status of the cluster
> > like
> > > the no of tables enabled, disabled and deleted.
> > > Enabled and deleted states are handled bit different in the 0.94
> version.
> > >
> > > ZK is used for various region assignments.
> > >
> > > Also the ZK is used to track the Active master and standby master.
> > >
> > > As you understand correctly that the master is responsible for the
> > overall
> > > maintenance of the no of tables and their respective states, it seeks
> the
> > > help of ZK to do it and that is where the states are persisted.
> > >
> > > Also there are few cases where the enable and disable table are having
> > some
> > > issues due to some race conditions in the 0.92 versions, In the latest
> > > version we are trying to resolve them.
> > > You can attach the master and RS logs to identify exactly what caused
> > this
> > > problem in your case which will be really help ful so that I can be
> fixed
> > > in
> > > the kernel.
> > >
> > > Regards
> > > Ram
> > >
> > > > -----Original Message-----
> > > > From: Mohit Anchlia [mailto:mohitanchlia-***@public.gmane.org]
> > > > Sent: Thursday, September 27, 2012 5:09 AM
> > > > To: user-50Pas4EWwPEyzMRdD/***@public.gmane.org
> > > > Subject: Re: disable table
> > > >
> > > > I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll
> > > > restart
> > > > hbase and see if it works.
> > > >
> > > > One thing I don't understand is why is zookeeper holding information
> > > > about
> > > > this table if it is enabled or disabled? Wouldn't this information be
> > > > with
> > > > master?
> > > >
> > > > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia
> > > > <mohitanchlia-***@public.gmane.org>wrote:
> > > >
> > > > > I don't see path like /hbase/SESSIONID_TIMELINE
> > > > > This is what I see
> > > > >
> > > > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> > > > > [SESSIONID_TIMELINE]
> > > > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
> > > > >
> > > > > cZxid = 0x1000000fe
> > > > > ctime = Mon Sep 10 15:31:45 PDT 2012
> > > > > mZxid = 0x1000000fe
> > > > > mtime = Mon Sep 10 15:31:45 PDT 2012
> > > > > pZxid = 0x5000008f1
> > > > > cversion = 3
> > > > > dataVersion = 0
> > > > > aclVersion = 0
> > > > > ephemeralOwner = 0x0
> > > > > dataLength = 0
> > > > > numChildren = 1
> > > > >
> > > > > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq
> > > > <dontariq-***@public.gmane.org>wrote:
> > > > >
> > > > >> In order to delete a znode you have to go to the ZK shell and
> issue
> > > > the
> > > > >> "delete" command along with the required path. For example :
> > > > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you can visit
> > > > the ZK
> > > > >> homepage at : zookeeper.apache.org
> > > > >>
> > > > >> Actually when we try to fetch data from an Hbase table, the client
> > > > or app
> > > > >> first contacts the ZK to get the location of server holding the
> > > > >> -ROOT- table. From this we come to know about the server hosting
> the
> > > > >> .META.
> > > > >> table. This tells us the location of the server which actually
> holds
> > > > the
> > > > >> rows of interest. Because of some reasons the znode which was
> > > > holding this
> > > > >> info has either faced some catastrophe or lost the info associated
> > > > with
> > > > >> this particular table. Or sometimes the znode remains unable to
> keep
> > > > >> itself
> > > > >> updated with the latest changes. That could also be a probable
> > > > reason. We
> > > > >> should always keep in mind that ZK is the centralized service that
> > > > >> actually
> > > > >> coordinating everything behind the scene. As a result, any problem
> > > > to the
> > > > >> ZK quorum means problem with Hbase custer.
> > > > >>
> > > > >> Regards,
> > > > >> Mohammad Tariq
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia
> > > > <mohitanchlia-***@public.gmane.org
> > > > >> >wrote:
> > > > >>
> > > > >> > Thanks! I do see Inconsistency. How do I remove the znode. And
> > > > also
> > > > >> could
> > > > >> > you please help me understand how this might have happened?
> > > > >> >
> > > > >> >
> > > > >> > ERROR: Region
> > > > >> >
> > > > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050.
> not
> > > > >> > deployed on any region server.
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq
> > > > <dontariq-***@public.gmane.org>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > A possible reason could be that the znode associated with this
> > > > >> particular
> > > > >> > > table is not behaving properly. In such case, you can try the
> > > > >> following:
> > > > >> > >
> > > > >> > > Stop Hbase
> > > > >> > > Stop ZK
> > > > >> > > Take a backup of ZK data
> > > > >> > > Restart ZK
> > > > >> > > Remove the znode
> > > > >> > > Start Hbase again
> > > > >> > >
> > > > >> > > After this hopefully your table would be enabled.
> > > > >> > >
> > > > >> > > Regards,
> > > > >> > > Mohammad Tariq
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq
> > > > <dontariq-***@public.gmane.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Yes. Also have a look at the logs of the problematic region
> if
> > > > hbck
> > > > >> > shows
> > > > >> > > > any inconsistency.
> > > > >> > > >
> > > > >> > > > Regards,
> > > > >> > > > Mohammad Tariq
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
> > > > >> mohitanchlia-***@public.gmane.org
> > > > >> > > >wrote:
> > > > >> > > >
> > > > >> > > >> Which node should I look at for logs? Is this the master
> > > > node? I'll
> > > > >> > try
> > > > >> > > >> hbck.
> > > > >> > > >>
> > > > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <
> > > > >> dontariq-***@public.gmane.org>
> > > > >> > > >> wrote:
> > > > >> > > >>
> > > > >> > > >> > Hello Mohit,
> > > > >> > > >> >
> > > > >> > > >> > Try hbck once and see if it shows any inconsistency.
> > > > Also,
> > > > >> you
> > > > >> > can
> > > > >> > > >> try
> > > > >> > > >> > restarting your cluster and deleting the table again.
> > > > Having a
> > > > >> look
> > > > >> > at
> > > > >> > > >> the
> > > > >> > > >> > logs could also be useful.
> > > > >> > > >> >
> > > > >> > > >> > Regards,
> > > > >> > > >> > Mohammad Tariq
> > > > >> > > >> >
> > > > >> > > >> >
> > > > >> > > >> >
> > > > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> > > > >> > > mohitanchlia-***@public.gmane.org
> > > > >> > > >> > >wrote:
> > > > >> > > >> >
> > > > >> > > >> > > When I try to disable table I get:
> > > > >> > > >> > >
> > > > >> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> > > > >> > > >> > > ERROR:
> org.apache.hadoop.hbase.TableNotEnabledException:
> > > > >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> > > > >> > SESSIONID_TIMELINE
> > > > >> > > >> > > Here is some help for this command:
> > > > >> > > >> > > Start disable of named table: e.g. "hbase> disable
> 't1'"
> > > > >> > > >> > >
> > > > >> > > >> > > But then I try to enable I get:
> > > > >> > > >> > >
> > > > >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > > > >> > > >> > > ERROR:
> org.apache.hadoop.hbase.TableNotDisabledException:
> > > > >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> > > > >> > > SESSIONID_TIMELINE
> > > > >> > > >> > > Here is some help for this command:
> > > > >> > > >> > > Start enable of named table: e.g. "hbase> enable 't1'"
> > > > >> > > >> > >
> > > > >> > > >> > > I've tried flush, major_compaction also. I tseems it's
> > > > stuck in
> > > > >> > > >> > > inconsistent state. Could someone point me to correct
> > > > >> direction? I
> > > > >> > > am
> > > > >> > > >> > using
> > > > >> > > >> > > 92.1
> > > > >> > > >> > >
> > > > >> > > >> >
> > > > >> > > >>
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > >
> > >
> >
>
Mohit Anchlia
2012-09-27 16:24:41 UTC
Permalink
I did restart entire cluster and still that didn't help. Looks like once I
get in this Race condition there is no way to come out of it?

On Thu, Sep 27, 2012 at 8:00 AM, rajesh babu chintaguntla <
chrajeshbabu32-***@public.gmane.org> wrote:

> Hi Mohit,
>
> We should not delete znode's manually which will cause inconsistencies like
> region may be shown as online on master, but it wont be on region server.
> That's put is failing in your case. Master restart will bring back your
> cluster to normal state(recovery any failures in enable/disable). Even hbck
> also wont solve this problem.
>
> FYI,
> Presently discussion is going on this issue. You can follow jira associated
> with this issue at
> https://issues.apache.org/jira/browse/HBASE-6469
>
> On Thu, Sep 27, 2012 at 8:11 PM, Mohit Anchlia <mohitanchlia-***@public.gmane.org
> >wrote:
>
> > Thanks everyone for the input, it's helpful. I did remove the znode from
> > /hbase/table/SESSIONID_TIMELINE and after that I was able to list the
> > table. At that point I tried to do a "put" but when I did a put I got a
> > message NoRegionServer online. I looked in the logs and it says the
> Failed
> > to open region server at nodexxx. When I went to nodexxx it complains
> > something about unable to run testcompression.
> >
> > I setup SNAPPY compression on my table and I also ran SNAPPY compression
> > test which was successful. Not sure what's going on in the cluster.
> > On Thu, Sep 27, 2012 at 1:10 AM, Mohammad Tariq <dontariq-***@public.gmane.org>
> > wrote:
> >
> > > Hello Mohit,
> > >
> > > It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies for
> the
> > > typo. For rest of the things, I feel Ramkrishna sir has provided a good
> > and
> > > proper explanation. Please let us know if you still have any doubt or
> > > question.
> > >
> > > Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to share
> > > space with you people.
> > >
> > > Regards,
> > > Mohammad Tariq
> > >
> > >
> > >
> > > On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan <
> > > ramkrishna.vasudevan-hv44wF8Li93QT0dZR+***@public.gmane.org> wrote:
> > >
> > > > Hi Mohith
> > > > First of all thanks to Tariq for his replies.
> > > >
> > > > Just to add on,
> > > > Basically HBase uses the Zookeeper to know the status of the cluster
> > > like
> > > > the no of tables enabled, disabled and deleted.
> > > > Enabled and deleted states are handled bit different in the 0.94
> > version.
> > > >
> > > > ZK is used for various region assignments.
> > > >
> > > > Also the ZK is used to track the Active master and standby master.
> > > >
> > > > As you understand correctly that the master is responsible for the
> > > overall
> > > > maintenance of the no of tables and their respective states, it seeks
> > the
> > > > help of ZK to do it and that is where the states are persisted.
> > > >
> > > > Also there are few cases where the enable and disable table are
> having
> > > some
> > > > issues due to some race conditions in the 0.92 versions, In the
> latest
> > > > version we are trying to resolve them.
> > > > You can attach the master and RS logs to identify exactly what caused
> > > this
> > > > problem in your case which will be really help ful so that I can be
> > fixed
> > > > in
> > > > the kernel.
> > > >
> > > > Regards
> > > > Ram
> > > >
> > > > > -----Original Message-----
> > > > > From: Mohit Anchlia [mailto:mohitanchlia-***@public.gmane.org]
> > > > > Sent: Thursday, September 27, 2012 5:09 AM
> > > > > To: user-50Pas4EWwPEyzMRdD/***@public.gmane.org
> > > > > Subject: Re: disable table
> > > > >
> > > > > I did /hbase/table/SESSIONID_TIMELINE and that seem to work. I'll
> > > > > restart
> > > > > hbase and see if it works.
> > > > >
> > > > > One thing I don't understand is why is zookeeper holding
> information
> > > > > about
> > > > > this table if it is enabled or disabled? Wouldn't this information
> be
> > > > > with
> > > > > master?
> > > > >
> > > > > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia
> > > > > <mohitanchlia-***@public.gmane.org>wrote:
> > > > >
> > > > > > I don't see path like /hbase/SESSIONID_TIMELINE
> > > > > > This is what I see
> > > > > >
> > > > > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> > > > > > [SESSIONID_TIMELINE]
> > > > > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
> > > > > >
> > > > > > cZxid = 0x1000000fe
> > > > > > ctime = Mon Sep 10 15:31:45 PDT 2012
> > > > > > mZxid = 0x1000000fe
> > > > > > mtime = Mon Sep 10 15:31:45 PDT 2012
> > > > > > pZxid = 0x5000008f1
> > > > > > cversion = 3
> > > > > > dataVersion = 0
> > > > > > aclVersion = 0
> > > > > > ephemeralOwner = 0x0
> > > > > > dataLength = 0
> > > > > > numChildren = 1
> > > > > >
> > > > > > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq
> > > > > <dontariq-***@public.gmane.org>wrote:
> > > > > >
> > > > > >> In order to delete a znode you have to go to the ZK shell and
> > issue
> > > > > the
> > > > > >> "delete" command along with the required path. For example :
> > > > > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you can
> visit
> > > > > the ZK
> > > > > >> homepage at : zookeeper.apache.org
> > > > > >>
> > > > > >> Actually when we try to fetch data from an Hbase table, the
> client
> > > > > or app
> > > > > >> first contacts the ZK to get the location of server holding the
> > > > > >> -ROOT- table. From this we come to know about the server hosting
> > the
> > > > > >> .META.
> > > > > >> table. This tells us the location of the server which actually
> > holds
> > > > > the
> > > > > >> rows of interest. Because of some reasons the znode which was
> > > > > holding this
> > > > > >> info has either faced some catastrophe or lost the info
> associated
> > > > > with
> > > > > >> this particular table. Or sometimes the znode remains unable to
> > keep
> > > > > >> itself
> > > > > >> updated with the latest changes. That could also be a probable
> > > > > reason. We
> > > > > >> should always keep in mind that ZK is the centralized service
> that
> > > > > >> actually
> > > > > >> coordinating everything behind the scene. As a result, any
> problem
> > > > > to the
> > > > > >> ZK quorum means problem with Hbase custer.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Mohammad Tariq
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia
> > > > > <mohitanchlia-***@public.gmane.org
> > > > > >> >wrote:
> > > > > >>
> > > > > >> > Thanks! I do see Inconsistency. How do I remove the znode. And
> > > > > also
> > > > > >> could
> > > > > >> > you please help me understand how this might have happened?
> > > > > >> >
> > > > > >> >
> > > > > >> > ERROR: Region
> > > > > >> >
> > > > > SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050.
> > not
> > > > > >> > deployed on any region server.
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq
> > > > > <dontariq-***@public.gmane.org>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > A possible reason could be that the znode associated with
> this
> > > > > >> particular
> > > > > >> > > table is not behaving properly. In such case, you can try
> the
> > > > > >> following:
> > > > > >> > >
> > > > > >> > > Stop Hbase
> > > > > >> > > Stop ZK
> > > > > >> > > Take a backup of ZK data
> > > > > >> > > Restart ZK
> > > > > >> > > Remove the znode
> > > > > >> > > Start Hbase again
> > > > > >> > >
> > > > > >> > > After this hopefully your table would be enabled.
> > > > > >> > >
> > > > > >> > > Regards,
> > > > > >> > > Mohammad Tariq
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq
> > > > > <dontariq-***@public.gmane.org>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Yes. Also have a look at the logs of the problematic
> region
> > if
> > > > > hbck
> > > > > >> > shows
> > > > > >> > > > any inconsistency.
> > > > > >> > > >
> > > > > >> > > > Regards,
> > > > > >> > > > Mohammad Tariq
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
> > > > > >> mohitanchlia-***@public.gmane.org
> > > > > >> > > >wrote:
> > > > > >> > > >
> > > > > >> > > >> Which node should I look at for logs? Is this the master
> > > > > node? I'll
> > > > > >> > try
> > > > > >> > > >> hbck.
> > > > > >> > > >>
> > > > > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <
> > > > > >> dontariq-***@public.gmane.org>
> > > > > >> > > >> wrote:
> > > > > >> > > >>
> > > > > >> > > >> > Hello Mohit,
> > > > > >> > > >> >
> > > > > >> > > >> > Try hbck once and see if it shows any
> inconsistency.
> > > > > Also,
> > > > > >> you
> > > > > >> > can
> > > > > >> > > >> try
> > > > > >> > > >> > restarting your cluster and deleting the table again.
> > > > > Having a
> > > > > >> look
> > > > > >> > at
> > > > > >> > > >> the
> > > > > >> > > >> > logs could also be useful.
> > > > > >> > > >> >
> > > > > >> > > >> > Regards,
> > > > > >> > > >> > Mohammad Tariq
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >> >
> > > > > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> > > > > >> > > mohitanchlia-***@public.gmane.org
> > > > > >> > > >> > >wrote:
> > > > > >> > > >> >
> > > > > >> > > >> > > When I try to disable table I get:
> > > > > >> > > >> > >
> > > > > >> > > >> > > hbase(main):011:0> disable 'SESSIONID_TIMELINE'
> > > > > >> > > >> > > ERROR:
> > org.apache.hadoop.hbase.TableNotEnabledException:
> > > > > >> > > >> > > org.apache.hadoop.hbase.TableNotEnabledException:
> > > > > >> > SESSIONID_TIMELINE
> > > > > >> > > >> > > Here is some help for this command:
> > > > > >> > > >> > > Start disable of named table: e.g. "hbase> disable
> > 't1'"
> > > > > >> > > >> > >
> > > > > >> > > >> > > But then I try to enable I get:
> > > > > >> > > >> > >
> > > > > >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > > > > >> > > >> > > ERROR:
> > org.apache.hadoop.hbase.TableNotDisabledException:
> > > > > >> > > >> > > org.apache.hadoop.hbase.TableNotDisabledException:
> > > > > >> > > SESSIONID_TIMELINE
> > > > > >> > > >> > > Here is some help for this command:
> > > > > >> > > >> > > Start enable of named table: e.g. "hbase> enable
> 't1'"
> > > > > >> > > >> > >
> > > > > >> > > >> > > I've tried flush, major_compaction also. I tseems
> it's
> > > > > stuck in
> > > > > >> > > >> > > inconsistent state. Could someone point me to correct
> > > > > >> direction? I
> > > > > >> > > am
> > > > > >> > > >> > using
> > > > > >> > > >> > > 92.1
> > > > > >> > > >> > >
> > > > > >> > > >> >
> > > > > >> > > >>
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>
Ramkrishna.S.Vasudevan
2012-09-28 03:57:20 UTC
Permalink
Hi Mohith

Before restarting again just disable the compression means set it to default
and restart the cluster. This is just to ensure that the cluster is able to
come back from the enable/disable problem.
The Snappy problem could be different. I am suggesting this so that we
can isolate the problem is because of compression.

Regards
Ram

> -----Original Message-----
> From: Mohit Anchlia [mailto:mohitanchlia-***@public.gmane.org]
> Sent: Thursday, September 27, 2012 9:55 PM
> To: user-50Pas4EWwPEyzMRdD/***@public.gmane.org
> Subject: Re: disable table
>
> I did restart entire cluster and still that didn't help. Looks like
> once I
> get in this Race condition there is no way to come out of it?
>
> On Thu, Sep 27, 2012 at 8:00 AM, rajesh babu chintaguntla <
> chrajeshbabu32-***@public.gmane.org> wrote:
>
> > Hi Mohit,
> >
> > We should not delete znode's manually which will cause
> inconsistencies like
> > region may be shown as online on master, but it wont be on region
> server.
> > That's put is failing in your case. Master restart will bring back
> your
> > cluster to normal state(recovery any failures in enable/disable).
> Even hbck
> > also wont solve this problem.
> >
> > FYI,
> > Presently discussion is going on this issue. You can follow jira
> associated
> > with this issue at
> > https://issues.apache.org/jira/browse/HBASE-6469
> >
> > On Thu, Sep 27, 2012 at 8:11 PM, Mohit Anchlia
> <mohitanchlia-***@public.gmane.org
> > >wrote:
> >
> > > Thanks everyone for the input, it's helpful. I did remove the znode
> from
> > > /hbase/table/SESSIONID_TIMELINE and after that I was able to list
> the
> > > table. At that point I tried to do a "put" but when I did a put I
> got a
> > > message NoRegionServer online. I looked in the logs and it says the
> > Failed
> > > to open region server at nodexxx. When I went to nodexxx it
> complains
> > > something about unable to run testcompression.
> > >
> > > I setup SNAPPY compression on my table and I also ran SNAPPY
> compression
> > > test which was successful. Not sure what's going on in the cluster.
> > > On Thu, Sep 27, 2012 at 1:10 AM, Mohammad Tariq
> <dontariq-***@public.gmane.org>
> > > wrote:
> > >
> > > > Hello Mohit,
> > > >
> > > > It should be /hbase/hbase/table/SESSIONID_TIMELINE..Apologies
> for
> > the
> > > > typo. For rest of the things, I feel Ramkrishna sir has provided
> a good
> > > and
> > > > proper explanation. Please let us know if you still have any
> doubt or
> > > > question.
> > > >
> > > > Ramkrishna.S.Vasudevan : You are welcome sir. It's my pleasure to
> share
> > > > space with you people.
> > > >
> > > > Regards,
> > > > Mohammad Tariq
> > > >
> > > >
> > > >
> > > > On Thu, Sep 27, 2012 at 9:59 AM, Ramkrishna.S.Vasudevan <
> > > > ramkrishna.vasudevan-hv44wF8Li93QT0dZR+***@public.gmane.org> wrote:
> > > >
> > > > > Hi Mohith
> > > > > First of all thanks to Tariq for his replies.
> > > > >
> > > > > Just to add on,
> > > > > Basically HBase uses the Zookeeper to know the status of the
> cluster
> > > > like
> > > > > the no of tables enabled, disabled and deleted.
> > > > > Enabled and deleted states are handled bit different in the
> 0.94
> > > version.
> > > > >
> > > > > ZK is used for various region assignments.
> > > > >
> > > > > Also the ZK is used to track the Active master and standby
> master.
> > > > >
> > > > > As you understand correctly that the master is responsible for
> the
> > > > overall
> > > > > maintenance of the no of tables and their respective states, it
> seeks
> > > the
> > > > > help of ZK to do it and that is where the states are persisted.
> > > > >
> > > > > Also there are few cases where the enable and disable table are
> > having
> > > > some
> > > > > issues due to some race conditions in the 0.92 versions, In the
> > latest
> > > > > version we are trying to resolve them.
> > > > > You can attach the master and RS logs to identify exactly what
> caused
> > > > this
> > > > > problem in your case which will be really help ful so that I
> can be
> > > fixed
> > > > > in
> > > > > the kernel.
> > > > >
> > > > > Regards
> > > > > Ram
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Mohit Anchlia [mailto:mohitanchlia-***@public.gmane.org]
> > > > > > Sent: Thursday, September 27, 2012 5:09 AM
> > > > > > To: user-50Pas4EWwPEyzMRdD/***@public.gmane.org
> > > > > > Subject: Re: disable table
> > > > > >
> > > > > > I did /hbase/table/SESSIONID_TIMELINE and that seem to work.
> I'll
> > > > > > restart
> > > > > > hbase and see if it works.
> > > > > >
> > > > > > One thing I don't understand is why is zookeeper holding
> > information
> > > > > > about
> > > > > > this table if it is enabled or disabled? Wouldn't this
> information
> > be
> > > > > > with
> > > > > > master?
> > > > > >
> > > > > > On Wed, Sep 26, 2012 at 4:27 PM, Mohit Anchlia
> > > > > > <mohitanchlia-***@public.gmane.org>wrote:
> > > > > >
> > > > > > > I don't see path like /hbase/SESSIONID_TIMELINE
> > > > > > > This is what I see
> > > > > > >
> > > > > > > [zk: pprfdaaha303:5181(CONNECTED) 5] ls /hbase/table
> > > > > > > [SESSIONID_TIMELINE]
> > > > > > > [zk: pprfdaaha303:5181(CONNECTED) 6] get /hbase/table
> > > > > > >
> > > > > > > cZxid = 0x1000000fe
> > > > > > > ctime = Mon Sep 10 15:31:45 PDT 2012
> > > > > > > mZxid = 0x1000000fe
> > > > > > > mtime = Mon Sep 10 15:31:45 PDT 2012
> > > > > > > pZxid = 0x5000008f1
> > > > > > > cversion = 3
> > > > > > > dataVersion = 0
> > > > > > > aclVersion = 0
> > > > > > > ephemeralOwner = 0x0
> > > > > > > dataLength = 0
> > > > > > > numChildren = 1
> > > > > > >
> > > > > > > On Wed, Sep 26, 2012 at 3:57 PM, Mohammad Tariq
> > > > > > <dontariq-***@public.gmane.org>wrote:
> > > > > > >
> > > > > > >> In order to delete a znode you have to go to the ZK shell
> and
> > > issue
> > > > > > the
> > > > > > >> "delete" command along with the required path. For example
> :
> > > > > > >> delete /hbase/SESSIONID_TIMELINE. For detailed info you
> can
> > visit
> > > > > > the ZK
> > > > > > >> homepage at : zookeeper.apache.org
> > > > > > >>
> > > > > > >> Actually when we try to fetch data from an Hbase table,
> the
> > client
> > > > > > or app
> > > > > > >> first contacts the ZK to get the location of server
> holding the
> > > > > > >> -ROOT- table. From this we come to know about the server
> hosting
> > > the
> > > > > > >> .META.
> > > > > > >> table. This tells us the location of the server which
> actually
> > > holds
> > > > > > the
> > > > > > >> rows of interest. Because of some reasons the znode which
> was
> > > > > > holding this
> > > > > > >> info has either faced some catastrophe or lost the info
> > associated
> > > > > > with
> > > > > > >> this particular table. Or sometimes the znode remains
> unable to
> > > keep
> > > > > > >> itself
> > > > > > >> updated with the latest changes. That could also be a
> probable
> > > > > > reason. We
> > > > > > >> should always keep in mind that ZK is the centralized
> service
> > that
> > > > > > >> actually
> > > > > > >> coordinating everything behind the scene. As a result, any
> > problem
> > > > > > to the
> > > > > > >> ZK quorum means problem with Hbase custer.
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Mohammad Tariq
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Sep 27, 2012 at 3:39 AM, Mohit Anchlia
> > > > > > <mohitanchlia-***@public.gmane.org
> > > > > > >> >wrote:
> > > > > > >>
> > > > > > >> > Thanks! I do see Inconsistency. How do I remove the
> znode. And
> > > > > > also
> > > > > > >> could
> > > > > > >> > you please help me understand how this might have
> happened?
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > ERROR: Region
> > > > > > >> >
> > > > > >
> SESSIONID_TIMELINE,,1348689726526.0e200aace5e81cead8d8714ed8076050.
> > > not
> > > > > > >> > deployed on any region server.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Sep 26, 2012 at 2:36 PM, Mohammad Tariq
> > > > > > <dontariq-***@public.gmane.org>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > A possible reason could be that the znode associated
> with
> > this
> > > > > > >> particular
> > > > > > >> > > table is not behaving properly. In such case, you can
> try
> > the
> > > > > > >> following:
> > > > > > >> > >
> > > > > > >> > > Stop Hbase
> > > > > > >> > > Stop ZK
> > > > > > >> > > Take a backup of ZK data
> > > > > > >> > > Restart ZK
> > > > > > >> > > Remove the znode
> > > > > > >> > > Start Hbase again
> > > > > > >> > >
> > > > > > >> > > After this hopefully your table would be enabled.
> > > > > > >> > >
> > > > > > >> > > Regards,
> > > > > > >> > > Mohammad Tariq
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Thu, Sep 27, 2012 at 2:59 AM, Mohammad Tariq
> > > > > > <dontariq-***@public.gmane.org>
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Yes. Also have a look at the logs of the problematic
> > region
> > > if
> > > > > > hbck
> > > > > > >> > shows
> > > > > > >> > > > any inconsistency.
> > > > > > >> > > >
> > > > > > >> > > > Regards,
> > > > > > >> > > > Mohammad Tariq
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Thu, Sep 27, 2012 at 2:55 AM, Mohit Anchlia <
> > > > > > >> mohitanchlia-***@public.gmane.org
> > > > > > >> > > >wrote:
> > > > > > >> > > >
> > > > > > >> > > >> Which node should I look at for logs? Is this the
> master
> > > > > > node? I'll
> > > > > > >> > try
> > > > > > >> > > >> hbck.
> > > > > > >> > > >>
> > > > > > >> > > >> On Wed, Sep 26, 2012 at 2:19 PM, Mohammad Tariq <
> > > > > > >> dontariq-***@public.gmane.org>
> > > > > > >> > > >> wrote:
> > > > > > >> > > >>
> > > > > > >> > > >> > Hello Mohit,
> > > > > > >> > > >> >
> > > > > > >> > > >> > Try hbck once and see if it shows any
> > inconsistency.
> > > > > > Also,
> > > > > > >> you
> > > > > > >> > can
> > > > > > >> > > >> try
> > > > > > >> > > >> > restarting your cluster and deleting the table
> again.
> > > > > > Having a
> > > > > > >> look
> > > > > > >> > at
> > > > > > >> > > >> the
> > > > > > >> > > >> > logs could also be useful.
> > > > > > >> > > >> >
> > > > > > >> > > >> > Regards,
> > > > > > >> > > >> > Mohammad Tariq
> > > > > > >> > > >> >
> > > > > > >> > > >> >
> > > > > > >> > > >> >
> > > > > > >> > > >> > On Thu, Sep 27, 2012 at 2:24 AM, Mohit Anchlia <
> > > > > > >> > > mohitanchlia-***@public.gmane.org
> > > > > > >> > > >> > >wrote:
> > > > > > >> > > >> >
> > > > > > >> > > >> > > When I try to disable table I get:
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > hbase(main):011:0> disable
> 'SESSIONID_TIMELINE'
> > > > > > >> > > >> > > ERROR:
> > > org.apache.hadoop.hbase.TableNotEnabledException:
> > > > > > >> > > >> > >
> org.apache.hadoop.hbase.TableNotEnabledException:
> > > > > > >> > SESSIONID_TIMELINE
> > > > > > >> > > >> > > Here is some help for this command:
> > > > > > >> > > >> > > Start disable of named table: e.g. "hbase>
> disable
> > > 't1'"
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > But then I try to enable I get:
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > hbase(main):012:0> enable 'SESSIONID_TIMELINE'
> > > > > > >> > > >> > > ERROR:
> > > org.apache.hadoop.hbase.TableNotDisabledException:
> > > > > > >> > > >> > >
> org.apache.hadoop.hbase.TableNotDisabledException:
> > > > > > >> > > SESSIONID_TIMELINE
> > > > > > >> > > >> > > Here is some help for this command:
> > > > > > >> > > >> > > Start enable of named table: e.g. "hbase>
> enable
> > 't1'"
> > > > > > >> > > >> > >
> > > > > > >> > > >> > > I've tried flush, major_compaction also. I
> tseems
> > it's
> > > > > > stuck in
> > > > > > >> > > >> > > inconsistent state. Could someone point me to
> correct
> > > > > > >> direction? I
> > > > > > >> > > am
> > > > > > >> > > >> > using
> > > > > > >> > > >> > > 92.1
> > > > > > >> > > >> > >
> > > > > > >> > > >> >
> > > > > > >> > > >>
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
Dhaval Shah
2012-09-26 21:28:40 UTC
Permalink
I have had similar problems and it seems like zookeeper and hbase master have different notions of whether the table is enabled or not.. Stopping the cluster, deleting zookeeper data and then starting it worked for me in this scenario

Regards,
Dhaval


________________________________
From: Mohit Anchlia <mohitanchlia-***@public.gmane.org>
To: user-50Pas4EWwPEyzMRdD/***@public.gmane.org
Sent: Wednesday, 26 September 2012 4:54 PM
Subject: disable table

When I try to disable table I get:

hbase(main):011:0> disable 'SESSIONID_TIMELINE'
ERROR: org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: SESSIONID_TIMELINE
Here is some help for this command:
Start disable of named table: e.g. "hbase> disable 't1'"

But then I try to enable I get:

hbase(main):012:0> enable 'SESSIONID_TIMELINE'
ERROR: org.apache.hadoop.hbase.TableNotDisabledException:
org.apache.hadoop.hbase.TableNotDisabledException: SESSIONID_TIMELINE
Here is some help for this command:
Start enable of named table: e.g. "hbase> enable 't1'"

I've tried flush, major_compaction also. I tseems it's stuck in
inconsistent state. Could someone point me to correct direction? I am using
92.1 
Continue reading on narkive:
Loading...