Discussion:
Table is disabled an no way to get it back online
Cecile, Adam
2016-11-22 12:16:44 UTC
Permalink
Hello,


We're having a table stuck in disabled state. First I'd like to start with what I tried already:


* Restart all machines involved in HBase cluster

* hbase hbck with varios arguments

* hdfs fsck

* Purge ZK /hbase and restart masters


Now more details anout what happens:

* When enabling from hbase shell:


hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running


The task ID changes every time I run the command so I think it's talking about itself (and it gets stuck for a while before saying anything)


In the log, all I can see is:

2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure: Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure: Table sentinel-meta isn't enabled; skipping disable


Please also find attached a complete log from startup to shutdown on a single active master. You'll see the table is found as well as the regions but it gets deactivated with no reason.


Thanks a lot for your help, we're kinda running out if ideas here.


Best regards,


Adam.
Ted Yu
2016-11-22 14:46:15 UTC
Permalink
Master log contained entries in the following form:

2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta

Note the procedure Id was around 43000, far lower than 147464.

Can you turn debug log on and repost master log ?

Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start with
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's talking
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
sentinel-meta
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
sentinel-meta
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a
single active master. You'll see the table is found as well as the regions
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Cecile, Adam
2016-11-22 15:16:39 UTC
Permalink
Hey Ted,

Thank you. Heading home right now but I'll start the laptop again. Not sure exactly how I should turn debug log on so if you have the information it'd be appreciated, otherwise I'll look at the xml files.

Regards, Adam.
________________________________________
De : Ted Yu <***@gmail.com>
Envoyé : mardi 22 novembre 2016 15:46
À : ***@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online

Master log contained entries in the following form:

2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta

Note the procedure Id was around 43000, far lower than 147464.

Can you turn debug log on and repost master log ?

Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start with
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's talking
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
sentinel-meta
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
sentinel-meta
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a
single active master. You'll see the table is found as well as the regions
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Ted Yu
2016-11-22 16:05:21 UTC
Permalink
In log4j.properties :

log4j.logger.org.apache.hadoop.hbase=DEBUG

On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).

Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again. Not
sure exactly how I should turn debug log on so if you have the information
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's talking
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Cecile, Adam
2016-11-22 17:42:34 UTC
Permalink
Hello,

Sadly I could not use the webui, it killed my firefox (probably way too much time). Here is the debug log... (11Mb uncompressed for maybe two minutes running !!)

Best regards, Adam.
________________________________________
De : Ted Yu <***@gmail.com>
Envoyé : mardi 22 novembre 2016 17:05
À : ***@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online

In log4j.properties :

log4j.logger.org.apache.hadoop.hbase=DEBUG

On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).

Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again. Not
sure exactly how I should turn debug log on so if you have the information
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's talking
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Cecile, Adam
2016-11-22 17:48:01 UTC
Permalink
Another one, because I'm not sure the log is overwritten when restarting. This one has been cleared before service start.
________________________________________
De : Cecile, Adam <***@hitec.lu>
Envoyé : mardi 22 novembre 2016 18:42
À : ***@hbase.apache.org
Objet : RE: Table is disabled an no way to get it back online

Hello,

Sadly I could not use the webui, it killed my firefox (probably way too much time). Here is the debug log... (11Mb uncompressed for maybe two minutes running !!)

Best regards, Adam.
________________________________________
De : Ted Yu <***@gmail.com>
Envoyé : mardi 22 novembre 2016 17:05
À : ***@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online

In log4j.properties :

log4j.logger.org.apache.hadoop.hbase=DEBUG

On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).

Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again. Not
sure exactly how I should turn debug log on so if you have the information
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's talking
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Ted Yu
2016-11-22 18:00:16 UTC
Permalink
Please take a look at HBASE-13415

From the log, you're using hbase 1.2.0 already. But I heard there is a
subtle bug which is being fixed.

Matteo is the person with best knowledge in this regard.
Post by Cecile, Adam
Another one, because I'm not sure the log is overwritten when restarting.
This one has been cleared before service start.
________________________________________
Envoyé : mardi 22 novembre 2016 18:42
Objet : RE: Table is disabled an no way to get it back online
Hello,
Sadly I could not use the webui, it killed my firefox (probably way too
much time). Here is the debug log... (11Mb uncompressed for maybe two
minutes running !!)
Best regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 17:05
Objet : Re: Table is disabled an no way to get it back online
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again. Not
sure exactly how I should turn debug log on so if you have the
information
Post by Cecile, Adam
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
sentinel-meta
Post by Cecile, Adam
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's
talking
Post by Cecile, Adam
Post by Cecile, Adam
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Cecile, Adam
2016-11-22 18:05:30 UTC
Permalink
Thanks for everything. As you said, this bug is supposed to be fixed in 1.2.0. Matteo is reading this list as well ?

Regards, Adam.
________________________________________
De : Ted Yu <***@gmail.com>
Envoyé : mardi 22 novembre 2016 19:00
À : ***@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online

Please take a look at HBASE-13415

From the log, you're using hbase 1.2.0 already. But I heard there is a
subtle bug which is being fixed.

Matteo is the person with best knowledge in this regard.
Post by Cecile, Adam
Another one, because I'm not sure the log is overwritten when restarting.
This one has been cleared before service start.
________________________________________
Envoyé : mardi 22 novembre 2016 18:42
Objet : RE: Table is disabled an no way to get it back online
Hello,
Sadly I could not use the webui, it killed my firefox (probably way too
much time). Here is the debug log... (11Mb uncompressed for maybe two
minutes running !!)
Best regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 17:05
Objet : Re: Table is disabled an no way to get it back online
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again. Not
sure exactly how I should turn debug log on so if you have the
information
Post by Cecile, Adam
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
sentinel-meta
Post by Cecile, Adam
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's
talking
Post by Cecile, Adam
Post by Cecile, Adam
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Matteo Bertozzi
2016-11-22 18:12:28 UTC
Permalink
I don't think this has anything to do with HBASE-13415 or the bugfix it
related to it i'm working on.

this is probably the usual case of mismatch state with zk. Disable is
saying that the table is already not disabled.
so, enableTable() is the one that should give any exception in case. but I
don't see any.
I suggest to just drop the MasterWALs directory, drop the znode
/hbase/table/sentinel-meta, restart the master and try disable and then
enable. that should bring you back with the table online

Matteo
Post by Cecile, Adam
Thanks for everything. As you said, this bug is supposed to be fixed in
1.2.0. Matteo is reading this list as well ?
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 19:00
Objet : Re: Table is disabled an no way to get it back online
Please take a look at HBASE-13415
From the log, you're using hbase 1.2.0 already. But I heard there is a
subtle bug which is being fixed.
Matteo is the person with best knowledge in this regard.
Post by Cecile, Adam
Another one, because I'm not sure the log is overwritten when restarting.
This one has been cleared before service start.
________________________________________
Envoyé : mardi 22 novembre 2016 18:42
Objet : RE: Table is disabled an no way to get it back online
Hello,
Sadly I could not use the webui, it killed my firefox (probably way too
much time). Here is the debug log... (11Mb uncompressed for maybe two
minutes running !!)
Best regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 17:05
Objet : Re: Table is disabled an no way to get it back online
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again. Not
sure exactly how I should turn debug log on so if you have the
information
Post by Cecile, Adam
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
sentinel-meta
Post by Cecile, Adam
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's
talking
Post by Cecile, Adam
Post by Cecile, Adam
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
hbase.
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
hbase.
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on
a
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Cecile, Adam
2016-11-22 18:42:30 UTC
Permalink
Can you explain me how to delete masterwals directory?



Sent from my Samsung device


-------- Original message --------
From: Matteo Bertozzi <***@gmail.com>
Date: 22/11/2016 19:12 (GMT+01:00)
To: ***@hbase.apache.org
Subject: Re: Table is disabled an no way to get it back online

I don't think this has anything to do with HBASE-13415 or the bugfix it
related to it i'm working on.

this is probably the usual case of mismatch state with zk. Disable is
saying that the table is already not disabled.
so, enableTable() is the one that should give any exception in case. but I
don't see any.
I suggest to just drop the MasterWALs directory, drop the znode
/hbase/table/sentinel-meta, restart the master and try disable and then
enable. that should bring you back with the table online

Matteo
Post by Cecile, Adam
Thanks for everything. As you said, this bug is supposed to be fixed in
1.2.0. Matteo is reading this list as well ?
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 19:00
Objet : Re: Table is disabled an no way to get it back online
Please take a look at HBASE-13415
From the log, you're using hbase 1.2.0 already. But I heard there is a
subtle bug which is being fixed.
Matteo is the person with best knowledge in this regard.
Post by Cecile, Adam
Another one, because I'm not sure the log is overwritten when restarting.
This one has been cleared before service start.
________________________________________
Envoyé : mardi 22 novembre 2016 18:42
Objet : RE: Table is disabled an no way to get it back online
Hello,
Sadly I could not use the webui, it killed my firefox (probably way too
much time). Here is the debug log... (11Mb uncompressed for maybe two
minutes running !!)
Best regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 17:05
Objet : Re: Table is disabled an no way to get it back online
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again. Not
sure exactly how I should turn debug log on so if you have the
information
Post by Cecile, Adam
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
sentinel-meta
Post by Cecile, Adam
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to start
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's
talking
Post by Cecile, Adam
Post by Cecile, Adam
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
hbase.
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
hbase.
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on
a
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Matteo Bertozzi
2016-11-22 18:48:17 UTC
Permalink
hadoop fs -rmr /hbase/MasterProcWALs

Matteo
Post by Cecile, Adam
Can you explain me how to delete masterwals directory?
Sent from my Samsung device
-------- Original message --------
Date: 22/11/2016 19:12 (GMT+01:00)
Subject: Re: Table is disabled an no way to get it back online
I don't think this has anything to do with HBASE-13415 or the bugfix it
related to it i'm working on.
this is probably the usual case of mismatch state with zk. Disable is
saying that the table is already not disabled.
so, enableTable() is the one that should give any exception in case. but I
don't see any.
I suggest to just drop the MasterWALs directory, drop the znode
/hbase/table/sentinel-meta, restart the master and try disable and then
enable. that should bring you back with the table online
Matteo
Post by Cecile, Adam
Thanks for everything. As you said, this bug is supposed to be fixed in
1.2.0. Matteo is reading this list as well ?
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 19:00
Objet : Re: Table is disabled an no way to get it back online
Please take a look at HBASE-13415
From the log, you're using hbase 1.2.0 already. But I heard there is a
subtle bug which is being fixed.
Matteo is the person with best knowledge in this regard.
Post by Cecile, Adam
Another one, because I'm not sure the log is overwritten when
restarting.
Post by Cecile, Adam
Post by Cecile, Adam
This one has been cleared before service start.
________________________________________
Envoyé : mardi 22 novembre 2016 18:42
Objet : RE: Table is disabled an no way to get it back online
Hello,
Sadly I could not use the webui, it killed my firefox (probably way too
much time). Here is the debug log... (11Mb uncompressed for maybe two
minutes running !!)
Best regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 17:05
Objet : Re: Table is disabled an no way to get it back online
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again.
Not
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sure exactly how I should turn debug log on so if you have the
information
Post by Cecile, Adam
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
sentinel-meta
Post by Cecile, Adam
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to
start
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's
talking
Post by Cecile, Adam
Post by Cecile, Adam
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
hbase.
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
hbase.
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown
on
Post by Cecile, Adam
a
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Cecile, Adam
2016-11-23 07:13:07 UTC
Permalink
Hello,

Thanks a lot, the table is back online. One last question ? Can you provide a log pattern to spot this ? Just in case it occurs again ;-)

Regards, Adam.
________________________________________
De : Matteo Bertozzi <***@gmail.com>
Envoyé : mardi 22 novembre 2016 19:48
À : ***@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online

hadoop fs -rmr /hbase/MasterProcWALs

Matteo
Post by Cecile, Adam
Can you explain me how to delete masterwals directory?
Sent from my Samsung device
-------- Original message --------
Date: 22/11/2016 19:12 (GMT+01:00)
Subject: Re: Table is disabled an no way to get it back online
I don't think this has anything to do with HBASE-13415 or the bugfix it
related to it i'm working on.
this is probably the usual case of mismatch state with zk. Disable is
saying that the table is already not disabled.
so, enableTable() is the one that should give any exception in case. but I
don't see any.
I suggest to just drop the MasterWALs directory, drop the znode
/hbase/table/sentinel-meta, restart the master and try disable and then
enable. that should bring you back with the table online
Matteo
Post by Cecile, Adam
Thanks for everything. As you said, this bug is supposed to be fixed in
1.2.0. Matteo is reading this list as well ?
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 19:00
Objet : Re: Table is disabled an no way to get it back online
Please take a look at HBASE-13415
From the log, you're using hbase 1.2.0 already. But I heard there is a
subtle bug which is being fixed.
Matteo is the person with best knowledge in this regard.
Post by Cecile, Adam
Another one, because I'm not sure the log is overwritten when
restarting.
Post by Cecile, Adam
Post by Cecile, Adam
This one has been cleared before service start.
________________________________________
Envoyé : mardi 22 novembre 2016 18:42
Objet : RE: Table is disabled an no way to get it back online
Hello,
Sadly I could not use the webui, it killed my firefox (probably way too
much time). Here is the debug log... (11Mb uncompressed for maybe two
minutes running !!)
Best regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 17:05
Objet : Re: Table is disabled an no way to get it back online
log4j.logger.org.apache.hadoop.hbase=DEBUG
On master UI, you can select the Procedures tab. Pastebin what you see
(text is enough).
Thanks
Post by Cecile, Adam
Hey Ted,
Thank you. Heading home right now but I'll start the laptop again.
Not
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sure exactly how I should turn debug log on so if you have the
information
Post by Cecile, Adam
it'd be appreciated, otherwise I'll look at the xml files.
Regards, Adam.
________________________________________
Envoyé : mardi 22 novembre 2016 15:46
Objet : Re: Table is disabled an no way to get it back online
2016-11-22 13:13:41,836 INFO [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43538 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
sentinel-meta
Post by Cecile, Adam
Note the procedure Id was around 43000, far lower than 147464.
Can you turn debug log on and repost master log ?
Thanks
Post by Cecile, Adam
Hello,
We're having a table stuck in disabled state. First I'd like to
start
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
with
Post by Cecile, Adam
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's
talking
Post by Cecile, Adam
Post by Cecile, Adam
about itself (and it gets stuck for a while before saying anything)
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.
hbase.
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0]
procedure2.ProcedureExecutor: Rolledback procedure
DisableTableProcedure
Post by Cecile, Adam
Post by Cecile, Adam
(table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.
hbase.
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.
Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown
on
Post by Cecile, Adam
a
Post by Cecile, Adam
Post by Cecile, Adam
Post by Cecile, Adam
single active master. You'll see the table is found as well as the
regions
Post by Cecile, Adam
but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
Loading...