Cecile, Adam
2016-11-22 12:16:44 UTC
Hello,
We're having a table stuck in disabled state. First I'd like to start with what I tried already:
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
Now more details anout what happens:
* When enabling from hbase shell:
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's talking about itself (and it gets stuck for a while before saying anything)
In the log, all I can see is:
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure: Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure: Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a single active master. You'll see the table is found as well as the regions but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.
We're having a table stuck in disabled state. First I'd like to start with what I tried already:
* Restart all machines involved in HBase cluster
* hbase hbck with varios arguments
* hdfs fsck
* Purge ZK /hbase and restart masters
Now more details anout what happens:
* When enabling from hbase shell:
hbase(main):002:0> enable "sentinel-meta"
ERROR: The procedure 147464 is still running
The task ID changes every time I run the command so I think it's talking about itself (and it gets stuck for a while before saying anything)
In the log, all I can see is:
2016-11-22 13:10:50,776 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
2016-11-22 13:10:50,781 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure: Table sentinel-meta isn't enabled; skipping disable
2016-11-22 13:10:51,084 INFO [ProcedureExecutor-0] procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta
2016-11-22 13:10:51,088 INFO [ProcedureExecutor-0] procedure.DisableTableProcedure: Table sentinel-meta isn't enabled; skipping disable
Please also find attached a complete log from startup to shutdown on a single active master. You'll see the table is found as well as the regions but it gets deactivated with no reason.
Thanks a lot for your help, we're kinda running out if ideas here.
Best regards,
Adam.