HBase Get Encounters java.lang.IndexOutOfBoundsException

Discussion:

William Shen

2018-12-05 19:21:57 UTC

Hi there,

We've recently encountered issue retrieving data from our HBase cluster,
and have not had much luck troubleshooting the issue. We narrowed down our
issue to a single GET, which appears to be caused by
FastDiffDeltaEncoder.java running into java.lang.IndexOutOfBoundsException.
Has anyone encountered similar issues before, or does anyone have
experience troubleshooting issues such as this one? Any help would be much
appreciated! We are running 1.2.0-cdh5.9.2, and the GET in question is:

hbase(main):004:0> get 'qa2.ADGROUPS',
"\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"

COLUMN CELL

ERROR: java.io.IOException

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)

at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)

at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)

at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

Caused by: java.lang.IndexOutOfBoundsException

at java.nio.Buffer.checkBounds(Buffer.java:567)

at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)

at
org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)

at
org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)

at
org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)

at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)

at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)

at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)

at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)

at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)

at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706)

at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865)

at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643)

at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620)

at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606)

at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801)

at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779)

at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)

at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)

... 3 more

Thank you very much in advance!

Stack

2018-12-05 20:53:14 UTC

Permalink

Looks like bug in FastDiffDeltaEncoder triggered by whatever the current
form of the target file. Can you figure which file it is (going by the Get
coordinates?). I suppose the compactor is running into the same problem
(was thinking a major compaction might get you over this hump). You could
make a copy of the problematic file and play with it standalone to see if
can figure the bug. Failing that, post to a JIRA if you yourself can't
figure it so someone else might have a go at it?

Thanks,
S

Post by William Shen
Hi there,
We've recently encountered issue retrieving data from our HBase cluster,
and have not had much luck troubleshooting the issue. We narrowed down our
issue to a single GET, which appears to be caused by
FastDiffDeltaEncoder.java running into java.lang.IndexOutOfBoundsException.
Has anyone encountered similar issues before, or does anyone have
experience troubleshooting issues such as this one? Any help would be much
hbase(main):004:0> get 'qa2.ADGROUPS',
"\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"
COLUMN CELL
ERROR: java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkBounds(Buffer.java:567)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
at
org.apache.hadoop.hbase.io
.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
at
org.apache.hadoop.hbase.io
.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
at
org.apache.hadoop.hbase.io
.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
at
org.apache.hadoop.hbase.io
.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
... 3 more
Thank you very much in advance!

William Shen

2018-12-05 22:24:55 UTC

Permalink

Thank you Stack.
I was able to isolate the specific Hfile causing the exception. Do you mind
teaching me how to play with the file standalone? I am not sure if I know
how to do that.
Thanks!

Post by Stack
Looks like bug in FastDiffDeltaEncoder triggered by whatever the current
form of the target file. Can you figure which file it is (going by the Get
coordinates?). I suppose the compactor is running into the same problem
(was thinking a major compaction might get you over this hump). You could
make a copy of the problematic file and play with it standalone to see if
can figure the bug. Failing that, post to a JIRA if you yourself can't
figure it so someone else might have a go at it?
Thanks,
S

Post by William Shen
Hi there,
We've recently encountered issue retrieving data from our HBase cluster,
and have not had much luck troubleshooting the issue. We narrowed down

our

Post by William Shen
issue to a single GET, which appears to be caused by
FastDiffDeltaEncoder.java running into

java.lang.IndexOutOfBoundsException.

Post by William Shen
Has anyone encountered similar issues before, or does anyone have
experience troubleshooting issues such as this one? Any help would be

much

Post by William Shen
hbase(main):004:0> get 'qa2.ADGROUPS',

"\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"

Post by William Shen
COLUMN CELL
ERROR: java.io.IOException
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
Caused by: java.lang.IndexOutOfBoundsException
at java.nio.Buffer.checkBounds(Buffer.java:567)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
at
org.apache.hadoop.hbase.io
.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
at
org.apache.hadoop.hbase.io

.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)

Post by William Shen
at
org.apache.hadoop.hbase.io

.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)

Post by William Shen
at
org.apache.hadoop.hbase.io
.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
at

org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)