Tuesday 12 August 2014

Hive FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired.

Our issue:

In Hive CLI

INSERT INTO TABLE parquet_table PARTITION(year, month) 
SELECT * FROM regular_table; 

After a long time, shows the error: 

Error in acquireLocks... 
FAILED: Error in acquiring locks: Locks on the underlying objects cannot be acquired. retry after some time 

Rerun by "hive --hiveconf hive.root.logger=ALL,console"

14/08/11 16:05:37 DEBUG zookeeper.ClientCnxn: Session establishment request sent on  14/08/11 16:05:37 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 14/08/11 16:05:37 INFO ql.Driver: <PERFLOG method=acquireReadWriteLocks> 14/08/11 16:05:38 INFO zookeeper.ClientCnxn: Opening socket connection to server. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration) 14/08/11 16:05:38 INFO zookeeper.ClientCnxn: Socket connection established to , initiating session 14/08/11 16:05:38 DEBUG zookeeper.ClientCnxn: Session establishment request sent on  14/08/11 16:05:38 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 14/08/11 16:05:38 WARN ZooKeeperHiveLockManager: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hive_zookeeper_namespace_hive1/default at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.createChild(ZooKeeperHiveLockManager.java:285) at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lockPrimitive(ZooKeeperHiveLockManager.java:354) at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lock(ZooKeeperHiveLockManager.java:303) at org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lock(ZooKeeperHiveLockManager.java:220) at org.apache.hadoop.hive.ql.Driver.acquireReadWriteLocks(Driver.java:858) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:975) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) 14/08/11 16:05:38 INFO zookeeper.ClientCnxn: Opening socket connection to server . Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration) 14/08/11 16:05:38 INFO zookeeper.ClientCnxn: Socket connection established to  initiating session 14/08/11 16:05:38 DEBUG zookeeper.ClientCnxn: Session establishment request sent on  14/08/11 16:05:38 INFO zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 14/08/11 16:05:38 DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown input java.net.SocketException: Transport endpoint is not connected at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:665) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:430) at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:189) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097) 14/08/11 16:05:38 DEBUG zookeeper.ClientCnxnSocketNIO: Ignoring exception during shutdown output java.net.SocketException: Transport endpoint is not connected at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:676) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:438) at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:196) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1157) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1097)


Hive uses ZooKeeper to implement its locking facility, which is then used as a basis of concurrency support, that is, to support concurrent read/write to a same table/partition without causing any corruption to table data/metadata. You can find more details about locking in Hive in following page: 

https://cwiki.apache.org/confluence/display/Hive/Locking
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_18_5.html



4 comments:

  1. what is the solution for this?

    ReplyDelete
  2. sorry I dont understand the error completely . why Locks on the underlying objects cannot be acquired?

    ReplyDelete
  3. This comment has been removed by the author.

    ReplyDelete
  4. Hi ,

    Please set this property below and run it.

    SET hive.support.concurrency=false;

    By default it is true.

    Regards
    Nitish Khanna

    ReplyDelete