Monday, November 3, 2014

Getting Hive metastore to work with HDInsight Emulator.

For some reason, Hive metastore.exe (which runs as a service) did not create the metastore db that is required for Hive to store table definitions. 

Hive reported the following exception at startup in the log files. 
"set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out"


Because of this error, "Show databases" command did not work and I could not use Hive with HDInsight emulator. 

After hours of investigation, I made the following edits to "C:\hdp\hive-0.13.0.2.1.3.0-1981\conf\hive-site.xml". As you can see, I changed the settings to EmbeddedDriver instead of ClientDriver. 

  javax.jdo.option.ConnectionURL
 
  jdbc:derby:;databaseName=metastore_db;create=true
  JDBC connect string for a JDBC metastore

  javax.jdo.option.ConnectionDriverName
 
  org.apache.derby.jdbc.EmbeddedDriver
  Driver class name for a JDBC metastore

Then I stopped the default metastore.exe service. 
And used the following command to start metastore
"Hive.cmd --service metastore" 

with the initial directory set to hive home.

This created a folder named "metadata_db" and finally hive started successfully!! 

By default, the metastore service listens for requests on port 9083.
You can confirm that the service is running, by executing netstat command as shown below.  
netstat -ano | find "9083"

What is success?

The journey of life takes us through varied experiences like landing an admission at a prestigious college, earning a degree, getting hired,...