Error:"MetaStoreClient lost connection. Attempting to reconnect (1 of 24) after 5s. getCurrentNotificationEventId" occurs as HiveServer2 fails to start as it cannot connect to Metastore in HDP 3.0

SupportKB

Problem Description:
After upgrading to HDP 3.0, the HiveServer2 fails to start and the following error is displayed in the HiveServer2 logs:

1. WARN [main]: metastore.RetryingMetaStoreClient (:()) -
2. MetaStoreClient lost connection. Attempting to reconnect (1 of 24) after 5s. getCurrentNotificationEventId
3. org.apache.thrift.TApplicationException: Internal error processing get_current_notificationEventId
4. at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
5. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
6. at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
7. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
8. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_current_notificationEventId
9. (ThriftHiveMetastore.java:5575) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
10. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_current_notificationEventId
11. (ThriftHiveMetastore.java:5563) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
12. at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getCurrentNotificationEventId
13. (HiveMetaStoreClient.java:2723) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
14. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
15. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
16. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
17. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
18. at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
19. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
20. at com.sun.proxy.$Proxy40.getCurrentNotificationEventId(Unknown Source) ~[?:?]
21. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
22. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112]
23. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
24. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
25. at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke
26. (HiveMetaStoreClient.java:2773) ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
27. at com.sun.proxy.$Proxy40.getCurrentNotificationEventId(Unknown Source) ~[?:?]
28. at org.apache.hadoop.hive.metastore.messaging.EventUtils$MSClientNotificationFetcher.
29. getCurrentNotificationEventId(EventUtils.java:73)
30. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
31. at org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.<init>
32. (NotificationEventPoll.java:103)
33. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
34. at org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll.initialize
35. (NotificationEventPoll.java:59)
36. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
37. at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:275)
38. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
39. at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1063)
40. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
41. at org.apache.hive.service.server.HiveServer2.access$1600(HiveServer2.java:140)
42. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
43. at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1332)
44. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
45. at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1176)
46. ~[hive-service-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
47. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112]
48. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
49. ~[?:1.8.0_112]
50. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112]
51. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
52. at org.apache.hadoop.util.RunJar.run(RunJar.java:318) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
53. at org.apache.hadoop.util.RunJar.main(RunJar.java:232) ~[hadoop-common-3.1.0.3.0.0.0-1634.jar:?]

In hive-metastore.log, the following message is displayed at the same timeline:

1. ERROR [pool-6-thread-200]: metastore.HiveMetaStore (HiveMetaStore.java:get_current_notificationEventId(7617))
2. - Not authorized to make the get_current_notificationEventId call.
3. You can try to disable metastore.metastore.event.db.notification.api.auth
4. org.apache.hadoop.hive.metastore.api.MetaException: User hive is not allowed to perform this API call
5. at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.authorizeProxyPrivilege
6. (HiveMetaStore.java:7655)
7. ~[hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
8. at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_current_notificationEventId(HiveMetaStore.java:7615)
9. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
10. at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) ~[?:?]
11. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
12. ~[?:1.8.0_112]
13. at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112]
14. at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
15. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
16. at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
17. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
18. at com.sun.proxy.$Proxy34.get_current_notificationEventId(Unknown Source) [?:?]
19. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_current_notificationEventId.getResult
20. (ThriftHiveMetastore.java:18364)
21. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
22. at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_current_notificationEventId.getResult
23. (ThriftHiveMetastore.java:18349) [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
24. at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
25. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
26. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
27. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
28. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
29. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
30. at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_112]
31. at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_112]
32. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)
33. [hadoop-common-3.1.0.3.0.0.0-1634.jar:?]
34. at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
35. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
36. at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
37. [hive-exec-3.1.0.3.0.0.0-1634.jar:3.1.0.3.0.0.0-1634]
38. at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
39. [?:1.8.0_112]
40. at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
41. [?:1.8.0_112]
42. at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

Cause:
The connection to Metastore from HiveServer2 fails after upgrading the cluster when hive.metastore.event.db.notification.api.auth property is set to true.
Solution:
This is a known issue (HIVE-19740). 

Workaround In HDFS > Configs > Custom core-site.xml, add the following:

1. hadoop.proxyuser.hive.hosts=HS2_HOST
2. hadoop.proxyuser.hive.groups=*

Or 

In Hive >  Configs > Custom hive-site.xml, add the following:

1. hive.metastore.event.db.notification.api.auth=false

Restart the required services. 

For more details about this property, see HiveReplicationv2Development

About:
This article created by Hortonworks Support (Article: 000007703) on 2018-08-06 07:32
OS: All
Type: Cluster_Administration
Version: H3.0.0

Support ID: 000007703