zoukankan      html  css  js  c++  java
  • 【hbase】使用thrift with python 访问HBase

    HBase 版本: 0.98.6

    thrift   版本: 0.9.0

    使用 thrift client with python 连接 HBase 报错:

     1 Traceback (most recent call last):
     2   File "D:workspacePythonpyhelloworld.py", line 27, in <module>
     3     tables = client.getTableNames()
     4   File "E:mazhongsoftpythonlibhbaseHbase.py", line 788, in getTableNames
     5     return self.recv_getTableNames()
     6   File "E:mazhongsoftpythonlibhbaseHbase.py", line 798, in recv_getTableNames
     7     (fname, mtype, rseqid) = self._iprot.readMessageBegin()
     8   File "E:mazhongsoftpythonlib	hriftprotocolTBinaryProtocol.py", line 126, in readMessageBegin
     9     sz = self.readI32()
    10   File "E:mazhongsoftpythonlib	hriftprotocolTBinaryProtocol.py", line 203, in readI32
    11     buff = self.trans.readAll(4)
    12   File "E:mazhongsoftpythonlib	hrift	ransportTTransport.py", line 58, in readAll
    13     chunk = self.read(sz-have)
    14   File "E:mazhongsoftpythonlib	hrift	ransportTTransport.py", line 155, in read
    15     self.__rbuf = StringIO(self.__trans.read(max(sz, self.DEFAULT_BUFFER)))
    16   File "E:mazhongsoftpythonlib	hrift	ransportTSocket.py", line 94, in read
    17     raise TTransportException('TSocket read 0 bytes')
    18 thrift.transport.TTransport.TTransportException: None

    查找原因,过程如下:
    1) 客户端代码

    1 transport = TTransport.TBufferedTransport(TSocket('192.168.0.10', 9090))
    2 protocol = TBinaryProtocol.TBinaryProtocol(transport)
    3 client = Hbase.Client(protocol)
    4 transport.open()

    2) hbase-site.xml 配置如下

    1   <property>
    2      <name>hbase.regionserver.thrift.framed</name>
    3      <value>true</value>
    4   </property>
    5   <property>
    6      <name>hbase.regionserver.thrift.compact</name>
    7      <value>true</value>
    8   </property>

    3) thrift server日志如下:(/var/log/hbase/hbase-hbase-thrift-<hostname>.log)

     1 Wed Jan 14 14:54:43 CST 2015 Starting thrift on <hostname>
     2 core file size          (blocks, -c) 0
     3 data seg size           (kbytes, -d) unlimited
     4 scheduling priority             (-e) 0
     5 file size               (blocks, -f) unlimited
     6 pending signals                 (-i) 191956
     7 max locked memory       (kbytes, -l) unlimited
     8 max memory size         (kbytes, -m) unlimited
     9 open files                      (-n) 32768
    10 pipe size            (512 bytes, -p) 8
    11 POSIX message queues     (bytes, -q) 819200
    12 real-time priority              (-r) 0
    13 stack size              (kbytes, -s) 10240
    14 cpu time               (seconds, -t) unlimited
    15 max user processes              (-u) 1024
    16 virtual memory          (kbytes, -v) unlimited
    17 file locks                      (-x) unlimited
    18 INFO  [main] util.VersionInfo: HBase 0.98.6-cdh5.3.0
    19 INFO  [main] util.VersionInfo: Subversion file:///data/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hbase-0.98.6-cdh5.3.0 -r Unknown
    20 INFO  [main] util.VersionInfo: Compiled by jenkins on Tue Dec 16 19:13:29 PST 2014
    21 INFO  [main] thrift.ThriftServerRunner: Using default thrift server type
    22 INFO  [main] thrift.ThriftServerRunner: Using thrift server type threadpool
    23 INFO  [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties
    24 INFO  [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
    25 INFO  [main] impl.MetricsSystemImpl: HBase metrics system started
    26 INFO  [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
    27 INFO  [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
    28 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context thrift
    29 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
    30 INFO  [main] http.HttpServer: Jetty bound to port 9095
    31 INFO  [main] mortbay.log: jetty-6.1.26.cloudera.4
    32 INFO  [main] mortbay.log: Started HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:9095
    33 DEBUG [main] thrift.ThriftServerRunner: Using compact protocol
    34 DEBUG [main] thrift.ThriftServerRunner: Using framed transport
    35 INFO  [main] thrift.ThriftServerRunner: starting TBoundedThreadPoolServer on /0.0.0.0:9090; min worker threads=16, max worker threads=1000, max queued requests=1000

    由以上红色文字部分可知,是因为thrift 的server端和client端的协议不匹配造成的。

    解决方案有如下两个:
    方案一:修改hbase-site.xml,禁用TFramedTransport和TCompactProtocol功能,即:

    1   <property>
    2      <name>hbase.regionserver.thrift.framed</name>
    3      <value>false</value>
    4   </property>
    5   <property>
    6      <name>hbase.regionserver.thrift.compact</name>
    7      <value>false</value>
    8   </property>

    重启thrift 服务器: service hbase-thrift restart

    方案二:修改客户端代码

    1 transport = TFramedTransport(TSocket('192.168.0.10', 9090))  
    2 protocol = TCompactProtocol.TCompactProtocol(transport)  
    3 client = Hbase.Client(protocol)  
    4 transport.open()  

    如果报错未找到TFramedTransport和TCompactProtocol,请查看/usr/lib/python2.6/site-packages/thrift目录下有没有protocol/TCompactProtocol.py文件,检查transport/TTransport.py文件中有没有类TFramedTransport。如果没有,可以在thrift官网下载源码包 thrift-0.9.2.tar.gz,用其中的lib/py/src/覆盖/usr/lib/python2.6/site-packages/thrift/目录即可。

  • 相关阅读:
    洛谷 1850 NOIP2016提高组 换教室
    2018牛客多校第三场 C.Shuffle Cards
    2018牛客多校第一场 B.Symmetric Matrix
    2018牛客多校第一场 A.Monotonic Matrix
    2018牛客多校第一场 D.Two Graphs
    2018宁夏邀请赛L Continuous Intervals
    2018宁夏邀请赛K Vertex Covers
    BZOJ
    HDU
    ACM International Collegiate Programming Contest, Egyptian Collegiate Programming Contest (ECPC 2015)
  • 原文地址:https://www.cnblogs.com/zhangningbo/p/4303422.html
Copyright © 2011-2022 走看看