GoldenGate实时投递数据到大数据平台（7）– Apache Hbase

GoldenGate实时投递数据到大数据平台（7）– Apache Hbase

2018-03-06 10:39:00 weixin_33851177 阅读数 52

 

Apache Hbase安装及运行

安装hbase1.4，确保在这之前hadoop是正常运行的。设置相应的环境变量，
export HADOOP_HOME=/u01/hadoop
export HBASE_HOME=/u01/hbase
export PATH=$PATH:$HADOOP_HOME/bin:$HBASE_HOME/bin

启动hbase
./start-hbase.sh

确保hadoop, hbase能正常启动，如有问题，可自行搜索文档解决。
[oracle@ol66 bin]$ jps
11685 NodeManager
11157 SecondaryNameNode
10844 NameNode
11405 ResourceManager
13135 HMaster
13455 Jps
10959 DataNode

确保hbase在hadoop上正常运行
[oracle@ol66 u01]$ hdfs dfs -ls /
Found 3 items
drwxr-xr-x   - oracle supergroup          0 2018-02-28 00:52 /hbase
drwxr-xr-x   - oracle supergroup          0 2018-02-27 23:14 /ogg
drwxr-xr-x   - oracle supergroup          0 2018-02-28 00:33 /tmp

[oracle@ol66 bin]$ ./hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 1.4.0, r10b9b9fae6b557157644fb9a0dc641bb8cb26e39, Fri Dec  8 16:09:13 PST 2017

hbase(main):001:0> list
TABLE                                                                                                                                                                 
0 row(s) in 0.3800 seconds

=> []
hbase(main):002:0>

可以看到，系统中还没有任何表。

OGG安装及测试

配置OGG for bigdata的环境变量
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/jre/lib/amd64/server

安装ogg for bigdata 12.3版本软件
进入ggsci
ggsci>create subdirs

退回到操作系统命令行，在OGG安装目录下执行如下命令，拷贝hbase示例投递参数后进行修改
cp AdapterExamples/big-data/hbase/* dirprm/
在dirprm目录下，编辑hbase.props文件。
根据安装的hbase的路径，修改gg.classpath中 hbase lib的路径；保存退出。
hbase.props的完整内容如下：

gg.handlerlist=hbase

gg.handler.hbase.type=hbase
gg.handler.hbase.hBaseColumnFamilyName=cf
gg.handler.hbase.keyValueDelimiter=CDATA[=]
gg.handler.hbase.keyValuePairDelimiter=CDATA[,]
gg.handler.hbase.encoding=UTF-8
gg.handler.hbase.pkUpdateHandling=abend
gg.handler.hbase.nullValueRepresentation=CDATA[NULL]
gg.handler.hbase.authType=none
gg.handler.hbase.includeTokens=false

gg.handler.hbase.mode=tx

goldengate.userexit.timestamp=utc
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE

gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec

gg.classpath=/u01/hbase/lib/*:/u01/hbase/conf/:

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar

重新进入GGSCI，使用示例参数和示例队列创建投递进程。
GGSCI>add replicat rhbase, exttrail AdapterExamples/trail/tr
rhbase.prm的内容如下：

REPLICAT rhbase
-- Trail file for this example is located in "AdapterExamples/trail" directory
-- Command to add REPLICAT
-- add replicat rhbase, exttrail AdapterExamples/trail/tr
TARGETDB LIBFILE libggjava.so SET property=dirprm/hbase.props
REPORTCOUNT EVERY 1 MINUTES, RATE
GROUPTRANSOPS 10000
MAP QASOURCE.*, TARGET QASOURCE.*;

测试

启动投递进程
GGSCI (ol66) 19> start rhbase

Sending START request to MANAGER ...
REPLICAT RHBASE starting

GGSCI (ol66) 20> info rhbase

REPLICAT   RHBASE    Initialized   2018-02-28 00:53   Status STARTING
Checkpoint Lag       00:00:00 (updated 00:02:16 ago)
Process ID           15424
Log Read Checkpoint  File AdapterExamples/trail/tr000000000
                      First Record  RBA 0

GGSCI (ol66) 22> info rhbase

REPLICAT   RHBASE    Last Started 2018-02-28 00:55   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:02:20 ago)
Process ID           15424
Log Read Checkpoint  File /u01/ogg4bd/AdapterExamples/trail/tr000000000
                      First Record  RBA 0

GGSCI (ol66) 27> info rhbase

REPLICAT   RHBASE    Last Started 2018-02-28 00:55   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:01 ago)
Process ID           15424
Log Read Checkpoint  File /u01/ogg4bd/AdapterExamples/trail/tr000000000
                      2015-11-06 02:45:39.000000  RBA 5660

投递完成，在OGG中检查投递结果

GGSCI (ol66) 28> stats rhbase, total

Sending STATS request to REPLICAT RHBASE ...

Start of Statistics at 2018-02-28 00:56:23.

Replicating from QASOURCE.TCUSTMER to QASOURCE.TCUSTMER:

*** Total statistics since 2018-02-28 00:55:43 ***
         Total inserts                                      5.00
         Total updates                                      1.00
         Total deletes                                      0.00
         Total discards                                     0.00
         Total operations                                   6.00

Replicating from QASOURCE.TCUSTORD to QASOURCE.TCUSTORD:

*** Total statistics since 2018-02-28 00:55:43 ***
         Total inserts                                      5.00
         Total updates                                      3.00
         Total deletes                                      2.00
         Total discards                                     0.00
         Total operations                                  10.00

End of Statistics.

在hbase上查看，已经比刚开始多了2张表

hbase(main):002:0> list
TABLE                                                                                                                                                                 
QASOURCE:TCUSTMER                                                                                                                                                     
QASOURCE:TCUSTORD                                                                                                                                                     
2 row(s) in 0.0190 seconds

=> ["QASOURCE:TCUSTMER", "QASOURCE:TCUSTORD"]

查看数据
hbase(main):005:0> scan 'QASOURCE:TCUSTMER'
ROW                                        COLUMN+CELL                                                                                                                
  ANN                                       column=cf:CITY, timestamp=1519750550592, value=NEW YORK                                                                    
  ANN                                       column=cf:CUST_CODE, timestamp=1519750550592, value=ANN                                                                    
  ANN                                       column=cf:NAME, timestamp=1519750550592, value=ANN'S BOATS                                                                 
  ANN                                       column=cf:STATE, timestamp=1519750550592, value=NY                                                                         
  BILL                                      column=cf:CITY, timestamp=1519750550592, value=DENVER                                                                      
  BILL                                      column=cf:CUST_CODE, timestamp=1519750550592, value=BILL                                                                   
  BILL                                      column=cf:NAME, timestamp=1519750550592, value=BILL'S USED CARS                                                            
  BILL                                      column=cf:STATE, timestamp=1519750550592, value=CO                                                                         
  DAVE                                      column=cf:CITY, timestamp=1519750550592, value=TALLAHASSEE                                                                 
  DAVE                                      column=cf:CUST_CODE, timestamp=1519750550592, value=DAVE                                                                   
  DAVE                                      column=cf:NAME, timestamp=1519750550592, value=DAVE'S PLANES INC.                                                          
  DAVE                                      column=cf:STATE, timestamp=1519750550592, value=FL                                                                         
  JANE                                      column=cf:CITY, timestamp=1519750550421, value=DENVER                                                                      
  JANE                                      column=cf:CUST_CODE, timestamp=1519750550421, value=JANE                                                                   
  JANE                                      column=cf:NAME, timestamp=1519750550421, value=ROCKY FLYER INC.                                                            
  JANE                                      column=cf:STATE, timestamp=1519750550421, value=CO                                                                         
  WILL                                      column=cf:CITY, timestamp=1519750550421, value=SEATTLE                                                                     
  WILL                                      column=cf:CUST_CODE, timestamp=1519750550421, value=WILL                                                                   
  WILL                                      column=cf:NAME, timestamp=1519750550421, value=BG SOFTWARE CO.                                                             
  WILL                                      column=cf:STATE, timestamp=1519750550421, value=WA                                                                         
5 row(s) in 0.3150 seconds

hbase(main):001:0> scan 'QASOURCE:TCUSTORD'
ROW                                        COLUMN+CELL                                                                                                                
  BILL|1995-12-31 15:00:00|CAR|765          column=cf:CUST_CODE, timestamp=1519750550614, value=BILL                                                                   
  BILL|1995-12-31 15:00:00|CAR|765          column=cf:ORDER_DATE, timestamp=1519750550614, value=1995-12-31 15:00:00                                                   
  BILL|1995-12-31 15:00:00|CAR|765          column=cf:ORDER_ID, timestamp=1519750550614, value=765                                                                     
  BILL|1995-12-31 15:00:00|CAR|765          column=cf:PRODUCT_AMOUNT, timestamp=1519750550614, value=3                                                                 
  BILL|1995-12-31 15:00:00|CAR|765          column=cf:PRODUCT_CODE, timestamp=1519750550614, value=CAR                                                                 
  BILL|1995-12-31 15:00:00|CAR|765          column=cf:PRODUCT_PRICE, timestamp=1519750550614, value=14000.00                                                           
  BILL|1995-12-31 15:00:00|CAR|765          column=cf:TRANSACTION_ID, timestamp=1519750550614, value=100                                                               
  BILL|1996-01-01 00:00:00|TRUCK|333        column=cf:CUST_CODE, timestamp=1519750550614, value=BILL                                                                   
  BILL|1996-01-01 00:00:00|TRUCK|333        column=cf:ORDER_DATE, timestamp=1519750550614, value=1996-01-01 00:00:00                                                   
  BILL|1996-01-01 00:00:00|TRUCK|333        column=cf:ORDER_ID, timestamp=1519750550614, value=333                                                                     
  BILL|1996-01-01 00:00:00|TRUCK|333        column=cf:PRODUCT_AMOUNT, timestamp=1519750550614, value=15                                                                
  BILL|1996-01-01 00:00:00|TRUCK|333        column=cf:PRODUCT_CODE, timestamp=1519750550614, value=TRUCK                                                               
  BILL|1996-01-01 00:00:00|TRUCK|333        column=cf:PRODUCT_PRICE, timestamp=1519750550614, value=25000.00                                                           
  BILL|1996-01-01 00:00:00|TRUCK|333        column=cf:TRANSACTION_ID, timestamp=1519750550614, value=100                                                               
  WILL|1994-09-30 15:33:00|CAR|144          column=cf:CUST_CODE, timestamp=1519750550614, value=WILL                                                                   
  WILL|1994-09-30 15:33:00|CAR|144          column=cf:ORDER_DATE, timestamp=1519750550614, value=1994-09-30 15:33:00                                                   
  WILL|1994-09-30 15:33:00|CAR|144          column=cf:ORDER_ID, timestamp=1519750550614, value=144                                                                     
  WILL|1994-09-30 15:33:00|CAR|144          column=cf:PRODUCT_AMOUNT, timestamp=1519750550453, value=3                                                                 
  WILL|1994-09-30 15:33:00|CAR|144          column=cf:PRODUCT_CODE, timestamp=1519750550614, value=CAR                                                                 
  WILL|1994-09-30 15:33:00|CAR|144          column=cf:PRODUCT_PRICE, timestamp=1519750550614, value=16520.00                                                           
  WILL|1994-09-30 15:33:00|CAR|144          column=cf:TRANSACTION_ID, timestamp=1519750550453, value=100                                                               
3 row(s) in 0.6770 seconds

可以看到，OGG配置投递到Hbase非常简单，可以根据DB中表的主键字段创建key，如果没有PK字段，则投递时会报错。以下是当前OGG版本支持的Hbase版本