GoldenGate实时投递数据到大数据平台（4）- ElasticSearch 2.x
ES 2.x
ES 2.x安装
下载elasticSearch 2.4.5, https://www.elastic.co/downloads/elasticsearch
解压下载后的压缩包，启动ES
./elasticsearch
| [2017-12-26 11:46:31,662][INFO ][node ] [Doctor Dorcas] version[2.4.5], pid[4424], build[c849dd1/2017-04-24T16:18:17Z]
[2017-12-26 11:46:31,663][INFO ][node ] [Doctor Dorcas] initializing ... [2017-12-26 11:46:32,665][INFO ][plugins ] [Doctor Dorcas] modules [reindex, lang-expression, lang-groovy], plugins [], sites [] [2017-12-26 11:46:32,748][INFO ][env ] [Doctor Dorcas] using [1] data paths, mounts [[/u02 (/dev/sdb1)]], net usable_space [36.3gb], net total_space [39.2gb], spins? [possibly], types [ext4] [2017-12-26 11:46:32,749][INFO ][env ] [Doctor Dorcas] heap size [1007.3mb], compressed ordinary object pointers [true] [2017-12-26 11:46:34,954][INFO ][node ] [Doctor Dorcas] initialized [2017-12-26 11:46:34,954][INFO ][node ] [Doctor Dorcas] starting ... [2017-12-26 11:46:35,197][INFO ][transport ] [Doctor Dorcas] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2017-12-26 11:46:35,204][INFO ][discovery ] [Doctor Dorcas] elasticsearch/R6MDYLdHRReeDckGGa9snw [2017-12-26 11:46:38,283][INFO ][cluster.service ] [Doctor Dorcas] new_master {Doctor Dorcas}{R6MDYLdHRReeDckGGa9snw}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2017-12-26 11:46:38,363][INFO ][http ] [Doctor Dorcas] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2017-12-26 11:46:38,363][INFO ][gateway ] [Doctor Dorcas] recovered [0] indices into cluster_state [2017-12-26 11:46:38,363][INFO ][node ] [Doctor Dorcas] started |
查看ES服务状态
curl -GET http://localhost:9200/
| {
"name" : "Doctor Dorcas", "cluster_name" : "elasticsearch", "cluster_uuid" : "RXsZqkc7RvKejoXO3pVbdg", "version" : { "number" : "2.4.5", "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276", "build_timestamp" : "2017-04-24T16:18:17Z", "build_snapshot" : false, "lucene_version" : "5.5.4" }, "tagline" : "You Know, for Search" } |
安装认证插件
默认GoldenGate是通过shield组件访问ES 2.x，所以要有认证模块才能投递数据。
停止ES进程
bin/plugin install license
bin/plugin install shield
重新启动ES进程
创建OGG访问的用户，并指定为admin用户。
bin/shield/esusers useradd elastic –r damin
Enter new password: changeme
Retype new password: changeme
查看用户及角色
[oracle@ol73 shield]$ ./esusers list
elastic : admin,user
确认新建的用户有权限访问
此时匿名访问将报错
curl -GET http://localhost:9200/
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm="shield" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication token for REST request [/]","header":{"WWW-Authenticate":"Basic realm="shield" charset="UTF-8""}},"status":401}
需要带上用户密码即可访问
curl -u elastic:changeme -GET http://localhost:9200/
| {
"name" : "Cardinal", "cluster_name" : "elasticsearch", "cluster_uuid" : "8DEhE0t0R_u9tN6ZtwagkA", "version" : { "number" : "2.4.5", "build_hash" : "c849dd13904f53e63e88efc33b2ceeda0b6a1276", "build_timestamp" : "2017-04-24T16:18:17Z", "build_snapshot" : false, "lucene_version" : "5.5.4" }, "tagline" : "You Know, for Search" } |
OGG投递测试
OGG配置
设置环境变量
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server
使用示例自带的队列文件投递数据
GGSCI> add replicat res, exttrail AdapterExamples/trail/tr
进程内容，用系统自带的示例即可
| REPLICAT res
-- Trail file for this example is located in "AdapterExamples/trail" directory -- Command to add REPLICAT -- add replicat res, exttrail AdapterExamples/trail/tr TARGETDB LIBFILE libggjava.so SET property=dirprm/elasticsearch2x.props REPORTCOUNT EVERY 1 MINUTES, RATE GROUPTRANSOPS 1000 MAP QASOURCE.*, TARGET QASOURCE.*; |
其中elasticsearch2x.props 内容如下
| gg.handlerlist=elasticsearch
gg.handler.elasticsearch.type=elasticsearch ## Handler properties for Elasticsearch 2.x gg.handler.elasticsearch.ServerAddressList=localhost:9300 gg.handler.elasticsearch.clientSettingsFile=client2x.properties gg.handler.elasticsearch.version=2.x #gg.handler.elasticsearch.bulkWrite=true gg.classpath=/u02/elastic/lib/*:/u02/elastic/plugins/shield/* goldengate.userexit.timestamp=utc goldengate.userexit.writers=javawriter javawriter.stats.display=TRUE javawriter.stats.full=TRUE gg.log=log4j gg.log.level=INFO gg.report.time=30sec javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm |
client2x.properties
cluster.name=elasticsearch
shield.user=elastic:changeme
启动ogg res
GGSCI (ol73) 16> start res
Sending START request to MANAGER ...
REPLICAT RES starting
查看状态
GGSCI (ol73) 28> info res
REPLICAT RES Last Started 2017-12-27 16:37 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:10 ago)
Process ID 21205
Log Read Checkpoint File /u01/ogg4bd_12.3/AdapterExamples/trail/tr000000000
First Record RBA 0
GGSCI (ol73) 29> info res
REPLICAT RES Last Started 2017-12-27 16:37 Status RUNNING
Checkpoint Lag 00:00:00 (updated 00:00:07 ago)
Process ID 21205
Log Read Checkpoint File /u01/ogg4bd_12.3/AdapterExamples/trail/tr000000000
2015-11-06 02:45:39.000000 RBA 5660
已经处理完示例队列文件，查看统计信息
GGSCI (ol73) 30> stats res, total
Sending STATS request to REPLICAT RES ...
Start of Statistics at 2017-12-27 16:37:57.
Replicating from QASOURCE.TCUSTMER to QASOURCE.TCUSTMER:
*** Total statistics since 2017-12-27 16:37:45 ***
Total inserts 5.00
Total updates 1.00
Total deletes 0.00
Total discards 0.00
Total operations 6.00
Replicating from QASOURCE.TCUSTORD to QASOURCE.TCUSTORD:
*** Total statistics since 2017-12-27 16:37:45 ***
Total inserts 5.00
Total updates 3.00
Total deletes 2.00
Total discards 0.00
Total operations 10.00
End of Statistics.
可以看到，数据有正常写入。
数据验证
通过ES Rest进行查询
curl -GET http://localhost:9200/_cat/indices?v
| health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open qasource_tcustmer 5 1 5 1 14.8kb 14.8kb yellow open qasource_tcustord 5 1 3 3 21.4kb 21.4kb |
可以看到有新建2个索引：qasource_tcustmer, qasource_tcustord.
查看索引的mapping信息
curl -XGET 'http://localhost:9200/_mapping?pretty=true'
|
{ "qasource_tcustord" : { "mappings" : { "TCUSTORD" : { "properties" : { "CUST_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "ORDER_DATE" : { "type" : "date" }, "ORDER_ID" : { "type" : "float" }, "PRODUCT_AMOUNT" : { "type" : "long" }, "PRODUCT_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "PRODUCT_PRICE" : { "type" : "float" }, "TRANSACTION_ID" : { "type" : "float" } } } } }, "qasource_tcustmer" : { "mappings" : { "TCUSTMER" : { "properties" : { "CITY" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "CUST_CODE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "NAME" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "STATE" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } } } |
可以看到每个索引的各个字段属性
查询其中一张表的所有数据
$ curl -s -XGET 'http://localhost:9200/qasource_tcustord/_search?q=*&pretty=true'
{
"took" : 83,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "BILL_1995-12-31 15:00:00_CAR_765",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 14000.0,
"ORDER_DATE" : "1995-12-31T15:00:00",
"ORDER_ID" : 765.0,
"CUST_CODE" : "BILL",
"PRODUCT_AMOUNT" : 3,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "CAR"
}
},
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "BILL_1996-01-01 00:00:00_TRUCK_333",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 25000.0,
"ORDER_DATE" : "1996-01-01T00:00:00",
"ORDER_ID" : 333.0,
"CUST_CODE" : "BILL",
"PRODUCT_AMOUNT" : 15,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "TRUCK"
}
},
{
"_index" : "qasource_tcustord",
"_type" : "TCUSTORD",
"_id" : "WILL_1994-09-30 15:33:00_CAR_144",
"_score" : 1.0,
"_source" : {
"PRODUCT_PRICE" : 16520.0,
"ORDER_DATE" : "1994-09-30T15:33:00",
"ORDER_ID" : 144.0,
"CUST_CODE" : "WILL",
"PRODUCT_AMOUNT" : 3,
"TRANSACTION_ID" : 100.0,
"PRODUCT_CODE" : "CAR"
}
}
]
}
}
ES中可正常查询写入的数据，测试完成。