zoukankan      html  css  js  c++  java
  • datax源代码编译安装

    what is DataX ?

    DataX是阿里巴巴集团内被广泛使用的离线数据同步工具/平台。
    实现包括MySQL,SQL Server,Oracle,PostgreSQL,HDFS,Hive,HBase,OTS,ODPS等各种异构数据源之间高效的数据同步功能。

    1.从Github下载源码,地址:https://github.com/alibaba/DataX?spm=a2c4e.11153940.blogcont642896.16.7d62ba62hHwiVO&file=DataX.git;

     DataX-master.zip

    2.安装好maven,通过maven打包源码编译:

     mvn -U clean package assembly:assembly -Dmaven.test.skip=true

    3.编译好的datax在 /target/datax/下:

    {YOUR_DATAX_HOME}/target/datax/

    4.编译过程中出现的报错解决办法:

    4.1.maven配置阿里云的maven私服

    <mirror> 
    <id>nexus-aliyun</id>
    <mirrorOf>central</mirrorOf>
    <name>Nexus aliyun</name>
    <url>https://maven.aliyun.com/repository/central</url>
    </mirror>

    4.2.本地编译报错无法找到工件com.aliyun.openservices:tablestore-streamclient:jar:1.0.0-SNAPSHOT;

    vim otsstreamreader/pom.xml


    <dependency>
    <groupId>com.aliyun.openservices</groupId>
    <artifactId>tablestore-streamclient</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    </dependency>

    此处把
    <version>1.0.0-SNAPSHOT</version>
    改成
    <version>1.0.0</version>

    4.3.编译datax odps插件模块会报错:

    ERROR] Failed to execute goal on project odpsreader: Could not resolve dependencies for project com.alibaba.datax:odpsreader:jar:0.0.1-SNAPSHOT:
    The following artifacts could not be resolved: com.alibaba.datax:datax-common:jar:0.0.1-SNAPSHOT,
    com.alibaba.external:bouncycastle.provider:jar:1.38-jdk15: Could not find artifact com.alibaba.datax:datax-common:jar:0.0.1-SNAPSHOT in
    dtwave (http://repo2.dtwave-inc.com/repository/public/) -> [Help 1] [ERROR] [ERROR]
    To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging.
    [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:
    [ERROR] [Help 1]
    http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

    比较过之前odps-sdk-core-0.19.3-public.pom的的依赖是
    org.bouncycastle bcprov-jdk15on 1.52
    现在是
    com.alibaba.external bouncycastle.provider 1.38-jdk15

    原因: 后来的这个依赖应该是阿里内部jar,外部仓库无法加载这个jar

       解决:修改pom.xml

    com.aliyun.odps
    odps-sdk-core
    换一下版本 :0.20.7-public 

    5.编译成功:

    6.测试:在eclipse中建立EngineTest.java,使用默认的job.json:

     
  • 相关阅读:
    hud 3336 count the string (KMP)
    JSOI2008星球大战(并查集)
    HAOI2006受欢迎的牛
    十二月个人考核
    十二月个人考核
    CentOS配置Tomcat监听80端口,虚拟主机
    CentOS配置Tomcat监听80端口,虚拟主机
    如何调试一个无法重现的错误?
    如何调试一个无法重现的错误?
    Highcharts的自适应DOM或者DIV,JS方法实现
  • 原文地址:https://www.cnblogs.com/gxc2015/p/9989009.html
Copyright © 2011-2022 走看看