| datax二次开发，非常详细[源码下载 | 您所在的位置：网站首页 › idea下载源码 › datax二次开发，非常详细[源码下载 |
datax二次开发，非常详细[源码下载
|
前序
参考文档： datax 官网中的DataX插件开发宝典 一、环境准备 jdk1.8 及以上idea 工具git 工具 一、项目搭建1、用idea下载源码 打开idea 选择 File->New -> Project from Version Control… 4.检查DataX 父目录下面pom.xml，注释掉无用的插件模块 4.0.0 com.alibaba.datax datax-all 0.0.1-SNAPSHOT org.hamcrest hamcrest-core 1.3 datax-all pom 1.8 0.0.1-SNAPSHOT 3.3.2 1.10 1.2 1.1.46.sec01 16.0.1 3.7.2.1-SNAPSHOT 1.7.10 1.0.13 2.4 4.11 5.1.22-1 1.0.0 UTF-8 UTF-8 UTF-8 UTF-8 common core transformer mongodbreader txtfilewriter hdfswriter streamwriter org.apache.commons commons-lang3 ${commons-lang3-version} com.alibaba fastjson ${fastjson-version} commons-io commons-io ${commons-io-version} org.slf4j slf4j-api ${slf4j-api-version} ch.qos.logback logback-classic ${logback-classic-version} com.taobao.tddl tddl-client ${tddl.version} com.google.guava guava com.taobao.diamond diamond-client com.taobao.diamond diamond-client ${diamond.version} com.alibaba.search.swift swift_client ${swift-version} junit junit ${junit-version} org.mockito mockito-all 1.9.5 test central Nexus aliyun https://maven.aliyun.com/repository/central true true central Nexus aliyun https://maven.aliyun.com/repository/central true true maven-assembly-plugin datax package.xml make-assembly package org.apache.maven.plugins maven-compiler-plugin 2.3.2 ${jdk-version} ${jdk-version} ${project-sourceEncoding}5.调bug 我的版本3.0 存在bug: ClickhouseWriter.java 中引用的类(import ru.yandex.clickhouse.ClickHouseTuple;)无效 import ru.yandex.clickhouse.ClickHouseTuple;解决办法：直接删除该行即可 6.打包发布： DataX使用assembly打包，assembly的使用方法请咨询谷哥或者度娘。打包命令如下： mvn clean package -DskipTests assembly:assembly编译后结果 D:\ProgramFiles\Java\jdk1.8.0_241\bin\java.exe -Dmaven.multiModuleProjectDirectory=D:\work\DataX -Dmaven.home=D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven3 -Dclassworlds.conf=D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven3\bin\m2.conf -Dmaven.ext.class.path=D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven-event-listener.jar -javaagent:D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\lib\idea_rt.jar=60517:D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\bin -Dfile.encoding=UTF-8 -classpath D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven3\boot\plexus-classworlds-2.6.0.jar;D:\ProgramFiles\JetBrains\IntelliJIDEA2020.1.2\plugins\maven\lib\maven3\boot\plexus-classworlds.license org.codehaus.classworlds.Launcher -Didea.version2020.1.3 -s D:\work\conf\settings.xml -Dmaven.repo.local=D:\work\repository -DskipTests=true clean package -DskipTests assembly:assembly [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] datax-all [pom] [INFO] datax-common [jar] [INFO] datax-transformer [jar] [INFO] datax-core [jar] [INFO] plugin-unstructured-storage-util [jar] [INFO] mongodbreader [jar] [INFO] txtfilewriter [jar] [INFO] hdfswriter [jar] [INFO] streamwriter [jar] [INFO] plugin-rdbms-util [jar] [INFO] hbase20xsqlreader [jar] [INFO] hbase20xsqlwriter [jar] [INFO] [INFO] --------------------< com.alibaba.datax:datax-all >--------------------- [INFO] Building datax-all 0.0.1-SNAPSHOT [1/12] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ datax-all --- [INFO] Deleting D:\work\DataX\target [INFO] [INFO] -------------------< com.alibaba.datax:datax-common >------------------- [INFO] Building datax-common 0.0.1-SNAPSHOT [2/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ datax-common --- [INFO] Deleting D:\work\DataX\common\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\common\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-common --- [INFO] Compiling 39 source files to D:\work\DataX\common\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\common\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-common --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-common --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-common --- [INFO] Building jar: D:\work\DataX\common\target\datax-common-0.0.1-SNAPSHOT.jar [INFO] [INFO] ----------------< com.alibaba.datax:datax-transformer >----------------- [INFO] Building datax-transformer 0.0.1-SNAPSHOT [3/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ datax-transformer --- [INFO] Deleting D:\work\DataX\transformer\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-transformer --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\transformer\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-transformer --- [INFO] Compiling 2 source files to D:\work\DataX\transformer\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-transformer --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\transformer\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-transformer --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-transformer --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-transformer --- [INFO] Building jar: D:\work\DataX\transformer\target\datax-transformer-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ datax-transformer --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\transformer\target\datax [WARNING] Assembly file: D:\work\DataX\transformer\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] --------------------< com.alibaba.datax:datax-core >-------------------- [INFO] Building datax-core 0.0.1-SNAPSHOT [4/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ datax-core --- [INFO] Deleting D:\work\DataX\core\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\core\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-core --- [INFO] Compiling 70 source files to D:\work\DataX\core\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\core\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-core --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-core --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-core --- [INFO] Building jar: D:\work\DataX\core\target\datax-core-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (default) @ datax-core --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\core\target\datax [WARNING] Assembly file: D:\work\DataX\core\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] ---------< com.alibaba.datax:plugin-unstructured-storage-util >--------- [INFO] Building plugin-unstructured-storage-util 0.0.1-SNAPSHOT [5/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ plugin-unstructured-storage-util --- [INFO] Deleting D:\work\DataX\plugin-unstructured-storage-util\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ plugin-unstructured-storage-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\plugin-unstructured-storage-util\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ plugin-unstructured-storage-util --- [INFO] Compiling 13 source files to D:\work\DataX\plugin-unstructured-storage-util\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ plugin-unstructured-storage-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\plugin-unstructured-storage-util\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ plugin-unstructured-storage-util --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ plugin-unstructured-storage-util --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ plugin-unstructured-storage-util --- [INFO] Building jar: D:\work\DataX\plugin-unstructured-storage-util\target\plugin-unstructured-storage-util-0.0.1-SNAPSHOT.jar [INFO] [INFO] ------------------< com.alibaba.datax:mongodbreader >------------------- [INFO] Building mongodbreader 0.0.1-SNAPSHOT [6/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ mongodbreader --- [INFO] Deleting D:\work\DataX\mongodbreader\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ mongodbreader --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ mongodbreader --- [INFO] Compiling 5 source files to D:\work\DataX\mongodbreader\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ mongodbreader --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\mongodbreader\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ mongodbreader --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ mongodbreader --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ mongodbreader --- [INFO] Building jar: D:\work\DataX\mongodbreader\target\mongodbreader-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ mongodbreader --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\mongodbreader\target\datax [WARNING] Assembly file: D:\work\DataX\mongodbreader\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] ------------------< com.alibaba.datax:txtfilewriter >------------------- [INFO] Building txtfilewriter 0.0.1-SNAPSHOT [7/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ txtfilewriter --- [INFO] Deleting D:\work\DataX\txtfilewriter\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ txtfilewriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ txtfilewriter --- [INFO] Compiling 3 source files to D:\work\DataX\txtfilewriter\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ txtfilewriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\txtfilewriter\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ txtfilewriter --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ txtfilewriter --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ txtfilewriter --- [INFO] Building jar: D:\work\DataX\txtfilewriter\target\txtfilewriter-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ txtfilewriter --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\txtfilewriter\target\datax [WARNING] Assembly file: D:\work\DataX\txtfilewriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] --------------------< com.alibaba.datax:hdfswriter >-------------------- [INFO] Building hdfswriter 0.0.1-SNAPSHOT [8/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hdfswriter --- [INFO] Deleting D:\work\DataX\hdfswriter\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hdfswriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hdfswriter --- [INFO] Compiling 6 source files to D:\work\DataX\hdfswriter\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hdfswriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\hdfswriter\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hdfswriter --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hdfswriter --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hdfswriter --- [INFO] Building jar: D:\work\DataX\hdfswriter\target\hdfswriter-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hdfswriter --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml Downloading from alimaven: http://maven.aliyun.com/nexus/content/groups/public/eigenbase/eigenbase-properties/1.1.4/eigenbase-properties-1.1.4.pom Downloading from alimaven: http://maven.aliyun.com/nexus/content/groups/public/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.pom [INFO] Copying files to D:\work\DataX\hdfswriter\target\datax [WARNING] Assembly file: D:\work\DataX\hdfswriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] -------------------< com.alibaba.datax:streamwriter >------------------- [INFO] Building streamwriter 0.0.1-SNAPSHOT [9/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ streamwriter --- [INFO] Deleting D:\work\DataX\streamwriter\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ streamwriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ streamwriter --- [INFO] Compiling 3 source files to D:\work\DataX\streamwriter\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ streamwriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\streamwriter\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ streamwriter --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ streamwriter --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ streamwriter --- [INFO] Building jar: D:\work\DataX\streamwriter\target\streamwriter-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ streamwriter --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\streamwriter\target\datax [WARNING] Assembly file: D:\work\DataX\streamwriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] ----------------< com.alibaba.datax:plugin-rdbms-util >----------------- [INFO] Building plugin-rdbms-util 0.0.1-SNAPSHOT [10/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ plugin-rdbms-util --- [INFO] Deleting D:\work\DataX\plugin-rdbms-util\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ plugin-rdbms-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\plugin-rdbms-util\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ plugin-rdbms-util --- [INFO] Compiling 24 source files to D:\work\DataX\plugin-rdbms-util\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ plugin-rdbms-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\plugin-rdbms-util\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ plugin-rdbms-util --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ plugin-rdbms-util --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ plugin-rdbms-util --- [INFO] Building jar: D:\work\DataX\plugin-rdbms-util\target\plugin-rdbms-util-0.0.1-SNAPSHOT.jar [INFO] [INFO] ----------------< com.alibaba.datax:hbase20xsqlreader >----------------- [INFO] Building hbase20xsqlreader 0.0.1-SNAPSHOT [11/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hbase20xsqlreader --- [INFO] Deleting D:\work\DataX\hbase20xsqlreader\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase20xsqlreader --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase20xsqlreader --- [INFO] Compiling 6 source files to D:\work\DataX\hbase20xsqlreader\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase20xsqlreader --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\hbase20xsqlreader\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase20xsqlreader --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hbase20xsqlreader --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hbase20xsqlreader --- [INFO] Building jar: D:\work\DataX\hbase20xsqlreader\target\hbase20xsqlreader-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hbase20xsqlreader --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\hbase20xsqlreader\target\datax [WARNING] Assembly file: D:\work\DataX\hbase20xsqlreader\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] ----------------< com.alibaba.datax:hbase20xsqlwriter >----------------- [INFO] Building hbase20xsqlwriter 0.0.1-SNAPSHOT [12/12] [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hbase20xsqlwriter --- [INFO] Deleting D:\work\DataX\hbase20xsqlwriter\target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase20xsqlwriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase20xsqlwriter --- [INFO] Compiling 7 source files to D:\work\DataX\hbase20xsqlwriter\target\classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase20xsqlwriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\hbase20xsqlwriter\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase20xsqlwriter --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hbase20xsqlwriter --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hbase20xsqlwriter --- [INFO] Building jar: D:\work\DataX\hbase20xsqlwriter\target\hbase20xsqlwriter-0.0.1-SNAPSHOT.jar [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hbase20xsqlwriter --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\hbase20xsqlwriter\target\datax [WARNING] Assembly file: D:\work\DataX\hbase20xsqlwriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] --------------------< com.alibaba.datax:datax-all >--------------------- [INFO] Building datax-all 0.0.1-SNAPSHOT [13/12] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] [INFO] >>> maven-assembly-plugin:2.2-beta-5:assembly (default-cli) > package @ datax-all >>> [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking datax-common 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\common\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-common --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-common --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\common\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-common --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-common --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-common --- [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking datax-transformer 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-transformer --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\transformer\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-transformer --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-transformer --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\transformer\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-transformer --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-transformer --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-transformer --- [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ datax-transformer --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\transformer\target\datax [WARNING] Assembly file: D:\work\DataX\transformer\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking datax-core 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ datax-core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\core\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ datax-core --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ datax-core --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\core\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ datax-core --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ datax-core --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ datax-core --- [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (default) @ datax-core --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\core\target\datax [WARNING] Assembly file: D:\work\DataX\core\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking plugin-unstructured-storage-util 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ plugin-unstructured-storage-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\plugin-unstructured-storage-util\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ plugin-unstructured-storage-util --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ plugin-unstructured-storage-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\plugin-unstructured-storage-util\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ plugin-unstructured-storage-util --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ plugin-unstructured-storage-util --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ plugin-unstructured-storage-util --- [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking mongodbreader 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ mongodbreader --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ mongodbreader --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ mongodbreader --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\mongodbreader\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ mongodbreader --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ mongodbreader --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ mongodbreader --- [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ mongodbreader --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\mongodbreader\target\datax [WARNING] Assembly file: D:\work\DataX\mongodbreader\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking txtfilewriter 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ txtfilewriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ txtfilewriter --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ txtfilewriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\txtfilewriter\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ txtfilewriter --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ txtfilewriter --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ txtfilewriter --- [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ txtfilewriter --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\txtfilewriter\target\datax [WARNING] Assembly file: D:\work\DataX\txtfilewriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking hdfswriter 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hdfswriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hdfswriter --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hdfswriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\hdfswriter\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hdfswriter --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hdfswriter --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hdfswriter --- [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hdfswriter --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\hdfswriter\target\datax [WARNING] Assembly file: D:\work\DataX\hdfswriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking streamwriter 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ streamwriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ streamwriter --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ streamwriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\streamwriter\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ streamwriter --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ streamwriter --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ streamwriter --- [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ streamwriter --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\streamwriter\target\datax [WARNING] Assembly file: D:\work\DataX\streamwriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking plugin-rdbms-util 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ plugin-rdbms-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\plugin-rdbms-util\src\main\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ plugin-rdbms-util --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ plugin-rdbms-util --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\plugin-rdbms-util\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ plugin-rdbms-util --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ plugin-rdbms-util --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ plugin-rdbms-util --- [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking hbase20xsqlreader 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase20xsqlreader --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase20xsqlreader --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase20xsqlreader --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\hbase20xsqlreader\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase20xsqlreader --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hbase20xsqlreader --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hbase20xsqlreader --- [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hbase20xsqlreader --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\hbase20xsqlreader\target\datax [WARNING] Assembly file: D:\work\DataX\hbase20xsqlreader\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] Forking hbase20xsqlwriter 0.0.1-SNAPSHOT [INFO] >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hbase20xsqlwriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase20xsqlwriter --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hbase20xsqlwriter --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory D:\work\DataX\hbase20xsqlwriter\src\test\resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase20xsqlwriter --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ hbase20xsqlwriter --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ hbase20xsqlwriter --- [INFO] [INFO] --- maven-assembly-plugin:2.2-beta-5:single (dwzip) @ hbase20xsqlwriter --- [INFO] Reading assembly descriptor: src/main/assembly/package.xml [INFO] Copying files to D:\work\DataX\hbase20xsqlwriter\target\datax [WARNING] Assembly file: D:\work\DataX\hbase20xsqlwriter\target\datax is not a regular file (it may be a directory). It cannot be attached to the project build for installation or deployment. [INFO] [INFO] private Configuration originalConfig = null; private MongoClient mongoClient; private String userName = null; private String password = null; @Override public List split(int adviceNumber) { return CollectionSplitUtil.doSplit(originalConfig,adviceNumber,mongoClient); } @Override public void init() { this.originalConfig = super.getPluginJobConf(); this.userName = originalConfig.getString(KeyConstant.MONGO_USER_NAME, originalConfig.getString(KeyConstant.MONGO_USERNAME)); this.password = originalConfig.getString(KeyConstant.MONGO_USER_PASSWORD, originalConfig.getString(KeyConstant.MONGO_PASSWORD)); String database = originalConfig.getString(KeyConstant.MONGO_DB_NAME, originalConfig.getString(KeyConstant.MONGO_DATABASE)); String authDb = originalConfig.getString(KeyConstant.MONGO_AUTHDB, database); if(!Strings.isNullOrEmpty(this.userName) && !Strings.isNullOrEmpty(this.password)) { this.mongoClient = MongoUtil.initCredentialMongoClient(originalConfig,userName,password,authDb); } else { this.mongoClient = MongoUtil.initMongoClient(originalConfig); } } @Override public void destroy() { } } public static class Task extends Reader.Task { private Configuration readerSliceConfig; private MongoClient mongoClient; private String userName = null; private String password = null; private String authDb = null; private String database = null; private String collection = null; private String query = null; private JSONArray mongodbColumnMeta = null; private Object lowerBound = null; private Object upperBound = null; private boolean isObjectId = true; @Override public void startRead(RecordSender recordSender) { if(lowerBound== null || upperBound == null || mongoClient == null || database == null || collection == null || mongodbColumnMeta == null) { throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE, MongoDBReaderErrorCode.ILLEGAL_VALUE.getDescription()); } MongoDatabase db = mongoClient.getDatabase(database); MongoCollection col = db.getCollection(this.collection); MongoCursor dbCursor = null; Document filter = new Document(); if (lowerBound.equals("min")) { if (!upperBound.equals("max")) { filter.append(KeyConstant.MONGO_PRIMARY_ID, new Document("$lt", isObjectId ? new ObjectId(upperBound.toString()) : upperBound)); } } else if (upperBound.equals("max")) { filter.append(KeyConstant.MONGO_PRIMARY_ID, new Document("$gte", isObjectId ? new ObjectId(lowerBound.toString()) : lowerBound)); } else { filter.append(KeyConstant.MONGO_PRIMARY_ID, new Document("$gte", isObjectId ? new ObjectId(lowerBound.toString()) : lowerBound).append("$lt", isObjectId ? new ObjectId(upperBound.toString()) : upperBound)); } if(!Strings.isNullOrEmpty(query)) { Document queryFilter = Document.parse(query); filter = new Document("$and", Arrays.asList(filter, queryFilter)); } dbCursor = col.find(filter).iterator(); while (dbCursor.hasNext()) { Document item = dbCursor.next(); Record record = recordSender.createRecord(); Iterator columnItera = mongodbColumnMeta.iterator(); while (columnItera.hasNext()) { JSONObject column = (JSONObject)columnItera.next(); Object tempCol = item.get(column.getString(KeyConstant.COLUMN_NAME)); if (tempCol == null) { if (KeyConstant.isDocumentType(column.getString(KeyConstant.COLUMN_TYPE))) { String[] name = column.getString(KeyConstant.COLUMN_NAME).split("\\."); if (name.length > 1) { Object obj; Document nestedDocument = item; for (String str : name) { obj = nestedDocument.get(str); if (obj instanceof Document) { nestedDocument = (Document) obj; } } if (null != nestedDocument) { Document doc = nestedDocument; tempCol = doc.get(name[name.length - 1]); } } } } if (tempCol == null) { //continue; 这个不能直接continue会导致record到目的端错位 record.addColumn(new StringColumn(null)); }else if (tempCol instanceof Double) { //TODO deal with Double.isNaN() record.addColumn(new DoubleColumn((Double) tempCol)); } else if (tempCol instanceof Boolean) { record.addColumn(new BoolColumn((Boolean) tempCol)); } else if (tempCol instanceof Date) { record.addColumn(new DateColumn((Date) tempCol)); } else if (tempCol instanceof Integer) { record.addColumn(new LongColumn((Integer) tempCol)); }else if (tempCol instanceof Long) { record.addColumn(new LongColumn((Long) tempCol)); } else { if(KeyConstant.isArrayType(column.getString(KeyConstant.COLUMN_TYPE))) { String splitter = column.getString(KeyConstant.COLUMN_SPLITTER); if(Strings.isNullOrEmpty(splitter)) { throw DataXException.asDataXException(MongoDBReaderErrorCode.ILLEGAL_VALUE, MongoDBReaderErrorCode.ILLEGAL_VALUE.getDescription()); } else { ArrayList array = (ArrayList)tempCol; String tempArrayStr = Joiner.on(splitter).join(array); record.addColumn(new StringColumn(tempArrayStr)); } } else { record.addColumn(new StringColumn(tempCol.toString())); } } } recordSender.sendToWriter(record); } } @Override public void init() { this.readerSliceConfig = super.getPluginJobConf(); this.userName = readerSliceConfig.getString(KeyConstant.MONGO_USER_NAME, readerSliceConfig.getString(KeyConstant.MONGO_USERNAME)); this.password = readerSliceConfig.getString(KeyConstant.MONGO_USER_PASSWORD, readerSliceConfig.getString(KeyConstant.MONGO_PASSWORD)); this.database = readerSliceConfig.getString(KeyConstant.MONGO_DB_NAME, readerSliceConfig.getString(KeyConstant.MONGO_DATABASE)); this.authDb = readerSliceConfig.getString(KeyConstant.MONGO_AUTHDB, this.database); if(!Strings.isNullOrEmpty(userName) && !Strings.isNullOrEmpty(password)) { mongoClient = MongoUtil.initCredentialMongoClient(readerSliceConfig,userName,password,authDb); } else { mongoClient = MongoUtil.initMongoClient(readerSliceConfig); } this.collection = readerSliceConfig.getString(KeyConstant.MONGO_COLLECTION_NAME); this.query = readerSliceConfig.getString(KeyConstant.MONGO_QUERY); this.mongodbColumnMeta = JSON.parseArray(readerSliceConfig.getString(KeyConstant.MONGO_COLUMN)); this.lowerBound = readerSliceConfig.get(KeyConstant.LOWER_BOUND); this.upperBound = readerSliceConfig.get(KeyConstant.UPPER_BOUND); this.isObjectId = readerSliceConfig.getBool(KeyConstant.IS_OBJECTID); } @Override public void destroy() { } } }通过上面代码找到类型判断逻辑如下 ... else if (tempCol instanceof Double) { //TODO deal with Double.isNaN() record.addColumn(new DoubleColumn((Double) tempCol)); } else if (tempCol instanceof Boolean) { record.addColumn(new BoolColumn((Boolean) tempCol)); }...增加一下逻辑 else if (tempCol instanceof Document || tempCol instanceof List) { record.addColumn(new StringColumn(JSONObject.toJSONString(tempCol))); }重新打包发布： 由于不会部分插件方式打包，所以只能先打包mongodbreader然后将其 打包mongodbreader模块： mvn clean package -pl mongodbreader -DskipTests assembly:assembly 如果其他模块，直接将mongodbreader配置成对应的模块名（pom.xml中 内名称）即可。 谢谢，阅读！ |
github地址：https://github.com/alibaba/DataX.git 在URL:填写gihub地址并Clone 
2.等待加载相应文件… 3.Engine.java 启动文件 修改main方法，增加两行代码 //设置系统参数，参数值：datax编译后的目录 System.setProperty(“datax.home”, “D:\work\DataX\target\datax\datax”); //设置启动参数：job的json文件的路径，和其他参数 String[] datxArgs = {"-job", “D:\work\DataX\core\src\main\job\mongo-file.json”, “-mode”, “standalone”, “-jobid”, “-1”};【本文地址】
公司简介
联系我们