Java操作kettle(ETL)

1. 安裝Jar到本地Maven倉庫

將pdi-ce-7.1.0.0-12/data-integration/lib下的kettle-core-7.1.0.0-12.jar、kettle-engine-7.1.0.0-12.jar、metastore-7.1.0.0-12.jar安裝的本地maven倉庫中。

<code># 切換到pdi-ce-7.1.0.0-12中的lib目錄
cd /Users/xxx/Softwares/pdi-ce-7.1.0.0-12/data-integration/lib


mvn install:install-file -Dfile=kettle-core-7.1.0.0-12.jar -DgroupId=org.pentaho -DartifactId=kettle-core -Dversion=7.1.0.0-12 -Dpackaging=jar 

mvn install:install-file -Dfile=kettle-engine-7.1.0.0-12.jar -DgroupId=org.pentaho -DartifactId=kettle-engine -Dversion=7.1.0.0-12 -Dpackaging=jar 

mvn install:install-file -Dfile=metastore-7.1.0.0-12.jar -DgroupId=org.pentaho -DartifactId=metastore -Dversion=7.1.0.0-12 -Dpackaging=jar 
/<code>

2. 引入依賴

<code>
<project>         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelversion>4.0.0/<modelversion>
    <parent>
        <groupid>org.springframework.boot/<groupid>
        <artifactid>spring-boot-starter-parent/<artifactid>
        <version>2.1.3.RELEASE/<version>
        <relativepath> 
    /<parent>
    <groupid>com.example/<groupid>
    <artifactid>springboot-etl-kettle/<artifactid>
    <version>0.0.1-SNAPSHOT/<version>
    <name>springboot-etl-kettle/<name>
    <description>Demo project for Spring Boot/<description>

    <properties>
        <java.version>1.8/<java.version>
        <kettle.version>7.1.0.0-12/<kettle.version>
    /<properties>

    <dependencies>
        <dependency>
            <groupid>org.springframework.boot/<groupid>
            <artifactid>spring-boot-starter/<artifactid>
        /<dependency>

        
        <dependency>
            <groupid>org.pentaho/<groupid>
            <artifactid>kettle-core/<artifactid>
            <version>${kettle.version}/<version>
        /<dependency>
        <dependency>
            <groupid>org.pentaho/<groupid>
            <artifactid>kettle-engine/<artifactid>
            <version>${kettle.version}/<version>
        /<dependency>
        <dependency>
            <groupid>org.pentaho/<groupid>
            <artifactid>metastore/<artifactid>
            <version>${kettle.version}/<version>
        /<dependency>

        <dependency>
            <groupid>org.apache.commons/<groupid>
            <artifactid>commons-vfs2/<artifactid>
            <version>2.3/<version>
        /<dependency>
        <dependency>
            <groupid>commons-lang/<groupid>
            <artifactid>commons-lang/<artifactid>
            <version>2.6/<version>
        /<dependency>
        <dependency>
            <groupid>commons-codec/<groupid>
            <artifactid>commons-codec/<artifactid>
            <version>1.12/<version>
        /<dependency>
        <dependency>
            <groupid>com.google.guava/<groupid>
            <artifactid>guava/<artifactid>
            <version>23.0/<version>
        /<dependency>
        

        <dependency>
            <groupid>mysql/<groupid>
            <artifactid>mysql-connector-java/<artifactid>
            <version>5.1.40/<version>
        /<dependency>

        <dependency>
            <groupid>org.springframework.boot/<groupid>
            <artifactid>spring-boot-starter-test/<artifactid>
            <scope>test/<scope>
        /<dependency>
    /<dependencies>

    <build>
        <plugins>
            <plugin>
                <groupid>org.springframework.boot/<groupid>
                <artifactid>spring-boot-maven-plugin/<artifactid>
            /<plugin>
        /<plugins>
    /<build>
/<project>
/<code>

3. Test

<code>@RunWith(SpringRunner.class)
@SpringBootTest
public class SpringbootEtlKettleApplicationTests {

    private String filename = "/Users/mengday/Desktop/HelloWorld.ktr";

    @Test
    public void testEtl() {
        try {
            KettleEnvironment.init();
            TransMeta transMeta = new TransMeta(filename);
            Trans trans = new Trans(transMeta);

            // 設置變量(這些變量可以在.ktl中通過${variableName}來引用)
            String beginTime = "2019-04-02 00:00:00";
            String endTime = "2019-04-06 23:59:59";
            trans.setVariable("beginTime", beginTime);
            trans.setVariable("endTime", endTime);

            trans.execute(null);
            trans.waitUntilFinished();
        } catch (KettleException e) {
            e.printStackTrace();
        }
    }
}
/<code>

實際開發中我們將testEtl()方法中的代碼封裝到定時任務中，在定時任務中設置定時執行時間，將.ktl文件名或者其它變量值作為參數傳遞到Job中來執行。

閱讀更多 Java實用技術 的文章

關鍵字: XML Java