DataX不同ETL任务json模板

原创 已于 2022-10-10 18:09:26 修改 · 1.2k 阅读 · 4 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#json #etl #java

于 2022-10-10 15:54:07 首次发布
本文介绍使用 DataX 进行不同数据源间的数据迁移配置方法，包括 MySQL 到 Kudu、HDFS 到 MySQL、HDFS 到 ClickHouse 等多种场景。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

项目场景：

datax同步任务json模板：

mysql -> kudu

kudu建表

DROP TABLE dim.content_test;
CREATE TABLE dim.content_test(
id string NOT NULL,
title string NOT NULL,
PRIMARY KEY (id)
)
PARTITION BY HASH (id) PARTITIONS 3
STORED AS KUDU TBLPROPERTIES ('kudu.master_addresses'='xxx:7051,xxx:7051,xxx:7051');

json

{
  "core": {
    "transport": {
      "channel": {
        "speed": {
          "byte": 104857600
        }
      }
    }
  },
  "job": {
    "setting": {
      "speed": {
        "channel": 10,
        "byte": 1048576000
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "xxx",
            "password": "xxx",
            "splitPk": "",
            "connection": [
              {
                "querySql": [
                  "select ID,TITLE from content_test;"
                ],
                "jdbcUrl": [
                  "jdbc:mysql://xxx:3306/yy"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "kuduwriter",
          "parameter": {
            "batchSize": 1024,
            "bufferSize": 2048,
            "skipFail": false,
            "encoding": "UTF-8",
            "kuduConfig": {
              "kudu.master_addresses": "xxx:7051,xxx:7051,xxx:7051"
            },
            "table": "impala::dim.content_test",
            "truncate": false,
            "writeMode": "upsert",
            "column": [
              {
                "index": 0,
                "name": "id",
                "type": "int",
                "comment": "内容id",
                "primaryKey": true
              },
              {
                "index": 1,
                "name": "title",
                "type": "string",
                "comment": "内容名称"
              }
            ],
            "primaryKey": [
              {
                "index": 0,
                "name": "id",
                "type": "int"
              }
            ]
          }
        }
      }
    ]
  }
}

参考网址：https://github.com/alibaba/DataX/blob/master/kuduwriter/doc/kuduwirter.md

hdfs -> mysql

json

{
  "core": {
    "transport": {
      "channel": {
        "speed": {
          "byte": 104857600
        }
      }
    }
  },
  "job": {
    "setting": {
      "speed": {
        "channel": 10,
        "byte": 1048576000
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "xxx",
            "password": "xxx",
            "column": [
              "`id`",
              "`ww`"
            ],
            "splitPk": "",
            "connection": [
              {
                "table": [
                  "comp_tx"
                ],
                "jdbcUrl": [
                  "jdbc:mysql://XXX:3312/ddd"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "hdfswriter",
          "parameter": {
            "defaultFS": "hdfs://name1",
            "nullFormat": "\\N",
            "hadoopConfig": {
              "dfs.nameservices": "nameservice1",
              "dfs.ha.namenodes.nameservice1": "aa,bb",
              "dfs.namenode.rpc-address.nameservice1.aa": "hdfs://XXX:8020",
              "dfs.namenode.rpc-address.nameservice1.bb": "hdfs://XXX:8020",
              "dfs.client.failover.proxy.provider.nameservice1": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
              "dfs.ha.automatic-failover.enabled.yournamespace": "true"
            },
            "fileType": "orc",
            "path": "/user/hive/dim/XXX",
            "fileName": "XXX",
            "writeMode": "append",
            "fieldDelimiter": "\u0001",
            "column": [
              {
                "name": "id",
                "type": "string"
              },
              {
                "name": "ww",
                "type": "string"
              }
            ]
          }
        }
      }
    ]
  }
}

参考网址：https://github.com/alibaba/DataX/blob/master/hdfswriter/doc/hdfswriter.md

hdfs -> clickhouse

json

{
  "core": {
    "transport": {
      "channel": {
        "speed": {
          "byte": 104857600
        }
      }
    }
  },
  "job": {
    "setting": {
      "speed": {
        "channel": 10,
        "byte": 1048576000
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "hdfsreader",
          "parameter": {
            "path": "/user/hive/test01",
            "nullFormat": "\\N",
            "defaultFS": "hdfs://name1",
            "hadoopConfig": {
              "dfs.nameservices": "xx",
              "dfs.ha.namenodes.nameservice1": "xx01,xx02",
              "dfs.namenode.rpc-address.nameservice1.xx01": "hdfs://xxx:8020",
              "dfs.namenode.rpc-address.nameservice1.xx02": "hdfs://xxx:8020",
              "dfs.client.failover.proxy.provider.nameservice1": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
              "dfs.ha.automatic-failover.enabled.yournamespace": "true"
            },
            "fileType": "orc",
            "fieldDelimiter": "\u0001",
            "skipHeader": true,
            "column": [
              {
                "index": "0",
                "type": "string"
              },
              {
                "index": "1",
                "type": "string"
              },
              {
                "index": "2",
                "type": "string"
              },
              {
                "index": "3",
                "type": "string"
              }
            ]
          }
        },
        "writer": {
          "name": "clickhousewriter",
          "parameter": {
            "username": "xxxx",
            "password": "xxxx",
            "column": [
              "event",
              "aa",
              "bb",
              "dt"
            ],
            "connection": [
              {
                "table": [
                  "ww"
                ],
                "jdbcUrl": "jdbc:clickhouse://xxx:8123/test"
              }
            ]
          }
        }
      }
    ]
  }
}

hdfs -> textfile

json

{
  "core": {
    "transport": {
      "channel": {
        "speed": {
          "byte": 104857600
        }
      }
    }
  },
  "job": {
    "setting": {
      "speed": {
        "channel": 10,
        "byte": 1048576000
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "hdfsreader",
          "parameter": {
            "path": "/user/hive/warehouse/*.0.",
            "defaultFS": "hdfs://NameServiceHA",
            "hadoopConfig": {
              "dfs.nameservices": "NameServiceHA",
              "dfs.ha.namenodes.NameServiceHA": "data01,data02",
              "dfs.namenode.rpc-address.NameServiceHA.data01": "hdfs://XXX:8020",
              "dfs.namenode.rpc-address.NameServiceHA.data02": "hdfs://XXX:8020",
              "dfs.client.failover.proxy.provider.NameServiceHA": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
              "dfs.ha.automatic-failover.enabled.yournamespace": "true"
            },
            "column": [
              "*"
            ],
            "fileType": "TEXT",
            "fieldDelimiter": "\u0001"
          }
        },
        "writer": {
          "name": "txtfilewriter",
          "parameter": {
            "path": "/data/team/",
            "fileName": "test",
            "writeMode": "truncate",
            "dateFormat": "yyyy-MM-dd"
          }
        }
      }
    ]
  }
}

hdfs -> ftp

json

  "core": {
    "transport": {
      "channel": {
        "speed": {
          "byte": 104857600
        }
      }
    }
  },
  "job": {
    "setting": {
      "speed": {
        "channel": 10,
        "byte": 1048576000
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "hdfsreader",
          "parameter": {
            "path": "/user/hive/warehouse/*.0.",
            "defaultFS": "hdfs://NameServiceHA",
            "hadoopConfig": {
              "dfs.nameservices": "NameServiceHA",
              "dfs.ha.namenodes.NameServiceHA": "data01,data02",
              "dfs.namenode.rpc-address.NameServiceHA.data01": "hdfs://xxx:8020",
              "dfs.namenode.rpc-address.NameServiceHA.data02": "hdfs://xxx:8020",
              "dfs.client.failover.proxy.provider.NameServiceHA": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
              "dfs.ha.automatic-failover.enabled.yournamespace": "true"
            },
            "column": [
              "*"
            ],
            "fileType": "TEXT",
            "fieldDelimiter": "\u0001"
          }
        },
        "writer": {
          "name": "ftpwriter",
          "parameter": {
            "protocol": "ftp",
            "host": "xxx",
            "port": 21,
            "username": "xxx",
            "password": "xxx",
            "connectPattern": "PASV",
            "path": "/user/20220705/",
            "fileName": "test",
            "writeMode": "truncate",
            "fieldDelimiter": ",",
            "timeout": "3600000",
            "encoding": "UTF-8",
            "nullFormat": "\\N",
            "dateFormat": "yyyy-MM-dd",
            "fileFormat": "text",
            "suffix": ".text",
            "header": []
          }
        }
      }
    ]
  }
}

mysql -> clickhouse

json

{
  "job": {
    "setting": {
      "speed": {
        "channel": 3,
        "byte": 1048576
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "xxx",
            "password": "xxx",
            "splitPk": "",
            "connection": [
              {
                "querySql": [
                  "select id,code FROM test.aa"
                ],
                "jdbcUrl": [
                  "jdbc:mysql://xxx:3306/test"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "clickhousewriter",
          "parameter": {
            "username": "xxx",
            "password": "xxx",
            "column": [
              "id",
              "code"
            ],
            "connection": [
              {
                "table": [
                  "test01"
                ],
                "jdbcUrl": "jdbc:clickhouse://xxx:8123/test"
              }
            ]
          }
        }
      }
    ]
  }
}

hdfs -> hdfs

json

{
  "job": {
    "setting": {
      "speed": {
        "channel": 10
      }
    },
    "content": [
      {
        "reader": {
          "name": "hdfsreader",
          "parameter": {
            "path": "/user/hive/warehouse/*.0.parq",
            "defaultFS": "hdfs://NameServiceHA",
            "hadoopConfig": {
              "dfs.nameservices": "NameServiceHA",
              "dfs.ha.namenodes.NameServiceHA": "test01,test02",
              "dfs.namenode.rpc-address.NameServiceHA.data01": "hdfs://xxx:8020",
              "dfs.namenode.rpc-address.NameServiceHA.data02": "hdfs://xxx:8020",
              "dfs.client.failover.proxy.provider.NameServiceHA": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
              "dfs.ha.automatic-failover.enabled.yournamespace": "true"
            },
            "column": [
              "*"
            ],
            "fileType": "PARQUET",
            "fieldDelimiter": "\u0001"
          }
        },
        "writer": {
          "name": "hdfswriter",
          "parameter": {
            "defaultFS": "hdfs://test",
            "hadoopConfig": {
              "dfs.nameservices": "test",
              "dfs.ha.namenodes.nameservice1": "test01,test02",
              "dfs.namenode.rpc-address.nameservice1.test01": "hdfs://xxx:8020",
              "dfs.namenode.rpc-address.nameservice1.test02": "hdfs://xxx:8020",
              "dfs.client.failover.proxy.provider.nameservice1": "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
              "dfs.ha.automatic-failover.enabled.yournamespace": "true"
            },
            "fileType": "TEXT",
            "path": "/newtv/hive/test_temp",
            "fileName": "test_temp",
            "column": [
              {
                "name": "aa",
                "type": "string"
              },
              {
                "name": "bb",
                "type": "string"
              }
            ],
            "writeMode": "append",
            "fieldDelimiter": "^"
          }
        }
      }
    ]
  }
}
【Datax同步】json模板
cui8483的博客
02-01 468
【Datax同步】json模板
ETL数据同步之DataX，附赠一套DataX通用模板
wowulita123的博客
07-17 930
ETL数据同步之DataX，附赠一套DataX通用模板
Datax json模板集合
weixin_47444415的博客
11-27 1294
{ "job": { "setting": { "speed": { "channel": 6, "byte": 8048576, "record": 10000000 }, "errorLimit": { "record": 0, "percentage": 0.02 } }, "content": [ { "reader": { "n
DataX的json配置文件，{}，[]讲解
最新发布
全局可见
05-25 482
通过DataX将文件系统迁移到MySQL配置文件如下，（json文件中是不允许有//注释的，这里为了理解配置项的含义所以给加上了）在 JSON（JavaScript Object Notation）中，{}和[]
DataX-json模板(hdfsToMysql、hdfsToOracle、mysqlToHdfs、oracleToDoris、oracleToHDFS...)【全】
weixin_42326851的博客
04-25 920
DataX-json模板(hdfsToMysql、hdfsToOracle、mysqlToHdfs、oracleToDoris)【全】
datax的jobjson的jinja2模板oracle2mysql
viviliving的专栏
11-16 520
{ "job": { "setting": { "speed": { "channel": $speedchannel, "byte": $speedbyte }, "errorLimit": { "record": $errrecord, "percentage": $errpercentage } }, "content": [ { ".
datax的json范例
08-26
datax的简单范例. 有一些解释,适用于初学者
阿里开源ETL工具DATAX
05-29
用户需要编写JSON格式的任务配置文件，定义数据源、数据目标以及转换规则。然后，通过执行DATAX提供的命令行工具启动数据同步任务。在任务运行过程中，DATAX会监控任务状态，记录日志，并在完成后提供详细报告。 5...
DataX篇—阿里开源ETL同步工具
qq_33451695的博客
03-21 2440
DataX篇—阿里开源ETL同步工具DataX插件简介DataX架构核心模块介绍DataX 执行流程编程接口模板：dataX-web 可视化管理工具DataX-web架构图Data-web 功能说明日志分析 DataX 插件简介 DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台，实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、DRDS 等各种异构数据源之间高效的数据
ETL可视化工具 DataX -- DataX-Web安装 (三)
dazhong2012的专栏
06-17 2377
1.任务管理–数据源管理 ：创建 数据库连接信息2.任务管理–datax 任务模板：创建 定时执行任务模板3.创建完成后，在任务管理中查看。
ETL可视化工具 DataX -- 安装部署 ( 二)
dazhong2012的专栏
06-14 861
可以通过命令查看配置模板： python datax.py -r {YOUR_READER} -w {YOUR_WRITER}
datax的json范例--datax的简单范例. 有一些解释,适用于初学者
07-03
datax的json范例--datax的简单范例. 有一些解释,适用于初学者
DataX Json配置.json
01-18
datax抽取mysql表到pgsql表的json配置
datax同步模板.rar
09-24
datax安装包下载，下载之后直接解压即可用
DataX
我的笔记
03-15 1556
大数据自学笔记——DataX学习笔记
0101模板生成任务与shell命令执行任务-datax-python工具
gaogzhen的博客
03-30 873
0101模板生成任务与shell命令执行任务-datax-python工具
DataxJson文件生成示例
weixin_45793879的博客
12-11 1717
小伙伴们可以参考，具体的实现根据自己的需求修改。
datax小模板
BackToMe的博客
09-21 571
从mysql到hdfs { "job": { "content": [ { "reader": { "name": "mysqlreader", "parameter": { "column"...
datax的jobjson的jinja2模板mysql2oracle
viviliving的专栏
11-16 504
{ "job": { "setting": { "speed": { "channel": $speedchannel, "byte": $speedbyte }, "errorLimit": { "record": $errrecord, "percentage": $errpercentage } }, "content": [ { ".
ETL抽取工具DATAX
02-13
### ETL 工具 DataX 使用指南和配置教程 #### 安装与环境准备 为了使用 DataX 进行数据同步操作，需先满足一些基本的前置条件。Java 开发工具包 (JDK) 是必需的组件之一[^2]。 安装完成后，可以通过简单的命令来验证 JDK 是否成功安装： ```bash java -version ``` #### 配置 DataX 项目结构 DataX 支持插件式的架构设计，这意味着可以根据需求灵活加载不同类型的读写器(reader/writer)[^1]。通常情况下，一个完整的 DataX 作业配置文件会包含以下几个部分: - **job**: 描述本次任务的整体属性。 - **setting**: 设置并发数、错误记录条目数量等参数。 - **content**: 列表形式定义多个具体的数据传输子任务(taskGroup)，每个 taskGroup 下又细分为 reader 和 writer 的设置项。 下面是一个基础 JSON 格式的配置模板示例： ```json { "job": { "content":[ { "reader":{ "name":"mysqlreader", "parameter":{ ... } }, "writer":{ "name":"streamwriter", "parameter":{ ... } } } ], "setting":{} } } ``` 此模板展示了如何指定 MySQL 数据库作为源端以及控制台输出流为目标端之间的简单迁移过程。 #### 提升稳定性措施 考虑到实际应用场景中的复杂性和不确定性，DataX 特别注重提高系统的健壮性。针对可能出现的各种异常状况，比如网络波动或资源不可达等问题，实现了多级别的自动恢复机制——包括但不限于线程内重试、跨线程的任务重启等功能特性。这些改进有助于确保即使遇到突发问题也能尽可能保障数据同步工作的连续性和可靠性。 #### 日志监控功能 除了强大的容错能力外，DataX 还提供了详尽的日志记录服务。通过内置的日志系统，不仅可以追踪每一步执行状态的变化，还能获取有关吞吐量统计、内存占用率等方面的关键指标信息，便于后续分析优化工作。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值