PALO
更新时间：2023-12-07
PALO DDL
CREATE TABLE sink_palo_table (
`field` STRING
) WITH (
'connector.type' = 'PALO',
'format.encode' = 'TXT',
'connector.cluster-id' = 'xxxxxxxxxxxx',
'connector.username' = 'admin',
'connector.password' = 'xxxxxx',
'connector.table' = 'bsc_test_palo_2',
'connector.database' = 'bsc',
'connector.mysql-host' = 'bscTestPalo-be.palo.su.baidubce.com',
'connector.mysql-port' = '9030',
'connector.compute-node-host' = 'bscTestPalo-be.palo.su.baidubce.com',
'connector.compute-node-port' = '8040'
);PALO 参数设置
| 名称 | 简称 | 是否必填 | 用例 | SPARK | FLINK | 说明 |
|---|---|---|---|---|---|---|
| connector.type | type | Y | PALO |
Y | Y | 服务类型 |
| format.encode | encode | 默认 TXT |
Y | Y | 数据编码 | |
| connector.cluster-id | cluster-id | Y | xxxxxxxxxxxx |
Y | Y | PALO 集群ID |
| connector.username | username | Y | admin |
Y | Y | PALO 用户名称 |
| connector.password | password | Y | xxxx |
Y | Y | PALO 用户密码 |
| connector.table | table | Y | bsc_test_palo_2 |
Y | Y | PALO 表名 |
| connector.database | database | Y | bsc |
Y | Y | PALO 数据库名 |
| connector.mysql-host | mysql-host | Y | xxxxxxx-be.palo.su.baidubce.com |
Y | Y | |
| connector.mysql-port | mysql-port | Y | 9030 |
Y | Y | |
| connector.compute-node-host | compute-node-host | Y | xxxxxxx-be.palo.su.baidubce.com |
Y | Y | |
| connector.compute-node-port | compute-node-port | Y | 8040 |
Y | Y | |
| connector.leader-node-host | leader-node-host | xxxxxxx-be.palo.su.baidubce.com |
Y | |||
| connector.leader-node-port | leader-node-port | 8040 |
Y | |||
| connector.properties.http-socket-timeout-ms | http-socket-timeout-ms | 60000 |
Y | |||
| connector.properties.http-connect-timeout-ms | http-connect-timeout-ms | 60000 |
Y | |||
| connector.properties.http-connect-request-timeout-ms | http-connect-request-timeout-ms | 60000 |
Y | |||
| connector.properties.http-request-max-retries | http-request-max-retries | 10000 |
Y | |||
| connector.write.max-message-num-per-batch | max-message-num-per-batch | 1000 |
Y | |||
| connector.write.max-message-size-per-batch | max-message-size-per-batch | 1000 |
Y | |||
| connector.write.message-batch-interval-millis | message-batch-interval-millis | 60000 |
Y |
PALO 示例说明
PALO 仅能接受一个字符串，通过指定的列分隔符，由 palo 自行将其拆分为 palo 数据库的各个字段值。 因此，PALO 的DDL只有一个 STRING 字段， 需要在 DML 当中完成对sink 数据的组装。
CREATE TABLE sink_palo_table (
`field` STRING,
) WITH (
'connector.type' = 'PALO',
'format.encode' = 'TXT',
'format.attributes.field-delimiter' = ';', -- 指定的分隔符，默认逗号
'connector.cluster-id' = 'xxxxxxxxxxx',
'connector.username' = 'admin',
'connector.password' = 'admin',
'connector.table' = 'bsc_test_palo_2',
'connector.database' = 'bsc',
'connector.mysql-host' = 'bscTestPalo-be.palo.su.baidubce.com',
'connector.mysql-port' = '9030',
'connector.compute-node-host' = 'bscTestPalo-be.palo.su.baidubce.com',
'connector.compute-node-port' = '8040'
);
INSERT INTO
sink_palo_table
SELECT
FORMAT_STRING("%s;%d",`field01`, `field02`) -- 组装 string 的时候，使用指定 palo的分隔符来分隔不同字段，需要与上文中 field-delimiter 参数保持一致
FROM
source_kafka_table;