目录
一、概要二、核心引擎部署 自动安装 验证服务 启动服务
手动安装
三、Web界面部署 初始化数据库 配置文件修改 配置下载数据源jar包 启动服务
四、web页面一览登录数据源任务用户管理虚拟表
五、其它问题记录总结
一、概要
安装包及源码包下载地址
Apache SeaTunnel 是一个分布式、高性能、易扩展的数据集成平台，用于实时和离线数据处理。它支持多种数据源之间的数据迁移和转换。其中，apache-seatunnel-web-1.0.0-bin.tar.gz 和 apache-seatunnel-2.3.3-bin.tar.gz 分别代表了 Apache SeaTunnel Web 界面的安装包和 Apache SeaTunnel 的核心引擎的安装包。
apache-seatunnel-web-1.0.0-bin.tar.gz：这是 Apache SeaTunnel Web 的安装包，提供了用户友好的 Web 界面，使得用户可以更方便地管理和监控 SeaTunnel 任务，包括创建、编辑和运行作业等操作。通过这个界面，用户可以无需直接接触命令行就能进行配置和管理。apache-seatunnel-2.3.3-bin.tar.gz：这是 Apache SeaTunnel 核心引擎的二进制发行版，包含了执行数据抽取、转换和加载的核心代码库。它是实际执行数据处理任务的部分，可以与各种数据源和目标（如数据库、文件系统、消息队列等）进行交互。
在部署 Apache SeaTunnel 时，通常需要先安装并配置好核心引擎，然后根据需要选择是否使用 Web 界面进行管理。如果你只需要在命令行下运行和管理任务，那么只需要安装核心引擎即可。如果你想拥有一个图形化的任务管理界面，那么就需要同时安装 Apache SeaTunnel Web。
二、核心引擎部署
上传解压安装包： tar xf apache-seatunnel-2.3.3-bin.tar.gz
自动安装
自动安装插件（全量安装，很慢，一直到下班，一个半小时都没结束，第二天早上来显示下载结束）
[opensource@bigdata02 bin]$ pwd
/home/opensource/app/apache-seatunnel-2.3.3/bin
[opensource@bigdata02 bin]$ ls
install-plugin.sh seatunnel.sh start-seatunnel-flink-15-connector-v2.sh start-seatunnel-spark-3-connector-v2.sh
seatunnel-cluster.sh start-seatunnel-flink-13-connector-v2.sh start-seatunnel-spark-2-connector-v2.sh stop-seatunnel-cluster.sh
[opensource@bigdata02 bin]$
[opensource@bigdata02 bin]$ ./install-plugin.sh
验证服务
执行系统自带的任务进行验证： sh seatunnel.sh --config ../config/v2.batch.config.template -m local
[opensource@bigdata02 bin]$ sh seatunnel.sh --config ../config/v2.batch.config.template -m local
十二月 08, 2023 9:09:51 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
十二月 08, 2023 9:09:51 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml
十二月 08, 2023 9:09:51 上午 org.apache.seatunnel.engine.common.config.SeaTunnelConfig
信息: seatunnel.home is /home/opensource/app/apache-seatunnel-2.3.3/bin
十二月 08, 2023 9:09:51 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
十二月 08, 2023 9:09:51 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml
2023-12-08 09:09:51,483 WARN com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-233929] [5.1] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2023-12-08 09:09:51,487 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-233929] [5.1] Resolving domain name 'localhost' to address(es): [127.0.0.1]
2023-12-08 09:09:51,487 INFO com.hazelcast.instance.AddressPicker - [LOCAL] [seatunnel-233929] [5.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [localhost/127.0.0.1]
2023-12-08 09:09:51,509 INFO org.apache.seatunnel.engine.server.SeaTunnelServer - SeaTunnel server start...
2023-12-08 09:09:51,510 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] Based on Hazelcast IMDG version: 5.1.0 (20220228 - 21f20e7)
2023-12-08 09:09:51,510 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] Cluster name: seatunnel-233929
2023-12-08 09:09:51,510 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1]
_____ _____ _
/ ___| |_ _| | |
\ `--. ___ __ _ | | _ _ _ __ _ __ ___ | |
`--. \ / _ \ / _` | | | | | | || '_ \ | '_ \ / _ \| |
/\__/ /| __/| (_| | | | | |_| || | | || | | || __/| |
\____/ \___| \__,_| \_/ \__,_||_| |_||_| |_| \___||_|
2023-12-08 09:09:51,511 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] Copyright © 2021-2022 The Apache Software Foundation. Apache SeaTunnel, SeaTunnel, and its feather logo are trademarks of The Apache Software Foundation.
2023-12-08 09:09:51,511 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] Integrity Checker is disabled. Fail-fast on corrupted executables will not be performed.
To enable integrity checker do one of the following:
- Change member config using Java API: config.setIntegrityCheckerEnabled(true);
- Change XML/YAML configuration property: Set hazelcast.integrity-checker.enabled to true
- Add system property: -Dhz.integritychecker.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_INTEGRITYCHECKER_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2023-12-08 09:09:51,513 INFO com.hazelcast.system - [localhost]:5801 [seatunnel-233929] [5.1] The Jet engine is disabled.
To enable the Jet engine on the members, do one of the following:
- Change member config using Java API: config.getJetConfig().setEnabled(true)
- Change XML/YAML configuration property: Set hazelcast.jet.enabled to true
- Add system property: -Dhz.jet.enabled=true (for Hazelcast embedded, works only when loading config via Config.load)
- Add environment variable: HZ_JET_ENABLED=true (recommended when running container image. For Hazelcast embedded, works only when loading config via Config.load)
2023-12-08 09:09:51,867 INFO com.hazelcast.system.security - [localhost]:5801 [seatunnel-233929] [5.1] Enable DEBUG/FINE log level for log category com.hazelcast.system.security or use -Dhazelcast.security.recommendations system property to see security recommendations and the status of current config.
2023-12-08 09:09:51,919 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-233929] [5.1] Using TCP/IP discovery
2023-12-08 09:09:51,920 WARN com.hazelcast.cp.CPSubsystem - [localhost]:5801 [seatunnel-233929] [5.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
十二月 08, 2023 9:09:52 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
十二月 08, 2023 9:09:52 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml
十二月 08, 2023 9:09:52 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
十二月 08, 2023 9:09:52 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml
2023-12-08 09:09:52,071 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] The Node is not ready yet, Node state STARTING,looking forward to the next scheduling
2023-12-08 09:09:52,072 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] Created new BusWork : 1217639236
2023-12-08 09:09:52,082 WARN org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - failed send heartbeat to resource manager, will retry later. this address: [localhost]:5801
2023-12-08 09:09:52,084 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 0
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 0
completedTaskCount : 0
taskCount : 0
***********************************************
2023-12-08 09:09:52,121 INFO com.hazelcast.internal.diagnostics.Diagnostics - [localhost]:5801 [seatunnel-233929] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-12-08 09:09:52,128 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5801 is STARTING
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5802 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5817 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5819 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5823 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5804 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5821 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5813 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5825 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5815 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5808 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5806 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5829 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5810 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5827 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5822 is added to the blacklist.
2023-12-08 09:09:52,166 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5820 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5814 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5812 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5826 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5805 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5807 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5818 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5816 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5830 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5809 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5811 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5824 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5828 is added to the blacklist.
2023-12-08 09:09:52,167 INFO com.hazelcast.internal.cluster.impl.TcpIpJoiner - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5803 is added to the blacklist.
2023-12-08 09:09:53,162 INFO com.hazelcast.internal.cluster.ClusterService - [localhost]:5801 [seatunnel-233929] [5.1]
Members {size:1, ver:1} [
Member [localhost]:5801 - 87c80e4a-80c5-4b82-b577-cb0649025b5e this
]
2023-12-08 09:09:53,172 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5801 is STARTED
十二月 08, 2023 9:09:53 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast-client.yaml' from System property 'hazelcast.client.config'
十二月 08, 2023 9:09:53 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast-client.yaml
2023-12-08 09:09:53,269 INFO com.hazelcast.client.impl.spi.ClientInvocationService - hz.client_1 [seatunnel-233929] [5.1] Running with 2 response threads, dynamic=true
2023-12-08 09:09:53,279 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTING
2023-12-08 09:09:53,279 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is STARTED
2023-12-08 09:09:53,285 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-233929] [5.1] Trying to connect to cluster: seatunnel-233929
2023-12-08 09:09:53,286 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-233929] [5.1] Trying to connect to [localhost]:5801
2023-12-08 09:09:53,299 INFO com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask - [localhost]:5801 [seatunnel-233929] [5.1] Received auth from Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55059, qualifier=null, endpoint=[127.0.0.1]:55059, remoteUuid=535a4a1b-58b7-40a4-8909-218d69fedfe5, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 535a4a1b-58b7-40a4-8909-218d69fedfe5, client name: hz.client_1, client version: 5.1
2023-12-08 09:09:53,302 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_CONNECTED
2023-12-08 09:09:53,302 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-233929] [5.1] Authenticated with server [localhost]:5801:87c80e4a-80c5-4b82-b577-cb0649025b5e, server version: 5.1, local address: /127.0.0.1:55059
2023-12-08 09:09:53,303 INFO com.hazelcast.internal.diagnostics.Diagnostics - hz.client_1 [seatunnel-233929] [5.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2023-12-08 09:09:53,308 INFO com.hazelcast.client.impl.spi.ClientClusterService - hz.client_1 [seatunnel-233929] [5.1]
Members [1] {
Member [localhost]:5801 - 87c80e4a-80c5-4b82-b577-cb0649025b5e
}
2023-12-08 09:09:53,329 INFO com.hazelcast.client.impl.statistics.ClientStatisticsService - Client statistics is enabled with period 5 seconds.
2023-12-08 09:09:53,453 INFO org.apache.seatunnel.engine.client.job.JobExecutionEnvironment - add common jar in plugins :[]
2023-12-08 09:09:53,466 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Loading config file from path: ../config/v2.batch.config.template
2023-12-08 09:09:53,504 INFO org.apache.seatunnel.core.starter.utils.ConfigShadeUtils - Load config shade spi: [base64]
2023-12-08 09:09:53,540 INFO org.apache.seatunnel.core.starter.utils.ConfigBuilder - Parsed config file: {
"env" : {
"execution.parallelism" : 2,
"job.mode" : "BATCH",
"checkpoint.interval" : 10000
},
"source" : [
{
"schema" : {
"fields" : {
"name" : "string",
"age" : "int"
}
},
"row.num" : 16,
"parallelism" : 2,
"result_table_name" : "fake",
"plugin_name" : "FakeSource"
}
],
"sink" : [
{
"plugin_name" : "Console"
}
]
}
2023-12-08 09:09:53,560 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,561 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,564 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel
2023-12-08 09:09:53,568 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
2023-12-08 09:09:53,568 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Console at: file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar
2023-12-08 09:09:53,571 INFO org.apache.seatunnel.engine.core.parse.ConfigParserUtil - Currently, incorrect configuration of source_table_name and result_table_name options don't affect job running. In the future we will ban incorrect configurations.
2023-12-08 09:09:53,571 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,571 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,571 WARN org.apache.seatunnel.engine.core.parse.ConfigParserUtil - This configuration is not recommended. A source/transform(FakeSource) is configured with 'result_table_name' option value of 'fake', but subsequent transform/sink(Console) is not configured with 'source_table_name' option.
2023-12-08 09:09:53,572 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sources.
2023-12-08 09:09:53,572 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,581 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel
2023-12-08 09:09:53,585 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: FakeSource at: file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar
2023-12-08 09:09:53,586 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='FakeSource'} from classpath
2023-12-08 09:09:53,603 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all transforms.
2023-12-08 09:09:53,603 INFO org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser - start generating all sinks.
2023-12-08 09:09:53,603 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,607 INFO org.apache.seatunnel.api.configuration.ReadonlyConfig - Config uses fallback configuration key 'plugin_name' instead of key 'factory'
2023-12-08 09:09:53,636 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Start submit job, job id: 785312093409640449, with plugin jar [file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar]
2023-12-08 09:09:53,640 WARN org.apache.seatunnel.engine.server.SeaTunnelServer - This is master node, waiting the coordinator service init finished
2023-12-08 09:09:54,086 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1] This node become a new active master node, begin init coordinator service
2023-12-08 09:09:54,114 INFO com.hazelcast.internal.partition.impl.PartitionStateManager - [localhost]:5801 [seatunnel-233929] [5.1] Initializing cluster partition table arrangement...
2023-12-08 09:09:54,141 WARN org.apache.seatunnel.engine.server.SeaTunnelServer - This is master node, waiting the coordinator service init finished
2023-12-08 09:09:54,646 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - Init ResourceManager
2023-12-08 09:09:54,646 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker...
2023-12-08 09:09:54,646 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker live nodes: [[localhost]:5801]
2023-12-08 09:09:54,648 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - registerWorker: {[localhost]:5801=WorkerProfile(address=[localhost]:5801, profile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=1029177344}}, unassignedResource=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=1029177344}}, assignedSlots=[], unassignedSlots=[])}
2023-12-08 09:09:54,662 INFO org.apache.seatunnel.engine.server.master.JobMaster - Init JobMaster for Job SeaTunnel_Job (785312093409640449)
2023-12-08 09:09:54,662 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (785312093409640449) needed jar urls [file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-fake-2.3.3.jar, file:/home/opensource/app/apache-seatunnel-2.3.3/connectors/seatunnel/connector-console-2.3.3.jar]
2023-12-08 09:09:54,874 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-12-08 09:09:54,976 INFO org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage - Path /tmp/seatunnel/checkpoint_snapshot/785312093409640449 is not a directory
2023-12-08 09:09:54,976 INFO org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage - No checkpoint found for job, job id is: 785312093409640449
2023-12-08 09:09:54,978 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] is in state CREATED when init state future
2023-12-08 09:09:54,979 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] is in state CREATED when init state future
2023-12-08 09:09:54,979 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] is in state CREATED when init state future
2023-12-08 09:09:54,981 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Submit job finished, job id: 785312093409640449, job name: SeaTunnel
2023-12-08 09:09:54,982 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job Job SeaTunnel_Job (785312093409640449) waiting for scheduler finished
2023-12-08 09:09:54,984 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (785312093409640449) turn from state CREATED to SCHEDULED.
2023-12-08 09:09:54,987 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] turn from state CREATED to SCHEDULED.
2023-12-08 09:09:54,989 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] state from CREATED to SCHEDULED
2023-12-08 09:09:54,990 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] turn from state CREATED to SCHEDULED.
2023-12-08 09:09:54,994 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] state from CREATED to SCHEDULED
2023-12-08 09:09:54,994 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] turn from state CREATED to SCHEDULED.
2023-12-08 09:09:54,995 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] state from CREATED to SCHEDULED
2023-12-08 09:09:54,996 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] turn from state CREATED to SCHEDULED.
2023-12-08 09:09:55,003 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] turn from state SCHEDULED to DEPLOYING.
2023-12-08 09:09:55,005 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] state from SCHEDULED to DEPLOYING
2023-12-08 09:09:55,006 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] turn from state SCHEDULED to DEPLOYING.
2023-12-08 09:09:55,006 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] state from SCHEDULED to DEPLOYING
2023-12-08 09:09:55,007 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] turn from state SCHEDULED to DEPLOYING.
2023-12-08 09:09:55,008 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] state from SCHEDULED to DEPLOYING
2023-12-08 09:09:55,009 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] turn from state SCHEDULED to DEPLOYING.
2023-12-08 09:09:55,009 WARN org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run
2023-12-08 09:09:55,016 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] received deploying task executionId [785312099889840129]
2023-12-08 09:09:55,025 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] received deploying task executionId [785312099889971201]
2023-12-08 09:09:55,025 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] received deploying task executionId [785312099889905665]
2023-12-08 09:09:55,034 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1}, executionId [785312099889840129]
2023-12-08 09:09:55,035 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1} init success
2023-12-08 09:09:55,040 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1} success
2023-12-08 09:09:55,040 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] state from DEPLOYING to RUNNING
2023-12-08 09:09:55,042 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: pipeline-1 [Source[0]-FakeSource-fake]
2023-12-08 09:09:55,042 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] turn from state DEPLOYING to RUNNING.
2023-12-08 09:09:55,047 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}, executionId [785312099889971201]
2023-12-08 09:09:55,048 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001} init success
2023-12-08 09:09:55,048 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}, executionId [785312099889905665]
2023-12-08 09:09:55,049 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001} success
2023-12-08 09:09:55,050 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] state from DEPLOYING to RUNNING
2023-12-08 09:09:55,051 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000} init success
2023-12-08 09:09:55,052 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] turn from state DEPLOYING to RUNNING.
2023-12-08 09:09:55,052 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] deploying TaskGroup TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000} success
2023-12-08 09:09:55,053 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Try to update the task Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] state from DEPLOYING to RUNNING
2023-12-08 09:09:55,054 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] turn from state DEPLOYING to RUNNING.
2023-12-08 09:09:55,054 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] turn from state DEPLOYING to RUNNING.
2023-12-08 09:09:55,054 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 0
2023-12-08 09:09:55,054 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 1
2023-12-08 09:09:55,060 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (785312093409640449) turn from state SCHEDULED to RUNNING.
2023-12-08 09:09:55,061 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (785312093409640449) scheduler finished
2023-12-08 09:09:55,064 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 0
2023-12-08 09:09:55,064 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 1
2023-12-08 09:09:55,068 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - output rowType: name
2023-12-08 09:09:55,068 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - output rowType: name
2023-12-08 09:09:55,166 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}, taskID=40001, index=1}
2023-12-08 09:09:55,166 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}, taskID=40000, index=0}
2023-12-08 09:09:55,245 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received enough reader, starting enumerator...
2023-12-08 09:09:55,245 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Starting to calculate splits.
2023-12-08 09:09:55,246 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Assigned [FakeSourceSplit(splitId=1, rowNum=16), FakeSourceSplit(splitId=0, rowNum=16)] to 2 readers.
2023-12-08 09:09:55,246 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Calculated splits successfully, the size of splits is 2.
2023-12-08 09:09:55,247 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Assigning splits to readers 0 [FakeSourceSplit(splitId=0, rowNum=16)]
2023-12-08 09:09:55,251 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceSplitEnumerator - Assigning splits to readers 1 [FakeSourceSplit(splitId=1, rowNum=16)]
2023-12-08 09:09:55,269 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceReader - 16 rows of data have been generated in split(0). Generation time: 1701997795267
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceReader - Closed the bounded fake source
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceReader - 16 rows of data have been generated in split(1). Generation time: 1701997795266
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.fake.source.FakeSourceReader - Closed the bounded fake source
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=1: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : kkTzr, 348993738
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=1: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : cHRlJ, 141341247
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=2: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : JoqKS, 949667288
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=2: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : wuRPB, 739004063
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=3: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : vzyWZ, 1185127730
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=3: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : MzsIj, 255990586
2023-12-08 09:09:55,270 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=4: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : pKInq, 1307779433
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=4: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : JmaUB, 255860364
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=5: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : MYeRA, 1997823969
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=5: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : uVILC, 995215612
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=6: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : DlVbk, 731828921
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=7: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : inYyz, 1847828742
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=6: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : cyjxD, 1832606505
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=7: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : eMzXN, 802135106
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=8: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : kPUMq, 1916092538
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=8: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : aySgR, 1299495336
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=9: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : vFyQU, 1895773363
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=9: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : pjYqn, 1385074915
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=10: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : dQmgk, 1234357286
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=10: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : zASIv, 1497863961
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=11: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : zQuTA, 903385081
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=11: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : IVXOt, 870847193
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=12: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : jrLTl, 705955925
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=13: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : YflRj, 293817756
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=12: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : DbvZU, 1486490099
2023-12-08 09:09:55,271 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=14: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : SeaoD, 1066109912
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=13: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : VsbMy, 910565833
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=15: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : vkEhB, 403795863
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=14: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : fegPV, 165600914
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=0 rowIndex=16: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : rXrEp, 1087983282
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=15: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : DokOA, 1281308771
2023-12-08 09:09:55,272 INFO org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter - subtaskIndex=1 rowIndex=16: SeaTunnelRow#tableId= SeaTunnelRow#kind=INSERT : VjDeD, 1091404164
2023-12-08 09:09:55,356 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 9223372036854775807
2023-12-08 09:09:55,411 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - pending checkpoint(9223372036854775807/1@785312093409640449) notify finished!
2023-12-08 09:09:55,411 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start notify checkpoint completed, checkpoint:org.apache.seatunnel.engine.server.checkpoint.CompletedCheckpoint@4d1c18a2
2023-12-08 09:09:55,416 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator completed.
2023-12-08 09:09:55,417 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - Turn checkpoint_state_785312093409640449_1 state from null to FINISHED
2023-12-08 09:09:55,453 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 20000, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1}
2023-12-08 09:09:55,453 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] Task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1} complete with state FINISHED
2023-12-08 09:09:55,454 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1] Received task end from execution TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=1}, state FINISHED
2023-12-08 09:09:55,455 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] turn to end state FINISHED.
2023-12-08 09:09:55,455 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SplitEnumerator (1/1)] end with state FINISHED
2023-12-08 09:09:55,464 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 50001, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}
2023-12-08 09:09:55,464 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 50000, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}
2023-12-08 09:09:56,271 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 40001, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}
2023-12-08 09:09:56,271 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] taskDone, taskId = 40000, taskGroup = TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}
2023-12-08 09:09:56,272 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] Task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000} complete with state FINISHED
2023-12-08 09:09:56,272 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-233929] [5.1] Task TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001} complete with state FINISHED
2023-12-08 09:09:56,272 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1] Received task end from execution TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30000}, state FINISHED
2023-12-08 09:09:56,272 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-233929] [5.1] Received task end from execution TaskGroupLocation{jobId=785312093409640449, pipelineId=1, taskGroupId=30001}, state FINISHED
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] turn to end state FINISHED.
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] turn to end state FINISHED.
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (1/2)] end with state FINISHED
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-FakeSource-fake]-SourceTask (2/2)] end with state FINISHED
2023-12-08 09:09:56,273 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] end with state FINISHED
2023-12-08 09:09:56,310 INFO org.apache.seatunnel.engine.server.master.JobMaster - release the pipeline Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] resource
2023-12-08 09:09:56,311 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 785312093409640449, slot: SlotProfile{worker=[localhost]:5801, slotID=1, ownerJobID=785312093409640449, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='fa7bad5a-cd2c-4d97-8368-88027b3b49b6'}
2023-12-08 09:09:56,311 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 785312093409640449, slot: SlotProfile{worker=[localhost]:5801, slotID=2, ownerJobID=785312093409640449, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='fa7bad5a-cd2c-4d97-8368-88027b3b49b6'}
2023-12-08 09:09:56,312 INFO org.apache.seatunnel.engine.server.service.slot.DefaultSlotService - received slot release request, jobID: 785312093409640449, slot: SlotProfile{worker=[localhost]:5801, slotID=3, ownerJobID=785312093409640449, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='fa7bad5a-cd2c-4d97-8368-88027b3b49b6'}
2023-12-08 09:09:56,314 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (785312093409640449), Pipeline: [(1/1)] turn to end state FINISHED.
2023-12-08 09:09:56,315 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (785312093409640449) end with state FINISHED
2023-12-08 09:09:56,328 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Job (785312093409640449) end with state FINISHED
2023-12-08 09:09:56,353 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand -
***********************************************
Job Statistic Information
***********************************************
Start Time : 2023-12-08 09:09:53
End Time : 2023-12-08 09:09:56
Total Time(s) : 2
Total Read Count : 32
Total Write Count : 32
Total Failed Count : 0
***********************************************
2023-12-08 09:09:56,354 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2023-12-08 09:09:56,357 INFO com.hazelcast.internal.server.tcp.TcpServerConnection - [localhost]:5801 [seatunnel-233929] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55059, qualifier=null, endpoint=[127.0.0.1]:55059, remoteUuid=535a4a1b-58b7-40a4-8909-218d69fedfe5, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2023-12-08 09:09:56,357 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-233929] [5.1] Removed connection to endpoint: [localhost]:5801:87c80e4a-80c5-4b82-b577-cb0649025b5e, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:55059->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2023-12-08 09:09:56.351, lastWriteTime=2023-12-08 09:09:56.328, closedTime=2023-12-08 09:09:56.355, connected server version=5.1}
2023-12-08 09:09:56,358 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2023-12-08 09:09:56,358 INFO com.hazelcast.client.impl.ClientEndpointManager - [localhost]:5801 [seatunnel-233929] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55059, qualifier=null, endpoint=[127.0.0.1]:55059, remoteUuid=535a4a1b-58b7-40a4-8909-218d69fedfe5, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=535a4a1b-58b7-40a4-8909-218d69fedfe5, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1701997793296, latest clientAttributes=lastStatisticsCollectionTime=1701997793329,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1701997793287,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=24435175424,os.freePhysicalMemorySize=50645405696,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=65536,os.openFileDescriptorCount=58,os.processCpuTime=9670000000,os.systemLoadAverage=2.31,os.totalPhysicalMemorySize=540422402048,os.totalSwapSpaceSize=0,runtime.availableProcessors=104,runtime.freeMemory=959249984,runtime.maxMemory=1029177344,runtime.totalMemory=1029177344,runtime.uptime=2770,runtime.usedMemory=69927360, labels=[]}
2023-12-08 09:09:56,360 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-233929] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2023-12-08 09:09:56,360 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2023-12-08 09:09:56,360 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5801 is SHUTTING_DOWN
2023-12-08 09:09:56,363 INFO com.hazelcast.internal.partition.impl.MigrationManager - [localhost]:5801 [seatunnel-233929] [5.1] Shutdown request of Member [localhost]:5801 - 87c80e4a-80c5-4b82-b577-cb0649025b5e this is handled
2023-12-08 09:09:56,367 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-233929] [5.1] Shutting down connection manager...
2023-12-08 09:09:56,369 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-233929] [5.1] Shutting down node engine...
2023-12-08 09:09:58,473 INFO com.hazelcast.instance.impl.NodeExtension - [localhost]:5801 [seatunnel-233929] [5.1] Destroying node NodeExtension.
2023-12-08 09:09:58,474 INFO com.hazelcast.instance.impl.Node - [localhost]:5801 [seatunnel-233929] [5.1] Hazelcast Shutdown is completed in 2111 ms.
2023-12-08 09:09:58,474 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel-233929] [5.1] [localhost]:5801 is SHUTDOWN
2023-12-08 09:09:58,474 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
2023-12-08 09:09:58,474 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2023-12-08 09:09:58,475 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal
[opensource@bigdata02 bin]$
启动服务
启动服务并且后台运行，用于web端连接：nohup sh seatunnel-cluster.sh 2>&1 &
[opensource@bigdata02 bin]$ ls
install-plugin.sh seatunnel.sh start-seatunnel-flink-15-connector-v2.sh start-seatunnel-spark-3-connector-v2.sh
seatunnel-cluster.sh start-seatunnel-flink-13-connector-v2.sh start-seatunnel-spark-2-connector-v2.sh stop-seatunnel-cluster.sh
[opensource@bigdata02 bin]$
[opensource@bigdata02 bin]$ nohup sh seatunnel-cluster.sh 2>&1 &
[1] 93523
[opensource@bigdata02 bin]$ nohup: 忽略输入并把输出追加到"nohup.out"
[opensource@bigdata02 bin]$ ls
install-plugin.sh seatunnel-cluster.sh start-seatunnel-flink-13-connector-v2.sh start-seatunnel-spark-2-connector-v2.sh stop-seatunnel-cluster.sh
nohup.out seatunnel.sh start-seatunnel-flink-15-connector-v2.sh start-seatunnel-spark-3-connector-v2.sh
[opensource@bigdata02 bin]$ tail -f nohup.out
十二月 08, 2023 9:30:58 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml
十二月 08, 2023 9:30:59 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml' from System property 'seatunnel.config'
十二月 08, 2023 9:30:59 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/seatunnel.yaml
十二月 08, 2023 9:30:59 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Loading configuration '/home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml' from System property 'hazelcast.config'
十二月 08, 2023 9:30:59 上午 com.hazelcast.internal.config.AbstractConfigLocator
信息: Using configuration file at /home/opensource/app/apache-seatunnel-2.3.3/config/hazelcast.yaml
^C
[opensource@bigdata02 bin]$ tail -f ../logs/seatunnel-engine-server.log
Member [localhost]:5801 - 9db58520-2335-441c-8999-ad78123f6509 this
]
2023-12-08 09:30:59,930 INFO com.hazelcast.core.LifecycleService - [localhost]:5801 [seatunnel] [5.1] [localhost]:5801 is STARTED
2023-12-08 09:31:01,865 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel] [5.1] This node become a new active master node, begin init coordinator service
2023-12-08 09:31:02,061 INFO com.hazelcast.internal.partition.impl.PartitionStateManager - [localhost]:5801 [seatunnel] [5.1] Initializing cluster partition table arrangement...
2023-12-08 09:31:04,863 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - Init ResourceManager
2023-12-08 09:31:04,863 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker...
2023-12-08 09:31:04,864 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker live nodes: [[localhost]:5801]
2023-12-08 09:31:04,865 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - registerWorker: {[localhost]:5801=WorkerProfile(address=[localhost]:5801, profile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=2058354688}}, unassignedResource=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=2058354688}}, assignedSlots=[], unassignedSlots=[])}
2023-12-08 09:31:59,863 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 0
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 0
completedTaskCount : 0
taskCount : 0
***********************************************
2023-12-08 09:31:59,867 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel] [5.1]
***********************************************
Job info detail
***********************************************
createdJobCount : 0
scheduledJobCount : 0
runningJobCount : 0
failingJobCount : 0
failedJobCount : 0
cancellingJobCount : 0
canceledJobCount : 0
finishedJobCount : 0
restartingJobCount : 0
suspendedJobCount : 0
reconcilingJobCount : 0
***********************************************
手动安装
不想全部安装，修改配置：vi config/plugin_config （具体保留哪些就不清楚了，只能说官方提供的安装包不好，无法直接启动，必须要先安装插件，但是这么多插件下载又很慢、也没必要） 无法在线安装，只能手动了 maven库下载地址 对于自动下载下来的jar包，已经全部打包上传到CSDN,大家可以免费下载，直接放到connectors/seatunnel 下载地址 CSDN资源大小不允许超过1000M,这里采用.tar.xz格式（解压会有些慢，小十分钟吧）
三种压缩命令：tar -cf seatunnel_connectors_2.3.3.tar seatunnel
tar -czf seatunnel_connectors_2.3.3.tar.gz seatunnel
tar -cJf seatunnel_connectors_2.3.3.tar.xz seatunnel
对应解压命令：tar -xf seatunnel_connectors_2.3.3.tar
tar -xzf seatunnel_connectors_2.3.3.tar.gz
tar -xJf seatunnel_connectors_2.3.3.tar.xz
三、Web界面部署
上传解压web部署包：tar xf apache-seatunnel-web-1.0.0-bin.tar.gz
初始化数据库
初始化SQL脚本script/seatunnel_server_mysql.sql
SET FOREIGN_KEY_CHECKS = 0;
-- ----------------------------
-- Table structure for role
-- ----------------------------
DROP TABLE IF EXISTS `role`;
CREATE TABLE `role` (
`id` int(20) NOT NULL AUTO_INCREMENT,
`type` int(2) NOT NULL,
`role_name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`description` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,
`create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of role
-- ----------------------------
INSERT INTO `role`(`type`,`role_name`,`description`) values (0, 'ADMIN_ROLE', 'Admin User');
INSERT INTO `role`(`type`,`role_name`,`description`) values (1, 'NORMAL_ROLE', 'Normal User');
-- ----------------------------
-- Table structure for role_user_relation
-- ----------------------------
DROP TABLE IF EXISTS `role_user_relation`;
CREATE TABLE `role_user_relation` (
`id` int(20) NOT NULL AUTO_INCREMENT,
`role_id` int(20) NOT NULL,
`user_id` int(20) NOT NULL,
`create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_datasource
-- ----------------------------
DROP TABLE IF EXISTS `t_st_datasource`;
CREATE TABLE `t_st_datasource` (
`id` bigint(20) NOT NULL,
`datasource_name` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`plugin_name` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`plugin_version` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT '1.0.0',
`datasource_config` varchar(1023) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`description` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(11) NOT NULL,
`update_user_id` int(11) NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `t_st_datasource_datasource_name_uindex`(`datasource_name`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_job_definition
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_definition`;
CREATE TABLE `t_st_job_definition` (
`id` bigint(20) NOT NULL,
`name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`description` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`job_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(11) NOT NULL,
`update_user_id` int(11) NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE,
UNIQUE INDEX `name`(`name`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_job_instance
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_instance`;
CREATE TABLE `t_st_job_instance` (
`id` bigint(20) NOT NULL,
`job_define_id` bigint(20) NOT NULL,
`job_status` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`job_config` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`engine_name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`engine_version` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`job_engine_id` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(20) NOT NULL,
`update_user_id` int(20) NULL DEFAULT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
`end_time` timestamp(3) NULL DEFAULT NULL,
`job_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_job_instance_history
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_instance_history`;
CREATE TABLE `t_st_job_instance_history` (
`id` bigint(20) NOT NULL,
`dag` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_job_line
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_line`;
CREATE TABLE `t_st_job_line` (
`id` bigint(20) NOT NULL,
`version_id` bigint(20) NOT NULL,
`input_plugin_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`target_plugin_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE,
INDEX `job_line_version_index`(`version_id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_job_metrics
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_metrics`;
CREATE TABLE `t_st_job_metrics` (
`id` bigint(20) NOT NULL,
`job_instance_id` bigint(20) NOT NULL,
`pipeline_id` int(20) NOT NULL,
`read_row_count` bigint(20) NOT NULL,
`write_row_count` bigint(20) NOT NULL,
`source_table_names` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`sink_table_names` varchar(200) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`read_qps` bigint(20) NULL DEFAULT NULL,
`write_qps` bigint(20) NULL DEFAULT NULL,
`record_delay` bigint(20) NULL DEFAULT NULL,
`status` varchar(20) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(20) NOT NULL,
`update_user_id` int(20) NULL DEFAULT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_job_task
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_task`;
CREATE TABLE `t_st_job_task` (
`id` bigint(20) NOT NULL,
`version_id` bigint(20) NOT NULL,
`plugin_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`config` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL,
`transform_options` varchar(5000) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`output_schema` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL,
`connector_type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`datasource_id` bigint(20) NULL DEFAULT NULL,
`datasource_option` varchar(5000) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`select_table_fields` varchar(5000) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`scene_mode` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`type` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE,
INDEX `job_task_plugin_id_index`(`plugin_id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_job_version
-- ----------------------------
DROP TABLE IF EXISTS `t_st_job_version`;
CREATE TABLE `t_st_job_version` (
`id` bigint(20) NOT NULL,
`job_id` bigint(20) NOT NULL,
`name` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`job_mode` varchar(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`env` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL,
`engine_name` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`engine_version` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`create_user_id` int(11) NOT NULL,
`update_user_id` int(11) NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for t_st_virtual_table
-- ----------------------------
DROP TABLE IF EXISTS `t_st_virtual_table`;
CREATE TABLE `t_st_virtual_table` (
`id` bigint(20) NOT NULL,
`datasource_id` bigint(20) NOT NULL,
`virtual_database_name` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`virtual_table_name` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`table_fields` varchar(1023) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`virtual_table_config` varchar(1023) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL,
`description` varchar(63) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL DEFAULT NULL,
`create_user_id` int(11) NOT NULL,
`update_user_id` int(11) NOT NULL,
`create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for user
-- ----------------------------
DROP TABLE IF EXISTS `user`;
CREATE TABLE `user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`username` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`password` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`status` tinyint(4) NOT NULL,
`type` tinyint(4) NOT NULL,
`create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 2 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Table structure for user_login_log
-- ----------------------------
DROP TABLE IF EXISTS `user_login_log`;
CREATE TABLE `user_login_log` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`token` mediumtext CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`token_status` tinyint(1) NOT NULL,
`create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
`update_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 106 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;
-- ----------------------------
-- Records of user_login_log
-- ----------------------------
INSERT INTO `user`(`username`,`password`,`status`,`type`) values ('admin', '7f97da8846fed829bb8d1fd9f8030f3b', 0, 0);
SET FOREIGN_KEY_CHECKS = 1;
配置文件修改
配置数据库信息： vi conf/application.yml 复制引擎服务中配置文件到web配置目录下面
[opensource@bigdata02 app]$ cp apache-seatunnel-2.3.3/config/hazelcast-client.yaml apache-seatunnel-web-1.0.0-bin/conf/
[opensource@bigdata02 app]$ cp apache-seatunnel-2.3.3/connectors/plugin-mapping.properties apache-seatunnel-web-1.0.0-bin/conf/
配置下载数据源jar包
选择的mysql作为元数据库，对应的驱动包mysql-connector-java-8.0.25.jar放到libs下 下载其它数据源资源包,首先下载源码，然后执行下载脚本download_datasource.sh(本地安装过git)
源码地址 修改download_datasource.sh相关配置 执行download_datasource.sh 开始下载jar包 下载完成 上传到libs目录
启动服务
sh bin/seatunnel-backend-daemon.sh start
四、web页面一览
登录
登录地址：http://xxxx:8801/ui/#/login ; 用户密码 admin/admin
数据源
任务
选不到数据源… 尝试创建整库同步
用户管理
虚拟表
五、其它
问题记录
服务启动失败：Application run failed org.yaml.snakeyaml.scanner.ScannerException: while scanning an anchor
14:29:08.032 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.yaml.snakeyaml.scanner.ScannerException: while scanning an anchor
in 'reader', line 30, column 15:
password: &R7#J*vDVWEuT48m
^
unexpected character found *(42)
in 'reader', line 30, column 20:
password: &R7#J*vDVWEuT48m
^
at org.yaml.snakeyaml.scanner.ScannerImpl.scanAnchor(ScannerImpl.java:1507)
at org.yaml.snakeyaml.scanner.ScannerImpl.fetchAnchor(ScannerImpl.java:958)
at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:389)
at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:248)
at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingValue.produce(ParserImpl.java:665)
at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:165)
at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:59)
at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:45)
at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:140)
at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:119)
at org.yaml.snakeyaml.composer.Composer.composeScalarN
解决办法，密码&R7#J*vDVWEuT48m增加双引号
总结
apache-seatunnel-web-1.0.0-bin还处在开发早期阶段，并不成熟，功能也不完善安装seatunnel的目的，就是想参考下web界面，寻求针对表的拖拽处理（减少sql编写），目前看效果不理想，也只是一个简单的输入、输出表字段映射，因此暂告一段落
参考链接
您阅读本篇文章共花了：
发表评论