手把手教你源码安装 Dolphin Scheduler 调度系统

本文介绍如何通过源码编译的方式,安装一个伪集群部署的 DolphinScheduler(下称 DS),使用 MySQL 8.0.30 作为 DolphinScheduler 的元数据库。


安装完 MySQL 8 数据库之后,为 DS 创建数据库和用户

注意:MySQL 8 默认不允许客户端从服务器获取公钥,如果在内网环境下可以通过配置: allowPublicKeyRetrieval=true 绕过

CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE USER 'dolphin'@'%' IDENTIFIED BY 'dolphin';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphin'@'%';

CREATE USER 'dolphin'@'localhost' IDENTIFIED BY 'dolphin';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphin'@'localhost';

FLUSH PRIVILEGES;

验证数据库和用户创建成功


安装并启动 Zookeeper

下载地址:https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz


克隆 DolphinScheduler 源码

DS 的源码地址是:https://github.com/apache/dolphinscheduler,通过 Fork 到自己的 Git Rrepositories 中,可以维护一个属于自己的 DolphinScheduler 项目。


Fork 成功之后,就可以通过 git clone 命令,将自己 Rrepositories 中的 DS 项目克隆下来

git@github.com:your_git/dolphinscheduler.git


使用 Intelli IDEA 打开上一步克隆好的 DS 项目


修改 pom.xml 文件

将 mysql-connector-java 的 test 删掉。因为我们使用 MySQL 8 作为 DS 元数据库,要用到 mysql-connector-java 这个 jar 包连接 MySQL,如果不删掉这行的话,打完包会少这个 jar 文件。


编辑环境配置文件

  • 编辑 script/env/install_env.sh

由于是伪集群部署的部署方式,将主机名称都改成 localhost。设置 installPath 的路径为 DS 的安装目录:/home/dolphin/software/apache-dolphinscheduler-3.0.1-SNAPSHOT-bin/

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips=${ips:-"localhost"}

# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort=${sshPort:-"22"}

# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters=${masters:-"localhost"}

# A comma separated list of machine : or :.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers=${workers:-"localhost:default"}

# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer=${alertServer:-"localhost"}

# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers=${apiServers:-"localhost"}

# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd). Do not add quotes to it if you using related path.
installPath=${installPath:-"/home/dolphin/software/apache-dolphinscheduler-3.0.1-SNAPSHOT-bin/"}

# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser=${deployUser:-"dolphin"}

# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
zkRoot=${zkRoot:-"/dolphinscheduler"}


  • 编辑 script/env/dolphinscheduler_env.sh

将 “export DATABASE=${DATABASE:-postgresql}” 改成 “export DATABASE=${DATABASE:-mysql}”,表示使用 mysql 作为元数据库。其它环境变量,根据情况进行相应的设置。

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-1.8.0}

# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-mysql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL="jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true"
export SPRING_DATASOURCE_USERNAME=dolphin
export SPRING_DATASOURCE_PASSWORD=dolphin

# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}

# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}

# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/apps/HADOOP-COMMON/hadoop-common-current/}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/apps/HADOOP-COMMON/hadoop-common-current/etc/hadoop}
export SPARK_HOME1=${SPARK_HOME1:-/opt/apps/SPARK2/spark2-current}
export SPARK_HOME2=${SPARK_HOME2:-/opt/apps/SPARK2/spark2-current}
export PYTHON_HOME=${PYTHON_HOME:-/usr/bin/python3}
export HIVE_HOME=${HIVE_HOME:-/opt/apps/HIVE/hive-current}
export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}

export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH


通过 Maven 命令打包

mvn clean install -Prelease -Dmaven.test.skip=true

打包成功后,可以在以下路径,找到 DS 安装包:dolphinscheduler-dist/target/apache-dolphinscheduler-3.0.1-SNAPSHOT-bin.tar.gz

将该安装包解压到与前面 script/env/install_env.sh 安装配置一致的目录:/home/dolphin/software/apache-dolphinscheduler-3.0.1-SNAPSHOT-bin/


安装 DolphinScheduler

将目录切换到上述 DS 安装路径下,执行以下命令,安装 DS。

bash bin/install.sh


通过 jps 命令,验证 DS 相关的服务都正常启动了。


通过浏览器,登录 Dolphin Scheduler 界面

默认用户名 / 密码:admin / dolphinscheduler123


参考文档:

https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/guide/installation/pseudo-cluster.html

发表评论
留言与评论(共有 0 条评论) “”
   
验证码:

相关文章

推荐文章