本文介绍如何通过源码编译的方式,安装一个伪集群部署的 DolphinScheduler(下称 DS),使用 MySQL 8.0.30 作为 DolphinScheduler 的元数据库。
注意:MySQL 8 默认不允许客户端从服务器获取公钥,如果在内网环境下可以通过配置: allowPublicKeyRetrieval=true 绕过
CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'dolphin'@'%' IDENTIFIED BY 'dolphin';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphin'@'%';
CREATE USER 'dolphin'@'localhost' IDENTIFIED BY 'dolphin';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphin'@'localhost';
FLUSH PRIVILEGES;
验证数据库和用户创建成功
下载地址:https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
DS 的源码地址是:https://github.com/apache/dolphinscheduler,通过 Fork 到自己的 Git Rrepositories 中,可以维护一个属于自己的 DolphinScheduler 项目。
Fork 成功之后,就可以通过 git clone 命令,将自己 Rrepositories 中的 DS 项目克隆下来
git@github.com:your_git/dolphinscheduler.git
将 mysql-connector-java 的
由于是伪集群部署的部署方式,将主机名称都改成 localhost。设置 installPath 的路径为 DS 的安装目录:/home/dolphin/software/apache-dolphinscheduler-3.0.1-SNAPSHOT-bin/
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips=${ips:-"localhost"}
# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort=${sshPort:-"22"}
# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters=${masters:-"localhost"}
# A comma separated list of machine : or :.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers=${workers:-"localhost:default"}
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer=${alertServer:-"localhost"}
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers=${apiServers:-"localhost"}
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd). Do not add quotes to it if you using related path.
installPath=${installPath:-"/home/dolphin/software/apache-dolphinscheduler-3.0.1-SNAPSHOT-bin/"}
# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser=${deployUser:-"dolphin"}
# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
zkRoot=${zkRoot:-"/dolphinscheduler"}
将 “export DATABASE=${DATABASE:-postgresql}” 改成 “export DATABASE=${DATABASE:-mysql}”,表示使用 mysql 作为元数据库。其它环境变量,根据情况进行相应的设置。
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-1.8.0}
# Database related configuration, set database type, username and password
export DATABASE=${DATABASE:-mysql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL="jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false&allowPublicKeyRetrieval=true"
export SPRING_DATASOURCE_USERNAME=dolphin
export SPRING_DATASOURCE_PASSWORD=dolphin
# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}
# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-localhost:2181}
# Tasks related configurations, need to change the configuration if you use the related tasks.
export HADOOP_HOME=${HADOOP_HOME:-/opt/apps/HADOOP-COMMON/hadoop-common-current/}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/apps/HADOOP-COMMON/hadoop-common-current/etc/hadoop}
export SPARK_HOME1=${SPARK_HOME1:-/opt/apps/SPARK2/spark2-current}
export SPARK_HOME2=${SPARK_HOME2:-/opt/apps/SPARK2/spark2-current}
export PYTHON_HOME=${PYTHON_HOME:-/usr/bin/python3}
export HIVE_HOME=${HIVE_HOME:-/opt/apps/HIVE/hive-current}
export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}
export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH
mvn clean install -Prelease -Dmaven.test.skip=true
打包成功后,可以在以下路径,找到 DS 安装包:dolphinscheduler-dist/target/apache-dolphinscheduler-3.0.1-SNAPSHOT-bin.tar.gz
将该安装包解压到与前面 script/env/install_env.sh 安装配置一致的目录:/home/dolphin/software/apache-dolphinscheduler-3.0.1-SNAPSHOT-bin/
将目录切换到上述 DS 安装路径下,执行以下命令,安装 DS。
bash bin/install.sh
通过 jps 命令,验证 DS 相关的服务都正常启动了。
默认用户名 / 密码:admin / dolphinscheduler123
https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/guide/installation/pseudo-cluster.html
留言与评论(共有 0 条评论) “” |