大数据存储-Doris部署

Doris简介

Doris 中文官方文档：doris官方文档

Doris概述

Apache Doris 是一个基于 MPP 架构的高性能、实时的分析型数据库，以极速易用的特点被人们所熟知，仅需亚秒级响应时间即可返回海量数据下的查询结果，不仅可以支持高并发的点查询场景，也能支持高吞吐的复杂分析场景。基于此，Apache Doris 能够较好的满足报表分析、即席查询、统一数仓构建、数据湖联邦查询加速等使用场景，用户可以在此之上构建用户行为分析、AB 实验平台、日志检索分析、用户画像分析、订单分析等应用

​ Apache Doris 最早是诞生于百度广告报表业务的 Palo 项目，2017 年正式对外开源，2018 年 7 月由百度捐赠给 Apache 基金会进行孵化，之后在 Apache 导师的指导下由孵化器项目管理委员会成员进行孵化和运营。目前 Apache Doris 社区已经聚集了来自不同行业数百家企业的 400 余位贡献者，并且每月活跃贡献者人数也超过 100 位。 2022 年 6 月，Apache Doris 成功从 Apache 孵化器毕业，正式成为 Apache 顶级项目（Top-Level Project，TLP）

​ Apache Doris 如今在中国乃至全球范围内都拥有着广泛的用户群体，截止目前， Apache Doris 已经在全球超过 2000 家企业的生产环境中得到应用，在中国市值或估值排行前 50 的互联网公司中，有超过 80% 长期使用 Apache Doris，包括百度、美团、小米、京东、字节跳动、腾讯、网易、快手、微博、贝壳等。同时在一些传统行业如金融、能源、制造、电信等领域也有着丰富的应用

1、服务器规划

2、部署

2.1、二进制部署包下载

#管理包doris-manager-23.7.4-x64-bin.tar.gz
wget https://selectdb-doris-1308700295.cos.ap-beijing.myqcloud.com/doris-manager/release/23.7.4/doris-manager-23.7.4-x64-bin.tar.gz
#核心包selectdb-core，selectdb-doris-2.0.0-bin-x64.tar.gz
wget https://download.selectdb.com/release/selectdb-core/2.0.0/selectdb-doris-2.0.0-bin-x64.tar.gz

2.2、服务器初始化

#生成密钥
[root@doris-fe doris]# ssh-keygen
#设置免密登陆
[root@doris-fe doris]# ssh-copy-id 172.31.16.3
[root@doris-fe doris]# ssh-copy-id 172.31.16.4
[root@doris-fe doris]# ssh-copy-id 172.31.16.5
[root@doris-fe doris]# ssh-copy-id 172.31.16.7
#所有服务器上执行
echo 'vm.max_map_count=2000000' >> /etc/sysctl.conf   #修改最大打开文件数
sed -i 's/65535/655536/g' /etc/security/limits.conf   #修改最大打开文件句柄数

2.3、服务部署

#部署config-tool工具
tar zxvf doris-manager-23.7.4-x64-bin.tar.gz
cp doris-manager-23.7.4-x64-bin/config-tool /usr/local/bin/
#验证
[root@doris-fe doris]# config-tool -v
build_version: 23.7.4
build_time:  2023-09-07T11:15:58+0800

#生成初始配置文件
config-tool template > init.conf
#修改配置文件
base_deploy_dir: /data/doris/manager                   # doris-manager's target deploy dir

webserver:
  version: 23.7.4                               # target installed doris-manager version
  host: 172.31.16.7                                   # localhost or 127.0.0.1 is not suggested
  port: 8004                                          # default webserver port, can change it

sdbctrl:
  port: 8003                                          # default sdbctrl port, can change it
  # log: /var/log/sdbctrl                             # default sdbctrl log path is {base_deploy_dir}/manager/sdbctrl/log

grafana:
  http_port: 3000                                     # default grafana port, can change it
  # datasource_name: DS_SELECTDB_ENTERPRISE           # default prometheus datasource name loaded in grafana
  # dashboard_tpl_name: SelectDB-Enterprise-Overview  # default dashboard template json file name(stored in grafana deployed dir)

prometheus:
  http_port: 9090                                     # default grafana port, can change it

alert_manager:
  web_port: 9093                                      # default grafana port, can change it

# db has only two types: 'h2' 'mysql'
# Example 1: h2
db:
  type: h2
  data_path: /data/doris/manager/manager-data                        # data.mv.db file's location

# Example 2: mysql
# db:
#   type: mysql
#   host: 127.0.0.1                                   # mysql host
#   port: 3306                                        # mysql port
#   user: root                                        # mysql user
#   password: doris                                   # mysql password
#   name: manager                                     # mysql database: manager (should be created first)

# mail config for alerting
#mail_config:
#  host: mailserver.com
#  username: name@company.com
#  password: Pass123456
#  port: 25
#  protocol: smtp

# doris and manger tar package path
path:
  doris: /data/doris/download                         # eg. doris/selectdb tar package path
  webui: /data/doris/download                         # eg. doris-manager tar package path
#备注：base_deploy_dir字段，doris manager服务整体的目标部署路径，默认为/opt/doris-manager，用户可修改为实际部署路径，部署的过程会将doris-manager解压包中的文件拷贝到base_deploy_dir目录
#webserver.version字段，应该修改为需要部署的doris-manager版本，比如23.7.1
#webserver.host字段，应该修改为 本机的IP ，不推荐填写127.0.0.1或者localhost, 如果用户使用的是云上主机，最好使用公网IP（如果想在本地浏览器访问云主机部署的管控服务平台时，使用内网IP等其他非公网的方案可能会导致grafana监控数据获取失败）
#path.doris字段，应该修改为具体存放doris安装包所在的绝对路径，默认为/opt/downloads/doris
#path.webui字段，应该修改为doris-manager安装包所在的绝对路径，默认为/opt/downloads/webui (如果用户一开始下载的doris-manager安装包不在此目录需要拷贝到该目录下)


#开始部署，部署持续大概5分钟左右
[root@doris-fe doris]# config-tool deploy -f init.conf
INFO[0002] [webserver] host = 172.31.16.7               
INFO[0002] [webserver] deploy_dir = /data/doris/manager/manager/webserver 
INFO[0002] [database] type = h2, data_path = /data/doris/manager/manager-data 
INFO[0002] [sdbctrl] deploy_dir = /data/doris/manager/manager/sdbctrl 
INFO[0002] [prometheus] deploy_dir = /data/doris/manager/manager/deps/prometheus 
INFO[0002] [alertmanager] deploy_dir = /data/doris/manager/manager/deps/alertmanager 
INFO[0002] [grafana] deploy_dir = /data/doris/manager/manager/deps/grafana 
INFO[0002] [path] doris package dir = /data/doris/download 
INFO[0002] [path] webui package dir = /data/doris/download 
INFO[0372] [grafana] ready to start ...                 
INFO[0372] [grafana] ready to start ...                 
INFO[0402] [grafana] grafana is up and running!         
INFO[0402] [grafana] start successfully, on port [3000] 
INFO[0402] [grafana] ready to add datasource...         
INFO[0402] [grafana] add datasource successfully        
INFO[0402] [grafana] ready to import dashboard...       
INFO[0402] [grafana] import dashboard done              
INFO[0402] [alertmanager] ready to configure...         
INFO[0402] [alertmanager] configure successfully        
INFO[0402] [alertmanager] ready to start...             
INFO[0405] [alertmanager] alertmanager is up and healthy! 
INFO[0405] [alertmanager] start successfully, on port [9093] 
INFO[0405] [prometheus] ready to configure...           
INFO[0405] [prometheus] configure successfully          
INFO[0405] [prometheus] ready to starting...            
INFO[0408] [prometheus] prometheus is up and healthy!   
INFO[0408] [prometheus] start successfully, on port [9090] 
INFO[0408] [sdbctrl] ready to configure...              
INFO[0408] [sdbctrl] configure successfully.            
INFO[0408] [sdbctrl] ready to start...                  
INFO[0411] [sdbctrl] start successfully, on port [8003] 
INFO[0411] [webserver] ready to configure...            
INFO[0412] [webserver] configure successfully           
INFO[0412] [webserver] ready to start...                
INFO[0421] [webserver] start successfully, on port [8004] 
INFO[0421] deploy doris manager successfully !!!        
INFO[0421] notice here! manager meta data is saved in [~/.config-tool/meta.yaml] 
INFO[0421] now you can visit [http://172.31.16.7:8004] to use doris-manager platform

3、访问manager