rancher1.6高可用集群搭建(笔记)


rancher1.6高可用集群搭建笔记

一、搭建环境

1.安装系统

  • 下载centos最新版, http://mirrors.sohu.com/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1708.iso
  • 最小化安装
  • 配置网络
    TYPE=Ethernet
    PROXY_METHOD=none
    BROWSER_ONLY=no
    BOOTPROTO=static
    DEFROUTE=yes
    IPV4_FAILURE_FATAL=no
    IPV6INIT=yes
    IPV6_AUTOCONF=yes
    IPV6_DEFROUTE=yes
    IPV6_FAILURE_FATAL=no
    IPV6_ADDR_GEN_MODE=stable-privacy
    NAME=enp0s3
    UUID=95a50ea2-3ad5-4601-9f80-93b7f1913eab
    DEVICE=enp0s3
    ONBOOT=yes
    IPADDR=192.168.0.106
    NETMASK=255.255.255.0
    GATEWAY=192.168.0.1
    
  • 安装常用工具,wget、vim、curl
  • 修改yum源,http://mirrors.163.com/.help/centos.html
    cd /etc/yum.repos.d/
    mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
    wget http://mirrors.163.com/.help/CentOS7-Base-163.repo
    yum clean all
    yum makecache
    

    2.安装docker

  • 安装docker,选择社区版https://docs.docker.com/install/linux/docker-ce/centos/
    #安装所需的包
    sudo yum install -y yum-utils \
    device-mapper-persistent-data \
    lvm2
    #添加源  
    sudo yum-config-manager \
      --add-repo \
      https://download.docker.com/linux/centos/docker-ce.repo
    # 安装docker-ce
    sudo yum install docker-ce
    #启动Docker
    sudo systemctl start docker
    #docker通过运行hello-world 映像验证安装是否正确
    sudo docker run hello-world
    

    如果被墙,不能连接download.docker.com,则可以下载该 .rpm文件手动安装

  • 修改镜像源
    vim /etc/docker/daemon.json
    #增加一行 {"registry-mirrors": ["https://fu1ctwn9.mirror.aliyuncs.com"]}
    sudo systemctl daemon-reload
    sudo systemctl restart docker
    #或者
    curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://你的的地址.m.daocloud.io
    

二、搭建高可用rancher

准备至少三台机器。

node0 192.168.0.106
node1 192.168.0.107
node2 192.168.0.108

1.搭建Galera Cluster高一致性MySQL集群架构

使用galera cluster保证高可用和高一致性,mariadb 10.1起默认安装了galera,

不要用mariadb 10.3,目前rancher sql语句有bug,已提给rancher官方。

docker run -d --name mariadb-cluster0 -p 3306:3306 -p 4567:4567 -p 4568:4568 -p 4444:4444 -v /home/docker/mariadb/conf:/etc/mysql/conf.d -v /home/docker/mariadb/cluster0/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=20053140 mariadb:10.2

mariadb默认配置文件

[mysqld]
server_id=100
pid-file    = /var/run/mysqld/mysqld.pid
socket        = /var/run/mysqld/mysqld.sock
port        = 3306
basedir        = /usr
datadir        = /var/lib/mysql
log-error=/var/log/mysqld.log
user=mysql

default-time-zone=UTC
#默认存储字符集编码格式
character-set-server=utf8
#禁止域名解析的
skip_name_resolve
#确保二进制日志格式设置为使用行级复制,而不是语句级复制
binlog_format=ROW
#确保mysqld没有绑定到127.0.0.1
bind-address=0.0.0.0 

#最大连接数
max_connections     = 500
connect_timeout     = 5
wait_timeout        = 600
max_allowed_packet  = 16M
thread_cache_size   = 128
sort_buffer_size    = 4M
bulk_insert_buffer_size =16M
tmp_table_size      = 32M
max_heap_table_size = 32M

#确保默认存储引擎是InnoDB,Galera不适用于MyISAM或其他非事务性存储引擎
default_storage_engine=innodb
#确保用于生成自动增量值的InnoDB锁定模式设置为交替锁定模式
innodb_autoinc_lock_mode=2
#确保将InnoDB日志缓冲区每秒写入一次文件,而不是每次提交,以提高性能。
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=2G

#log-bin=/app/galera/mysql-bin  #如果不接从库,注释掉  
#log_slave_updates=1         #如果不接从库,注释掉

[galera]
#节点应用完事务才返回查询请求  
wsrep_causal_reads=ON
#同步复制缓冲池
wsrep_provider_options="gcache.size=300M;gcache.page_size=300M"  
#为没有显式申明主键的表生成一个用于certificationtest的主键,默认为ON  
wsrep_certify_nonPK=ON   
#开启全同步复制模式
wsrep_on=ON
#galera library  
wsrep_provider=/usr/lib/galera/libgalera_smm.so 
wsrep_sst_auth=syncuser:syncuser
#wsrep_sst_method=xtrabackup-v2
wsrep_sst_method=rsync
#开启并行复制线程,考虑使用两倍的CPU内核数量,默认1
wsrep_slave_threads=1
wsrep_cluster_name=MariaDB-Galera-Cluster
#galera cluster URL 
#wsrep_cluster_address="gcomm://192.168.0.106:4567,192.168.0.107:4567,192.168.0.108:4567"
#节点名称
wsrep_node_name=mariadb-0 
wsrep_node_address=192.168.0.106

进去容器添加用户

[root@localhost cluster0]# docker exec -it 929 bash
root@92902e6ff803:/# mysql -uroot -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.3.5-MariaDB-10.3.5+maria~jessie mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'IDENTIFIED BY '20053140' WITH GRANT OPTION;
Query OK, 0 rows affected (0.001 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'syncuser'@'%'IDENTIFIED BY 'syncuser' WITH GRANT OPTION;
Query OK, 0 rows affected (0.001 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.001 sec)

其他节点启动容器后,重复添加用户操作

docker run -d --name mariadb-cluster0 -p 3306:3306 -p 4567:4567 -p 4568:4568 -p 4444:4444 -v /home/docker/mariadb/conf:/etc/mysql/conf.d -v /home/docker/mariadb/cluster0/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=20053140 mariadb:10.2

解开 wsrep_cluster_address 注释,修改 wsrep_node_namewsrep_node_address

删除所有mariadb容器,重新run.

--wsrep-new-cluster 只有在第一次集群,第一个节点,才需要加上,下次启动不需要此参数

cluster0

docker run -d --name mariadb-cluster0 -p 3306:3306 -p 4567:4567 -p 4568:4568 -p 4444:4444 -v /home/docker/mariadb/conf:/etc/mysql/conf.d -v /home/docker/mariadb/cluster0/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=20053140 mariadb:10.2 --wsrep-new-cluster

cluster1

docker run -d --name mariadb-cluster1 -p 3306:3306 -p 4567:4567 -p 4568:4568 -p 4444:4444 -v /home/docker/mariadb/conf:/etc/mysql/conf.d -v /home/docker/mariadb/cluster0/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=20053140 mariadb:10.2

cluster2

docker run -d --name mariadb-cluster2 -p 3306:3306 -p 4567:4567 -p 4568:4568 -p 4444:4444 -v /home/docker/mariadb/conf:/etc/mysql/conf.d -v /home/docker/mariadb/cluster0/data:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=20053140 mariadb:10.2

进入任意一台容器的mysql

MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+
1 row in set (0.000 sec)

配合外部负载均衡,保证外部使用高可用

添加haproxy用户用于haproxy监控检查

MariaDB [(none)]> SET sql_mode = 'STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION';
Query OK, 0 rows affected (0.000 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'haproxy'@'192.168.0.104'IDENTIFIED BY '' WITH GRANT OPTION;
Query OK, 0 rows affected (0.005 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.013 sec)

haproxy配置文件

global  
    chroot /usr/local  
    daemon  
    nbproc 1   
    pidfile /opt/haproxy/logs/haproxy.pid  
    ulimit-n 65536  
    #spread-checks 5m   
    #stats timeout 5m  
    #stats maxconn 100  

########默认配置############  
defaults  
    mode tcp              
    retries 3              #两次连接失败就认为是服务器不可用,也可以通过后面设置  
    option redispatch      #当serverId对应的服务器挂掉后,强制定向到其他健康的服务器  
    option abortonclose    #当服务器负载很高的时候,自动结束掉当前队列处理比较久的链接  
    maxconn 32000          #默认的最大连接数  
    timeout connect 5000ms #连接超时  
    timeout client 30000ms #客户端超时  
    timeout server 30000ms #服务器超时  
    timeout check 2000    #心跳检测超时  
    log 127.0.0.1 local0 err #[err warning info debug]  

listen stats     #监控
   mode http
   bind 0.0.0.0:8888
   stats enable
   stats uri /stats
   stats realm Global\ statistics
   stats auth admin:admin

########MariaDB配置#################  
listen mariadb  
    bind 0.0.0.0:23306  
    mode tcp 
    option mysql-check user haproxy #mysql健康检查  haproxy为mysql登录用户名
    balance roundrobin  
    server s1 192.168.0.106:3306 weight 1 maxconn 10000 check inter 10s  
    server s2 192.168.0.107:3306 weight 1 maxconn 10000 check inter 10s  
    server s3 192.168.0.108:3306 weight 1 maxconn 10000 check inter 10s

启动haproxy容器

docker run -d --name haproxy -p 23306:23306 -p 8888:8888 -v /data/syncthing/dongxu/haproxy:/usr/local/etc/haproxy:ro haproxy:1.7

添加测试数据,测试数据同步是否正常,恶意kill掉容器,或关掉一台机器,测试数据是否正常,恢复机器恢复容器查看数据是否恢复正常,负载均衡是否正常。

image

停掉一台node3,查看负载均衡健康检查是否正常

image

2.搭建rancher集群

创建rancher需使用的数据库

CREATE DATABASE IF NOT EXISTS cattle COLLATE = 'utf8_general_ci' CHARACTER SET = 'utf8';
GRANT ALL ON cattle.* TO 'cattle'@'%' IDENTIFIED BY 'cattle';
GRANT ALL ON cattle.* TO 'cattle'@'localhost' IDENTIFIED BY 'cattle';

启动rancher

node1

docker run -d --restart=unless-stopped -p 8080:8080 -p 9345:9345 rancher/server \
     --db-host 192.168.0.104 --db-port 23306 --db-user root --db-pass 20053140 --db-name cattle \
     --advertise-address 192.168.0.106

访问192.168.0.106:8080,增加访问控制。

node2

docker run -d --restart=unless-stopped -p 8080:8080 -p 9345:9345 rancher/server \
     --db-host 192.168.0.104 --db-port 23306 --db-user root --db-pass 20053140 --db-name cattle \
     --advertise-address 192.168.0.107

node3

docker run -d --restart=unless-stopped -p 8080:8080 -p 9345:9345 rancher/server \
     --db-host 192.168.0.104 --db-port 23306 --db-user root --db-pass 20053140 --db-name cattle \
     --advertise-address 192.168.0.108

给rancher配置负载均衡,haproxy.cfg增加配置

frontend http-in
    mode http
    bind *:28080
    #bind *:8080 ssl crt /etc/haproxy/certificate.pem
    default_backend rancher_servers

    # Add headers for SSL offloading
    #http-request set-header X-Forwarded-Proto https if { ssl_fc }
    #http-request set-header X-Forwarded-Ssl on if { ssl_fc }

    acl is_websocket hdr(Upgrade) -i WebSocket
    acl is_websocket hdr_beg(Host) -i ws
    use_backend rancher_servers if is_websocket

backend rancher_servers
    mode http
    option httpchk HEAD /login HTTP/1.0
    server websrv106 192.168.0.106:8080 weight 1 maxconn 1024 check
    server websrv107 192.168.0.107:8080 weight 1 maxconn 1024 check
    server websrv108 192.168.0.108:8080 weight 1 maxconn 1024 check

重启haproxy

docker run -d --restart=unless-stopped --name haproxy -p 23306:23306 -p 8888:8888 -p 28080:28080 -v /data/syncthing/dongxu/haproxy:/usr/local/etc/haproxy:ro haproxy:1.7

访问192.168.0.104:28080添加主机,添加容器,测试rancher是否正常,恶意kill掉rancher server容器或者关闭机器,查看整个集群是否正常。


常常是最后一把钥匙打开了门