Harbor私有镜像仓库（下）

Harbor私有镜像仓库（下）

链接：https://pan.baidu.com/s/1MAb0dllUwmoOk7TeVCZOVQ
提取码：ldt5
复制这段内容后打开百度网盘手机App，操作更方便哦

7. Harbor HA:环境与准备

• 全新安装两台harbor，但不要着急执行 ./install （如果用旧的要清理下数据）
• 要做Harbor高可用，我们需要准备一台NFS服务器共享两台Harbor的数据存储目录,在生产环境中，我们可以选择 Ceph或Glusterfs
• 我们需要一台Postgres数据库给Harbor的clair漏洞扫描组件使用
• 我们需要一台MySQL数据库给Harbor使用
• 我们需要一台redis数据库给session使用

主机名	IP	用途	VIP
Harbor-master	192.168.200.16	Harbor镜像仓库-主	192.168.200.20
Harbor-slave	192.168.200.18	Harbor镜像仓库-备
Docker-client	192.168.200.17	NFS服务器端（Harbor HA文件存储）
LDNS	192.168.200.19	DNS服务器

7.1 在Docker-client上搭建nfs服务端

[root@Docker-client ~]# mkdir -p /data/nfs
[root@Docker-client ~]# yum -y install nfs-utils
[root@Docker-client ~]# rpm -qa nfs-utils
nfs-utils-1.3.0-0.61.el7.x86_64

[root@Docker-client ~]# cat /etc/exports
/data/nfs 192.168.200.0/24(rw,no_root_squash)
[root@Docker-client ~]# systemctl start nfs

7.2 在harbor主和备上创建数据挂载目录，并安装nfs节点支持包

#harbor主备都进行如下操作
[root@Harbor-master ~]# yum -y install nfs-utils
[root@Harbor-master ~]# rpm -qa nfs-utils
nfs-utils-1.3.0-0.61.el7.x86_64
[root@Harbor-master ~]# mkdir -p /data/storage
[root@Harbor-master ~]# mount 192.168.200.17:/data/nfs /data/storage

[root@Harbor-master ~]# df -hT | grep /data/nfs
192.168.200.17:/data/nfs    nfs4       17G  2.8G   15G   17% /data/storage
[root@Harbor-slave ~]# df -hT | grep /data/nfs
192.168.200.17:/data/nfs    nfs4       17G  2.8G   15G   17% /data/storage

7.3 在docker-client（NFS服务器端）启动一个redis容器

#下载一个redis镜像（alpine系统目前docker领域很火，因为它容量很小，比centos小很多）
[root@Docker-client ~]# docker pull redis:alpine
[root@Docker-client ~]# docker images
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
mongo                             latest              9c02a5a12c52        41 hours ago        413MB
www.yunjisuan.com/library/mongo   latest              9c02a5a12c52        41 hours ago        413MB
redis                             alpine              d975eaec5f68        13 days ago         51.1MB

#启动redis镜像，映射端口
[root@Docker-client ~]# docker run -dit --name redis_test -p 6379:6379 redis:alpine
448de2a11cf1677c20e7280301ce869d878c2a0a6627019082e44cc337a6d71f
[root@Docker-client ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
448de2a11cf1        redis:alpine        "docker-entrypoint.s…"   5 seconds ago       Up 3 seconds        0.0.0.0:6379->6379/tcp   redis_test

7.4 在docker-client（NFS服务器端）启动一个postgreSQL数据库容器

#下载postgres
[root@Docker-client ~]# docker pull postgres
[root@Docker-client ~]# docker images
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
mongo                             latest              9c02a5a12c52        43 hours ago        413MB
www.yunjisuan.com/library/mongo   latest              9c02a5a12c52        43 hours ago        413MB
postgres                          latest              53912975086f        7 days ago          312MB
redis                             alpine              d975eaec5f68        13 days ago         51.1MB

[root@Docker-client ~]# docker run -dit --name postgres_test -p 5432:5432 -e POSTGRES_PASSWORD=123123 postgres
04c883f32fdc8fffb6c9f90539a0093ffb302cbb9d2ec4c4bcb73b90133d3952
[root@Docker-client ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS                    NAMES
04c883f32fdc        postgres            "docker-entrypoint.s…"   12 seconds ago      Up 11 seconds               0.0.0.0:5432->5432/tcp   postgres_test
448de2a11cf1        redis:alpine        "docker-entrypoint.s…"   2 hours ago         Exited (0) 52 minutes ago                            redis_test

7.5 在docker-client（NFS）服务器端启动一个MySQL数据库容器

#下载MySQL5.6版镜像
[root@Docker-client ~]# docker pull mysql:5.6
[root@Docker-client ~]# docker images
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
mongo                             latest              9c02a5a12c52        43 hours ago        413MB
www.yunjisuan.com/library/mongo   latest              9c02a5a12c52        43 hours ago        413MB
mysql                             5.6                 7b01f1418bd7        2 days ago          256MB
postgres                          latest              53912975086f        7 days ago          312MB
redis                             alpine              d975eaec5f68        13 days ago         51.1MB

#启动MySQL容器，并映射端口
[root@Docker-client ~]# docker run -dit --name mysql_test -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123123 mysql:5.6 --character-set-server=utf8
bfe4d57f424e27e48553a735aee8e2e1f0d65dc51691069db43bc92986ca4b70
[root@Docker-client ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS                    NAMES
bfe4d57f424e        mysql:5.6           "docker-entrypoint.s…"   11 seconds ago      Up 10 seconds               0.0.0.0:3306->3306/tcp   mysql_test
04c883f32fdc        postgres            "docker-entrypoint.s…"   5 minutes ago       Up 5 minutes                0.0.0.0:5432->5432/tcp   postgres_test
448de2a11cf1        redis:alpine        "docker-entrypoint.s…"   2 hours ago        Up 1 seconds       0.0.0.0:6379->6379/tcp   session

7.6 最后按照用途，我们分别给数据库改一下名称

[root@Docker-client ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                      PORTS                    NAMES
bfe4d57f424e        mysql:5.6           "docker-entrypoint.s…"   50 seconds ago      Up 49 seconds               0.0.0.0:3306->3306/tcp   mysql_test
04c883f32fdc        postgres            "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes                0.0.0.0:5432->5432/tcp   postgres_test
448de2a11cf1        redis:alpine        "docker-entrypoint.s…"   2 hours ago         Exited (0) 58 minutes ago                            redis_test

[root@Docker-client ~]# docker rename postgres_test clair_db
[root@Docker-client ~]# docker rename mysql_test harbor_db
[root@Docker-client ~]# docker rename redis_test session

[root@Docker-client ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                      PORTS                    NAMES
bfe4d57f424e        mysql:5.6           "docker-entrypoint.s…"   About a minute ago   Up About a minute           0.0.0.0:3306->3306/tcp   harbor_db
04c883f32fdc        postgres            "docker-entrypoint.s…"   6 minutes ago        Up 6 minutes                0.0.0.0:5432->5432/tcp   clair_db
448de2a11cf1        redis:alpine        "docker-entrypoint.s…"   2 hours ago          Up 10 seconds       0.0.0.0:6379->6379/tcp   session

8. Harbor HA：修改配置

8.1 向mysql_db容器里导入数据表（192.168.200.17）

#在解压后的harbor目录里的ha目录下的registry.sql表导入到我们之前在NFS服务端上创建的MySQL容器里
[root@Harbor-master ~]# cd /data/install/harbor
[root@Harbor-master harbor]# ls
common                    docker-compose.notary.yml  ha          harbor.cfg.bak        install.sh  NOTICE
docker-compose.clair.yml  docker-compose.yml         harbor.cfg  harbor.v1.5.0.tar.gz  LICENSE     prepare

[root@Harbor-master harbor]# tree ha
ha
├── docker-compose.clair.tpl
├── docker-compose.clair.yml
├── docker-compose.tpl
├── docker-compose.yml     #需要修改的配置文件
├── registry.sql           #需要导入的mysql表格
└── sample
    ├── active_active
    │   ├── check.sh
    │   └── keepalived_active_active.conf
    └── active_standby
        ├── check_harbor.sh
        └── keepalived_active_standby.conf

3 directories, 9 files

#Harbor-master本地安装mysql客户端连接程序
[root@Harbor-master harbor]# yum -y install mysql
[root@Harbor-master harbor]# which mysql
/usr/bin/mysql

#远程连接到192.168.200.17（NFS服务器端）的3306端口，导入表registry.sql
[root@Harbor-master harbor]# mysql -uroot -p123123 -h192.168.200.17 -P3306
Welcome to the MariaDB monitor.  Commands end with ; or g.
Your MySQL connection id is 1
Server version: 5.6.45 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
+--------------------+
3 rows in set (0.01 sec)

MySQL [(none)]> source ha/registry.sql    #导入表格
#以下省略若干。。。
MySQL [registry]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| registry           |
+--------------------+
4 rows in set (0.00 sec)

MySQL [registry]> 

特别提示：如果导入表格出现如下错误
specified key was too long max key length is 767bytes
这是因为导入的表格建立的索引超过mysql默认上线767bytes >=254,因此我们需要修改导入的表

[root@harbor harbor]# cat -n ha/registry.sql | sed -n '220p;291p'
220 repository varchar(256) NOT NULL,    #将256修改成254
291 resource_name varchar(256),          #将256修改成254

#改完之后再进行表格导入就不会报错了

8.2 修改配置文件

#修改/data/install/ha/docker-compose.yml配置文件
[root@Harbor-master harbor]# cat -n ha/docker-compose.yml | sed -n '19p'
    19	      - /data/registry:/storage:z

#修改成我们的nfs共享目录
[root@Harbor-master harbor]# cat -n ha/docker-compose.yml | sed -n '19p'
    19	      - /data/storage:/storage:z

#修改/data/install/harbor.cfg文件
[root@Harbor-master harbor]# cat -n harbor.cfg.bak | sed -n '7p;11p;23p;24p;68p;130p;133p;136p;139p;145p;150p;154p;157p;160p;163p'
    7 hostname = reg.mydomain.com             #harbor的访问域名（不能用IP地址）
   11 ui_url_protocol = http                  #web访问连接方式
   23 ssl_cert = /data/cert/server.crt        #ca证书路径
   24 ssl_cert_key = /data/cert/server.key    #ca密钥路径
   68 harbor_admin_password = Harbor12345     #harbor默认初始密码
  130 db_host = mysql                         #harbor连接的mysql_db的IP
  133 db_password = root123                   #MySQL连接密码
  136 db_port = 3306                          #Mysql连接端口
  139 db_user = root                          #MySQL连接用户
  145 redis_url = redis:6379                  #session连接的redis数据库路径
  150 clair_db_host = postgres                #clair漏洞检测组件连接的数据库IP
  154 clair_db_password = password            #postgres数据库连接密码
  157 clair_db_port = 5432                    #postgres数据库连接端口
  160 clair_db_username = postgres            #默认的连接用户名
  163 clair_db = postgres                     #默认的库名

#修改成如下所示
[root@Harbor-master harbor]# cat -n harbor.cfg | sed -n '7p;11p;23p;24p;68p;130p;133p;136p;139p;145p;150p;154p;157p;160p;163p'
     7	hostname = www.yunjisuan.com
    11	ui_url_protocol = https
    23	ssl_cert = /etc/ssl/harbor/www.yunjisuan.com.crt
    24	ssl_cert_key = /etc/ssl/harbor/www.yunjisuan.com.key
    68	harbor_admin_password = Harbor12345
   130	db_host = 192.168.200.17
   133	db_password = 123123
   136	db_port = 3306
   139	db_user = root
   145	redis_url = 192.168.200.17:6379
   150	clair_db_host = 192.168.200.17
   154	clair_db_password = 123123
   157	clair_db_port = 5432
   160	clair_db_username = postgres
   163	clair_db = postgres

9. Harbor HA：启动Harbor

[root@Harbor-master harbor]# pwd
/data/install/harbor
[root@Harbor-master harbor]# ./install.sh --with-clair --ha
#因此使用了自定义存储路径，安装中途需要yes确认

浏览器进行访问测试：http://192.168.200.16

#在docker-client（NFS共享存储服务器端）进行镜像上传测试：
[root@Docker-client ~]# docker login www.yunjisuan.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[root@Docker-client ~]# docker tag redis:alpine www.yunjisuan.com/library/redis
[root@Docker-client ~]# docker push www.yunjisuan.com/library/redis
The push refers to repository [www.yunjisuan.com/library/redis]
8fdd7da74c31: Pushed 
2166e8ad934d: Pushed 
c921f5478449: Pushed 
9b8719029b64: Pushed 
bd23b36e1125: Pushed 
1bfeebd65323: Pushed 
latest: digest: sha256:6eed01a8bd56b7b400ddd6c232567b05aa9295e58c92f48b1377642b80a3dfd4 size: 1571

#查看NFS服务器共享目录
[root@Docker-client ~]# tree /data/nfs/
/data/nfs/
└── docker
    └── registry
        └── v2
            ├── blobs
            │   └── sha256
            │       ├── 05
            │       │   └── 0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa
            │       │       └── data
            │       ├── 33
            │       │   └── 3348f84e43d019f5288bf0f3143725683ec3e95d771af1dc60b2ec08ab33e919
            │       │       └── data
            │       ├── 6e
            │       │   └── 6eed01a8bd56b7b400ddd6c232567b05aa9295e58c92f48b1377642b80a3dfd4
            │       │       └── data
            │       ├── 7a
            │       │   └── 7a3fdc0143e12cb63356b93af0fae6daacaf9fda239e776a8ab5c121ff184dc7
            │       │       └── data
            │       ├── ab
            │       │   └── ab70e0f222721845b57e1a38fa16eee604153e6430df3e209ffc47b2874f3d5d
            │       │       └── data
            │       ├── d4
            │       │   └── d48f315c369d800f68a3c7b2ed1b713df08231f439f3dcdcb7110fa87609fe4e
            │       │       └── data
            │       ├── d9
            │       │   └── d975eaec5f68eddceab6bbc3f8c96fa3418978acd431c2a8cab1d7860372b1d1
            │       │       └── data
            │       └── ec
            │           └── ecf40235d2c75d0220ad5f7c654d05cff5b527ca9f231f4a0203f0c19e5fc519
            │               └── data
            └── repositories
                └── library
                    └── redis
                        ├── _layers
                        │   └── sha256
                        │       ├── 0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa
                        │       │   └── link
                        │       ├── 3348f84e43d019f5288bf0f3143725683ec3e95d771af1dc60b2ec08ab33e919
                        │       │   └── link
                        │       ├── 7a3fdc0143e12cb63356b93af0fae6daacaf9fda239e776a8ab5c121ff184dc7
                        │       │   └── link
                        │       ├── ab70e0f222721845b57e1a38fa16eee604153e6430df3e209ffc47b2874f3d5d
                        │       │   └── link
                        │       ├── d48f315c369d800f68a3c7b2ed1b713df08231f439f3dcdcb7110fa87609fe4e
                        │       │   └── link
                        │       ├── d975eaec5f68eddceab6bbc3f8c96fa3418978acd431c2a8cab1d7860372b1d1
                        │       │   └── link
                        │       └── ecf40235d2c75d0220ad5f7c654d05cff5b527ca9f231f4a0203f0c19e5fc519
                        │           └── link
                        ├── _manifests
                        │   ├── revisions
                        │   │   └── sha256
                        │   │       └── 6eed01a8bd56b7b400ddd6c232567b05aa9295e58c92f48b1377642b80a3dfd4
                        │   │           └── link
                        │   └── tags
                        │       └── latest
                        │           ├── current
                        │           │   └── link
                        │           └── index
                        │               └── sha256
                        │                   └── 6eed01a8bd56b7b400ddd6c232567b05aa9295e58c92f48b1377642b80a3dfd4
                        │                       └── link
                        └── _uploads

44 directories, 18 files

10. Harbor HA：Keepalived安装配置与测试

10.1 首先我们再安装一个harbor-slave备库（192.168.200.18）

重复之前的操作，过程略（harbor HA，域名需要相同）

10.2 进行上传测试

[root@Docker-client ~]# cd /data/nfs/
[root@Docker-client nfs]# docker tag mysql:5.6 www.yunjisuan.com/library/mysql
[root@Docker-client nfs]# docker push www.yunjisuan.com/library/mysql
The push refers to repository [www.yunjisuan.com/library/mysql]
a1e3e0513114: Pushed 
6c621d0720e2: Pushed 
d86d34816513: Pushed 
b314ec235321: Pushed 
812e5f94ac49: Pushed 
d355dacb791d: Pushed 
2f1b41b24201: Pushed 
007a7f930352: Pushed 
c6926fcee191: Pushed 
b78ec9586b34: Pushed 
d56055da3352: Pushed 
latest: digest: sha256:ce58204b5f01bac11838b2ce2f379492841a11206a71a379bb47a68f63d057bf size: 2621

浏览器访问测试:
https://192.168.200.16
https://192.168.200.18

10.3 Harbor-master和Harbor-slave安装keepalived

#在Harbor-master进行如下操作
[root@Harbor-master harbor]# yum -y install keepalived
[root@Harbor-master harbor]# which keepalived
/usr/sbin/keepalived

[root@Harbor-master harbor]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    router_id harbor01
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32
    virtual_router_id 55
    priority 150     
    advert_int 1
    authentication {
       auth_type PASS
       auth_pass 1111
   }
   virtual_ipaddress {
   192.168.200.20 dev ens32
   }  
}

[root@Harbor-master harbor]# systemctl start keepalived
[root@Harbor-master harbor]# systemctl enable keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.

[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
    inet 192.168.200.20/32 scope global ens32

#在Harbor-slave进行如下操作
[root@Harbor-slave harbor]# yum -y install keepalived
[root@Harbor-slave harbor]# which keepalived
/usr/sbin/keepalived

[root@Harbor-slave harbor]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    router_id harbor01
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32
    virtual_router_id 55
    priority 100
    advert_int 1
    authentication {
       auth_type PASS
       auth_pass 1111
   }
   virtual_ipaddress {
   192.168.200.20 dev ens32
   }  
}


[root@Harbor-slave harbor]# systemctl start keepalived
[root@Harbor-slave harbor]# systemctl enable keepalived
Created symlink from /etc/systemd/system/multi-user.target.wants/keepalived.service to /usr/lib/systemd/system/keepalived.service.

[root@Harbor-slave harbor]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since 五 2019-07-26 11:56:38 CST; 1min 31s ago
 Main PID: 69765 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─69765 /usr/sbin/keepalived -D
           ├─69766 /usr/sbin/keepalived -D
           └─69767 /usr/sbin/keepalived -D

7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: Registering gratuitous ARP shared channel
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: Opening file '/etc/keepalived/keepalived.conf'.
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP_Instance(VI_1) removing protocol VIPs.
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: Using LinkWatch kernel netlink reflector...
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP sockpool: [ifindex(2), proto(112), unicast(0...1)]
7月 26 11:56:38 Harbor-slave Keepalived_healthcheckers[69766]: Opening file '/etc/keepalived/keepalived...'.
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP_Instance(VI_1) Transition to MASTER STATE
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP_Instance(VI_1) Received advert with higher p...100
7月 26 11:56:38 Harbor-slave Keepalived_vrrp[69767]: VRRP_Instance(VI_1) Entering BACKUP STATE
7月 26 11:56:38 Harbor-slave systemd[1]: Started LVS and VRRP High Availability Monitor.
Hint: Some lines were ellipsized, use -l to show in full.

10.4 进行VIP切换测试

在Harbor-master上操作
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
    inet 192.168.200.20/32 scope global ens32
[root@Harbor-master harbor]# systemctl stop keepalived
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20

在Harbor-slave上验证
[root@Harbor-slave harbor]# ip addr | grep 192.168.200.20
    inet 192.168.200.20/32 scope global ens32

在Harbor-master上操作
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
[root@Harbor-master harbor]# systemctl start keepalived
[root@Harbor-master harbor]# ip addr | grep 192.168.200.20
    inet 192.168.200.20/32 scope global ens32

在Harbor-slave上验证
[root@Harbor-slave harbor]# ip addr | grep 192.168.200.20