codis部署安装及性能测试

git clone https://github.com/CodisLabs/codis.git

1:下载go环境 官网：https://golang.org/doc/install
wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz

tar -C /usr/local -xzf go1.6.2.linux-amd64.tar.gz

cd /usr/local/go

cat >> /etc/profile <<EOF
export GOROOT=/usr/local/go
export GOPATH=/workspace/golang
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
export CODIS_HOME=$GOPATH/src/github.com/CodisLabs/codis
export GO15VENDOREXPERIMENT=0
EOF



source /etc/profile



2:下载安装codis 官网：https://github.com/CodisLabs/codis/blob/master/doc/tutorial_zh.md  根据官网推荐采用这种方式，而不是git clone ...codis
go get -u -d github.com/CodisLabs/codis
go get github.com/tools/godep

cd $GOPATH/src/github.com/CodisLabs/codis
make
make gotest
#mkdir -p /usr/local/codis/{logs,conf,scripts}
#mkdir -p /data/codis_server/{logs,conf,data}

cd $GOPATH/src/github.com/CodisLabs/codis/bin


3:安装zookeeper
wget http://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz  -P /opt
tar -C /opt -xvf /opt/zookeeper-3.4.6.tar.gz  
cd /opt/zookeeper-3.4.6/conf
cp zoo_sample.cfg zoo.cfg
/opt/zookeeper-3.4.6/bin/zkServer.sh start

read -p "请输入zookeeper的地址: " zookeeper-ip

sed -i  's/192.168.0.123/$zookeeper-ip/g' config.ini


###assets //dashboard的静态文件目录
###codis-config codis命令式配置管理组件
###codis-proxy codis 代理也就是核心组件
###codis-server codis基于redis2.8.21定制版redis


mkdir /workspace/golang/src/github.com/CodisLabs/codis/wk
cp /workspace/golang/src/github.com/CodisLabs/codis/config.ini  /workspace/golang/src/github.com/CodisLabs/codis/wk/

4:启动 dashboard（codis-config上操作）
cd $CODIS_HOME && ./bin/codis-config -c $CODIS_HOME/wk/config.ini dashboard &

可以从浏览器打开http://192.168.16.239:18087/admin/

5:初始化 slot,该命令会在zookeeper上创建slot相关信息
cd $CODIS_HOME &&  ./bin/codis-config -c $CODIS_HOME/wk/config.ini slot init &

6:启动 Codis Redis , 和官方的Redis Server参数一样（codis-server上操作）
cp extern/redis-2.8.21/redis.conf wk/
cd $CODIS_HOME && ./codis-server ../extern/redis-2.8.21/redis.conf                    (端口6379)作为16.241的6380的master
                  ./codis-server ../extern/redis-2.8.21/redis6380.conf                   （端口6380)作为16.241的6379的slave

7:添加 Redis Server Group , 每一个 Server Group 作为一个 Redis 服务器组存在, 只允许有一个 master,
     可以有多个 slave, group id 仅支持大于等于1的整数（codis-config上操作）
    ./codis-config -c ../wk/config.ini server add 1 192.168.16.239:6379 master
    ./codis-config -c ../wk/config.ini server add 1 192.168.16.241:6380 slave
    
    
    ./codis-config -c ../wk/config.ini server add 2 192.168.16.241:6379 master
    ./codis-config -c ../wk/config.ini server add 2 192.168.16.239:6380 slave
    

8: 启动241上面的redis服务，作为slave
/codis-server ../extern/redis-2.8.21/redis.conf                作为16.239的6380 的master
/codis-server ../extern/redis-2.8.21/redis6380.conf              作为16.239的6379 的slave

****************注意，不需要从redis.conf 去配置slave，必须用codis-server 去启动redis***************







[root@shell bin]# 2016/04/28 14:16:48 dashboard.go:160: [INFO] dashboard listening on addr: :18087
2016/04/28 14:16:49 dashboard.go:234: [PANIC] create zk node failed
[error]: dashboard already exists: {"addr": "192.168.16.239:18087", "pid": 2280}
[stack]:
    3   /workspace/golang/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard.go:234
            main.runDashboard
    2   /workspace/golang/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard.go:54
            main.cmdDashboard
    1   /workspace/golang/src/github.com/CodisLabs/codis/cmd/cconfig/main.go:85
            main.runCommand
    0   /workspace/golang/src/github.com/CodisLabs/codis/cmd/cconfig/main.go:152
            main.main
        ... ...
        
解决：
./zkCli.sh
[zk: localhost:2181(CONNECTED) 11] rmr  /zk/codis

原因：
2016/04/28 14:52:44 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_wk/dashboard, {"addr": "192.168.16.239:18087", "pid": 8578}
2016/04/28 14:52:44 dashboard.go:144: [WARN] ********** Attention **********
2016/04/28 14:52:44 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me,
2016/04/28 14:52:44 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually


9:设置 server group 服务的 slot 范围

Codis 采用 Pre-sharding 的技术来实现数据的分片, 默认分成 1024 个 slots (0-1023), 对于每个key来说, 通过以下公式确定所属的
Slot Id : SlotId = crc32(key) % 1024 每一个 slot 都会有一个且必须有一个特定的 server group id 来表示这个 slot 的数据由哪个 server group 来提供

[root@shell bin]# ./codis-config  -c ../wk/config.ini slot -h
usage:
        codis-config slot init [-f]
        codis-config slot info <slot_id>
        codis-config slot set <slot_id> <group_id> <status>
        codis-config slot range-set <slot_from> <slot_to> <group_id> <status>
        codis-config slot migrate <slot_from> <slot_to> <group_id> [--delay=<delay_time_in_ms>]
        codis-config slot rebalance [--delay=<delay_time_in_ms>]
        
[root@shell bin]# ./codis-config  -c ../wk/config.ini slot  range-set 0 511 1 online
{
  "msg": "OK",
  "ret": 0
}

到浏览器上看看效果

或者
[root@shell bin]# ./codis-config  -c ../wk/config.ini slot  info 1
{
  "group_id": 1,
  "id": 1,
  "product_name": "wk",
  "state": {
    "last_op_ts": "0",
    "migrate_status": {
      "from": -1,
      "to": -1
    },
    "status": "online"
  }
 
10：启动 codis-proxy

[root@shell bin]# ./codis-proxy -c ../wk/config.ini -L ./log/proxy.log  --cpu=4 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &

  _____  ____    ____/ /  (_)  _____
 / ___/ / __   / __  /  / /  / ___/
/ /__  / /_/ / / /_/ /  / /  (__  )
\___/  \____/  \__,_/  /_/  /____/


在241上启动第二个代理服务器
 ./codis-proxy -c ../wk/config.ini -L ./log/proxy.log  --cpu=4 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &
 
 在浏览器上看到proxy_2是offile状态，如果用命令去启动proxy_2 online状态，则提示下面错误，需要去启动dashboard,我们直接在界面上启动为online状态

[root@zabbix bin]# ./codis-config -c ../wk/config.ini proxy online proxy_2
2016/05/03 13:43:12 utils.go:49: [ERROR] can't connect to dashboard, please check 'dashboard_addr' is corrent in config file
[stack]:


11：用单个proxy进行测试，16.239proxy,可以用redis客户端连接proxy 的端口
[root@shell src]# ./redis-cli -p 19000
127.0.0.1:19000> info
...
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.16.241,port=6380,state=online,offset=722094,lag=1
master_repl_offset:722094
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:23019
repl_backlog_histlen:699076
...

插入数据
[root@shell opt]# vim redis-setkey.sh

#!/bin/bash

n=100000

for (( i=n ;i>=1 ; i-- ))
do
        echo $i

        redis-cli -p 19000 set zhao$i aa

done

根据下面脚本查看key的分布情况
#!/bin/bash

n=100000

for ((i=n;i>=0;i--))
do
        echo $i >> /opt/a.txt
        redis-cli -h 192.168.16.241 -p 6379 get zhao$i >>/opt/a.txt
done

sed 'N;s# # #g' a.txt  | awk '{if($2=="aa") print $1}' | wc -l
结果为：50029


       result=`redis-cli -h 192.168.16.241 -p 6379 get zhao$i`
        if [ ! -z $result ];then
                echo $i >>/opt/b.txt
        fi
[root@shell opt]# cat b.txt | wc -l
50029

结果都一样，发现192.168.16.241 -p 6379  存了500029条数据，再查看下16.239上的6379
[root@shell opt]# cat b.txt | wc -l
49971

正好结果相加是对的


查看主从复制及数据是否一致：
    result=`redis-cli -h 192.168.16.239 -p 6380 get zhao$i`
        if [ ! -z $result ];then
                echo $i >>/opt/b.txt
        fi
[root@shell opt]# cat b.txt  | wc -l
50029
结果跟16.241 6379上的数据保持一致。



12：性能测试：
1）10000请求，200并发，上是codis，下是单机
[root@shell src]# ./redis-benchmark -p 19000 -n 10000 -c 200 -q
PING_INLINE: 52083.33 requests per second
PING_BULK: 52083.33 requests per second
SET: 38314.18 requests per second
GET: 41666.67 requests per second
INCR: 37174.72 requests per second
LPUSH: 36496.35 requests per second
LPOP: 38610.04 requests per second
SADD: 37313.43 requests per second
SPOP: 36101.08 requests per second
LPUSH (needed to benchmark LRANGE): 34843.21 requests per second
LRANGE_100 (first 100 elements): 14430.01 requests per second
LRANGE_300 (first 300 elements): 5241.09 requests per second
LRANGE_500 (first 450 elements): 3713.33 requests per second
LRANGE_600 (first 600 elements): 2627.43 requests per second
MSET (10 keys): 16556.29 requests per second

[root@shell src]# ./redis-benchmark -p  6379 -n 10000 -c 200 -q      
PING_INLINE: 67567.57 requests per second
PING_BULK: 73529.41 requests per second
SET: 73529.41 requests per second
GET: 73529.41 requests per second
INCR: 59523.81 requests per second
LPUSH: 68493.15 requests per second
LPOP: 62893.08 requests per second
SADD: 76335.88 requests per second
SPOP: 63291.14 requests per second
LPUSH (needed to benchmark LRANGE): 68965.52 requests per second
LRANGE_100 (first 100 elements): 36496.35 requests per second
LRANGE_300 (first 300 elements): 15748.03 requests per second
LRANGE_500 (first 450 elements): 11441.65 requests per second
LRANGE_600 (first 600 elements): 8928.57 requests per second

MSET (10 keys): 40160.64 requests per second
2）10000请求，500并发，上是codis，下是单机
[root@shell src]# ./redis-benchmark -p 19000 -n 10000 -c 500 -q
PING_INLINE: 55865.92 requests per second
PING_BULK: 55248.62 requests per second
SET: 35087.72 requests per second
GET: 49751.24 requests per second
INCR: 38167.94 requests per second
LPUSH: 34246.57 requests per second
LPOP: 36101.08 requests per second
SADD: 33112.59 requests per second
SPOP: 42553.19 requests per second
LPUSH (needed to benchmark LRANGE): 38461.54 requests per second
LRANGE_100 (first 100 elements): 15151.51 requests per second
LRANGE_300 (first 300 elements): 5238.34 requests per second
LRANGE_500 (first 450 elements): 3716.09 requests per second
LRANGE_600 (first 600 elements): 2680.25 requests per second
MSET (10 keys): 15948.96 requests per second

[root@shell src]# ./redis-benchmark -p  6379 -n 10000 -c 500 -q  
PING_INLINE: 70422.53 requests per second
PING_BULK: 69444.45 requests per second
SET: 58823.53 requests per second
GET: 70921.98 requests per second
INCR: 66666.66 requests per second
LPUSH: 65789.48 requests per second
LPOP: 63694.27 requests per second
SADD: 68493.15 requests per second
SPOP: 67114.09 requests per second
LPUSH (needed to benchmark LRANGE): 66225.17 requests per second
LRANGE_100 (first 100 elements): 35714.29 requests per second
LRANGE_300 (first 300 elements): 15847.86 requests per second
LRANGE_500 (first 450 elements): 11918.95 requests per second
LRANGE_600 (first 600 elements): 9107.47 requests per second
MSET (10 keys): 38314.18 requests per second

3）10000请求，1000并发，上是codis，下是单机
[root@shell src]# ./redis-benchmark -p 19000 -n 10000 -c 1000 -q
PING_INLINE: 47846.89 requests per second
PING_BULK: 41493.78 requests per second
SET: 30769.23 requests per second
GET: 40322.58 requests per second
INCR: 38910.51 requests per second
LPUSH: 39370.08 requests per second
LPOP: 37453.18 requests per second
SADD: 40000.00 requests per second
SPOP: 42553.19 requests per second
LPUSH (needed to benchmark LRANGE): 36764.70 requests per second
LRANGE_100 (first 100 elements): 15384.62 requests per second
LRANGE_300 (first 300 elements): 5353.32 requests per second
LRANGE_500 (first 450 elements): 3776.44 requests per second
LRANGE_600 (first 600 elements): 2677.38 requests per second
MSET (10 keys): 14641.29 requests per second

[root@shell src]# ./redis-benchmark -p  6379 -n 10000 -c 1000 -q  
PING_INLINE: 65789.48 requests per second
PING_BULK: 65789.48 requests per second
SET: 63694.27 requests per second
GET: 67114.09 requests per second
INCR: 66666.66 requests per second
LPUSH: 64516.13 requests per second
LPOP: 66666.66 requests per second
SADD: 67114.09 requests per second
SPOP: 65789.48 requests per second
LPUSH (needed to benchmark LRANGE): 63694.27 requests per second
LRANGE_100 (first 100 elements): 34843.21 requests per second
LRANGE_300 (first 300 elements): 15243.90 requests per second
LRANGE_500 (first 450 elements): 11682.24 requests per second
LRANGE_600 (first 600 elements): 9082.65 requests per second
MSET (10 keys): 37174.72 requests per second

4）10000 请求 2000并发
[root@shell src]# ./redis-benchmark -p 19000 -n 10000 -c 2000 -q
PING_INLINE: 37313.43 requests per second
PING_BULK: 42735.04 requests per second
SET: 29940.12 requests per second
GET: 31250.00 requests per second
INCR: 29411.76 requests per second
LPUSH: 29585.80 requests per second
LPOP: 23752.97 requests per second
SADD: 21978.02 requests per second
SPOP: 34129.69 requests per second
LPUSH (needed to benchmark LRANGE): 6600.66 requests per second
LRANGE_100 (first 100 elements): 10582.01 requests per second
LRANGE_300 (first 300 elements): 2314.81 requests per second
LRANGE_500 (first 450 elements): 2180.07 requests per second
LRANGE_600 (first 600 elements): 2630.19 requests per second
MSET (10 keys): 8481.76 requests per second

[root@shell src]# ./redis-benchmark -p  6379 -n 10000 -c 2000 -q  
PING_INLINE: 49504.95 requests per second
PING_BULK: 64102.56 requests per second
SET: 58823.53 requests per second
GET: 64102.56 requests per second
INCR: 51020.41 requests per second
LPUSH: 43859.65 requests per second
LPOP: 15948.96 requests per second
SADD: 55248.62 requests per second
SPOP: 62500.00 requests per second
LPUSH (needed to benchmark LRANGE): 61349.69 requests per second
LRANGE_100 (first 100 elements): 30211.48 requests per second
LRANGE_300 (first 300 elements): 16000.00 requests per second
LRANGE_500 (first 450 elements): 11123.47 requests per second
LRANGE_600 (first 600 elements): 8271.30 requests per second
MSET (10 keys): 31746.03 requests per second