zoukankan      html  css  js  c++  java

  • 哈，又一款超级简单的队列（MQ）实现方案来了~

    开源的消息队列已经很多了，但大部分很重，实际环境下，很多可能只是使用到了一点功能而已，杀鸡使用牛刀，着实有些浪费了。很多时候，我们只想要一片绿叶，但它们给了我们整个的春天，很难消化。本着DIR精神， 也琢磨了一个超级简单的队列实现。
    说是超级简单，嗯，绝对是超级简单，队列的存储采用Redis进行持久化存储，采用Netty提供HTTP方式的队列的出/入。Redis的客户端采用的Jedis。然后呢，然后就没了啊。

    一。Redis

    Redis内置订阅发布模型(Publish/Subscribe),其缺陷是，不存储，一旦订阅者断线，将无法接收到消息，再次连接上之后，在断线期间发布者发布的消息都是无法获取到的。只能采用list数组实现，采用rpush/lpop组合命令来实现先进先出的队列模型，当然redis也提供了阻塞版本的blpush/brpush/blpop/brpop等，就看我们实际环境下如何使用了。
    JAVA客户端使用Jedis，提供接口也很丰富。但要注意的是，需要使用连接池，否则在大数量的情况下，有可能jedis的连接不够用。 
    private static JedisPool pool;
 static {
  ResourceBundle bundle = ResourceBundle.getBundle("redis");
  if (bundle == null) {
   throw new IllegalArgumentException(
     "cannot find the SignVerProp.properties");
  }
  JedisPoolConfig config = new JedisPoolConfig();
  config.setMaxActive(Integer.valueOf(bundle
    .getString("redis.pool.maxActive")));
  config.setMaxIdle(Integer.valueOf(bundle
    .getString("redis.pool.maxIdle")));
  config.setMaxWait(Integer.valueOf(bundle
    .getString("redis.pool.maxWait")));
  pool = new JedisPool(config, bundle.getString("redis.server"),
    Integer.valueOf(bundle.getString("redis.port")));
 }

    二。Netty

    很成熟的NIO框架，用它来提供HTTP方式的队列的出入。嗯，目前只提供HTTP方式的入队列，出队列等。
    HTTP形式为：http://服务器名称:端口/队列操作原语/队列名称?msg=消息内容 
    队列操作原语，只有get和put；get为出队列，put为入队列。
    返回为json： {s:0, m:'错误消息/消息内容'} 
    s为success的缩写，值为0，意味着false，1对应成功。 m为message的缩写，错误消息/消息内容，添加消息时，会返回插入消息对应的数目 默认采用UTF-8。
    入队列：http://localhost:8080/put/demo?msg=消息内容 
    出队列：http://localhost:8080/get/demo HTTP方式，特别适合局域网之间，消息数据的推送。
     

    三。入口解读

    采用了QueueDaemon封装了netty代码： 调用很简单，程序启动的入口： 而HTTP方式队列请求处理器为HttpRequestHandler：

    四。队列处理器

    这个很简单，直接采用Jedis客户端即可，这里一直占用一个连接，不释放。

    五。ab压力测试

    本机配置：Pentium(R) Dual-Core CPU E5200 2.50GHz，2.00 GB内存，Windows XP系统，redis-2.4.5-win32 32版本（非linux版本），都在一台机器上运行。 插入 512 bytes 文本消息队列: 1781.24 requests/sec 
    D:Apache2.2in>ab -c 10 -n 100000 "http://localhost:8080/put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Document Length:        12 bytes
Concurrency Level:      10
Time taken for tests:   56.141 seconds
Complete requests:      100000
Failed requests:        99991
   (Connect: 0, Receive: 0, Length: 99991, Exceptions: 0)
Write errors:           0
Total transferred:      8188895 bytes
HTML transferred:       1588895 bytes
Requests per second:    1781.24 [#/sec] (mean)
Time per request:       5.614 [ms] (mean)
Time per request:       0.561 [ms] (mean, across all concurrent requests)
Transfer rate:          142.45 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.7      0      16
Processing:     0    5  10.7      0     781
Waiting:        0    5  10.4      0     766
Total:          0    6  10.8      0     781
Percentage of the requests served within a certain time (ms)
  50%      0
  66%     16
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%    781 (longest request)
    插入 512 bytes 文本消息队列(添加Keep-Alive支持): 1875.18 requests/sec 
    D:Apache2.2in>ab -k -c 10 -n 100000 "http://localhost:8080/put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /put/demo?msg=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Document Length:        17 bytes
Concurrency Level:      10
Time taken for tests:   53.328 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      8300000 bytes
HTML transferred:       1700000 bytes
Requests per second:    1875.18 [#/sec] (mean)
Time per request:       5.333 [ms] (mean)
Time per request:       0.533 [ms] (mean, across all concurrent requests)
Transfer rate:          151.99 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.5      0      16
Processing:     0    5   7.8      0     203
Waiting:        0    5   7.8      0     203
Total:          0    5   7.9      0     203
Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%    203 (longest request)
    获取 512 bytes 消息(With Keep-Alive): 1875.73 requests/sec 
    D:Apache2.2in>ab -k -c 10 -n 100000 "http://localhost:8080/get/demo"
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:
Server Hostname:        localhost
Server Port:            8080
Document Path:          /get/demo
Document Length:        523 bytes
Concurrency Level:      10
Time taken for tests:   53.313 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    0
Total transferred:      58900000 bytes
HTML transferred:       52300000 bytes
Requests per second:    1875.73 [#/sec] (mean)
Time per request:       5.331 [ms] (mean)
Time per request:       0.533 [ms] (mean, across all concurrent requests)
Transfer rate:          1078.91 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.9      0      16
Processing:     0    5   7.5      0      94
Waiting:        0    4   6.9      0      94
Total:          0    5   7.6      0      94
Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%     94 (longest request)

    其它问题

    1. 暂时对安全/授权没有支持，这个其实很容易实现
    2. 队列处理器很简单，直接使用jedis即可
    3. 对队列数据进行分片(sharding)，写一个QueueService实现即可
    4. 对Redis的内存持久化不放心，采用diskstore存储模式好了，在Redis中配置，不涉及到程序
    5. 对队列分布式存储，写一个QueueService实现，其它可不用变化
    6. 不适合严格意义上的订阅/发布模型，这里适合多个发布者/单个订阅者环境
    7. HTTP请求返回内容为json格式，xml格式，暂时不需要
    8. 局域网环境下，系统之间进行消息的推送/通知，
    项目下载地址：http://code.google.com/p/nettyqueue/ 参考资料：
    1. httpsqs

    http://www.blogjava.net/yongboy/archive/2012/03/20/372308.html
  • 相关阅读:
    IDEA创建test测试类
    SpringBoot Unable to find a @SpringBootConfiguration, you need to use @ContextConfiguration
    Mysql在线加索引锁表归纳
    工作感悟--对上一份工作总结
    ESP8266获取网络NTP时间（转）
    Python中的CGI编程 config配置（windows、Apache） 以及后期的编写（转）
    CGI与FastCGI（转）
    JSON-RPC轻量级远程调用协议介绍及使用
    java插件化编程(动态加载)
    PF4J入门指南
  • 原文地址：https://www.cnblogs.com/softidea/p/5568904.html
Copyright © 2011-2022 走看看