JBoss Data Grid 7.2 在OpenShift环境中的Quick Start

 为了在容器环境中运行，首先需要弄清楚的是在传统环境下如何运行，所以我们从传统环境开始。

先去http://access.redhat.com下载相应介质,主要是 jboss-datagrid-7.2.0-server.zip和jboss-datagrid-7.2.0-tomcat8-session-client.zip

前者用于jboss data grid的启动，后者用于客户端tomcat通过Client-Server方式去连接和操作

1. 安装

直接解压就是安装，但要注意如果是需要多个server构成一个集群，需要建立两个目录分别解压，我试过只修改配置不成，因为还有

其他文件在进程启动以后需要进行同时写入。所以最佳办法是每个实例分别建立一个目录。

修改配置文件cluster.xml,如果需要加入定义的Cache,可以添加下面这一段

<subsystem xmlns="urn:infinispan:server:endpoint:6.0">
        <hotrod-connector socket-binding="hotrod"  cache-container="clusteredcache">
         <topology-state-transfer lazy-retrieval="false" lock-timeout="1000" replication-timeout="5000"/>
        </hotrod-connector>
        .........
      <subsystem xmlns="urn:infinispan:server:core:6.0" default-cache-container="clusteredcache">
                   <cache-container name="clusteredcache" default-cache="default" statistics="true">
                       <transport executor="infinispan-transport" lock-timeout="60000"/>
                    ......
               <distributed-cache name="directory-dist-cache" mode="SYNC" owners="2" remote-                   timeout="30000" start="EAGER">
              <locking isolation="READ_COMMITTED" acquire-timeout="30000" striping="false"/>
              <eviction strategy="LRU" max-entries="20" />
              <transaction mode="NONE"/>
              </distributed-cache>
             ..............
  </cache-container>

如果不需要定义，可以用缺省的配置，也就是default,配置为分布式

<distributed-cache name="default"/>

修改server2的端口，主要是标黑的port-offset,标黑的那段

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:100}">
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
        <socket-binding name="hotrod" port="11222"/>
        <socket-binding name="hotrod-internal" port="11223"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" port="57600"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:234.99.54.14}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="54200"/>
        <socket-binding name="memcached" port="11211"/>
        <socket-binding name="rest" port="8080"/>
        <socket-binding name="rest-multi-tenancy" port="8081"/>
        <socket-binding name="rest-ssl" port="8443"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="remote-store-hotrod-server">
            <remote-destination host="remote-host" port="11222"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="remote-store-rest-server">
            <remote-destination host="remote-host" port="8080"/>
        </outbound-socket-binding>
    </socket-binding-group>

2.启动

standalone.bat -c=clustered1.xml -Djboss.node.name=server1

standalone.bat -c=clustered2.xml -Djboss.node.name=server2

 从日志中可以看到server2的加入，并进行数据的rebalance.

3.监控和操作

• CLI操作

可以通过bin/cli.sh或者cli.bat进行直接对缓存的读取。windows环境中基本命令如下

[disconnected /] connect 127.0.0.1:9990
[standalone@127.0.0.1:9990 /] container clustered
[standalone@127.0.0.1:9990 cache-container=clustered] cache
ISPN019029: No cache selected yet
[standalone@127.0.0.1:9990 cache-container=clustered] cache default
[standalone@127.0.0.1:9990 distributed-cache=default] cache
default
[standalone@127.0.0.1:9990 distributed-cache=default] put 1 ericnie
[standalone@127.0.0.1:9990 distributed-cache=default] get 1
ericnie

container的值，可以从cluster.xml的配置中找到，截取一段，cache值也是一样，缺省为default.

<subsystem xmlns="urn:infinispan:server:core:8.5" default-cache-container="clustered">
            <cache-container name="clustered" default-cache="default" statistics="true">
                <transport lock-timeout="60000"/>
                <global-state/>
                <distributed-cache-configuration name="transactional">
                    <transaction mode="NON_XA" locking="PESSIMISTIC"/>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="async" mode="ASYNC"/>
                <replicated-cache-configuration name="replicated"/>
                <distributed-cache-configuration name="persistent-file-store">
                    <file-store shared="false" fetch-state="true" passivation="false"/>
                </distributed-cache-configuration>
                <distributed-cache-configuration name="indexed">
                    <indexing index="LOCAL" auto-config="true"/>
                </distributed-cache-configuration>

• 监控层面

惊闻Jboss ON要end of life,以后更多需要走prometheus或者openshift容器化的监控手段了，所以果断来个最基本的jmx监控。

启动jconsole, 基于jmx连接本地或者远程端口(9990),在MBean中找到jboss.datagrid-infinispan

• 查看集群属性，CacheManager->clustered

•  查看Cache Entry

4.客户端访问

在tomcat的webapp下建立一个项目jdg,然后建立WEB-INF,在lib下面把之前的jar包拷入。

写一段客户端访问代码.

<%@ page language="java" import="java.util.*" pageEncoding="gbk"%>
<%@ page import="org.infinispan.client.hotrod.RemoteCache,org.infinispan.client.hotrod.RemoteCacheManager,org.infinispan.client.hotrod.configuration.ConfigurationBuilder,com.redhat.lab.jdg.*,java.utils.*" %>
<html>
  <head>
    <title>My JSP starting page</title>
  </head>
  
  <body>
    <h1>
        
     <%
       try {
              ConfigurationBuilder builder = new ConfigurationBuilder();
                  builder.addServer().host("127.0.0.1")
                   .port(Integer.parseInt("11322"));
                  RemoteCacheManager  cacheManager = new RemoteCacheManager(builder.build());
                  RemoteCache<String, User> cache = cacheManager.getCache();
          
          
        
                    User user = new User();
                    user.setFirstName("John");
                    user.setLastName("Doe");
                    cache.put("jdoe", user);
                    System.out.println("John Doe has been put into the cache");
                    out.println("John Doe has been put into the cache");
            
          
                    if (cache.containsKey("jdoe")) {
                            System.out.println("jdoe key is indeed in the cache");
                            out.println("jdoe key is indeed in the cache");
                    }
        
                    if (cache.containsKey("jane")) {
                            System.out.println("jane key is indeed in the cache");
                            out.println("jane key is indeed in the cache");
                    }
        
                    user = cache.get("jdoe");
                    System.out.println("jdoe's firstname is " +
                    user.getFirstName());
                    
                    out.println("jdoe's firstname is " +
                    user.getFirstName());
                    
            
        } catch (Exception e) {    
            e.printStackTrace();
    }
    
    
     %>
    </h1>
  </body>
</html>

 然后是各种验证

5.OpenShift部署

首先找到官方镜像地址

https://github.com/openshift/library/tree/master/official/datagrid

打开imagestreams/jboss-datagrid72-openshift-rhel7.json，然后pull到本地

docker pull registry.redhat.io/jboss-datagrid-7/datagrid72-openshift:1.2

pull前先用docker login登录网站redhat.io(3.11的新特征 :()

然后查看Service catalog

 咱们就来搞这个7.2的Ephemeral, no https了

oc get templates -n openshift

.....
datagrid72-basic                                    An example Red Hat JBoss Data Grid application. For more information about us...   17 (11 blank)     6

........

然后修改镜像地址

oc edit template datagrid72-basic -n openshift

 切换到openshift命名空间，导入ImageStream

oc project openshift

oc import-image datagrid72-openshift:1.2   --from=registry.example.com/jboss-datagrid-7/datagrid72-openshift:1.2  --insecure --confirm

一切就绪，开始建立

输入一个Cache名，然后创建.

创建完成

Scale Pod,然后检查Pod日志，可见新的pod已经加入集群。

6.OpenShift环境中的验证

Openshift环境中JDG提供了访问的三种模式

• memcached, 基于memcache协议
• hotrod,基于TCP,适合Client Server
• Rest(对应datagrid-app),适合基于http协议，因此暴露对外路由。

原本想法是，修改hotrod，加入nodePort,然后通过OpenShift外面的tomcat或Java Client进行访问，但尝试了一下，发现不行，

Client端会直接找JDG Pod的实际地址建立连接，然后发现无法访问。因此需要将tomcat部署到OpenShift内部进行尝试。

• 在同一项目中访问，修改jsp代码为

                ConfigurationBuilder builder = new ConfigurationBuilder();
                                builder.addServer().host(System.getenv("DATAGRID_APP_HOTROD_SERVICE_HOST"))
                                .port(Integer.parseInt(System.getenv("DATAGRID_APP_HOTROD_SERVICE_PORT"));
                                RemoteCacheManager  cacheManager = new RemoteCacheManager(builder.build());

这里hotrod的地址是通过存放在tomcat pod中的环境变量DATAGRID_APP_HOTROD_SERVICE_HOST已及ATAGRID_APP_HOTROD_SERVICE_PORT获取

访问成功。

• 不同项目中访问，需要在java代码中根据服务名获取服务地址。

 不同的项目访问，在pod的环境变量中不会有其他项目的环境变量，因此需要根据服务名获取服务地址，核心代码为

                        InetAddress address = InetAddress.getByName("datagrid-app-hotrod.jdg");
        System.out.println(address.getHostAddress());



                ConfigurationBuilder builder = new ConfigurationBuilder();
                                builder.addServer().host(address.getHostAddress())
                                .port(Integer.parseInt("11333"));

访问hotrod的地址，通过InetAddress.getByName("datagrid-app-hotrod.jdg"),带上服务名以及项目名获取。验证无误。

贴一个jdg-write.jsp的完整代码:

<%@ page language="java" import="java.util.*" pageEncoding="gbk"%>
<%@ page import="org.infinispan.client.hotrod.RemoteCache,org.infinispan.client.hotrod.RemoteCacheManager,org.infinispan.client.hotrod.configuration.ConfigurationBuilder,com.redhat.lab.jdg.*,java.net.*,java.utils.*" %>
<html>
  <head>
    <title>My JSP starting page</title>
  </head>

  <body>
    <h1>

     <%
       try {

                        InetAddress address = InetAddress.getByName("datagrid-app-hotrod.jdg");
        System.out.println(address.getHostAddress());



                ConfigurationBuilder builder = new ConfigurationBuilder();
                                builder.addServer().host(address.getHostAddress())
                                .port(Integer.parseInt("11333"));
                                RemoteCacheManager  cacheManager = new RemoteCacheManager(builder.build());


                RemoteCache<String, User> cache = cacheManager.getCache("samples");

                                        User user = new User();
                                        user.setFirstName("John");
                                        user.setLastName("Doe");
                                        cache.put("jdoe", user);
                                        System.out.println("John Doe has been put into the cache");
                                        out.println("John Doe has been put into the cache");


                                        if (cache.containsKey("jdoe")) {
                                                        System.out.println("jdoe key is indeed in the cache");
                                                        out.println("jdoe key is indeed in the cache");
                                        }

                                        if (cache.containsKey("jane")) {
                                                        System.out.println("jane key is indeed in the cache");
                                                        out.println("jane key is indeed in the cache");
                                        }

                                        user = cache.get("jdoe");
                                        System.out.println("jdoe's firstname is " +
                                        user.getFirstName());

                                        out.println("jdoe's firstname is " +
                                        user.getFirstName());


                } catch (Exception e) {
            e.printStackTrace();
    }


     %>
    </h1>
  </body>
</html>

调试到了第11个版本终于解决了各种低级错误，成功运行，反复调试修改后主要执行的三个命令记录如下

docker build -t registry.example.com/jdg/tomcatsample:v1 .

docker push registry.example.com/jdg/tomcatsample:v1

oc import-image tomcatsample:v1   --from=registry.example.com/jdg/tomcatsample:v1  --insecure --confirm

然后就依赖于Deployconfig的根据镜像变化从而触发Pod更新的trigger了。

形成客户端镜像的Tomcat 的Dockerfile

[root@master client]# cat Dockerfile 
FROM registry.example.com/tomcat:8-slim

RUN mkdir -p /usr/local/tomcat/webapps/jdg
COPY samples/jdg/* /usr/local/tomcat/webapps/jdg/


USER root

RUN unzip -d /usr/local/tomcat/webapps/jdg/ /usr/local/tomcat/webapps/jdg/WEB-INF.zip

CMD [ "/usr/local/tomcat/bin/catalina.sh","run" ]