Ehcache不仅支持基本的内存缓存，还支持多种方式将本地内存中的缓存同步到其他使用Ehcache的服务器中，形成集群。如下图所示：
Ehcache支持多种集群方式，下面以RMI通信方式为例，来具体分析一下Ehcache集群的源码。
1服务Provider
Ehcache支持两种服务发现方式：一种是通过广播的方式，服务间自动发现，动态更新存活服务的列表；另一种就是在配置文件中配置好静态服务列表。
1.1自动发现配置
Server1和2的配置都一样，广播地址为230.0.0.1：
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=automatic, multicastGroupAddress=230.0.0.1,
multicastGroupPort=4446, timeToLive=32"/>
1.2手动发现配置
Server1的配置，rmiUrls为server2上的两个cache：
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=manual,rmiUrls=//server2:40001/sampleCache11|//server2:40001/sampleCache12"/>
Server2的配置，rmiUrls为server1上的两个cache：
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=manual,rmiUrls=//server1:40001/sampleCache11|//server1:40001/sampleCache12"/>
1.3源码分析-RMICacheManagerPeerProviderFactory
对应上面两种配置方式，根据peerDiscovery属性的值，创建自动或手动两种Provider。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
public CacheManagerPeerProvider createCachePeerProvider(CacheManager cacheManager, Properties properties)
throws CacheException {
String peerDiscovery = PropertyUtil.extractAndLogProperty(PEER_DISCOVERY, properties);
if (peerDiscovery == null || peerDiscovery.equalsIgnoreCase(AUTOMATIC_PEER_DISCOVERY)) {
try {
return createAutomaticallyConfiguredCachePeerProvider(cacheManager, properties);
} catch (IOException e) {
throw new CacheException("Could not create CacheManagerPeerProvider. Initial cause was " + e.getMessage(), e);
}
} else if (peerDiscovery.equalsIgnoreCase(MANUALLY_CONFIGURED_PEER_DISCOVERY)) {
return createManuallyConfiguredCachePeerProvider(properties);
} else {
return null;
}
}
protected CacheManagerPeerProvider createManuallyConfiguredCachePeerProvider(Properties properties) {
String rmiUrls = PropertyUtil.extractAndLogProperty(RMI_URLS, properties);
if (rmiUrls == null || rmiUrls.length() == 0) {
LOG.info("Starting manual peer provider with empty list of peers. " +
"No replication will occur unless peers are added.");
rmiUrls = new String();
}
rmiUrls = rmiUrls.trim();
StringTokenizer stringTokenizer = new StringTokenizer(rmiUrls, PayloadUtil.URL_DELIMITER);
RMICacheManagerPeerProvider rmiPeerProvider = new ManualRMICacheManagerPeerProvider();
while (stringTokenizer.hasMoreTokens()) {
String rmiUrl = stringTokenizer.nextToken();
rmiUrl = rmiUrl.trim();
rmiPeerProvider.registerPeer(rmiUrl);
LOG.debug("Registering peer {}", rmiUrl);
}
return rmiPeerProvider;
}
|
以创建手动发现服务的Provider为例，新建ManualRMICacheManagerPeerProvider实例后，会调用其registerPeer方法将配置文件中配置的集群服务都注册上。
例如rmiUrls=//server2:40001/sampleCache11|//server2:40001/sampleCache12。
注册代码如下。注册方法仅仅将服务器地址保存到Map中，当后面要讲到的Replicator想要与集群中其他结点通信时，会调用Provider的listRemoteCachePeers()方法，
通过RMI的Naming.lookup()方法找到远程结点并返回。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
|
public final synchronized void registerPeer(String rmiUrl) {
peerUrls.put(rmiUrl, new Date());
}
public final synchronized List listRemoteCachePeers(Ehcache cache) throws CacheException {
List remoteCachePeers = new ArrayList();
List staleList = new ArrayList();
for (Iterator iterator = peerUrls.keySet().iterator(); iterator.hasNext();) {
String rmiUrl = (String) iterator.next();
String rmiUrlCacheName = extractCacheName(rmiUrl);
if (!rmiUrlCacheName.equals(cache.getName())) {
continue;
}
Date date = (Date) peerUrls.get(rmiUrl);
if (!stale(date)) {
CachePeer cachePeer = null;
try {
cachePeer = lookupRemoteCachePeer(rmiUrl);
remoteCachePeers.add(cachePeer);
} catch (Exception e) {
if (LOG.isDebugEnabled()) {
LOG.debug("Looking up rmiUrl " + rmiUrl + " through exception " + e.getMessage()
+ ". This may be normal if a node has gone offline. Or it may indicate network connectivity"
+ " difficulties", e);
}
}
} else {
LOG.debug("rmiUrl {} should never be stale for a manually configured cluster.", rmiUrl);
staleList.add(rmiUrl);
}
}
//Remove any stale remote peers. Must be done here to avoid concurrent modification exception.
for (int i = 0; i < staleList.size(); i++) {
String rmiUrl = (String) staleList.get(i);
peerUrls.remove(rmiUrl);
}
return remoteCachePeers;
}
public CachePeer lookupRemoteCachePeer(String url) throws MalformedURLException, NotBoundException, RemoteException {
LOG.debug("Lookup URL {}", url);
CachePeer cachePeer = (CachePeer) Naming.lookup(url);
return cachePeer;
}
|
广播方式的自动发现Provider与上面源码很像，只是多了两个心跳线程，一个用来监听服务器列表的变化，并动态更新Provider中的列表，一个用来发送自己的心跳。
|
1
2
3
4
5
6
7
8
9
10
11
|
public MulticastRMICacheManagerPeerProvider(CacheManager cacheManager, InetAddress groupMulticastAddress,
Integer groupMulticastPort, Integer timeToLive, InetAddress hostAddress) {
super(cacheManager);
heartBeatReceiver = new MulticastKeepaliveHeartbeatReceiver(this, groupMulticastAddress,
groupMulticastPort, hostAddress);
heartBeatSender = new MulticastKeepaliveHeartbeatSender(cacheManager, groupMulticastAddress,
groupMulticastPort, timeToLive, hostAddress);
}
|
2服务Listener
服务Listener用来监听集群中其他服务器Ehcache的消息，所以Listener要监听本机上端口。
2.1配置文件
server1和server2配置一样，都是监听本机上40001端口：
<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="hostName=localhost, port=40001,
socketTimeoutMillis=2000"/>
2.2源码分析
取出当前配置，然后新建一个RMICacheManagerPeerListener实例。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
|
public final CacheManagerPeerListener createCachePeerListener(CacheManager cacheManager, Properties properties)
throws CacheException {
String hostName = PropertyUtil.extractAndLogProperty(HOSTNAME, properties);
String portString = PropertyUtil.extractAndLogProperty(PORT, properties);
Integer port = null;
if (portString != null && portString.length() != 0) {
port = Integer.valueOf(portString);
} else {
port = Integer.valueOf(0);
}
//0 means any port in UnicastRemoteObject, so it is ok if not specified to make it 0
String remoteObjectPortString = PropertyUtil.extractAndLogProperty(REMOTE_OBJECT_PORT, properties);
Integer remoteObjectPort = null;
if (remoteObjectPortString != null && remoteObjectPortString.length() != 0) {
remoteObjectPort = Integer.valueOf(remoteObjectPortString);
} else {
remoteObjectPort = Integer.valueOf(0);
}
String socketTimeoutMillisString = PropertyUtil.extractAndLogProperty(SOCKET_TIMEOUT_MILLIS, properties);
Integer socketTimeoutMillis;
if (socketTimeoutMillisString == null || socketTimeoutMillisString.length() == 0) {
socketTimeoutMillis = DEFAULT_SOCKET_TIMEOUT_MILLIS;
} else {
socketTimeoutMillis = Integer.valueOf(socketTimeoutMillisString);
}
return doCreateCachePeerListener(hostName, port, remoteObjectPort, cacheManager, socketTimeoutMillis);
}
protected CacheManagerPeerListener doCreateCachePeerListener(String hostName,
Integer port,
Integer remoteObjectPort,
CacheManager cacheManager,
Integer socketTimeoutMillis) {
try {
return new RMICacheManagerPeerListener(hostName, port, remoteObjectPort, cacheManager, socketTimeoutMillis);
} catch (UnknownHostException e) {
throw new CacheException("Unable to create CacheManagerPeerListener. Initial cause was " + e.getMessage(), e);
}
}
|
之后RMICacheManagerPeerListener的init()方法会调用RMI的API，提供RMI服务：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
|
public void init() throws CacheException {
if (!status.equals(Status.STATUS_UNINITIALISED)) {
return;
}
RMICachePeer rmiCachePeer = null;
try {
startRegistry();
int counter = 0;
populateListOfRemoteCachePeers();
synchronized (cachePeers) {
for (Iterator iterator = cachePeers.values().iterator(); iterator.hasNext();) {
rmiCachePeer = (RMICachePeer) iterator.next();
bind(rmiCachePeer.getUrl(), rmiCachePeer);
counter++;
}
}
LOG.debug(counter + " RMICachePeers bound in registry for RMI listener");
status = Status.STATUS_ALIVE;
} catch (Exception e) {
String url = null;
if (rmiCachePeer != null) {
url = rmiCachePeer.getUrl();
}
throw new CacheException("Problem starting listener for RMICachePeer "
+ url + ". Initial cause was " + e.getMessage(), e);
}
}
protected void startRegistry() throws RemoteException {
try {
registry = LocateRegistry.getRegistry(port.intValue());
try {
registry.list();
} catch (RemoteException e) {
//may not be created. Let's create it.
registry = LocateRegistry.createRegistry(port.intValue());
registryCreated = true;
}
} catch (ExportException exception) {
LOG.error("Exception starting RMI registry. Error was " + exception.getMessage(), exception);
}
}
protected void populateListOfRemoteCachePeers() throws RemoteException {
String[] names = cacheManager.getCacheNames();
for (int i = 0; i < names.length; i++) {
String name = names[i];
Ehcache cache = cacheManager.getEhcache(name);
synchronized (cachePeers) {
if (cachePeers.get(name) == null) {
if (isDistributed(cache)) {
RMICachePeer peer = new RMICachePeer(cache, hostName, port, remoteObjectPort, socketTimeoutMillis);
cachePeers.put(name, peer);
}
}
}
}
}
|
3 事件Listener
通过Provider记录集群中其他服务器的地址，通过Listener在40001端口监听RMI消息，就差配置Replicator监听本地缓存增删改查的事件并发送到集群中其他服务器了。
3.1配置文件
可以在每个<cache>配置中提供一个EventListener。可以配置Replicator是同步还是异步的，并配置Put、Update、Remove等哪些事件需要同步：
<!-- Sample cache named sampleCache2. -->
<cache name ="sampleCache2"
maxEntriesLocalHeap ="10"
eternal="false"
timeToIdleSeconds ="100"
timeToLiveSeconds ="100"
overflowToDisk="false" >
<cacheEventListenerFactory
class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"
properties="replicateAsynchronously=true, replicatePuts=true, replicateUpdates=true,
replicateUpdatesViaCopy=false, replicateRemovals=true "/>
</cache>
另一种简单配法是将EventListener配置到Cache的属性上，EventListener的所有属性都采用默认值。
<!-- Sample cache named sampleCache4. All missing RMICacheReplicatorFactory properties
default to true --><cachename="sampleCache4"maxEntriesLocalHeap="10"eternal="true"overflowToDisk="false"memoryStoreEvictionPolicy="LFU"><cacheEventListenerFactoryclass="net.sf.ehcache.distribution.RMICacheReplicatorFactory"/></cache>RMICacheReplicatorFactory会根据replicateAsynchronously属性创建同步或异步的Replicator。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
public final CacheEventListener createCacheEventListener(Properties properties) {
boolean replicatePuts = extractReplicatePuts(properties);
boolean replicatePutsViaCopy = extractReplicatePutsViaCopy(properties);
boolean replicateUpdates = extractReplicateUpdates(properties);
boolean replicateUpdatesViaCopy = extractReplicateUpdatesViaCopy(properties);
boolean replicateRemovals = extractReplicateRemovals(properties);
boolean replicateAsynchronously = extractReplicateAsynchronously(properties);
int asynchronousReplicationIntervalMillis = extractReplicationIntervalMilis(properties);
if (replicateAsynchronously) {
return new RMIAsynchronousCacheReplicator(
replicatePuts,
replicatePutsViaCopy,
replicateUpdates,
replicateUpdatesViaCopy,
replicateRemovals,
asynchronousReplicationIntervalMillis);
} else {
return new RMISynchronousCacheReplicator(
replicatePuts,
replicatePutsViaCopy,
replicateUpdates,
replicateUpdatesViaCopy,
replicateRemovals);
}
}
|
先以同步Replicator的ElementPut事件为例，看同步Replicator是如何处理事件的。replicatePutsViaCopy属性的JavaDoc文档解释的很清楚，这个属性是用来说明，当发生Put事件时，是通知集群中其他服务器结点更新该Element，还是直接置为失效。前者适合Element的新建很耗时，而后者适合重新同步数据库中的数据。要通知的服务器列表就来自上面配置的CacheManagerPeerProviderFactory创建出的Provider对象。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
|
/**
* Whether a put should replicated by copy or by invalidation, (a remove).
* <p/>
* By copy is best when the entry is expensive to produce. By invalidation is best when
* we are really trying to force other caches to sync back to a canonical source like a database.
* An example of a latter usage would be a read/write cache being used in Hibernate.
* <p/>
* This setting only has effect if <code>#replicateUpdates</code> is true.
*/
protected boolean replicatePutsViaCopy;
public void notifyElementPut(final Ehcache cache, final Element element) throws CacheException {
if (notAlive()) {
return;
}
if (!replicatePuts) {
return;
}
if (!element.isSerializable()) {
if (LOG.isWarnEnabled()) {
LOG.warn("Object with key " + element.getObjectKey() + " is not Serializable and cannot be replicated");
}
return;
}
if (replicatePutsViaCopy) {
replicatePutNotification(cache, element);
} else {
replicateRemovalNotification(cache, (Serializable) element.getObjectKey());
}
}
protected static void replicatePutNotification(Ehcache cache, Element element) throws RemoteCacheException {
List cachePeers = listRemoteCachePeers(cache);
for (Object cachePeer1 : cachePeers) {
CachePeer cachePeer = (CachePeer) cachePeer1;
try {
cachePeer.put(element);
} catch (Throwable t) {
LOG.error("Exception on replication of putNotification. " + t.getMessage() + ". Continuing...", t);
}
}
}
static List listRemoteCachePeers(Ehcache cache) {
CacheManagerPeerProvider provider = cache.getCacheManager().getCacheManagerPeerProvider("RMI");
return provider.listRemoteCachePeers(cache);
}
|
异步Replicator的实现方式也很简单。以ElementPut事件为例，之前的同步Replicator是直接通知其他结点，异步Replicator将事件保存到队列中，
后台线程ReplicationThread会定时将队列中积压的事件发送到集群中其他结点。之前同步Replicator调用的CachePeer的单条增删改查方法，
这次ReplicationThread调用的是CachePeer的批量方法send()。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
|
public final void notifyElementPut(final Ehcache cache, final Element element) throws CacheException {
if (notAlive()) {
return;
}
if (!replicatePuts) {
return;
}
if (replicatePutsViaCopy) {
if (!element.isSerializable()) {
if (LOG.isWarnEnabled()) {
LOG.warn("Object with key " + element.getObjectKey() + " is not Serializable and cannot be replicated.");
}
return;
}
addToReplicationQueue(new CacheEventMessage(EventMessage.PUT, cache, element, null));
} else {
if (!element.isKeySerializable()) {
if (LOG.isWarnEnabled()) {
LOG.warn("Object with key " + element.getObjectKey()
+ " does not have a Serializable key and cannot be replicated via invalidate.");
}
return;
}
addToReplicationQueue(new CacheEventMessage(EventMessage.REMOVE, cache, null, element.getKey()));
}
}
protected void addToReplicationQueue(CacheEventMessage cacheEventMessage) {
if (!replicationThread.isAlive()) {
LOG.error("CacheEventMessages cannot be added to the replication queue because the replication thread has died.");
} else {
synchronized (replicationQueue) {
replicationQueue.add(cacheEventMessage);
}
}
}
private final class ReplicationThread extends Thread {
public ReplicationThread() {
super("Replication Thread");
setDaemon(true);
setPriority(Thread.NORM_PRIORITY);
}
public final void run() {
replicationThreadMain();
}
}
private void replicationThreadMain() {
while (true) {
// Wait for elements in the replicationQueue
while (alive() && replicationQueue != null && replicationQueue.size() == 0) {
try {
Thread.sleep(asynchronousReplicationInterval);
} catch (InterruptedException e) {
LOG.debug("Spool Thread interrupted.");
return;
}
}
if (notAlive()) {
return;
}
try {
if (replicationQueue.size() != 0) {
flushReplicationQueue();
}
} catch (Throwable e) {
LOG.error("Exception on flushing of replication queue: " + e.getMessage() + ". Continuing...", e);
}
}
}
private void flushReplicationQueue() {
List replicationQueueCopy;
synchronized (replicationQueue) {
if (replicationQueue.size() == 0) {
return;
}
replicationQueueCopy = new ArrayList(replicationQueue);
replicationQueue.clear();
}
Ehcache cache = ((CacheEventMessage) replicationQueueCopy.get(0)).cache;
List cachePeers = listRemoteCachePeers(cache);
List resolvedEventMessages = extractAndResolveEventMessages(replicationQueueCopy);
for (int j = 0; j < cachePeers.size(); j++) {
CachePeer cachePeer = (CachePeer) cachePeers.get(j);
try {
cachePeer.send(resolvedEventMessages);
} catch (UnmarshalException e) {
String message = e.getMessage();
if (message.indexOf("Read time out") != 0) {
LOG.warn("Unable to send message to remote peer due to socket read timeout. Consider increasing" +
" the socketTimeoutMillis setting in the cacheManagerPeerListenerFactory. " +
"Message was: " + e.getMessage());
} else {
LOG.debug("Unable to send message to remote peer. Message was: " + e.getMessage());
}
} catch (Throwable t) {
LOG.warn("Unable to send message to remote peer. Message was: " + t.getMessage(), t);
}
}
if (LOG.isWarnEnabled()) {
int eventMessagesNotResolved = replicationQueueCopy.size() - resolvedEventMessages.size();
if (eventMessagesNotResolved > 0) {
LOG.warn(eventMessagesNotResolved + " messages were discarded on replicate due to reclamation of " +
"SoftReferences by the VM. Consider increasing the maximum heap size and/or setting the " +
"starting heap size to a higher value.");
}
}
}
|
这里注意因为使用的是RMI通信方式，实际上CachePeer就是实现了RMI的Remote接口的存根对象。对CachePeer方法的调用就是对远程方法的调用。所以上面两种Replicator调用CachePeer时，就是将缓存事件同步到远程了。
结束语
RMI方式的Ehcache集群的实现比较简单、易理解，但对于前端用Nginx做负载均衡时，连续的几次调用可能是转发到不同的后端Ehcache服务器上，
异步方式的Ehcache缓存同步会不会有问题呢？