论分布式系统中单一锁控制的优化

文章目录

• 前言
• 单一锁控制的弊端
• 单一锁的优化：锁粒度的细化
• 锁的管理
• 锁的容量控制
• 锁的资源优先级问题
• 锁管理的demo样例

前言

---

在分布式系统中，为了保持数据操作的一致性，我们会看到锁在里面会有广泛的应用。简单一点的，我们可能就用一个简单的对象锁来做线程安全的同步。再细粒度一些的，我们会用到读写锁，然后对具体操作的属性（Read Or Write）在进行读写锁的分离控制。但今天笔者想对此话题再展开细致的探讨，相比较于使用单一锁模式，我们还是可以做出更进一步的改善的。

单一锁控制的弊端

---

如前文所提到的，不管说是普通的object对象锁还是说ReadWriteLock来说，它们本质上还是Single Lock（单一锁）。这种lock在客户端请求巨大的情况下会有严重的锁竞争问题，一个典型的例子是HDFS NN里的FSN锁。一旦某个请求获得了这个锁，那么其它的请求就会被block住了。所以说单一锁的问题在于，它的粒度有些时候还是太粗了。所以这里就涉及到一个锁粒度的问题了。

单一锁的优化：锁粒度的细化

---

那么怎么取做锁粒度的细化呢？读写锁的分离是一种常见的手段。但更好的一种做法是在实际资源上进行锁控制。因为我们使用锁去做线程安全同步控制时的初衷，是为了避免同时又多个操作在修改某个共同的资源（比如元数据的更新）。但其实不同的资源的相互更新其实是不影响的，那这时我们可以将大的锁拆分为A资源锁以及B资源锁。

用一个更具象的例子来解释这个问题：比如在分布式存储系统的元数据更新里，我们可以将DB级别的lock控制拆分为以table级别的lock。这里的Resource就从DB细粒度为Table。

锁的管理

---

如果锁的粒度已经按照对于实际的资源进行控制的话，这时我们可以考虑在上面包装出一个锁的管理器，由此管理器对外“借出”或“归还”锁。在这个管理类里，我们还能对当前的活跃锁做引用计数的更新。如果某个锁在某次unlock后没有再被外部引用了，我们就可以将其从锁管理器中移除出去了。

锁的容量控制

---

当然我们还需要对锁管理器做一个最大容量的控制，如果当前活跃锁达到容量阈值时，则后面的新锁申请也将会被Block住。这在一定程度上控制的是客户端的concurrency的行为度。

锁的资源优先级问题

---

这里还有一个隐蔽的问题：锁的资源优先级问题。比如说我们有3个资源，它们的资源范围关系如下：

• A，资源最大(包含多个B类资源）
• B，资源适中（包含多个C类资源）
• C，最小粒度资源

假设我们对上述3类资源做锁控制的话，可以遵循以下原则：

在持有低优先级资源锁的情况下，不能继续再获取其上资源的锁。持有高优先级资源锁的情况下，还可以继续获取其下低一级资源的锁。

这个原则对应上面ABC资源的情况就是，用户在获取A锁资源的情况下，可以继续获取B或C资源的锁，而假如说目前已经获取C锁的情况下，是无法获取B锁和A锁的。

锁管理的demo样例

---

以下是摘自Hadoop Ozone里的一个锁管理器的实现，里面涉及到了上文提到的一些要点，供大家学习参考。

/**
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with this
 * work for additional information regarding copyright ownership.  The ASF
 * licenses this file to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 * <p>
 * http://www.apache.org/licenses/LICENSE-2.0
 * <p>
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations under
 * the License.
 */

package org.apache.hadoop.ozone.lock;

import org.apache.commons.pool2.impl.GenericObjectPool;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdds.HddsConfigKeys;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

/**
 * Manages the locks on a given resource. A new lock is created for each
 * and every unique resource. Uniqueness of resource depends on the
 * {@code equals} implementation of it.
 */
public class LockManager<T> {

  private static final Logger LOG = LoggerFactory.getLogger(LockManager.class);

  private final Map<T, ActiveLock> activeLocks = new ConcurrentHashMap<>();
  private final GenericObjectPool<ActiveLock> lockPool =
      new GenericObjectPool<>(new PooledLockFactory());

  /**
   * Creates new LockManager instance.
   *
   * @param conf Configuration object
   */
  public LockManager(Configuration conf) {
    int maxPoolSize = conf.getInt(HddsConfigKeys.HDDS_LOCK_MAX_CONCURRENCY,
        HddsConfigKeys.HDDS_LOCK_MAX_CONCURRENCY_DEFAULT);
    lockPool.setMaxTotal(maxPoolSize);
  }


  /**
   * Acquires the lock on given resource.
   *
   * <p>If the lock is not available then the current thread becomes
   * disabled for thread scheduling purposes and lies dormant until the
   * lock has been acquired.
   */
  public void lock(T resource) {
    activeLocks.compute(resource, (k, v) -> {
      ActiveLock lock;
      try {
        if (v == null) {
          // 锁为空，则在锁对象池中创建并借出
          lock = lockPool.borrowObject();
        } else {
          lock = v;
        }
        // 增加锁引用计数
        lock.incrementActiveCount();
      } catch (Exception ex) {
        LOG.error("Unable to obtain lock.", ex);
        throw new RuntimeException(ex);
      }
      return lock;
      //得到锁后进行锁的lock操作
    }).lock();
  }

  /**
   * Releases the lock on given resource.
   */
  public void unlock(T resource) {
    ActiveLock lock = activeLocks.get(resource);
    if (lock == null) {
      // Someone is releasing a lock which was never acquired. Log and return.
      LOG.warn("Trying to release the lock on {}, which was never acquired.",
          resource);
      return;
    }
    lock.unlock();
    activeLocks.computeIfPresent(resource, (k, v) -> {
      v.decrementActiveCount();
      if (v.getActiveLockCount() != 0) {
        return v;
      }
      // 如果锁没有被引用了，则在锁对象池中移出此锁
      lockPool.returnObject(v);
      return null;
    });
  }
}

上面的ActiveLock实质上是一个经过简单包装的ReentrantLock，

/**
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with this
 * work for additional information regarding copyright ownership.  The ASF
 * licenses this file to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 * <p>
 * http://www.apache.org/licenses/LICENSE-2.0
 * <p>
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
 * License for the specific language governing permissions and limitations under
 * the License.
 */

package org.apache.hadoop.ozone.lock;

import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

/**
 * Lock implementation which also maintains counter.
 */
public final class ActiveLock {

  private Lock lock;
  private AtomicInteger count;

  /**
   * Use ActiveLock#newInstance to create instance.
   */
  private ActiveLock() {
    this.lock = new ReentrantLock();
    this.count = new AtomicInteger(0);
  }

  /**
   * Creates a new instance of ActiveLock.
   *
   * @return new ActiveLock
   */
  public static ActiveLock newInstance() {
    return new ActiveLock();
  }

  /**
   * Acquires the lock.
   *
   * <p>If the lock is not available then the current thread becomes
   * disabled for thread scheduling purposes and lies dormant until the
   * lock has been acquired.
   */
  public void lock() {
    lock.lock();
  }

  /**
   * Releases the lock.
   */
  public void unlock() {
    lock.unlock();
  }

  /**
   * Increment the active count of the lock.
   */
  void incrementActiveCount() {
    count.incrementAndGet();
  }

  /**
   * Decrement the active count of the lock.
   */
  void decrementActiveCount() {
    count.decrementAndGet();
  }

  /**
   * Returns the active count on the lock.
   *
   * @return Number of active leases on the lock.
   */
  int getActiveLockCount() {
    return count.get();
  }

  /**
   * Resets the active count on the lock.
   */
  void resetCounter() {
    count.set(0);
  }

  @Override
  public String toString() {
    return lock.toString();
  }
}