Graph database_neo4j 底层存储结构分析(3)

3.3  通用的Store 类型

3.3.1    id 类型

下面是 neo4j db 中,每种Store都有自己的ID文件(即后缀.id 文件)，它们的格式都是一样的。

[test00]$ls -lh target/neo4j-test00.db/ |grep .id

-rw-r–r–9 04-11 13:28 neostore.id

-rw-r–r–9 04-11 13:28 neostore.labeltokenstore.db.id

-rw-r–r–9 04-11 13:28 neostore.labeltokenstore.db.names.id

-rw-r–r–9 04-11 13:28 neostore.nodestore.db.id

-rw-r–r–9 04-11 13:28 neostore.nodestore.db.labels.id

-rw-r–r–9 04-11 13:28 neostore.propertystore.db.arrays.id

-rw-r–r–9 04-11 13:28 neostore.propertystore.db.id

-rw-r–r–9 04-11 13:28 neostore.propertystore.db.index.id

-rw-r–r–9 04-11 13:28 neostore.propertystore.db.index.keys.id

-rw-r–r–9 04-11 13:28 neostore.propertystore.db.strings.id

-rw-r–r–9 04-11 13:28 neostore.relationshipgroupstore.db.id

-rw-r–r–9 04-11 13:28 neostore.relationshipstore.db.id

-rw-r–r–9 04-11 13:28 neostore.relationshiptypestore.db.id

-rw-r–r–9 04-11 13:28 neostore.relationshiptypestore.db.names.id

-rw-r–r–9 04-11 13:28 neostore.schemastore.db.id

3.3.1.1        ID类型文件的存储格式

neo4j 中后缀为 “.id”的文件格式如上图所示，由文件头（9 Bytes）和 long类型 数组 2部分构成：

• sticky(1 byte) : if sticky the id generator wasn’t closed properly so it has to berebuilt (go through the node, relationship, property, rel type etc files).
• nextFreeId(long) : 保存最大的ID,该值与对应类型的存储数组的数组大小相对应。
• reuseId(long):用来保存已经释放且可复用的ID值。通过复用ID ,可以减少资源数组的空洞，提高磁盘利用率。

3.3.1.2        IdGeneratorImpl.java

每一种资源类型的ID 分配 neo4j 中是通过 IdGeneratorImpl 来实现的，其功能是负责ID管理分配和回收复用。对于节点，关系，属性等每一种资源类型，都可以生成一个IdGenerator  实例来负责其ID管理分配和回收复用。

3.3.1.2.1       读取id 文件进行初始化

下面试 IdGeneratorImpl.java 中， 读取id 文件进行初始化的过程，IdGeneratorImpl 会从 id 文件中读取grabSize 个可复用的ID (reuseId) 到idsReadFromFile(LinkedList<Long>) 中，在需要申请id 时优先分配 idsReadFromFile中的可复用ID。

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141

<div>   // initialize the id generator and performs a simple validation   private synchronized void initGenerator()   {   try   {   fileChannel = fs.open( fileName, "rw" );   ByteBuffer buffer = ByteBuffer.allocate( HEADER_SIZE );   readHeader( buffer );   markAsSticky( buffer );   fileChannel.position( HEADER_SIZE );   maxReadPosition = fileChannel.size();   defraggedIdCount = (int) (maxReadPosition - HEADER_SIZE) / 8;   readIdBatch();   }   catch ( IOException e )   {   throw new UnderlyingStorageException(   "Unable to init id generator " + fileName, e );   }   }   private void readHeader( ByteBuffer buffer ) throws IOException   {   readPosition = fileChannel.read( buffer );   if ( readPosition != HEADER_SIZE )   {   fileChannel.close();   throw new InvalidIdGeneratorException(   "Unable to read header, bytes read: " + readPosition );   }   buffer.flip();   byte storageStatus = buffer.get();   if ( storageStatus != CLEAN_GENERATOR )   {   fileChannel.close();   throw new InvalidIdGeneratorException( "Sticky generator[ " +   fileName + "] delete this id file and build a new one" );   }   this.highId.set( buffer.getLong() );   }   private void readIdBatch()   {   if ( !canReadMoreIdBatches() )   return;   try   {   int howMuchToRead = (int) Math.min( grabSize*8, maxReadPosition-readPosition );   ByteBuffer readBuffer = ByteBuffer.allocate( howMuchToRead );   fileChannel.position( readPosition );   int bytesRead = fileChannel.read( readBuffer );   assert fileChannel.position() <= maxReadPosition;   readPosition += bytesRead;   readBuffer.flip();   assert (bytesRead % 8) == 0;   int idsRead = bytesRead / 8;   defraggedIdCount -= idsRead;   for ( int i = 0; i < idsRead; i++ )   {   long id = readBuffer.getLong();   if ( id != INTEGER_MINUS_ONE )   {   idsReadFromFile.add( id );   }   }   }   catch ( IOException e )   {   throw new UnderlyingStorageException(   "Failed reading defragged id batch", e );   }   }

3.3.1.2.2       释放id(freeId)

用户释放一个 id 后，会先放入 releasedIdList （LinkedList<Long>），当releasedIdList 中回收的 id 个数超过 grabSize 个时， 写入到 id 文件的末尾。所以可见，对于一个 IdGeneratorImpl， 最多有 2 * grabSize 个 id 缓存(releasedIdList 和 idsReadFromFile)。

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

<div>   /**   * Frees the <CODE>id</CODE> making it a defragged id that will be   * returned by next id before any new id (that hasn't been used yet) is   * returned.   * <p>   * This method will throw an <CODE>IOException</CODE> if id is negative or   * if id is greater than the highest returned id. However as stated in the   * class documentation above the id isn't validated to see if it really is   * free.   */   @Override   public synchronized void freeId( long id )   {   if ( id == INTEGER_MINUS_ONE )   {   return;   }   if ( fileChannel == null )   {   throw new IllegalStateException( "Generator closed " + fileName );   }   if ( id < 0 || id >= highId.get() )   {   throw new IllegalArgumentException( "Illegal id[" + id + "]" );   }   releasedIdList.add( id );   defraggedIdCount++;   if ( releasedIdList.size() >= grabSize )   {   writeIdBatch( ByteBuffer.allocate( grabSize*8 ) );   }   }

3.3.1.2.3       申请id ( nextId)

当用户申请一个 id  时，IdGeneratorImpl 在分配时，有2种分配策略： “正常的分配策略” 和激进分配策略”（aggressiveReuse），可以根据配置进行选择。

n  “正常的分配策略”：

a)        首先从idsReadFromFile 中分配; 如果 idsReadFromFile 为空，则先从对应的 id 文件中读取已释放且可复用的 id 到idsReadFromFile.

b)        如果 idsReadFromFile 及 id 文件中没有已释放且可复用的 id了，则分配全新的id,即id = highId.get()  并将highId 加1；

n   “激进分配策略”（aggressiveReuse）:

a)        首先从releasedIdList（刚回收的ID List）中分配。

b)        releasedIdList分配光了，则从idsReadFromFile 中分配; 如果 idsReadFromFile 为空，则先从对应的 id 文件中读取已释放且可复用的 id 到idsReadFromFile.

c)        如果 idsReadFromFile 及 id 文件中没有已释放且可复用的 id了，则分配全新的id,即id = highId.get()  并将highId 加1；

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

<div>   /**   * Returns the next "free" id. If a defragged id exist it will be returned   * else the next free id that hasn't been used yet is returned. If no id   * exist the capacity is exceeded (all values <= max are taken) and a   * {@link UnderlyingStorageException} will be thrown.   */   @Override   public synchronized long nextId()   {   assertStillOpen();   long nextDefragId = nextIdFromDefragList();   if ( nextDefragId != -1 ) return nextDefragId;   long id = highId.get();   if ( id == INTEGER_MINUS_ONE )   {   // Skip the integer -1 (0xFFFFFFFF) because it represents   // special values, f.ex. the end of a relationships/property chain.   id = highId.incrementAndGet();   }   assertIdWithinCapacity( id );   highId.incrementAndGet();   return id;   }