1,SimHash
https://yanyiwu.com/work/2014/01/30/simhash-shi-xian-xiang-jie.html
64位Hash为什么海明距离选3?
http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/33026.pdf
上链接右上precision-recall 曲线图,3是最平衡点,即不会错判太多重复,也不会漏掉很多。
SimHash第一步需抽关键词(feature),并有权重(weight,这个一般是基于统计的?没有词库的默认就用1了)
Feature算法,选 slide windows方法,golang代码如下:
func (t *OverlappingStringTokeniser) Tokenise(input string) []string {
var chunks []string
inputLen := len(input)
for position := 0; position < inputLen-int(t.chunkSize); position += int(t.chunkSize - t.overlapSize) {
chunks = append(chunks, input[position:position+int(t.chunkSize)])
}
return chunks
}
1)简单
2)不比其实算法效果差
计算海明距离:
// Compare calculates the Hamming distance between two 64-bit integers
//
// Currently, this is calculated using the Kernighan method [1]. Other methods
// exist which may be more efficient and are worth exploring at some point
//
// [1] http://graphics.stanford.edu/~seander/bithacks.html#CountBitsSetKernighan
func Compare(a uint64, b uint64) uint8 {
v := a ^ b
var c uint8
for c = 0; v != 0; c++ {
v &= v - 1
}
return c
}