文本分段的两个指标

1. Pk

   • def pk(ref, hyp, k=None, boundary='1'):
         """
         Compute the Pk metric for a pair of segmentations A segmentation
         is any sequence over a vocabulary of two items (e.g. "0", "1"),
         where the specified boundary value is used to mark the edge of a
         segmentation.
     
         >>> '%.2f' % pk('0100'*100, '1'*400, 2)
         '0.50'
         >>> '%.2f' % pk('0100'*100, '0'*400, 2)
         '0.50'
         >>> '%.2f' % pk('0100'*100, '0100'*100, 2)
         '0.00'
     
         :param ref: the reference segmentation
         :type ref: str or list
         :param hyp: the segmentation to evaluate
         :type hyp: str or list
         :param k: window size, if None, set to half of the average reference segment length
         :type boundary: str or int or bool
         :param boundary: boundary value
         :type boundary: str or int or bool
         :rtype: float
         """
     
         if k is None:
             k = int(round(len(ref) / (ref.count(boundary) * 2.)))
     
         err = 0
         for i in range(len(ref)-k +1):
             r = ref[i:i+k].count(boundary) > 0
             h = hyp[i:i+k].count(boundary) > 0
             if r != h:
                err += 1
         return err / (len(ref)-k +1.)

2. WindowDiff
   • 　　

     def windowdiff(seg1, seg2, k, boundary="1", weighted=False):
         """
         Compute the windowdiff score for a pair of segmentations.  A
         segmentation is any sequence over a vocabulary of two items
         (e.g. "0", "1"), where the specified boundary value is used to
         mark the edge of a segmentation.
     
             >>> s1 = "000100000010"
             >>> s2 = "000010000100"
             >>> s3 = "100000010000"
             >>> '%.2f' % windowdiff(s1, s1, 3)
             '0.00'
             >>> '%.2f' % windowdiff(s1, s2, 3)
             '0.30'
             >>> '%.2f' % windowdiff(s2, s3, 3)
             '0.80'
     
         :param seg1: a segmentation
         :type seg1: str or list
         :param seg2: a segmentation
         :type seg2: str or list
         :param k: window width
         :type k: int
         :param boundary: boundary value
         :type boundary: str or int or bool
         :param weighted: use the weighted variant of windowdiff
         :type weighted: boolean
         :rtype: float
         """
     
         if len(seg1) != len(seg2):
             raise ValueError("Segmentations have unequal length")
         if k > len(seg1):
             raise ValueError("Window width k should be smaller or equal than segmentation lengths")
         wd = 0
         for i in range(len(seg1) - k + 1):
             ndiff = abs(seg1[i:i + k].count(boundary) - seg2[i:i + k].count(boundary))
             if weighted:
                 wd += ndiff
             else:
                 wd += min(1, ndiff)
         return wd / (len(seg1) - k + 1.)

     这两个指标观看文献，还真是有点玄学！还好，找到了nltk中对应的实现，极其简单明了！