推荐系统（四）

(原创文章，转载请注明出处！)

用户对物品的推荐数据通常形成一个巨大的矩阵，而且通常用户的数量比物品的数量多，可以通过SVD（奇异值分解）来将矩阵分解，减少计算中使用的数据量，降低计算的复杂度。假设数据R是m x n矩阵，m个用户，n个物品，通过奇异值分解，R=U∑V^T。那么将R投影到低维的k（k < min(m,n)）空间：R_k=R^TU_k∑_k，R^T是R的转置 n x m矩阵， U_k是m x k矩阵，∑_k是 k x k 对角阵，所以投影完成后的矩阵R_k是 n x k 矩阵，每一行代表一个物品。

计算过程：

1. 对原始数据进行normalization（z-Score）

2. 对normalization后的数据矩阵进行SVD分解，将数据矩阵投影到新的低维空间

3. 使用IBCF来计算推荐结果

实现代码如下：

## Decompose the rating matrix with SVD and project the rating matix 
## to lower dimension space. Find the top n items as the item recommendation 
## list with the Item-Based Collaborative Filtering algorithm over the 
## lower dimension data.
## Args :
##      x  -  a matrix, contain all rating reslut. 
##            Each colum is the rating by one user, each row is the rating of one movie.
##            If a movie hasn't been rated by a user, the corresponding postion in the matrix is NA.
##      userI - index of specified user
##      k  -  k nearest neigbour of useriI
##      n  -  top n items that will be recommended to user-I
##      pc_threshold  -  principal component threshold
## Returns :
##      a list, contains recommendation result
svdRecommendationIBCF <- function(x, userI, k, n, pc_threshold=0.9)
{
    # todo: how to calculate the Pearson correlation coefficient between two vectors
    # sum((x - u_x)*(y - u_y)) / (sd_x*sd_y)
    
    x[which(is.na(x))] <- 0
    ## normalize the data
    normlizedResult <- zScoreNormalization( x )
    x <- t( normlizedResult$xNormalized )
    
    ## svd decomposition
    svd_x <- svd(x)
    # find the top-k singular value
    numTopSV <- 0
    for(sv in svd_x$d) {
        numTopSV <- numTopSV + 1
        if ( (sum(svd_x$d[1:numTopSV]) / sum(svd_x$d)) >= pc_threshold ) {
            break
        }
    }
    # project the rating data to lower dimension
    # x_lowDim is a n-by-numTopSV matrix
    # n is the number of items
    # numTopSV is less than or equal to min(m , n)
    x_lowDim <- t(x) %*% svd_x$u[,1:numTopSV] %*% diag(svd_x$d[1:numTopSV])
    
    
    ## predicting the rating of user-I's un-rated items
    unRatedIdx <- which(x[,userI] == 0)
    ratedIdx <- which(x[,userI] != 0)
    ratingOfUnRatedItems <- numeric( dim(x)[1] )
    for (i in unRatedIdx) {        
        # calculate the Pearson correlation coefficient to each item
        itemSim <- cor( x = x_lowDim[i,], y = t(x_lowDim[ratedIdx,]), use = "everything", method = "pearson" )
        itemSim <- 0.5 + 0.5*itemSim # keep the similarity in [0,1]
        # find the k nearest items to item-i
        KSimilarItemIdx <- apply( matrix(itemSim,nrow=1), 
                                  MARGIN=1,  # apply the function to each row
                                  FUN=function(x) head(  order(x, decreasing=TRUE, na.last=TRUE), k)
                                )
        KSimilarItemIdx <- as.vector(KSimilarItemIdx)                              

        r <- x[ratedIdx,]
        ratingOfUnRatedItems[i] <- sum( r[KSimilarItemIdx,userI] * itemSim[KSimilarItemIdx] )   
                                   /   sum( itemSim[KSimilarItemIdx] )
        if ( is.na(normlizedResult$meanOfcol[i]) || is.na(normlizedResult$sdOfcol[i]) ) {
            next
        }
        ratingOfUnRatedItems[i] <- zScoreNormalizationInverse( ratingOfUnRatedItems[i], 
                                                            normlizedResult$meanOfcol[i], 
                                                            normlizedResult$sdOfcol[i] )
    }
    
    ## find the Top-N items
    topnIdx <- apply( matrix(ratingOfUnRatedItems,nrow=1), MARGIN=1, 
                     FUN=function(x) head(  order(x, decreasing=TRUE, na.last=TRUE), n )  )
    topnIdx <- as.vector(topnIdx)
    recommendList <- list(ratingResult = ratingOfUnRatedItems[topnIdx], topnIndex = topnIdx)
    return( recommendList )
}

以上代码中使用到的zScoreNormalization，与zScoreNormalizationInverse函数在文章推荐系统（三）中有给出。

代码与推荐系统（三）中给出的IBCF代码的主要差别是在24-38行使用SVD对评分矩阵进行了分解，并将原始的评分矩阵投影到低维空间，47行在计算物品间相似性时使用了低维矩阵，可以在一定程度上降低计算的复杂度。