Spark 逻辑回归LogisticRegression

1、概念

逻辑回归是预测分类相应的常用方法。广义线性回归的一个特例是预测结果的概率。在spark.ml逻辑回归中，可以使用二项逻辑回归来预测二元结果，

或者可以使用多项逻辑回归来预测多类结果。使用该family参数在这两种算法之间选择，或者保持不设置（缺省auto），Spark将推断出正确的变量。

通过将family参数设置为“多项式”，可以将多项逻辑回归用于二进制分类。它将产生两组系数和两个截距.

在分类问题中，我们尝试预测的是结果是否属于某一个类(例如正确或错误)。分类问题的例子有：判断一封电子邮件是否是垃圾邮件；判断一次金融交易是否是欺诈；

2、code，参考地址：https://github.com/asker124143222/spark-demo

package com.home.spark.ml

import org.apache.spark.SparkConf
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
import org.apache.spark.ml.feature.{IndexToString, StringIndexer, VectorIndexer}
import org.apache.spark.ml.linalg.{Vector, Vectors}
import org.apache.spark.sql.{Dataset, Row, SparkSession}

/**
  * @Description: 逻辑回归，二项分类预测
  *
  **/
object Ex_BinomialLogisticRegression {
  def main(args: Array[String]): Unit = {
    val conf = new SparkConf(true).setMaster("local[*]").setAppName("spark ml label")
    val spark = SparkSession.builder().config(conf).getOrCreate()
    //rdd转换成df或者ds需要SparkSession实例的隐式转换
    //导入隐式转换，注意这里的spark不是包名，而是SparkSession的对象名
    import spark.implicits._

    val data = spark.sparkContext.textFile("input/iris.data.txt")
      .map(_.split(","))
      .map(a => Iris(
        Vectors.dense(a(0).toDouble, a(1).toDouble, a(2).toDouble, a(3).toDouble),
        a(4))
      ).toDF()
    data.show()

    data.createOrReplaceTempView("iris")

    val TotalCount = spark.sql("select count(*) from iris")
    println("记录数： " + TotalCount.collect().take(1).mkString)

    //二项预测，由于样本数据有三类数据，排除Iris-setosa
    val df = spark.sql("select * from iris where label!='Iris-setosa'")
    df.map(r => r(1) + " : " + r(0)).collect().take(10).foreach(println)
    println("过滤后的记录数： " + df.count())


    /* VectorIndexer
    提高决策树或随机森林等ML方法的分类效果。
    VectorIndexer是对数据集特征向量中的类别（离散值）特征（index categorical features categorical features ）进行编号。
    它能够自动判断那些特征是离散值型的特征，并对他们进行编号，
    具体做法是通过设置一个maxCategories，特征向量中某一个特征不重复取值个数小于maxCategories，则被重新编号为0～K（K<=maxCategories-1）。
    某一个特征不重复取值个数大于maxCategories，则该特征视为连续值，不会重新编号（不会发生任何改变）
    假设maxCategories=5，那么特征列中非重复取值小于等于5的列将被重新索引
    为了索引的稳定性，规定如果这个特征值为0，则一定会被编号成0，这样可以保证向量的稀疏度
    maxCategories缺省是20
    */
    //对特征列和标签列进行索引转换
    val labelIndexer = new StringIndexer().setInputCol("label").setOutputCol("indexedLabel").fit(df)
    val featureIndexer = new VectorIndexer()
//      .setMaxCategories(5) //设置为5后，由于特征列的非重复值个数都大于5，所以不会发生任何转换，也就没有意义
      .setInputCol("features").setOutputCol("indexedFeatures")
      .fit(df)


    //对原数据集划分训练数据（70%）和测试数据（30%）
    val Array(trainingData, testData): Array[Dataset[Row]] = df.randomSplit(Array(0.7, 0.3))

    /**
      * LR建模
      * setMaxIter设置最大迭代次数(默认100),具体迭代次数可能在不足最大迭代次数停止
      * setTol设置容错(默认1e-6),每次迭代会计算一个误差,误差值随着迭代次数增加而减小,当误差小于设置容错,则停止迭代
      * setRegParam设置正则化项系数(默认0),正则化主要用于防止过拟合现象,如果数据集较小,特征维数又多,易出现过拟合,考虑增大正则化系数
      * setElasticNetParam正则化范式比(默认0),正则化有两种方式:L1(Lasso)和L2(Ridge),L1用于特征的稀疏化,L2用于防止过拟合
      * setLabelCol设置标签列
      * setFeaturesCol设置特征列
      * setPredictionCol设置预测列
      * setThreshold设置二分类阈值
      */
    //设置逻辑回归参数
    val lr = new LogisticRegression().setLabelCol("indexedLabel").setFeaturesCol("indexedFeatures").setFamily()
      .setMaxIter(100).setRegParam(0.3).setElasticNetParam(0.8)

    //转换器，将预测的类别重新转成字符型
    val labelConverter = new IndexToString()
      .setInputCol("prediction")
      .setOutputCol("predectionLabel")
      .setLabels(labelIndexer.labels)


    //建立工作流
    val lrPipeline = new Pipeline().setStages(Array(labelIndexer, featureIndexer, lr, labelConverter))

    //生成模型
    val model = lrPipeline.fit(trainingData)

    //预测
    val result = model.transform(testData)

    //打印结果
    result.show(200, false)

    //模型评估，预测准确性和错误率
    val evaluator = new MulticlassClassificationEvaluator().setLabelCol("indexedLabel").setPredictionCol("prediction")
    val lrAccuracy: Double = evaluator.evaluate(result)

    println("Test Error = " + (1.0 - lrAccuracy))

    spark.stop()
  }
}


case class Iris(features: Vector, label: String)

3、result

+-----------------+---------------+------------+-------------------+--------------------------------------------+----------------------------------------+----------+---------------+
|features         |label          |indexedLabel|indexedFeatures    |rawPrediction                               |probability                             |prediction|predectionLabel|
+-----------------+---------------+------------+-------------------+--------------------------------------------+----------------------------------------+----------+---------------+
|[4.9,2.4,3.3,1.0]|Iris-versicolor|0.0         |[4.9,3.0,3.3,0.0]  |[1.0071037675553336,-1.0071037675553336]    |[0.7324529695042751,0.2675470304957249] |0.0       |Iris-versicolor|
|[5.0,2.0,3.5,1.0]|Iris-versicolor|0.0         |[5.0,0.0,3.5,0.0]  |[0.938177922699384,-0.938177922699384]      |[0.7187314594034615,0.2812685405965385] |0.0       |Iris-versicolor|
|[5.6,2.5,3.9,1.1]|Iris-versicolor|0.0         |[5.6,4.0,3.9,1.0]  |[0.7107814076350716,-0.7107814076350716]    |[0.6705737993354417,0.3294262006645583] |0.0       |Iris-versicolor|
|[5.6,2.9,3.6,1.3]|Iris-versicolor|0.0         |[5.6,8.0,3.6,3.0]  |[0.6350805242141693,-0.6350805242141693]    |[0.6536405613705153,0.3463594386294846] |0.0       |Iris-versicolor|
|[5.8,2.7,4.1,1.0]|Iris-versicolor|0.0         |[5.8,6.0,4.1,0.0]  |[0.7314003881315354,-0.7314003881315354]    |[0.6751125028597408,0.32488749714025916]|0.0       |Iris-versicolor|
|[6.1,2.8,4.7,1.2]|Iris-versicolor|0.0         |[6.1,7.0,4.7,2.0]  |[0.34553320285886,-0.34553320285886]        |[0.5855339747983552,0.41446602520164466]|0.0       |Iris-versicolor|
|[6.2,2.2,4.5,1.5]|Iris-versicolor|0.0         |[6.2,1.0,4.5,5.0]  |[0.14582457165756946,-0.14582457165756946]  |[0.5363916772629104,0.46360832273708963]|0.0       |Iris-versicolor|
|[6.4,2.9,4.3,1.3]|Iris-versicolor|0.0         |[6.4,8.0,4.3,3.0]  |[0.39384006721834597,-0.39384006721834597]  |[0.597206774507057,0.40279322549294305] |0.0       |Iris-versicolor|
|[6.6,3.0,4.4,1.4]|Iris-versicolor|0.0         |[6.6,9.0,4.4,4.0]  |[0.2698323194379575,-0.2698323194379575]    |[0.5670517391689078,0.43294826083109217]|0.0       |Iris-versicolor|
|[6.7,3.0,5.0,1.7]|Iris-versicolor|0.0         |[6.7,9.0,5.0,7.0]  |[-0.20557969118713126,0.20557969118713126]  |[0.44878532413929256,0.5512146758607075]|1.0       |Iris-virginica |
|[6.7,3.1,4.4,1.4]|Iris-versicolor|0.0         |[6.7,10.0,4.4,4.0] |[0.2698323194379575,-0.2698323194379575]    |[0.5670517391689078,0.43294826083109217]|0.0       |Iris-versicolor|
|[7.0,3.2,4.7,1.4]|Iris-versicolor|0.0         |[7.0,11.0,4.7,4.0] |[0.16644355215403328,-0.16644355215403328]  |[0.5415150896404186,0.4584849103595813] |0.0       |Iris-versicolor|
|[4.9,2.5,4.5,1.7]|Iris-virginica |1.0         |[4.9,4.0,4.5,7.0]  |[-0.033265079047257284,0.033265079047257284]|[0.49168449702809164,0.5083155029719083]|1.0       |Iris-virginica |
|[5.4,3.0,4.5,1.5]|Iris-versicolor|0.0         |[5.4,9.0,4.5,5.0]  |[0.14582457165756946,-0.14582457165756946]  |[0.5363916772629104,0.46360832273708963]|0.0       |Iris-versicolor|
|[5.6,2.8,4.9,2.0]|Iris-virginica |1.0         |[5.6,7.0,4.9,10.0] |[-0.43975124481639627,0.43975124481639627]  |[0.39180024423019144,0.6081997557698086]|1.0       |Iris-virginica |
|[5.6,3.0,4.1,1.3]|Iris-versicolor|0.0         |[5.6,9.0,4.1,3.0]  |[0.4627659120742955,-0.4627659120742955]    |[0.6136701219061476,0.38632987809385244]|0.0       |Iris-versicolor|
|[5.8,2.7,3.9,1.2]|Iris-versicolor|0.0         |[5.8,6.0,3.9,2.0]  |[0.6212365822826582,-0.6212365822826582]    |[0.6504997376392441,0.34950026236075604]|0.0       |Iris-versicolor|
|[5.8,2.7,5.1,1.9]|Iris-virginica |1.0         |[5.8,6.0,5.1,9.0]  |[-0.419132264319932,0.419132264319932]      |[0.3967244102962335,0.6032755897037665] |1.0       |Iris-virginica |
|[5.9,3.0,5.1,1.8]|Iris-virginica |1.0         |[5.9,9.0,5.1,8.0]  |[-0.32958743896751885,0.32958743896751885]  |[0.4183410089972438,0.5816589910027563] |1.0       |Iris-virginica |
|[6.0,2.9,4.5,1.5]|Iris-versicolor|0.0         |[6.0,8.0,4.5,5.0]  |[0.14582457165756946,-0.14582457165756946]  |[0.5363916772629104,0.46360832273708963]|0.0       |Iris-versicolor|
|[6.1,3.0,4.6,1.4]|Iris-versicolor|0.0         |[6.1,9.0,4.6,4.0]  |[0.20090647458200817,-0.20090647458200817]  |[0.5500583546439539,0.4499416453560461] |0.0       |Iris-versicolor|
|[6.2,3.4,5.4,2.3]|Iris-virginica |1.0         |[6.2,13.0,5.4,13.0]|[-0.8807003330135101,0.8807003330135101]    |[0.29303267372325625,0.7069673262767437]|1.0       |Iris-virginica |
|[6.7,3.1,4.7,1.5]|Iris-versicolor|0.0         |[6.7,10.0,4.7,5.0] |[0.07689872680162013,-0.07689872680162013]  |[0.5192152136737482,0.48078478632625177]|0.0       |Iris-versicolor|
|[6.7,3.3,5.7,2.5]|Iris-virginica |1.0         |[6.7,12.0,5.7,15.0]|[-1.163178751002261,1.163178751002261]      |[0.23809016943453823,0.7619098305654617]|1.0       |Iris-virginica |
|[6.8,3.0,5.5,2.1]|Iris-virginica |1.0         |[6.8,9.0,5.5,11.0] |[-0.7360736047366578,0.7360736047366578]    |[0.32386333429517283,0.6761366657048272]|1.0       |Iris-virginica |
|[6.9,3.1,5.4,2.1]|Iris-virginica |1.0         |[6.9,10.0,5.4,11.0]|[-0.7016106823086834,0.7016106823086834]    |[0.33145521561995817,0.6685447843800418]|1.0       |Iris-virginica |
|[7.2,3.6,6.1,2.5]|Iris-virginica |1.0         |[7.2,14.0,6.1,15.0]|[-1.3010304407141597,1.3010304407141597]    |[0.21399164655179387,0.7860083534482062]|1.0       |Iris-virginica |
|[7.7,2.8,6.7,2.0]|Iris-virginica |1.0         |[7.7,7.0,6.7,10.0] |[-1.0600838485199424,1.0600838485199424]    |[0.2572934314622856,0.7427065685377143] |1.0       |Iris-virginica |
|[7.7,3.0,6.1,2.3]|Iris-virginica |1.0         |[7.7,9.0,6.1,13.0] |[-1.1219407900093334,1.1219407900093334]    |[0.24565146441425778,0.7543485355857422]|1.0       |Iris-virginica |
|[7.9,3.8,6.4,2.0]|Iris-virginica |1.0         |[7.9,15.0,6.4,10.0]|[-0.9566950812360182,0.9566950812360182]    |[0.2775403823663211,0.7224596176336789] |1.0       |Iris-virginica |
+-----------------+---------------+------------+-------------------+--------------------------------------------+----------------------------------------+----------+---------------+

Test Error = 0.03314285714285714