使用weka进行Cross-validation实验

Generating cross-validation folds (Java approach)

文献：

http://weka.wikispaces.com/Generating+cross-validation+folds+%28Java+approach%29

This article describes how to generate train/test splits for cross-validation using the Weka API directly. 

The following variables are given:

 Instances data =  ...;   // contains the full dataset we wann create train/test sets from

 int seed = ...;          // the seed for randomizing the data

 int folds = ...;         // the number of folds to generate, >=2

 Randomize the data

First, randomize your data:

 Random rand = new Random(seed);   // create seeded number generator

 randData = new Instances(data);   // create copy of original data

 randData.randomize(rand);         // randomize data with number generator

In case your data has a nominal class and you wanna perform stratified cross-validation:

 randData.stratify(folds);

 Generate the folds

 Single run

Next thing that we have to do is creating the train and the test set:

 for (int n = 0; n < folds; n++) {

   Instances train = randData.trainCV(folds, n);

   Instances test = randData.testCV(folds, n);

 

   // further processing, classification, etc.

   ...

 }

Note:

• the above code is used by the weka.filters.supervised.instance.StratifiedRemoveFolds filter
• the weka.classifiers.Evaluation class and the Explorer/Experimenter would use this method for obtaining the train set:

 Instances train = randData.trainCV(folds, n, rand);

 Multiple runs

The example above only performs one run of a cross-validation. In case you want to run 10 runs of 10-fold cross-validation, use the following loop:

 Instances data = ...;  // our dataset again, obtained from somewhere

 int runs = 10;

 for (int i = 0; i < runs; i++) {

   seed = i+1;  // every run gets a new, but defined seed value

 

   // see: randomize the data

   ...

 

   // see: generate the folds

   ...

 }

一个简单的小实验：

继续对上一节中的红酒和白酒进行分类。分类器没有变化，只是增加了重复试验过程

package assignment2;

import weka.core.Instances;

import weka.core.converters.ConverterUtils.DataSource;

import weka.core.Utils;

import weka.classifiers.Classifier;

import weka.classifiers.Evaluation;

import weka.classifiers.trees.J48;

import weka.filters.Filter;

import weka.filters.unsupervised.attribute.Remove;

 

import java.io.FileReader;

import java.util.Random;

public class cv_rw {

    public static Instances getFileInstances(String filename) throws Exception{

       FileReader frData =new FileReader(filename);

       Instances data = new Instances(frData);

       int length= data.numAttributes();

       String[] options = new String[2];

       options[0]="-R";

       options[1]=Integer.toString(length);

       Remove remove =new Remove();

       remove.setOptions(options);

       remove.setInputFormat(data);

       Instances newData= Filter.useFilter(data, remove);

       return newData;

    }

    public static void main(String[] args) throws Exception {

        // loads data and set class index

       Instances data = getFileInstances("D://Weka_tutorial//WineQuality//RedWhiteWine.arff");

//     System.out.println(instances);

       data.setClassIndex(data.numAttributes()-1);

 

        // classifier

//      String[] tmpOptions;

//      String classname;

//      tmpOptions     = Utils.splitOptions(Utils.getOption("W", args));

//      classname      = tmpOptions[0];

//      tmpOptions[0]  = "";

//      Classifier cls = (Classifier) Utils.forName(Classifier.class, classname, tmpOptions);

//

//      // other options

//      int runs  = Integer.parseInt(Utils.getOption("r", args));//重复试验

//      int folds = Integer.parseInt(Utils.getOption("x", args));

       int runs=1;

       int folds=10;

       J48 j48= new J48();

//     j48.buildClassifier(instances);

 

        // perform cross-validation

        for (int i = 0; i < runs; i++) {

          // randomize data

          int seed = i + 1;

          Random rand = new Random(seed);

          Instances randData = new Instances(data);

          randData.randomize(rand);

//        if (randData.classAttribute().isNominal())    //没看懂这里什么意思，往高手回复，万分感谢

//          randData.stratify(folds);

 

          Evaluation eval = new Evaluation(randData);

          for (int n = 0; n < folds; n++) {

            Instances train = randData.trainCV(folds, n);

            Instances test = randData.testCV(folds, n);

            // the above code is used by the StratifiedRemoveFolds filter, the

            // code below by the Explorer/Experimenter:

            // Instances train = randData.trainCV(folds, n, rand);

 

            // build and evaluate classifier

            Classifier j48Copy = Classifier.makeCopy(j48);

            j48Copy.buildClassifier(train);

            eval.evaluateModel(j48Copy, test);

          }

 

          // output evaluation

          System.out.println();

          System.out.println("=== Setup run " + (i+1) + " ===");

          System.out.println("Classifier: " + j48.getClass().getName());

          System.out.println("Dataset: " + data.relationName());

          System.out.println("Folds: " + folds);

          System.out.println("Seed: " + seed);

          System.out.println();

          System.out.println(eval.toSummaryString("=== " + folds + "-fold Cross-validation run " + (i+1) + "===", false));

        }

 

    }

}

运行程序得到实验结果：

 

=== Setup run 1 ===

Classifier: weka.classifiers.trees.J48

Dataset: RedWhiteWine-weka.filters.unsupervised.instance.Randomize-S42-weka.filters.unsupervised.instance.Randomize-S42-weka.filters.unsupervised.attribute.Remove-R13

Folds: 10

Seed: 1

 

=== 10-fold Cross-validation run 1===

Correctly Classified Instances        6415               98.7379 %

Incorrectly Classified Instances        82                1.2621 %

Kappa statistic                          0.9658

Mean absolute error                      0.0159

Root mean squared error                  0.1109

Relative absolute error                  4.2898 %

Root relative squared error             25.7448 %

Total Number of Instances             6497