sparkR的一个运行的例子

   在sparkR在配置完成的基础上，本例采用Spark on yarn模式，介绍sparkR运行的一个例子。

     在spark的安装目录下，/examples/src/main/r，有一个dataframe.R文件。该文件默认是在本地的模式下运行的，不与hdfs交互。可以将脚本进行相应修改，提交到yarn模式下。

      在提交之前，要先将${SPARK_HOME}/examples/src/main/resources/people.json 文件上传到hdfs上，我上传到了hdfs://data-mining-cluster/data/bigdata_mining/sparkr/people.json 目录下。

dataframe.R文件内容：

# Convert local data frame to a SparkR DataFrame
df <- createDataFrame(sqlContext, localDF)

# Print its schema
printSchema(df)
# root
#  |-- name: string (nullable = true)
#  |-- age: double (nullable = true)

# Create a DataFrame from a JSON file
#path <- file.path("hdfs://data-mining-cluster/data/bigdata_mining/sparkr/people.json")
path <- file.path("hdfs://data-mining-cluster/data/bigdata_mining/sparkr/people.json")
peopleDF <- read.json(sqlContext, path)
printSchema(peopleDF)

# Register this DataFrame as a table.
registerTempTable(peopleDF, "people")

# SQL statements can be run by using the sql methods provided by sqlContext
teenagers <- sql(sqlContext, "SELECT name FROM people WHERE age >= 13 AND age <= 19")

# Call collect to get a local data.frame
teenagersLocalDF <- collect(teenagers)

# Print the teenagers in our dataset
print(teenagersLocalDF)

# Stop the SparkContext now
sparkR.stop()

sparkR  --master yarn-client dataframe.R  这样就可以将任务提交到yarn上了。

     另外，有些集群会报如下错误：

      

16/06/16 11:40:35 ERROR RBackendHandler: json on 15 failed
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) :
  java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:185)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:198)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(
Calls: read.json -> callJMethod -> invokeJava

       这种情况一般是由于启用了lzo压缩导致的。可以通过--jars  添加lzo的jar包，就可以了。例如：sparkR  --master yarn-client dataframe.R --jars /usr/local/Hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar