zoukankan      html  css  js  c++  java
  • sparkR读取csv文件

    sparkR读取csv文件

    The general method for creating SparkDataFrames from data sources is read.df. This method takes in the path for the file to load and the type of data source, and the currently active SparkSession will be used automatically. SparkR supports reading JSON, CSV and Parquet files natively, and through packages available from sources like Third Party Projects, you can find data source connectors for popular file formats like Avro. These packages can either be added by specifying --packages with spark-submit or sparkR commands, or if initializing SparkSession with sparkPackages parameter when in an interactive R shell or from RStudio.

    http://spark.apache.org/docs/latest/sparkr.html

    那spark-csv_2.11-1.4.0.jar包并不是一个R包,不需要安装,在我们的机器没有网的情况下,你下载的jar包根本不知道要放置在哪里?然后我通过在有网的环境下下载并使用该jar包,得知应该放在如下路径:

    (1)   你的R用户的工作目录下的一个子目录下,如:

    /home/summer/.ivy2/cache/com.databricks/spark-csv_2.11/jars/spark-csv_2.11-1.4.0.jar

    (2)   /root/.ivy2/cache/com.databricks/spark-csv_2.11/jars/spark-csv_2.11-1.4.0.jar

    注意安装的scala版本与上面的jar包的对应,此处scala应为2.11版本。

    .// bin/spark-shell

    bin/spark-shell --packages com.databricks:spark-csv_2.11:1.4.0

    =====================================

    .libPaths(c(file.path(Sys.getenv('SPARK_HOME'), 'R', 'lib'), .libPaths()))
    library(SparkR)

     Sys.setenv('SPARKR_SUBMIT_ARGS'='"--packages" "com.databricks:spark-csv_2.11:1.4.0" "sparkr-shell"')

    sc <- sparkR.init(master="local[*]",sparkPackages=”com.databricks:spark-csv_2.11:1.4.0”, sparkEnvir = list(spark.driver.memory="2g"))
    sqlContext <- sparkRSQL.init(sc)

    setwd(“~/hgData”)
    hgdata<-read.csv(sqlContext ,"db1014.csv",header = TRUE,colClasses=list('character','character','character','character','character','character','numeric','Date'))
  • 相关阅读:
    内网穿透之 ssh 转发
    如何解决pytorch 编译时CUDA版本与运行时CUDA版本不对应
    如何监控GPU使用情况并杀死指定其中进程
    Screen 常用命令+VNC 启动停止命令总结
    C++快速文件输入输出
    Pycharm使用宝典
    Docker软件安装系列。
    Docker自动清理日志
    consul集群搭建说明
    jenkins搭建可以支持【FTP】上传功能的项目
  • 原文地址:https://www.cnblogs.com/llphhl/p/6102969.html
Copyright © 2011-2022 走看看