zoukankan      html  css  js  c++  java
  • win7下本地运行spark以及spark.sql.warehouse.dir设置

    1  SparkSession spark = SparkSession
    2                   .builder()
    3                   .master("local[*]")
    4                   .enableHiveSupport()
    5                   .config("spark.sql.warehouse.dir", "file:///E:/git/bigdata_sparkIDE/spark-ide/workspace/functionModule/spark-warehouse")
    6                   .getOrCreate();
    在使用spark2.0的时候遇到如下错误:
    
    16/09/21 14:12:22 INFO SharedState: Warehouse path is 'file:E:scalacode_v2Spark2Pro/spark-warehouse'.
    Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:E:/scalacode_v2/Spark2Pro/spark-warehouse
        at org.apache.hadoop.fs.Path.initialize(Path.java:206)
        at org.apache.hadoop.fs.Path.<init>(Path.java:172)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeQualifiedPath(SessionCatalog.scala:114)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:145)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(SessionCatalog.scala:89)
        at org.apache.spark.sql.internal.SessionState.catalog$lzycompute(SessionState.scala:95)
        at org.apache.spark.sql.internal.SessionState.catalog(SessionState.scala:95)
        at org.apache.spark.sql.internal.SessionState$$anon$1.<init>(SessionState.scala:112)
        at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:112)
        at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:111)
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:161)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:167)
        at org.apache.spark.sql.Dataset$.apply(Dataset.scala:59)
        at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:441)
        at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:395)
        at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
        at com.yisa.test.Test$.main(Test.scala:24)
        at com.yisa.test.Test.main(Test.scala)
    Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:E:/scalacode_v2/Spark2Pro/spark-warehouse
        at java.net.URI.checkPath(Unknown Source)
        at java.net.URI.<init>(Unknown Source)
        at org.apache.hadoop.fs.Path.initialize(Path.java:203)
        ... 18 more
    
    
    
    google了一下后发现如下解析:
    
    The default value of `spark.sql.warehouse.dir` is `System.getProperty("user.dir")/spark-warehouse`. Since `System.getProperty("user.dir")` is a local dir, we should explicitly set the scheme to local filesystem.
    就是说我们需要添加一个配置spark.sql.warehouse.dir,如果不添加上该配置,默认是找的user.dir下面的目录。这个其实是没有的。所以报错。
  • 相关阅读:
    bzoj1415 NOI2005聪聪和可可
    Tyvj1952 Easy
    poj2096 Collecting Bugs
    COGS 1489玩纸牌
    COGS1487 麻球繁衍
    cf 261B.Maxim and Restaurant
    cf 223B.Two Strings
    cf 609E.Minimum spanning tree for each edge
    cf 187B.AlgoRace
    cf 760B.Frodo and pillows
  • 原文地址:https://www.cnblogs.com/lyy-blog/p/9620294.html
Copyright © 2011-2022 走看看