zoukankan      html  css  js  c++  java
  • win7下本地运行spark以及spark.sql.warehouse.dir设置

    1  SparkSession spark = SparkSession
    2                   .builder()
    3                   .master("local[*]")
    4                   .enableHiveSupport()
    5                   .config("spark.sql.warehouse.dir", "file:///E:/git/bigdata_sparkIDE/spark-ide/workspace/functionModule/spark-warehouse")
    6                   .getOrCreate();
    在使用spark2.0的时候遇到如下错误:
    
    16/09/21 14:12:22 INFO SharedState: Warehouse path is 'file:E:scalacode_v2Spark2Pro/spark-warehouse'.
    Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:E:/scalacode_v2/Spark2Pro/spark-warehouse
        at org.apache.hadoop.fs.Path.initialize(Path.java:206)
        at org.apache.hadoop.fs.Path.<init>(Path.java:172)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeQualifiedPath(SessionCatalog.scala:114)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:145)
        at org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(SessionCatalog.scala:89)
        at org.apache.spark.sql.internal.SessionState.catalog$lzycompute(SessionState.scala:95)
        at org.apache.spark.sql.internal.SessionState.catalog(SessionState.scala:95)
        at org.apache.spark.sql.internal.SessionState$$anon$1.<init>(SessionState.scala:112)
        at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:112)
        at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:111)
        at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:161)
        at org.apache.spark.sql.Dataset.<init>(Dataset.scala:167)
        at org.apache.spark.sql.Dataset$.apply(Dataset.scala:59)
        at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:441)
        at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:395)
        at org.apache.spark.sql.SQLImplicits.rddToDatasetHolder(SQLImplicits.scala:163)
        at com.yisa.test.Test$.main(Test.scala:24)
        at com.yisa.test.Test.main(Test.scala)
    Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:E:/scalacode_v2/Spark2Pro/spark-warehouse
        at java.net.URI.checkPath(Unknown Source)
        at java.net.URI.<init>(Unknown Source)
        at org.apache.hadoop.fs.Path.initialize(Path.java:203)
        ... 18 more
    
    
    
    google了一下后发现如下解析:
    
    The default value of `spark.sql.warehouse.dir` is `System.getProperty("user.dir")/spark-warehouse`. Since `System.getProperty("user.dir")` is a local dir, we should explicitly set the scheme to local filesystem.
    就是说我们需要添加一个配置spark.sql.warehouse.dir,如果不添加上该配置,默认是找的user.dir下面的目录。这个其实是没有的。所以报错。
  • 相关阅读:
    【POJ1958】汉诺塔+
    hdu 5067(暴力搜索)
    hdu 5063(思路题-反向操作数组)
    hdu 5062(水题)
    hdu 2227(树状数组+dp)
    hdu 5480(维护前缀和+思路题)
    hdu 2492(树状数组)
    hdu 1394(树状数组)
    poj 2299(离散化+树状数组)
    poj 3321(树状数组)
  • 原文地址:https://www.cnblogs.com/lyy-blog/p/9620294.html
Copyright © 2011-2022 走看看