zoukankan      html  css  js  c++  java
  • spark优化之临时目录

    官方是这样说的:

    Directory to use for "scratch" space in Spark, including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks. NOTE: In Spark 1.0 and later this will be overriden by SPARK_LOCAL_DIRS (Standalone, Mesos) or LOCAL_DIRS (YARN) environment variables set by the cluster manager.
    

    大概就是MAP和RDD的临时数据可能存放在spark.local.dir(SPARK_LOCAL_DIRS)指定的目录中,默认是在/tmp中。

    该目录的磁盘一定是快速的本地磁盘系统,我们可以考虑把这个目录来单独放成一个磁盘,或者以都好隔开放入多个磁盘,来提高i/o读写速度

  • 相关阅读:
    IOC
    软件问题
    POJO和JavaBean
    tail命令
    实现质数遍历并输出所需时间
    完数
    break、continue
    *各种形状
    for、while、do-while
    jenkins实现maven项目自动化部署tomcat
  • 原文地址:https://www.cnblogs.com/hark0623/p/4512506.html
Copyright © 2011-2022 走看看