zoukankan      html  css  js  c++  java
  • Spark 大数据平台

    Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write.

    BDAS, the Berkeley Data Analytics Stack, is an open source software stack that integrates software components being built by the AMPLab to make sense of Big Data.

    Berkeley Data Analytics Stack

    Vision of spark

    Spark Components VS. Hadoop Components
    Spark Core <------> Apache Hadoop MR
    Spark Streaming <------> Apache Storm
    Spark SQL <------> Apache Hive
    Spark GraphX <------> MPI(taobao)
    Spark MLlib <------> Apache Mahout

    BlinkDB is a massively parallel, approximate query engine for running interactive SQL queries on large volumes of data. It allows users to +, enabling interactive queries over massive data by running queries on data samples and presenting results annotated with meaningful error bars.
    Two key ideas:

    • An adaptive optimization framework that builds and maintains a set of multi-dimensional samples from original data over time
    • A dynamic sample selection strategy that selects an appropriately sized sample based on a query’s accuracy and/or response time requirements.

    Why spark is fast:

    • in-memory computing
    • Directed Acyclic Graph (DAG) engine, compiler can see the whole computing graph in advance so that it can optimize it. Delay Scheduling

    Resilient Distributed Dataset

    • A list of partitions
    • A function for computing each split
    • A list of dependencies on other RDDs
    • Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)
    • Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)

    Storage Strategy

    class StorageLevel private(
        private var useDisk_ : Boolean,
        private var useMemory_ : Boolean,
        private var deserialized_ : Boolean,
        private var replication_ : Int = 1)
        
    val MEMORY_ONLY_ = new StorageLevel(false, true, true)
    

    RDD, transformation & action

    lazy evaluation
    transformation and actions

  • 相关阅读:
    一些恶作剧的vbs程序代码
    The Weather
    UEditor (富文本编译器)
    Redis 存储数组
    QQ第三方登录(完结篇)
    QQ第三方登录(预备节)
    谷歌浏览器添加JSON-Handle插件
    Nginx 之防盗链配置
    noi,ac第五场部分题解 By cellur925
    NOIp2002神经网络 【拓扑排序】By cellur925
  • 原文地址:https://www.cnblogs.com/rainbow203/p/Spark-da-shu-ju-ping-tai.html
Copyright © 2011-2022 走看看