zoukankan      html  css  js  c++  java
  • Spark 大数据平台

    Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write.

    BDAS, the Berkeley Data Analytics Stack, is an open source software stack that integrates software components being built by the AMPLab to make sense of Big Data.

    Berkeley Data Analytics Stack

    Vision of spark

    Spark Components VS. Hadoop Components
    Spark Core <------> Apache Hadoop MR
    Spark Streaming <------> Apache Storm
    Spark SQL <------> Apache Hive
    Spark GraphX <------> MPI(taobao)
    Spark MLlib <------> Apache Mahout

    BlinkDB is a massively parallel, approximate query engine for running interactive SQL queries on large volumes of data. It allows users to +, enabling interactive queries over massive data by running queries on data samples and presenting results annotated with meaningful error bars.
    Two key ideas:

    • An adaptive optimization framework that builds and maintains a set of multi-dimensional samples from original data over time
    • A dynamic sample selection strategy that selects an appropriately sized sample based on a query’s accuracy and/or response time requirements.

    Why spark is fast:

    • in-memory computing
    • Directed Acyclic Graph (DAG) engine, compiler can see the whole computing graph in advance so that it can optimize it. Delay Scheduling

    Resilient Distributed Dataset

    • A list of partitions
    • A function for computing each split
    • A list of dependencies on other RDDs
    • Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned)
    • Optionally, a list of preferred locations to compute each split on (e.g. block locations for an HDFS file)

    Storage Strategy

    class StorageLevel private(
        private var useDisk_ : Boolean,
        private var useMemory_ : Boolean,
        private var deserialized_ : Boolean,
        private var replication_ : Int = 1)
        
    val MEMORY_ONLY_ = new StorageLevel(false, true, true)
    

    RDD, transformation & action

    lazy evaluation
    transformation and actions

  • 相关阅读:
    20200721训练记录
    20200717训练记录
    打家劫舍III(力扣第337题)
    HBase API的删除数据操作的分析
    相交链表(第160题)
    删除排序链表中的重复元素(第83题)
    合并两个有序链表(力扣第21题)
    删除链表的倒数第N个节点(第19题)
    HBase的架构原理
    回文链表、链表求和(234、445)
  • 原文地址:https://www.cnblogs.com/rainbow203/p/Spark-da-shu-ju-ping-tai.html
Copyright © 2011-2022 走看看