zoukankan      html  css  js  c++  java
  • Amazon Redshift and Massively Parellel Processing

    Today, Yelp held a tech talk in Columbia University about the data warehouse adopted by Yelp.

    Yelp used Amazon Redshift as data warehouse.

    There are several features for Redshift:

    1. Massively Parellel Processing

    2. SQL access

    3. Column-based Datastore

    Benefits are:

    1. Data is structured, accessible and well documented.
    2. Architecture allows for easy extensibility and sharing across teams.
    3. Allows use of entire SQL-compatible tool ecosystem.

    Details:

    Massively Parellel Processing (MMP)

    Traditional BigData always uses Hadoop + MapReduce. MapReduce's native control mechanism is Java code (to implement the Map and Reduce logic), whereas MPP products are queried with SQL(Structural Query Language). You can refer detail here.

    Below is the structure for implementing MMP.

    Similarly, Data is distributed across each segment database to achieve data and processing parallelism. This is achieved by creating a database table with DISTRIBUTED BY clause. By using this clause data is automatically distributed across segment databases. (referrence: Introduction to MMP)

    Typical query sentence in MMP

    Column-based Datastore

    Enables sparse table definitions
    Enables compact storage
    Improve scanning/filtering

    (Benefits: wiki)

    Column-based Datastore

    1. Column-oriented organizations are more efficient when an aggregate needs to be computed over many rows but only for a notably smaller subset of all columns of data, because reading that smaller subset of data can be faster than reading all data.
    2. Column-oriented organizations are more efficient when new values of a column are supplied for all rows at once, because that column data can be written efficiently and replace old column data without touching any other columns for the rows.
    3. Row-oriented organizations are more efficient when many columns of a single row are required at the same time, and when row-size is relatively small, as the entire row can be retrieved with a single disk seek.
    4. Row-oriented organizations are more efficient when writing a new row if all of the row data is supplied at the same time, as the entire row can be written with a single disk seek.

    In practice, row-oriented storage layouts are well-suited for OLTP-like workloads which are more heavily loaded with interactive transactions. Column-oriented storage layouts are well-suited for OLAP-like workloads (e.g., data warehouses) which typically involve a smaller number of highly complex queries over all data (possibly terabytes).

  • 相关阅读:
    【流量劫持】SSLStrip 终极版 —— location 瞒天过海
    【流量劫持】沉默中的狂怒 —— Cookie 大喷发
    【流量劫持】SSLStrip 的未来 —— HTTPS 前端劫持
    Web 前端攻防(2014版)
    流量劫持 —— 浮层登录框的隐患
    流量劫持能有多大危害?
    流量劫持是如何产生的?
    XSS 前端防火墙 —— 整装待发
    XSS 前端防火墙 —— 天衣无缝的防护
    XSS 前端防火墙 —— 无懈可击的钩子
  • 原文地址:https://www.cnblogs.com/ireneyanglan/p/4856666.html
Copyright © 2011-2022 走看看