zoukankan      html  css  js  c++  java
  • A Realtime BigData Dashboad

    Purpose

    Reference and mock the course practice, http://dblab.xmu.edu.cn/post/8274/

    Its business flow is as below picture.

    Inspired by this project,  and reuse the technologies to implement an feature:

    (1) scrawl information from the URL

    https://en.wikipedia.org/wiki/COVID-19_pandemic

    (2) leverage spark streaming to count words times(words stats).

    (3) push word stats to browser to display the imediate info to end user.

    Take an exercise for bigdata realtime streaming app for improving competence.

    https://github.com/fanqingsong/Realtime-BigData-Dashboard

    A demo for realtime dashboard, based on bigdata technology and popular realtime comunication web technology.

    Architect

    • scrawler.py ---> kafka
    • kafka ---> wordCounter.py
    • wordCounter.py ---> kafka
    • kafka ---> app.py
    • app.py ---> browser

    Technology

    bigdata techs:

    • kafka -- tranfer all data between components
    • spark streaming -- data statistics
    • scrawler -- get raw data from url.

     

    web techs:

    • flask -- python web framework
    • socket.io -- frontend/backend data exchange tunnel
    • vue -- popular frontend JS framework

    Install & Run

    reference:

    https://github.com/fanqingsong/Realtime-BigData-Dashboard

    Demo

    run srawler app, then go to browser to see the imediate statistics.

    python3 backend/scrawler.py

  • 相关阅读:
    CentOS 7 虚拟机的安装
    2 MySQL rpm
    01-在实体类上加了lombok的@Data注解
    02-myBatisPlus的wrapper接口的使用
    2 MySQL rpm 安装 --下载
    1-MySQL介绍
    MySQL的不归路
    电脑型号4 1500 内存大 机械大硬盘
    电脑型号3 1200 大硬盘
    电脑概览 2 1200 固态SSD
  • 原文地址:https://www.cnblogs.com/lightsong/p/13285226.html
Copyright © 2011-2022 走看看