实时数据分析Realtime data analysis frameworks (or stream system)

最近的工作中涉及要设计一个系统可以实时的监控系统的状态，比如hadoop任务的执行情况，服务器的健康等。这个系统需要实时的处理对象产生的信息，并发送给用户。

这个系统显然需要具备如下特性：

1. 可靠性
2. 大数据处理
3. 实时性
   

显然这将是一个基于Hadoop上的项目，目前可供参考的有

Kafka: Kafka is a messaging system that was originally developed at LinkedIn to serve as the foundation for LinkedIn’s activity stream processing pipeline. Nice talk

S4: S4 is a general-purpose, distributed, scalable, partially fault-tolerant, pluggable platform that allows programmers to easily develop applications for processing continuous unbounded streams of data.

Hedwig: Hedwig is a publish-subscribe system designed to carry large amounts of data across the internet in a guaranteed-delivery fashion from those who produce it (publishers) to those who are interested in it (subscribers).

Storm: Storm is a distributed, reliable, and fault-tolerant stream processing system. Its use cases are so broad that we consider it to be a fundamental new primitive for data processing. Introduction slide

Flume: Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. Its main goal is to deliver data from applications to Apache Hadoop’s HDFS.

Scribe: Scribe is a server for aggregating streaming log data. It is designed to scale to a very large number of nodes and be robust to network and node failures.

随着项目的跟进，我会继续更新。