Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark (a new framework for low-latency interactive and iterative jobs), and other applications. Mesos is open source in the Apache Incubator.
You can use Mesos to:
- Run Hadoop, MPI, Spark, and other frameworks on a dynamically shared pool of nodes.
- Run multiple instances of Hadoop on the same cluster to isolate production and experimental jobs, or even multiple versions of Hadoop.
- Run long-lived services (e.g. Hypertable and HBase) on the same nodes as batch applications and share resources between them.
- Build new cluster computing frameworks without reinventing low-level facilities for farming out tasks, and have them coexist with existing ones.
Features
- Fault-tolerant replicated master using ZooKeeper.
- Scalability to 10,000s of nodes using fast, event-driven C++ implementation.
- Isolation between tasks with Linux Containers.
- Multi-resource scheduling (memory and CPU aware).
- Efficient application-controlled scheduling mechanism (resource offers) that lets frameworks achieve their own placement goals (e.g. data locality).
- Java, Python and C++ APIs for developing new parallel applications.
- Web UI for viewing cluster state.
Users
Mesos is being used to manage clusters at Twitter, Conviva, UC Berkeley, and UC San Francisco.
Research Project
Mesos started out as a research project in the UC Berkeley AMP Lab before being open sourced and joining the Apache Incubator. You can find more about the research behind Mesos, including our papers, on our research page.
Apache Incubator Notice
Apache Mesos is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.