日志搜集平台ELK-系统架构及部署

备注:所有软件已经统一放到 http://op.jieyue.com/softdown/ELK/

1.系统架构

2.logstash-安装

软件二进制安装包，解压启动

软件安装路径：/usr/local/logstash

input{
redis {
type => "h3c"
host => "127.0.0.1"
port => 6379
password => "xxxx"
db => 1
data_type => "list"
key => "h3c"
}

redis {
type => "cisco"
host => "127.0.0.1"
port => 6379
password => "xxxx"
db => 4
data_type => "list"
key => "cisco"
}

redis {
type => "netflow"
host => "127.0.0.1"
port => 6379
password => "xxxxx"
db => 2
data_type => "list"
key => "netflow"
}

redis {
type => "sflow"
host => "127.0.0.1"
port => 6379
password => "xxxx"
db => 3
data_type => "list"
key => "sflow"
}

redis {
type => "nginx-log"
host => "127.0.0.1"
port => 6379
password => "xxxxx"
db => 10
data_type => "list"
key => "nginx-log"
}

}

output{
if[type] == "h3c" {
elasticsearch {
hosts=> ["10.50.194.x:9200","10.50.194.x:9200","10.50.194.x:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}

if[type] == "cisco" {
elasticsearch {
hosts=> ["10.50.194.x:9200","10.50.194.x:9200","10.50.194.x:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}

if[type] == "netflow" {
elasticsearch {
hosts=> ["10.50.194.x:9200","10.50.194.x:9200","10.50.194.x:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}

if[type] == "sflow" {
elasticsearch {
hosts=> ["10.50.194.x:9200","10.50.194.x:9200","10.50.194.x:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}

}

3.ES安装

软件二进制安装包，解压启动（3个节点，都负责数据存储和管理，5个分片，2个副本）

软件安装路径：/usr/local/elasticsearch

cat elasticsearch.yml |egrep -v "^#|^$"
cluster.name: cloud-ops
node.name: node-138
path.data: /data/esdata
path.logs: /data/eslogs
network.host: 10.50.194.x
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.50.194.x", "10.50.194.x", "10.50.194.x"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#head 插件安装

1.node install
yum install xz -y
tar -xf node-v6.11.0-linux-x64.tar.xz
vi /etc/profile
export PATH=$PATH:/usr/local/node-v6.11.0-linux-x64/bin

cd /usr/local/
git clone https://github.com/mobz/elasticsearch-head.git
npm install grunt --registry=https://registry.npm.taobao.org
vi Gruntfile.js +92
hostname: '0.0.0.0',
##
cd /usr/local/elasticsearch-head/node_modules
npm install grunt-contrib-clean --registry=https://registry.npm.taobao.org
npm install grunt-contrib-concat --registry=https://registry.npm.taobao.org
npm install grunt-contrib-watch --registry=https://registry.npm.taobao.org
npm install grunt-contrib-connect --registry=https://registry.npm.taobao.org
npm install grunt-contrib-copy --registry=https://registry.npm.taobao.org
npm install grunt-contrib-jasmine --registry=https://registry.npm.taobao.org

#start
cd /usr/local/elasticsearch-head/node_modules/grunt/bin
nohup grunt server &

4.kibana 安装

软件二进制安装包，解压启动

软件位置： /usr/local/kibana

cat kibana.yml |egrep -v "^#|^$"
server.port: 9300
server.host: "10.50.194.x"
elasticsearch.url: "http://10.50.194.x:9200"
kibana.index: ".kibana"
logging.dest: /data/kibana-logs/kibana.log
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

5.服务器信息

IP	用户名		功能
10.50.194.138	jyapp		es
10.50.194.139	jyapp		es
10.50.194.140	jyapp		es
10.50.194.141	jyapp		logstash    redis
10.50.194.142	jyapp		logstash kibana
10.50.194.204	jyapp		logstash、 es-data
10.50.194.205	jyapp		logstash   es-data
10.50.194.161	jyapp		logstash