elk 平台搭建:
ELK平台搭建
系统环境
System: Centos release 6.7 (Final)
ElasticSearch: 2.1.0
Logstash: 2.1.1
Kibana: 4.3.0
Java: openjdk version "1.8.0_65"
注：由于Logstash的运行依赖于Java环境， 而Logstash 1.5以上版本不低于java 1.7，因此推荐使用最新版本的Java。因为我们只需要Java的运行环境，所以可以只安装JRE，不过这里我依然使用JDK，请自行搜索安装。
配置ElasticSearch：
tar -zxvf elasticsearch-2.1.0.tar.gz
cd elasticsearch-2.1.0
zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/plugin install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Plugins directory [/usr/local/elasticsearch-2.3.4/plugins] does not exist. Creating...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip ...
Downloading .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
Installed head into /usr/local/elasticsearch-2.3.4/plugins/head
然后编辑ES的配置文件：
vi config/elasticsearch.yml
添加配置:
cluster.name=es_cluster
node.name=node0
path.data=/tmp/elasticsearch/data
path.logs=/tmp/elasticsearch/logs
#当前hostname或IP，我这里是centos2
network.host=192.168.32.80
network.port=9200
启动报错:
zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/elasticsearch
Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: ElasticsearchParseException[malformed, expected settings to start with 'object', instead was [VALUE_STRING]];
Likely root cause: ElasticsearchParseException[malformed, expected settings to start with 'object', instead was [VALUE_STRING]]
at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:65)
at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:107)
at org.elasticsearch.common.cli.CliTool.<init>(CliTool.java:100)
at org.elasticsearch.bootstrap.BootstrapCLIParser.<init>(BootstrapCLIParser.java:48)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:226)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
原因 需要这么写:
cluster.name: es_cluster
node.name: node01
path.data: /tmp/elasticsearch/data
path.logs: /tmp/elasticsearch/logs
network.host: 192.168.32.80
network.port: 9200
zjtest7-redis:/usr/local/elasticsearch-2.3.4# ./bin/elasticsearch
Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.
创建 elk 用户和group:
[elk@zjtest7-redis local]$ cd elasticsearch-2.3.4
[elk@zjtest7-redis elasticsearch-2.3.4]$ ./bin/elasticsearch
[2016-07-22 14:03:22,797][WARN ][bootstrap ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
[2016-07-22 14:03:23,728][INFO ][node ] [node01] version[2.3.4], pid[1756], build[e455fd0/2016-06-30T11:24:31Z]
[2016-07-22 14:03:23,728][INFO ][node ] [node01] initializing ...
[2016-07-22 14:03:25,787][INFO ][plugins ] [node01] modules [reindex, lang-expression, lang-groovy], plugins [head], sites [head]
[2016-07-22 14:03:25,878][INFO ][env ] [node01] using [1] data paths, mounts [[/ (/dev/mapper/vg00-lv_root)]], net usable_space [86.7gb], net total_space [96.2gb], spins? [possibly], types [ext4]
[2016-07-22 14:03:25,882][INFO ][env ] [node01] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-07-22 14:03:25,882][WARN ][env ] [node01] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-07-22 14:03:33,299][INFO ][node ] [node01] initialized
[2016-07-22 14:03:33,302][INFO ][node ] [node01] starting ...
[2016-07-22 14:03:33,561][INFO ][transport ] [node01] publish_address {192.168.32.80:9300}, bound_addresses {192.168.32.80:9300}
[2016-07-22 14:03:33,576][INFO ][discovery ] [node01] es_cluster/gYNVInstR16CujdeJ1T6YQ
[2016-07-22 14:03:36,717][INFO ][cluster.service ] [node01] new_master {node01}{gYNVInstR16CujdeJ1T6YQ}{192.168.32.80}{192.168.32.80:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-07-22 14:03:36,759][INFO ][http ] [node01] publish_address {192.168.32.80:9200}, bound_addresses {192.168.32.80:9200}
[2016-07-22 14:03:36,759][INFO ][node ] [node01] started
[2016-07-22 14:03:36,867][INFO ][gateway ] [node01] recovered [0] indices into cluster_state
可以看到，它跟其他的节点的传输端口为9300，接受HTTP请求的端口为9200。
使用ctrl+C停止。当然，也可以使用后台进程的方式启动ES：
./bin/elasticsearch &
然后可以打开页面localhost:9200，将会看到以下内容：
http://192.168.32.80:9200/
返回；
{
"name" : "node01",
"cluster_name" : "es_cluster",
"version" : {
"number" : "2.3.4",
"build_hash" : "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",
"build_timestamp" : "2016-06-30T11:24:31Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
返回展示了配置的cluster_name和name，以及安装的ES的版本等信息。
刚刚安装的head插件，它是一个用浏览器跟ES集群交互的插件，可以查看集群状态、集群的doc内容、执行搜索和普通的Rest请求等。
现在也可以使用它打开localhost:9200/_plugin/head页面来查看ES集群状态：
Logstash
Logstash的功能如下：
[elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash
No command given
Usage: logstash <command> [command args]
Run a command with the --help flag to see the arguments.
For example: logstash agent --help
Available commands:
agent - runs the logstash agent
version - emits version info about this logstash
[elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.conf
Settings: Default pipeline workers: 1
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Pipeline main started
[elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.conf &
输入和输出日志:
代码如下：
input {
file {
type => "nginx_access"
path => ["/usr/share/nginx/logs/test.access.log"]
}
}
output {
redis {
host => "localhost"
data_type => "list"
key => "logstash:redis"
}
}
output {
# For detail config for elasticsearch as output,
# See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
elasticsearch {
action => "index" #The operation on ES
hosts => "192.168.32.80:9200" #ElasticSearch host, can be array.
index => "applog" #The index to write data to.
}
}
logstash,elasticsearch,kibana 怎么进行nginx的日志分析呢？首先，架构方面，nginx是有日志文件的，它的每个请求的状态等都有日志文件进行记录。其次，需要有个队列，redis的list结构正好可以作为队列使用。然后分析使用elasticsearch就可以进行分析和查询了。
我们需要的是一个分布式的，日志收集和分析系统。logstash有agent和indexer两个角色。对于agent角色，放在单独的web机器上面，然后这个agent不断地读取nginx的日志文件，每当它读到新的日志信息以后，就将日志传送到网络上的一台redis队列上。对于队列上的这些未处理的日志，有不同的几台logstash indexer进行接收和分析。分析之后存储到elasticsearch进行搜索分析。再由统一的kibana进行日志web界面的展示。
启动/usr/local/elasticsearch-2.3.4# ./bin/elasticsearch
[elk@zjtest7-redis logstash-2.3.4]$ ./bin/logstash agent -f config/log4j_to_es.conf
logstash日志收集分发到elastic集群，
elasticsearch进行数据索引，
kibana进行结构化查询展示，redis做缓存队列。
装：
logstash，新版下载即可用，需加配置文件agent.conf和index.conf，当然最好途径是官网sample即得即用。
/usr/local/redis/bin/redis-server *:6379
input {
file {
type => "nginx_access"
path => ["/usr/local/nginx/logs/test.access.log"]
}
}
output {
redis {
host => "localhost"
data_type => "list"
key => "logstash:redis"
port=>6379
password => "1234567"
}
}
~
~
zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_agent.conf
Settings: Default pipeline workers: 1
Pipeline main started
日志格式:
log_format logstash '$http_host $server_addr $remote_addr [$time_local] "$request" '
'$request_body $status $body_bytes_sent "$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time';
redis :
127.0.0.1:6379> llen "logstash:redis"
(integer) 328
到这里数据已经存放到redis队列:
logstash 数据写入到redis 队列:
zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_agent.conf ### 存数据
zjtest7-redis:/usr/local/logstash-2.3.4# ./bin/logstash -f /usr/local/logstash-2.3.4/config/logstash_indexer.conf
logstash分为 index和aget ，agent负责监控、过滤日志，
index负责收集日志并将日志交给ElasticSearch 做搜索
Shipper：发送事件（events）至LogStash；
通常，远程代理端（agent）只需要运行这个组件即可；
Broker and Indexer：接收并索引化事件；
Search and Storage：允许对事件进行搜索和存储；Web Interface：
基于Web的展示界面
192.168.32.162为日志查看服务器，该机器需要安装redis, elasticserch, logstatsh, kibana这四个应用程序。
192.168.32.161?为应用程序nginx应用程序，我们这次只收集他的日志进行分析。
这个配置文件,是读取nginx日志写入到redis
zjtest7-redis:/usr/local/logstash-2.3.4/config# cat logstash_agent.conf
input {
file {
type => "nginx_access"
path => ["/usr/local/nginx/logs/test.access.log"]
}
}
output {
redis {
host => "localhost"
data_type => "list"
key => "logstash:redis"
port=>"6379"
password => "1234567"
}
}
这个配置文件是读取本地的redis数据,交给elasticsearch
zjtest7-redis:/usr/local/logstash-2.3.4/config# cat logstash_indexer.conf
input {
redis {
host => "localhost"
data_type => "list"
key => "logstash:redis"
type => "redis-input"
password => "1234567"
port =>"6379"
}
}
output {
elasticsearch {
embedded => false
protocol => "http"
host => "localhost"
port => "9200"
index => "access-%{+YYYY.MM.dd}"
document_type="access"
}
stdout {
codec => rubydebug
}
}
Kibana
配置Kibana:
tar -zxvf kibana-4.3.0-linux-x86.tar.gz
cd kibana-4.3.0-linux-x86
vi config/kibana.yml
server.port: 5601
server.host: "192.168.32.80"
elasticsearch.url: http://192.168.32.80:9200
kibana.index:".kibana'