基于MaxMind的GeoIP数据库统计Nginx客户端IP所在城市

　　　　　　基于MaxMind的GeoIP数据库统计Nginx客户端IP所在城市

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

版权声明：原创作品，谢绝转载！否则将追究法律责任。

　　MaxMind是IP智能和在线欺诈预防工具的领先提供商。我们可以通过该公司的GeoIP数据库来统计Nginx访问日志中记录的客户端IP来自哪个国家及城市。

一.安装GeoIP数据库

1>.访问Maxmind官网

　　官网地址:
　　　　https://www.maxmind.com/en/home

2>.查看GeoIP2数据库产品

　　GeoIP2链接:
　　　　https://dev.maxmind.com/geoip/geoip2/geolite2/

3>.如下图所示，点击"Download_Access"

4>.自行注册一个账号

　　注册链接:
　　　　https://www.maxmind.com/en/geolite2/signup

5>.下载"GeoLite2 City"版本

6>.下载地址库文件并解压

[root@es103.yinzhengjie.com ~]# ls
GeoLite2-City_20200616.tar.gz  logstash-6.8.9.deb  mysql-connector-java_8.0.20-1ubuntu18.04_all.deb
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# gunzip GeoLite2-City_20200616.tar.gz 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ls
GeoLite2-City_20200616.tar  logstash-6.8.9.deb  mysql-connector-java_8.0.20-1ubuntu18.04_all.deb
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# 

[root@es103.yinzhengjie.com ~]# ls
GeoLite2-City_20200616.tar  logstash-6.8.9.deb  mysql-connector-java_8.0.20-1ubuntu18.04_all.deb
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# tar -xf GeoLite2-City_20200616.tar -C /etc/logstash/
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ll /etc/logstash/
total 52
drwxrwxr-x  4 root root 4096 Jun 22 23:25 ./
drwxr-xr-x 93 root root 4096 Jun 22 20:56 ../
drwxrwxr-x  2 root root 4096 Jun 22 23:18 conf.d/
drwxrwxr-x  2 root root 4096 Jun 16 11:19 GeoLite2-City_20200616/
-rw-r--r--  1 root root 1915 May  4 18:23 jvm.options
-rw-r--r--  1 root root 4568 May  4 18:23 log4j2.properties
-rw-r--r--  1 root root  342 May  4 18:23 logstash-sample.conf
-rw-r--r--  1 root root 8435 Jun 22 04:03 logstash.yml
-rw-r--r--  1 root root  285 May  4 18:23 pipelines.yml
-rw-------  1 root root 1696 May  4 18:23 startup.options
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# ll /etc/logstash/GeoLite2-City_20200616/
total 61676
drwxrwxr-x 2 root root     4096 Jun 16 11:19 ./
drwxrwxr-x 4 root root     4096 Jun 22 23:25 ../
-rw-r--r-- 1 root root       55 Jun 16 11:19 COPYRIGHT.txt
-rw-r--r-- 1 root root 63135716 Jun 16 11:19 GeoLite2-City.mmdb
-rw-r--r-- 1 root root      398 Jun 16 11:19 LICENSE.txt
-rw-r--r-- 1 root root      116 Jun 16 11:19 README.txt
[root@es103.yinzhengjie.com ~]# 

二.为logstash添加filter组件(使用geoip插件)

1>.启动nginx服务并使用filebeat收集日志到logstash

[root@es103.yinzhengjie.com ~]# egrep -v "^*#|^$" /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/syslog
  fields:
    host: "172.200.5.103"
    type: "filebeat-syslog-172-200-5-103"
    app: "ubuntu-syslog"
- type: log
  enable: true
  paths:
    - /var/log/nginx/access.log
  fields:
    host: "172.200.5.103"
    type: "filebeat-nginx-accesslog-172-200-5-103"
    app: "nginx"
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  
output.logstash:
  hosts: ["logstash105.yinzhengjie.com:8888"]
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
[root@es103.yinzhengjie.com ~]# 

2>.logstash将多个filebeats日志聚合并写入到redis集群

[root@logstash105.yinzhengjie.com ~]# vim /etc/logstash/conf.d/beats-to-redis.conf 
[root@logstash105.yinzhengjie.com ~]# 
[root@logstash105.yinzhengjie.com ~]# cat /etc/logstash/conf.d/beats-to-redis.conf 
input {
    beats {
        host  => "logstash105.yinzhengjie.com"
        port  => 8888
        codec => "json"
    }
}

output {
    if [fields][app] == "ubuntu-syslog" {
        redis {
            host => "redis104.yinzhengjie.com"
            port => "6379"
            password => "yinzhengjie"
            db => "0"
            key => "filebeat-syslog-172-200-5-103"
            data_type => "list"
            codec => "json"
        }
    }

    if [fields][app] == "nginx" {
        redis {
            host => "redis104.yinzhengjie.com"
            port => "6379"
            password => "yinzhengjie"
            db => "0"
            key => "filebeat-nginx-172-200-5-103"
            data_type => "list"
            codec => "json"
        }
    }
}
[root@logstash105.yinzhengjie.com ~]# 

3>.使用logstash将redis数据写入到elasticsearch集群并引入GeoIP数据库(并在kibana界面创建索引并查看日志信息是否有"geoip"字段，如下图所示)

[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/redis-to-elasticsearch.conf 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/redis-to-elasticsearch.conf 
input {
    redis {
        host => "redis104.yinzhengjie.com"
        port => "6379"
        password => "yinzhengjie"
        db => "0"
        key => "filebeat-syslog-172-200-5-103"
        data_type => "list"
        #codec => "json"
    }

    redis {
        host => "redis104.yinzhengjie.com"
        port => "6379"
        password => "yinzhengjie"
        db => "0"
        key => "filebeat-nginx-172-200-5-103"
        data_type => "list"
        codec => "json"
    }
}

filter {
    if [fields][app] == "nginx" {
        geoip {
            source => "clientip"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City_20200616/GeoLite2-City.mmdb"
            add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
            add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
        }
    }
}

output {
    if [fields][app] == "ubuntu-syslog" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200"]
            index => "yinzhengjie-logstash-syslog-%{+YYYY.MM.dd}"
        }
    }

    if [fields][app] == "nginx" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200"]
            index => "yinzhengjie-logstash-nginx-%{+YYYY.MM.dd}"
        }
    }
}
[root@es103.yinzhengjie.com ~]# 

 

三.新键可视化坐标地图实操案例(配置kibana显示nginx客户端的IP地址所在区域)

1>.如下图所示,依次点击"可视化" ---> "+"

2>.选择"坐标地图"

3>.选择索引

4>.添加索引必须是以logstash开头，否则会添加失败，如下图所示

[root@es103.yinzhengjie.com ~]# vim /etc/logstash/conf.d/redis-to-elasticsearch.conf 
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# cat /etc/logstash/conf.d/redis-to-elasticsearch.conf 
input {
    redis {
        host => "redis104.yinzhengjie.com"
        port => "6379"
        password => "yinzhengjie"
        db => "0"
        key => "filebeat-syslog-172-200-5-103"
        data_type => "list"
        #codec => "json"
    }

    redis {
        host => "redis104.yinzhengjie.com"
        port => "6379"
        password => "yinzhengjie"
        db => "0"
        key => "filebeat-nginx-172-200-5-103"
        data_type => "list"
        codec => "json"
    }
}

filter {
    if [fields][app] == "nginx" {
        geoip {
            source => "clientip"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City_20200616/GeoLite2-City.mmdb"
            add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
            add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
        }
    }
}

output {
    if [fields][app] == "ubuntu-syslog" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200"]
            index => "logstash-yinzhengjie-syslog-%{+YYYY.MM.dd}"
        }
    }

    if [fields][app] == "nginx" {
        elasticsearch {
            hosts => ["http://es101.yinzhengjie.com:9200","http://es102.yinzhengjie.com:9200"]
            index => "logstash-yinzhengjie-nginx-%{+YYYY.MM.dd}"
        }
    }
}
[root@es103.yinzhengjie.com ~]# 
[root@es103.yinzhengjie.com ~]# systemctl restart logstash.service 
[root@es103.yinzhengjie.com ~]# 

5>.通过视图查看数据

6>.保存视图

7>.视图保存成功