CentOS ELK5.0日志分析平台搭建

之前我们在Windows平台上安装过Elasticsearch+X-Pack+Kibana工具（具体参考：Windows 安装Elasticsearch&Kibana&X-Pack），这里我们在Linux系统中做一个日志分析平台。

一.安装Elasticsearch

#wget -c https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.0.tar.gz
#tar -zxvf elasticsearch-5.4.0.tar.gz
#mkdir  /usr/elk
#mv elasticsearch-5.4.0 /usr/elk/elasticsearch
#cd /usr/elk/elasticsearch/bin

然后我们可以使用如下命令启动

./elasticsearch

启动过程中可能遇到如下问题

①.jvm内存不足

解决方法，修改如下配置文件，调整-xms2g -Xmx2g为-xms1g -Xmx1g

vim /usr/elk/elasticsearch/config/jvm.options

②.要求openjdk版本至少1.8，oracle jdk 1.7

这个时候需要升级openjdk或者使用oracle jdk替换

#查询本地安装的数据库,并且将其删除

#rpm -qa | grep jdk  
java-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.x86_64
java-1.7.0-openjdk-1.7.0.0-1.32.1.11.1.el6.x86_64 

#yum remove java-1.6.0-openjdk-1.6.0.0-1.45.1.11.1.el6.x86_64 
#yum remove java-1.7.0-openjdk-1.7.0.0-1.32.1.11.1.el6.x86_64

#当然，如果你觉得上述操作麻烦，建议直接使用如下方式

#yum remove java-1.6.0-openjdk*
#yum remove java-1.7.0-openjdk*

#然后通过如下方式检索java-1.8.0-openjdk

#yum search java-1.8.0-openjdk

============== N/S Matched: java-1.8.0-openjdk ==============
java-1.8.0-openjdk.x86_64 : OpenJDK Runtime Environment
java-1.8.0-openjdk-debug.x86_64 : OpenJDK Runtime Environment with full debug on
java-1.8.0-openjdk-demo.x86_64 : OpenJDK Demos
java-1.8.0-openjdk-demo-debug.x86_64 : OpenJDK Demos with full debug on
java-1.8.0-openjdk-devel.x86_64 : OpenJDK Development Environment
java-1.8.0-openjdk-devel-debug.x86_64 : OpenJDK Development Environment with full debug on
java-1.8.0-openjdk-headless.x86_64 : OpenJDK Runtime Environment
java-1.8.0-openjdk-headless-debug.x86_64 : OpenJDK Runtime Environment with full debug on
java-1.8.0-openjdk-javadoc.noarch : OpenJDK API Documentation
java-1.8.0-openjdk-javadoc-debug.noarch : OpenJDK API Documentation for packages with debug on
java-1.8.0-openjdk-src.x86_64 : OpenJDK Source Bundle
java-1.8.0-openjdk-src-debug.x86_64 : OpenJDK Source Bundle for packages with debug on

一般都能检索出来，如果检索不出来，建议去下载安装oracle jdk

#如果检索到java-1.8.0-openjdk，直接安装即可

#yum install  java-1.8.0-openjdk java-1.8.0-openjdk-devel java-1.8.0-openjdk-headless

③.root用户不允许运行

修改运行文件

#groupadd elkstack                       #创建组
#useradd elkstack -g elkstack -d /usr/elk -s /bin/bash  #创建用户
#passwd  elkstack                        #给用户创建密码
#chown -R elkstack:elkstack /usr/elk     #目录的拥有者

#########################################################
#cd /etc/skel/                           #进入用户登录状态管理目录，如果不执行此操作，则登录界面在sh中
#ls -a  
. ..  .bash_logout  .bash_profile .bashrc  .mozilla  
#复制文件到新用户的创建目录
#cp .bash_logout  /home/MysqL/  
#cp .bash_profile  /home/MysqL/  
#cp .bashrc  /home/MysqL
#cd /

#########################################################
#su -l elkstack                          #切换用户 
#./elasticsearch/bin/elasticsearch       #启动elasticsearch

④.当前用户创建文件的大小，可使用的内存有限制

max file descriptors [4096] for elasticsearch process is too low,increase to at least [65536]
max number of threads [1024] for user [elkstack] is too low,increase to at least [2048]
max virtual memory areas vm.max_map_count [65530] is too low,increase to at least [262144]
system call filters Failed to install; check the logs and fix your configuration or disable system call filters at your own risk

Q:max file descriptors [4096] for elasticsearch process is too low,increase to at least [65536]

#vi /etc/security/limits.conf  
添加如下内容:

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

Q:max number of threads [1024] for user [elkstack] is too low,increase to at least [2048]

#vi /etc/security/limits.d/90-nproc.conf 

修改如下内容：
* soft nproc 1024

#修改为
* soft nproc 2048

Q:max virtual memory areas vm.max_map_count [65530] is too low,increase to at least [262144]

#vi /etc/sysctl.conf 

添加下面配置：
vm.max_map_count=655360

system call filters Failed to install; check the logs and fix your configuration or 
disable system call filters at your own risk

#vim config/elasticsearch.yml

添加
bootstrap.system_call_filter: false

配置完成之后，执行命令

sysctl -p

以上是安装过程中遇到的比较多的问题

使用curl 检查是否成功启动

#curl -i http://127.0.0.1:9200

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 327

{
  "name" : "rGlFyHB","cluster_name" : "elasticsearch","cluster_uuid" : "7sEFicrvQW-RPbJTjekbHg","version" : {
    "number" : "5.4.0","build_hash" : "780f8c4","build_date" : "2017-04-28T17:43:27.229Z","build_snapshot" : false,"lucene_version" : "6.5.0"
  },"tagline" : "You KNow,for Search"
}

二.安装kibana

#wget -c https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz
#tar -zxvf kibana-5.4.0-linux-x86_64.tar.gz
#mv kibana-5.4.0-linux-x86_64 /usr/elk/kibana
#cd /usr/elk/
#chown -R elkstack:elkstack kibana
#./kibana/bin/kibana

使用curl检测是否成功启动（注意：必须先启动elasticsearch）

#curl -i http://localhost:5601

HTTP/1.1 200 OK
kbn-name: kibana
kbn-version: 5.4.0
cache-control: no-cache
content-type: text/html; charset=utf-8
content-length: 217
accept-ranges: bytes
Date: Mon,22 May 2017 06:45:26 GMT
Connection: keep-alive

<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}

此外，kibana中需要配置elasticsearch的信息，如果elasticsearch的访问信息更新了，同样也需要更新kibana中的配置信息

#vim kibana/config/kibana.yml

#elasticsearch默认配置信息如下
elasticsearch.url: "http://localhost:9200"

三.安装logstash

#wget -c https://artifacts.elastic.co/downloads/logstash/logstash-5.4.0.tar.gz
#tar -zxvf logstash-5.4.0.tar.gz
#mv logstash-5.4.0 /usr/elk/logstash
#cd /usr/elk/
#chown -R elkstack:elkstack logstash
#./logstash/bin/logstash

测试是否安装成功

注意：最好以root用户运行，或者在sudoers中添加用户的sudo命令权限，否则可能产生好多问题

#./logstash/bin/logstash -e 'input{stdin{}} output{stdout{}}'

#启动之后，执行如下操作，测试是否有回显，如果有回显，则表示正确
#Hello World

四.配置&插件安装

①.远程访问

kibana.yml:

server.port:  5601
server.host:  "192.168.1.210"
elasticsearch.url: "http://192.168.1.210:9200"

elasticsearch.yml:

network.host: "192.168.1.210"
http.port: 9200

#加入新集群时使用的ip地址，默认是回环地址
#discovery.zen.ping.unicast.hosts: ["192.168.1.210"]  
#集群中最少的master数量
#discovery.zen.minimum_master_nodes: 3  

#bootstrap.system_call_filter: false

logstash.yml

http.host: "172.20.11.62"

②.安装X-Pack

注意：安装前必须停止elasticsearch与kibana服务

#cd /usr/elk/elasticsearch/bin
#./elasticsearch-plugin install x-pack

#cd /usr/elk/kibana/bin
#./kibana-plugin install x-pack

安装X-Pack完成之后，穷elasticsearch与kibana ，会进行用户登录校验，默认用户名和密码如下

username : elastic
passowrd : changeme

但是，对于logstash来说，需要在配置文件中配置用户名才行，否则无法链接到elasticsearch

input {
  file {
    type =>"syslog"
     path => ["/var/log/messages","/var/log/secure" ]
  }
  syslog {
    type =>"syslog"
    port =>"5544"
  }
}
output {
  stdout { codec=> rubydebug }
  elasticsearch {
	hosts => ["192.168.1.210:9200"]
	 user => elastic
         password => changeme
         index => "syslogstash-%{+YYYY.MM.dd}"
         template_overwrite => true	
	}
}

③.Kibana创建索引

Kibana创建索引的前提是logstash的pipline配置文件中存索引，并且logstash已经向elasticsearch注册了索引

index => "syslogstash-%{+YYYY.MM.dd}"

input {
  file {
    type =>"syslog"
     path => ["/var/log/messages","/var/log/secure" ]
  }
  syslog {
    type =>"syslog"
    port =>"5544"
  }
}
output {
  stdout { codec=> rubydebug }
  elasticsearch {
	hosts => ["192.168.1.210:9200"]
	 user => elastic
         password => changeme
         index => "logstash-%{+YYYY.MM.dd}"
         template_overwrite => true	
	}
}

检测配置文件是否正确

#logstash/bin/logstash -f test_logstash.conf -t

启动logstash

#logstash/bin/logstash -f test_logstash.conf

触发Input Event，让logstash主动注册index到elasticsearch

#logger -p info "hello,remote rsyslog"

然后登录Kibana，点击Management->Index Patterns打开索引注册页面，点击左侧菜单栏中的【+】，新增索引。

如果你看不到Create按钮，那么很可能意味着索引没有注册成功，注册可能需要一个Input Event输入触发才行。

如果索引注册成功，那么点击Kibana菜单 discover，选择syslogstash-*索引，便能看到相应的事件。

参考：

ELK集群部署及收集nginx日志

Elasticsearch5.0 版本安装错误

记录Linux下安装elasticSearch时遇到的一些错误

centos7虚拟机安装elasticsearch5.0.x-安装篇

ElasticSearch 5.0.0 安装部署常见错误或问题

已解决：登录Linux的 -bash-4.2$ 问题

原文地址：https://www.jb51.cc/centos/377511.html

CentOS ELK5.0日志分析平台搭建

一.安装Elasticsearch

二.安装kibana

三.安装logstash

四.配置&插件安装

相关推荐