通过Logstash收集日志

配置logstash

#安装
rpm -ivh logstash-5.6.5.rpm 

#配置软链
ln -s /etc/logstash /usr/share/logstash/config

#配置
vim /etc/logstash/logstash.yml 
path.config: /usr/share/logstash/config/conf.d     #修改path.config到这个目录


#启动
systemctl start logstash
systemctl status logstash
systemctl enable logstash

#软链启动命令
ln -s /usr/share/logstash/bin/logstash  /sbin  

收集单个系统日志并输出至文件

[root@localhost ~]# cat /etc/logstash/conf.d/system-log.conf 
input {
  file {
    type => "messagelog"   #
    path => "/var/log/messages"
    start_position => "beginning"     #第一次从头收集,之后从新添加的日志收集
  }
}

output {
  file {
    path => "/tmp/%{type}.%{+YYYY.MM.dd}"  #注：此处dd必须为小写
  }
}

• 检测配置文件语法

logstash -f /usr/share/logstash/config/conf.d/system-log.conf -t

• 生成数据并验证

echo "test" >> /var/log/messages
echo "测试" >> /var/log/messages

[root@localhost ~]# cat /tmp/messagelog.2017.12.12
{"@version":"1","host":"localhost.localdomain","path":"/var/log/messages","@timestamp":"2017-12-12T14:02:22.374Z","message":"Hello Logstash","type":"messagelog"}
{"@version":"1","host":"localhost.localdomain","path":"/var/log/messages","@timestamp":"2017-12-12T14:04:15.441Z","message":"测试","type":"messagelog"}
{"@version":"1","host":"localhost.localdomain","path":"/var/log/messages","@timestamp":"2017-12-12T14:04:37.454Z","message":"测试","type":"messagelog"}

• 排错

如果没有出现上面的测试结果，则需要查看logstash日志
tailf /var/log/logstash/logstash-plain.log

如果提示/var/log/messages Permission denied无权限，则需要给messages文件授权
chmod  644 /var/log/messages

通过logstash 收集多个日志文件

• 编辑logstash日志收集规则配置文件

/etc/logstash/conf.d/system-log.conf

[root@localhost conf.d]# cat system-log.conf 
input {
  file {
    path => "/var/log/messages"       #日志"收集源"的文件路径
    type => "systemlog"               #定义事件的唯一类型
    start_position => "beginning"     #定义第一次收集日志从头开始
    stat_interval => "3"              #定义日志收集时间间隔
  }

  file {
    path => "/var/log/secure"
    type => "securelog"
    start_position => "beginning"
    stat_interval => "3"
  }
}

output {
  if [type] == "systemlog" {          #判断type类型的value，并写入相同(或不同)的Elasticsearch集群
    elasticsearch {               
      hosts => ["10.0.10.21:9200"]    #指定elasticsearch集群节点地址
      index => "system-log-%{+YYYY.MM.dd}"  #指定收集日志到elasticsearch后的格式(注：dd必须小写)
  }}
  if [type] == "securelog" {
    elasticsearch {
      hosts => ["10.0.10.21:9200"]
      index => "secure-log-%{+YYYY.MM.dd}"
  }}
  
}

• 检验配置文件语法是否有误

[root@localhost conf.d]# logstash -f /usr/share/logstash/config/conf.d/system-log.conf -t
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK

• 重启logstash并查看日志是否有报错

systemctl restart logstash
systemctl status logstash

注：如果验证后没有“Configuration OK”说明配置一定有问题。可以查看日志来确定并修改具体的错误字段
tailf /var/log/logstash/logstash-plain.log

• 向被收集的日志中写入测试数据

echo "logstash收集多个日志文件-测试"  >> /var/log/secure
echo "logstash收集多个日志文件-测试"  >> /var/log/messages

• kibana界面添加system-messagess 索引

首先访问elasticsearch-head, 查看索引名称或者通过定义的logstash日志收集规则配置文件查看索引名称

访问kibana, Management -- Index Patterns -- 添加索引(如下图)

• kibana展示

• 总结

1. logstash收集谁的日志，就把logstash安装在哪台机器上，之间使用file模块指定在input即可。
2. logstash语法大多数为重复性的，记住常用的几个模块和if判断规则即可。特殊使用的话就去官网查找用法。

• 本文作者： GaryWu
• 本文链接： https://garywu520.github.io/2017/12/12/通过Logstash收集日志/
• 版权声明： 本博客所有文章除特别声明外，均采用 MIT 许可协议。转载请注明出处！