joppot

The program absolutely runs in copy paste

Server

How to get nginx error.log and access.log with td-agent

投稿日:2017年3月21日 更新日:

Pocket


Abstract

Hello everyone, It’s candle.
In this time, we collect nginx log with td-agent.
I explain with td-agent, but basic configuration is similer with fluentd.


Precondition

Td-agent or fluentd is installed
You have a nginx server.

SPONSORED LINK


Change the td-agent user

The user who execute td-agent server is the td-agent by default setting.
But only root user can gets nginx log.

If td-agent try to get nginx access.log and error.log, the error would appear about “Permission denied @ rb_sysopen – /var/log/nginx/access.log”.

To solve this, you would change the user of the td-agent server to root.
Open the “/etc/init.d/td-agent” file with your favorite editor.

sudo emacs /etc/init.d/td-agent

Change the value of “TD_AGENT_USER” in the file from “td-agent” to “root”.

Restart td-agent.

sudo service td-agent restart

The td-agent now can access to nginx log.

Get nginx access.log

Though there is the following format that takes nginx access.log,

format nginx

The result log contains “partter not match”
This is because the regular expression defined in “format nginx” and the result log are not match.

This is the sample log.

2017-03-05 05:21:39 +0000 [warn]: pattern not match: "192.168.33.1 - - [05/Mar/2017:05:21:39 +0000] "GET /assets/application.css HTTP/1.1" 304 0 "http://local-hogehogecom/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/602.4.8 (KHTML, like Gecko) Version/10.0.3 Safari/602.4.8" "-""

When I look into another format, many articles use ltsv format.
However, I don’t recommend this way too much.
The form of json is not so good.
This is the sample.

2017-03-05 05:17:00 +0000 nginx.access: {"192.168.33.1 - - [05/Mar/2017":"05:17:00 +0000] "GET /assets/application.js HTTP/1.1" 304 0 "http://local-hogehoge.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/602.4.8 (KHTML, like Gecko) Version/10.0.3 Safari/602.4.8" "-""}

Therefore, I refer to the following site and define the format oneself.

http://qiita.com/liubin/items/92a4e7e3917143ae4aaf

Open the td-agent configuration file.

/etc/td-agent/td-agent.conf

Write this code.

<source>
  @type tail
  path /var/log/nginx/access.log
  tag nginx.access
  pos_file /var/log/td-agent/nginx.access.pos
  format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)" "(?<forwarder>[^"]*)")?/
  time_key time_local
</source>

“type tail” tails a file defined by path
“path /var/log/nginx/access.log” is the target file of “type tail”
“tag nginx.access” tags the fetched logs.
“pos_file /var/log/td-agent/nginx.access.pos” defines a file that temporarily holds fetched logs
“Format” is written by regular expressions matching nginx access.log.
“time time_local” inserts time into the generated json.

For debugging, write the directive to display the fetched log.
If it matches the tag “nginx.access”, it will output to td-agent as standard output.

<match nginx.access>
  type stdout
</match>

Save the file and restart td-agent.

sudo service td-agent restart

The log of td-agent is /var/log/td-agent/td-agent.log.
Let’s watching it.

sudo tail -f /var/log/td-agent/td-agent.log

Access to nginx server with the browser and check whether the log is displayed.

The next text is a result log.

2017-03-05 14:21:08 +0000 nginx.access: {"remote":"192.168.33.1","host":"-","user":"-","time":"05/Mar/2017:14:21:08 +0000","method":"GET","path":"/assets/application-89224d7948f4a0ac2f79b292da0.js","code":"304","size":"0","referer":"http://local-hogehoge.com/","agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/602.4.8 (KHTML, like Gecko) Version/10.0.3 Safari/602.4.8","forwarder":"-"}

It’s success.


Get nginx error.log

And next, we will get the nginx error.log.
Open the td-agent conf file.

/etc/td-agent/td-agent.conf

Comment out the debugging directive of the access.log.

#<match nginx.access>
#  type stdout
#</match>

Write this code.

<source>
  type tail
  path /var/log/nginx/error.log
  tag nginx.error
  pos_file /var/log/td-agent/nginx.error.pos
  format /^(?<time>d{4}/d{2}/d{2} d{2}:d{2}:d{2}) [(?<log_level>w+)] (?<pid>d+).(?<tid>d+): (?<message>.*)$/
</source>

The basic setting is the same as access.log.
About the “format”, it is described original regular expression.
It is a regular expression matching to nginx error.log.
I took it from the official website

http://docs.fluentd.org/v0.12/articles/common-log-formats

Also write the following td-agent standard output for debugging.

<match nginx.error>
  type stdout
</match>

Save it and restart

sudo service td-agent restart

Let’s watch the td-agent log using “tail -f” as before.

sudo tail -f /var/log/td-agent/td-agent.log

I know It is difficult that you make nginx occur an error intentionally.
The under image is the error that is I access to the rails application when unicorn is stopped.

2017-03-05 08:28:00 +0000 nginx.error: {"log_level":"error","pid":"1970","tid":"1970","message":"*104 connect() to unix:/tmp/hogehoge_unicorn.sock failed (111: Connection refused) while connecting to upstream, client: 192.168.33.1, server: local-hogehoge.com, request: "GET / HTTP/1.1", upstream: "http://unix:/tmp/hogehoge_unicorn.sock:/", host: "local-hogehoge.com""}

It seems good.

Conclusion

We tried to get access.log and error.log of nginx.

スポンサードリンク

If you think this article is good, share it please

-Server
-,

執筆者:


comment

Your email address will not be published. Required fields are marked *

関連記事

Build up a td-agent server on centos 6.5

English 日本語 Abstract Hello everyone, It’s candle. In this time, we build up a td-agent server on the centos 6.5. Td-agent is service based on fluentd. It daemonizes fluentd and creates a GUI. Premise You have Centos 6.5 server. I use Centos 6.5 on the vagrant for explanation.

How to define environment variables for each virtual host in apache 2.4 and use it in php

English 日本語 Abstract Hello everyone, It’s candle This time I’ll show you how to define environment variables for each virtualhost in apache 2.4 and use it in php For example, when you are developing with wordpress, you don’t want to put password or access key or anything else under the control of git. So, you can define it on apache and require it with getenv for use. Premise apache2.4 anything php project you have already built some kind of virtual host.

Collect the Rails 4 production.log with td-agent

English 日本語 Abstract Hello everybody, It’s me candle. In this time, let’s get the rails 4 production.log by td-agent. I have been written same article, but there was lack of explanation and etc, I will rewrite it again. Also, I will show you how to get Rails 5 production.log at a later date. Precondition You must be able to execute ruby on rails in production mode. It is ok in any environment as long as it has executable. Such as webrick, nginx, puma and apache. td-agent or fluentd is instlled. It is assumed that td-agent or fluentd is installed.

Setup of td-agent-ui and operation check

English 日本語 Abstract Hello everyone, It’s candle. In this time, I will show you how to set up td-agent-ui and operation check. Td-agent is easier than fluentd to operate on the server. Precondition Td-agent is installed Please refer to the following article when building td-agent to Centos 6.5. Build up a td-agent server on centos 6.5 For EC2, here Build up td-agent server on the EC2 of aws

Monitor the memory state of ec2 with watch and free command

English 日本語 Abstract Hello everyone, It’s candle. In this time, I will show you how to monitor ec2 memory in real time. EC2 server of t2.micro has only 1GB of memory, If you run composer, rails, mysql on it, memory may be barely. It is useful if you can monitor the memory and tune the server. Precondition EC2 server

Profile


I work in the venture company as a CTO. I start to write programing in University, first I learned java, C++ and PHP. In the company, I'm developing web services by Rails. I do like to automation.

twitter

スポンサードリンク

Archives