Collect the Rails 4 production.log with td-agent



Hello everybody, It’s me candle.
In this time, let’s get the rails 4 production.log by td-agent.
I have been written same article, but there was lack of explanation and etc, I will rewrite it again.
Also, I will show you how to get Rails 5 production.log at a later date.


You must be able to execute ruby on rails in production mode.
It is ok in any environment as long as it has executable. Such as webrick, nginx, puma and apache.

td-agent or fluentd is instlled.
It is assumed that td-agent or fluentd is installed.

You may change the rails log format.

I think that the format of the production log of rails is the next image by default.

You can change the this log format easily.
The below is a code without any change from the default format.

class Logger::Formatter
  def call(severity, time, progname, msg)
    format = "%s, [%s #%d] %5s -- %s: %sn"
    format % [severity[0], "#{time.strftime('%Y-%m-%dT%H:%M:%S')}.#{'%06d' % time.usec.to_s}", $$, severity, progname, msg2str(msg)]

There is information that is not necessary for rails production log.
It is “Render partial” information.
The below text is sample.

I, [2017-03-24T06:19:15.314646 #5689] INFO — : Rendered users/shared/_links.html.haml (0.5ms)
I, [2017-03-24T06:19:15.315840 #5689] INFO — : Rendered users/registrations/new.html.haml within layouts/application (6.6ms)
I, [2017-03-24T06:19:15.318778 #5689] INFO — : Rendered layouts/_header.html.haml (0.6ms)
I, [2017-03-24T06:19:15.320581 #5689] INFO — : Rendered layouts/_footer.html.haml (0.8ms)

I think that this information increase the log file size
and this information is not necessary in the production mode.
It is ok if you don’t change that or not.

Open the config/environment.rb

Change the format like this.
If you are using erb please change “.haml” part to “.erb”.

class Logger::Formatter
  def call(severity, time, progname, msg)
    if !msg.include?('.haml')
      format = "%s, [%s #%d] %5s -- %s: %sn"
      format % [severity[0], "#{time.strftime('%Y-%m-%dT%H:%M:%S')}.#{'%06d' % time.usec.to_s}", $$, severity, progname, msg2str(msg)]

Setting of rails is over.

Preparing to get rails log with td-agent

Alright, let’s prepare for td-agent side.

My rails and td-agent environment is like this.
Td-agent is running with CentOS 6.5.
Rails application is running on the unicorn + nginx.
Please prepare those servers environment.

Set up td-agent

The biggest problem with rails 4 production log is that Fatal errors are displayed over multiple lines.
Fluentd gets a log each one line.
The multiple line errors are collected as separate logs.

F, [2017-03-24T08:24:51.329266 #6165] FATAL — :
NoMethodError (undefined method `each’ for nil:NilClass):
app/controllers/pages_controller.rb:7:in `about’

So, we use “format multiline”.
This means that the part not matching the regular expression written with “format_firstline” is regarded as the same as the matched part even if it is new line, and it is stored in one log.

Open the /etc/td-agent/td-agent.conf and write it.

  @type tail
  path /usr/share/nginx/alice4/log/production.log
  tag rails.production
  pos_file /var/log/td-agent/rails.production.pos
  format multiline
  format_firstline /^.,/
  format1 /^(?<log_initial>.*), [(?<date>d{1,4}-d{1,2}-d{1,2}Td{1,2}:d{1,2}:d{1,2}.d{1,6}) (?<process_id>[^ ]*)] *(?<log_level>[^ ]*) -- : (?<message>[^']*[^]]*)/

<match rails.production>
  @type stdout

With the “path” reference the production.log in the log folder of the rails project.
Please rewrite the reference path to your environment.

“tag rails.production” tagged a collected log.

With the “pos_file /var/log/td-agent/rails.production.pos” specifies the location of temporary files.

In the “format” declares “multiline”. It is possible to collect logs of multiple lines.

the meaning of “format_firstline / ^., /” regular expression is “1 character +,” at the beginning of the line.
This corresponds to the “I,” part of the rails production log.

“format1” matches the format of rails production.log.

In the “<match rails.production>” gets the tagged log and output it.

This is the setting for td-agent to get rails log.

Getting log and check it

Restart the td-agent.

sudo serice td-agent restart

Lauch the rails application in production.
Let’s access.

The td-agent log is output to the following location.


Check with the tail command.

sudo tail -f /var/log/td-agent/td-agent.log

This is the log.

The most important thing is the part of “FATAL”.

2017-03-24 08:30:11.203171123 +0000 rails.production: {“log_initial”:”F”,”date”:”2017-03-24T08:24:51.329266″,”process_id”:”#6165″,”log_level”:”FATAL”,”message”:”nNoMethodError (undefined method `each’ for nil:NilClass):n app/controllers/pages_controller.rb:7:in `about’nn”}

The messages are gathered to one.


We tried to collect Rails 4 production.log.


If you think this article is good, share it please

-, ,



Your email address will not be published. Required fields are marked *


How to get nginx error.log and access.log with td-agent

English 日本語 Abstract Hello everyone, It’s candle. In this time, we collect nginx log with td-agent. I explain with td-agent, but basic configuration is similer with fluentd. Precondition Td-agent or fluentd is installed You have a nginx server.

Setup of td-agent-ui and operation check

English 日本語 Abstract Hello everyone, It’s candle. In this time, I will show you how to set up td-agent-ui and operation check. Td-agent is easier than fluentd to operate on the server. Precondition Td-agent is installed Please refer to the following article when building td-agent to Centos 6.5. Build up a td-agent server on centos 6.5 For EC2, here Build up td-agent server on the EC2 of aws

How to define environment variables for each virtual host in apache 2.4 and use it in php

English 日本語 Abstract Hello everyone, It’s candle This time I’ll show you how to define environment variables for each virtualhost in apache 2.4 and use it in php For example, when you are developing with wordpress, you don’t want to put password or access key or anything else under the control of git. So, you can define it on apache and require it with getenv for use. Premise apache2.4 anything php project you have already built some kind of virtual host.

Collect Rails 5 production.log with td-agent

English 日本語 Abstract Hello everybody It’s me candle. In this article I will show you how to collect rails 5 production.log with td-agent. Rails 5 was changed log format and it cause some problem. I will solve it and explain. relational article If you are using rails 4 please refer to the following article. td-agentでRails 4のproduction.logを取得する Precondition You can work ruby on rails in production mode Before read this article you setup the rails production environment such as webrick, apache, puma and nginx. You already installed td-agent or fluentd. You need td-agent or fluentd. It is installed on the ruby …

Allocate original domain of Route53 to React hosted at S3

Abstract Hello everyone it’s me candle. We will allocate original domain of Route53 to React hosted in S3. Condition You have a domain at Route53 You have a bucket with the same name as domain in S3 Your React is already hosting on S3 Importantly, the bucket of S3 and the domain name acquired by Route53 must be the same name.If you have not created a bucket yet in S3 or React web app is not hosted in S3, please refer to the previous article. Checking The domain of Route53 I have domain at Route53. I will allocate “” …

I work in the venture company as a CTO. I start to write program in University, first I learned java, C++ and PHP. In the company, I'm developing web services by Rails. I do like to automation.