logo

Log Rotation

When developing server-side software it’s important that you use logs. Just as importantly though, you need to know how to “rotate” your logs, or prevent them from overtaking your server space.

Since we use Ubuntu 13.10 currently we use the logrotate feature to help set up our log files and automatically manage log rotations so that our logs won’t overtake the server. It’s fairly simple, but not everyone knows how to do this. Logrotate should be installed on your server, if it’s not, first install it.


sudo apt-get update
sudo apt-get install logrotate

To verify it’s now installed run the following


logrotate

Now cd into the logrotate.d folder to see the logrotate scripts

cd /etc/logrotate.d

This folder contains your logrotate scripts. Since we use Node.JS and Forever to keep our servers running we set up this as our configuration file:


vi /etc/logrotate.d/forever

This creates your file, and then copy paste the following into your file, where the first line is the location of your forever log file


/nodejs/.forever/service.log {
rotate 7
size 100M
daily
missingok
notifempty
sharedscripts
copytruncate
compress
}

Run this to verify your new logrotate script is running


cat /var/lib/logrotate/status

This will tell you what logs are being rotated, and your logs should show up there now!

Here’s a breakdown of what the logrotate commands mean:

compress
Old versions of log files are compressed with gzip by default. See also nocompress.

compresscmd
Specifies which command to use to compress log files. The default is gzip. See alsocompress.

uncompresscmd
Specifies which command to use to uncompress log files. The default is gunzip.

compressext
Specifies which extension to use on compressed logfiles, if compression is enabled. The default follows that of the configured compression command.

compressoptions
Command line options may be passed to the compression program, if one is in use. The default, for gzip, is “-9” (maximum compression).

copy
Make a copy of the log file, but don’t change the original at all. This option can be used, for instance, to make a snapshot of the current log file, or when some other utility needs to truncate or pare the file. When this option is used, the create option will have no effect, as the old log file stays in place.

copytruncate
Truncate the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one, It can be used when some program can not be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.

create mode owner group

Immediately after rotation (before the postrotate script is run) the log file is created (with the same name as the log file just rotated). mode specifies the mode for the log file in octal (the same as chmod(2)), owner specifies the user name who will own the log file, and groupspecifies the group the log file will belong to. Any of the log file attributes may be omitted, in which case those attributes for the new file will use the same values as the original log file for the omitted attributes. This option can be disabled using the nocreate option.

daily

Log files are rotated every day.

delaycompress
Postpone compression of the previous log file to the next rotation cycle. This has only effect when used in combination with compress. It can be used when some program can not be told to close its logfile and thus might continue writing to the previous log file for some time.

extension ext
Log files are given the final extension ext after rotation. If compression is used, the compression extension (normally .gz) appears after ext.

ifempty

Rotate the log file even if it is empty, overiding the notifempty option (ifempty is the default).

include file_or_directory
Reads the file given as an argument as if it was included inline where the include directive appears. If a directory is given, most of the files in that directory are read in alphabetic order before processing of the including file continues. The only files which are ignored are files which are not regular files (such as directories and named pipes) and files whose names end with one of the taboo extensions, as specified by the tabooext directive. The includedirective may not appear inside of a log file definition.

mail address
When a log is rotated out-of-existence, it is mailed to address. If no mail should be generated by a particular log, the nomail directive may be used.

mailfirst
When using the mail command, mail the just-rotated file, instead of the about-to-expire file.

maillast
When using the mail command, mail the about-to-expire file, instead of the just-rotated file (this is the default).

missingok
If the log file is missing, go on to the next one without issuing an error message. See alsonomissingok.

monthly
Log files are rotated the first time logrotate is run in a month (this is normally on the first day of the month).

nocompress
Old versions of log files are not compressed with gzip. See also compress.

nocopy
Do not copy the original log file and leave it in place. (this overrides the copy option).

nocopytruncate
Do not truncate the original log file in place after creating a copy (this overrides thecopytruncate option).

nocreate
New log files are not created (this overrides the create option).

nodelaycompress
Do not postpone compression of the previous log file to the next rotation cycle (this overrides the delaycompress option).

nomail
Don’t mail old log files to any address.

nomissingok
If a log file does not exist, issue an error. This is the default.

noolddir
Logs are rotated in the same directory the log normally resides in (this overrides the olddiroption).

nosharedscripts
Run prerotate and postrotate scripts for every script which is rotated (this is the default, and overrides the sharedscripts option).

notifempty
Do not rotate the log if it is empty (this overrides the ifempty option).

olddir directory
Logs are moved into directory for rotation. The directory must be on the same physical device as the log file being rotated. When this option is used all old versions of the log end up indirectory. This option may be overriden by the noolddir option.

postrotate/endscript
The lines between postrotate and endscript (both of which must appear on lines by themselves) are executed after the log file is rotated. These directives may only appear inside of a log file definition. See prerotate as well.

prerotate/endscript
The lines between prerotate and endscript (both of which must appear on lines by themselves) are executed before the log file is rotated and only if the log will actually be rotated. These directives may only appear inside of a log file definition. See postrotate as well.

firstaction/endscript
The lines between firstaction and endscript (both of which must appear on lines by themselves) are executed once before all log files that match the wildcarded pattern are rotated, before prerotate script is run and only if at least one log will actually be rotated. These directives may only appear inside of a log file definition. See lastaction as well.

lastaction/endscript
The lines between lastaction and endscript (both of which must appear on lines by themselves) are executed once after all log files that match the wildcarded pattern are rotated, after postrotate script is run and only if at least one log is rotated. These directives may only appear inside of a log file definition. See lastaction as well.

rotate count
Log files are rotated times before being removed or mailed to the address specified in a maildirective. If count is 0, old versions are removed rather then rotated.

size size
Log files are rotated when they grow bigger then size bytes. If size is followed by M, the size if assumed to be in megabytes. If the k is used, the size is in kilobytes. So size 100size 100k, and size 100M are all valid.

sharedscripts
Normally, prescript and postscript scripts are run for each log which is rotated, meaning that a single script may be run multiple times for log file entries which match multiple files (such as the /var/log/news/* example). If sharedscript is specified, the scripts are only run once, no matter how many logs match the wildcarded pattern. However, if none of the logs in the pattern require rotating, the scripts will not be run at all. This option overrides the nosharedscripts option.

start count
This is the number to use as the base for rotation. For example, if you specify 0, the logs will be created with a .0 extension as they are rotated from the original log files. If you specify 9, log files will be created with a .9, skipping 0-8. Files will still be rotated the number of times specified with the count directive.

tabooext [+] list
The current taboo extension list is changed (see the include directive for information on the taboo extensions). If a + precedes the list of extensions, the current taboo extension list is augmented, otherwise it is replaced. At startup, the taboo extension list contains .rpmorig, .rpmsave, ,v, .swp, .rpmnew, and ~.

weekly
Log files are rotated if the current weekday is less then the weekday of the last rotation or if more then a week has passed since the last rotation. This is normally the same as rotating logs on the first day of the week, but it works better if logrotate is not run every night.

  1. Jason

    Does this actually work for you? I find that if I move a file that forever is logging to, forever continues to log to the file in the new location.

    When using logrotate with, e.g., nginx, you have to send nginx a signal telling it to reload its log configuration. See e.g. http://www.nginxtips.com/how-to-rotate-nginx-logs/

    and the postrotate hook with “kill -USR1 `cat /var/run/nginx.pid`”

  2. admin

    Yes, this is working for me with Forever – I’ll need to look closer, but my servers have not been growing exponentially in disk space usage since implementing this.

  3. Richard Friedman

    Brandon, thanks for this. The copytruncate option I believe what makes it work well with nodejs/forever. Just using truncate would probably be an issue because of the way the stream is kept open.

Leave a Reply

*

captcha *