Modes

This processor is used to tail a file or multiple files according to multiple modes. The mode to choose depends of the logging pattern followed by the file(s) to tail. In any case, if there is a rolling pattern, the rolling files must be plain text files (compression is not supported at the moment).

Rolling filename pattern

In case the 'Rolling filename pattern' property is used, when the processor detects that the file to tail has rolled over, the processor will look for possible missing messages in the rolled file. To do so, the processor will use the pattern to find the rolling files in the same directory as the file to tail.

In order to keep this property available in the 'Multiple files' mode when multiples files to tail are in the same directory, it is possible to use the ${filename} tag to reference the name (without extension) of the file to tail. For example, if we have:

/my/path/directory/my-app.log.1
/my/path/directory/my-app.log
/my/path/directory/application.log.1
/my/path/directory/application.log

the 'rolling filename pattern' would be ${filename}.log.*.

Descriptions for different modes and strategies

The 'Single file' mode assumes that the file to tail has always the same name even if there is a rolling pattern. Example:

/my/path/directory/my-app.log.2
/my/path/directory/my-app.log.1
/my/path/directory/my-app.log

and new log messages are always appended in my-app.log file.

In case recursivity is set to 'true'. The regular expression for the files to tail must embrace the possible intermediate directories between the base directory and the files to tail. Example:

/my/path/directory1/my-app1.log
/my/path/directory2/my-app2.log
/my/path/directory3/my-app3.log

Base directory: /my/path
Files to tail: directory[1-3]/my-app[1-3].log
Recursivity: true

In the 'Multiple files' mode, it is possible to specify if the file to tail has always the same name or not. It is done through the property 'Rolling strategy'. The strategy can be 'Fixed name' in case the files to tail have always the same name (see example above) or can be 'Changing name' in case the files to tail do not always have the same name. Example:

/my/path/directory/my-app-2016-09-06.log
/my/path/directory/my-app-2016-09-07.log
/my/path/directory/my-app-2016-09-08.log

and new log messages are always appended in log file of the current day.

If the processor is configured with 'Multiple files' mode, two additional properties are relevant:

It is necessary to pay attention to 'Lookup frequency' and 'Maximum age' properties, as well as the frequency at which the processor is triggered, in order to achieve high performance. It is recommended to keep 'Maximum age' > 'Lookup frequency' > processor scheduling frequency to avoid missing data. It also recommended not to set 'Maximum Age' too low because if messages are appended in a file after this file has been considered "too old", all the messages in the file may be read again, leading to data duplication.

If the processor is configured with 'Multiple files' mode, the 'Rolling filename pattern' property must be specific enough to ensure that only the rolling files will be listed and not other currently tailed files in the same directory (this can be achieved using ${filename} tag).