File (om_file)

This module writes data to one or more output files. You can specify a static or a dynamic filename that NXLog Agent evaluates as an expression for each record, allowing you to route records to different files based on field values such as $Hostname. The module does not manage log rotation or limit file size on its own, so you must configure rotation to prevent output files from growing indefinitely; see Rotating files based on size below for an example.

To examine the supported platforms, see the list of installation packages.

Configuration

The om_file module accepts the following directives in addition to the common module directives.

Required directives

The following directive is required for the module to start.

File

Set this directive to specify the output file. The value must be a string expression.

Quote the filename to form a valid string literal; File does not accept an unquoted value. For relative filenames, NXLog Agent changes its working directory to / unless you change the global SpoolDir value. Below are three variations that specify the same output file on a Windows system:

File    'C:\Logs\nxlog-output.log'
File    "C:\\Logs\\nxlog-output.log"
File    'C:/Logs/nxlog-output.log'

If the File value is not a literal string but contains functions, field names, or operators, the module evaluates the expression for each record it processes. Processing high event rates increases CPU usage and can significantly degrade performance. If you notice performance issues with a dynamic File, use the simplest possible expression in your file-naming strategy.

If you write to a mapped network drive while NXLog Agent is running as a service, it might not be able to see the drive. See Cannot write logs to a mapped network drive for more information.

File permissions directives

The following directives are for specifying permissions for the created socket, pipe, or file.

Group

Set this directive to specify the group who owns the created socket, pipe, or file.

The default is the group set by the global Group directive. This directive does not work on Windows.

Perms

Set this directive to specify the permissions for the created socket, pipe, or file. Enter the value as a four-digit octal number starting with zero.

The default is the operating system’s default permissions. This directive does not work on Windows.

If the path includes non-existent directories, NXLog Agent adds execute permission (allowing directory browsing) to each newly created directory along the path for the current user.

User

Set this directive to specify the user who owns the created socket, pipe, or file.

The default is the user set by the global User directive. This directive does not work on Windows.

Optional directives

CacheSize

Set this directive to cache open files when using dynamic filenames, reducing the overhead of repeatedly opening and closing files. Set the value to the number of files you expect the module to write concurrently, but do not exceed the number of open files allowed by your system. This caching only improves performance on Windows.

By default, the module does not cache open files.

CreateDir

Set this directive to TRUE to create the output directory before opening the file for writing, if it does not already exist.

The default is FALSE.

OutputType

Set this directive to specify how the module formats output records. See the common OutputType directive for the available values, including support for data converters.

The default is LineBased, terminating each record with CRLF on Windows or LF on Unix.

RenameCheck

Set this directive to TRUE to check if an external application has rotated the open file. If the module detects that the file was rotated, it reopens the original file set by the File directive.

The default is FALSE.

Sync

Set this directive to TRUE to synchronize the file to disk after writing each record, flushing it from the buffers. This can degrade performance in write-heavy environments.

The default is FALSE.

Truncate

Set this directive to TRUE to truncate the file before each write, keeping only the most recent record.

The default is FALSE; the module appends records to the output file.

Functions

The following functions are exported by om_file.

type: string file_name()

Returns the current output filename, as specified by the File directive.

type: integer file_size()

Returns the size of the current output file in bytes. The function returns undef if the file doesn’t exist, which can happen if no event records have been written to the file yet.

The File Operations extension also provides a file_size() function with different functionality and parameters. Specify the module instance name with the arrow operator to ensure you call the correct function, e.g., MyInstanceName->file_size().

Procedures

The following procedures are exported by om_file.

reopen();: Reopens the current file. Call this procedure if the file is removed or renamed, for example with the file_cycle(), file_remove(), or file_rename() procedures of the File Operations extension. You do not need to call this procedure after rotate_to() because that procedure reopens the file automatically.
rotate_to(type: string filename);: Rotates the current output file to the specified filename. The target folder must exist, or the procedure fails to rotate the file. The module then recreates the original file set by the File directive.

This procedure uses the rename(2) system call, which does not support moving files across different devices on some platforms. To address this, first rotate the file on the same device, then use the exec_async() procedure of the External Programs extension to copy it to another device. Alternatively, use the file_copy() procedure of the File Operations extension.

Examples

Example 1. Basic om_file configuration

The following is a basic configuration that writes logs to a file on Linux.

nxlog.conf

<Output file>
    Module    om_file
    File      '/var/log/nxlog-output.log'
</Output>

Example 2. Rotating files based on size

This configuration rotates the output file once it exceeds 10 MB and retains up to two log files.

If you do not configure rotation, the module keeps writing to the same file indefinitely, which can fill up your disk.

nxlog.conf

define OUTPUTFILE  '/var/log/nxlog-output.log'

<Extension fileop>
    Module       xm_fileop
</Extension>

<Output file>
    Module       om_file
    File         %OUTPUTFILE%
    <Schedule>
        Every    60 sec
        <Exec>
            if (file_size() >= 10M)
            {
                fileop->file_cycle(%OUTPUTFILE%, 2);
                reopen();
            }
        </Exec>
    </Schedule>
</Output>

Example 3. Using dynamic filenames and compression

This configuration writes records to a file, using the $Hostname field as the filename. When the output file size exceeds 15 MB, it rotates it to a new file in a separate folder and compresses the new file.

If you use unsanitized fields for filenames, you risk exposing NXLog Agent to injection attacks.

nxlog.conf

<Extension exec> (1)
    Module    xm_exec
</Extension>

<Extension fileop> (2)
    Module    xm_fileop
</Extension>

<Output file>
    Module    om_file
    File      'tmp/' + $SanitizedHostname
    <Exec>
        $SanitizedHostname = $Hostname;
        if $SanitizedHostname =~ s/[\\\/:*?"<>|]//g {}; (3)

        if (file->file_size() > 15M) {
            $targetDir = 'tmp/' + month(now());

            if not fileop->dir_exists($targetDir) { (4)
                fileop->dir_make($targetDir);
            }

            $newfile = $targetDir + '/' + $SanitizedHostname + '_' + strftime(now(), '%Y%m%d%H%M%S');
            file->rotate_to($newfile);
            exec_async("/bin/bzip2", $newfile);
        }
    </Exec>
</Output>

1	The External Programs extension provides the exec_async() procedure.
2	The File Operations extension provides the dir_exists() function and the dir_make() procedure.
3	Removes illegal characters.
4	Creates the target folder if it does not exist.