The release of DTail 4.0.0

Published at 2022-03-06T18:11:39+00:00

                              ,_---~~~~~----._
                        _,,_,*^____      _____``*g*\"*,
  ____ _____     _ _   / __/ /'     ^.  /      \ ^@q   f
 |  _ \_   _|_ _(_) |   @f | @))    |  | @))   l  0 _/
 | | | || |/ _` | | |  \`/   \~____ / __ \_____/    \
 | |_| || | (_| | | |   |           _l__l_           I
 |____/ |_|\__,_|_|_|   }          [______]           I
                        ]            | | |            |
                        ]             ~ ~             |
                        |                            |
                         |                           |

I have recently released DTail 4.0.0 and this blog post goes through all the new goodies. If you want to jump directly to DTail, do it here (there are nice animated gifs which demonstrates the usage pretty well):

https://dtail.dev

So, what's new in 4.0.0?

Rewritten logging

For DTail 4, logging has been completely rewritten. The new package name is "internal/io/dlog". I rewrote the logging because DTail is a special case here: There are logs processed by DTail, there are logs produced by the DTail server itself, there are logs produced by a DTail client itself, there are logs only logged by a DTail client, there are logs only logged by the DTail server, and there are logs logged by both, server and client. There are also different logging levels and outputs involved.

As you can imagine, it becomes fairly complex. There is no ready Go off-shelf logging library which suits my needs and the logging code in DTail 3 was just one big source code file with global variables and it wasn't sustainable to maintain anymore. So why not rewrite it for profit and fun?

There's a are new log level structure now (The log level now can be specified with the "-logLevel" command line flag):

// Available log levels.
const (
	None    level = iota
	Fatal   level = iota
	Error   level = iota
	Warn    level = iota
	Info    level = iota
	Default level = iota
	Verbose level = iota
	Debug   level = iota
	Devel   level = iota
	Trace   level = iota
	All     level = iota
)

DTail also supports multiple log outputs (e.g. to file or to stdout). More are now easily pluggable with the new logging package. The output can also be "enriched" (default) or "plain" (read more about that further below).

Configurable terminal color codes

A complaint I received from the users of DTail 3 were the terminal colors used for the output. Under some circumstances (terminal configuration) it made the output difficult to read so that users defaulted to "--noColor" (disabling colored output completely). I toke it by heart and also rewrote the color handling. It's now possible to configure the foreground and background colors and an attribute (e.g. dim, bold, ...).

The example "dtail.json" configuration file represents the default (now, more reasonable default) color codes used, and it is free to the user to customize them:

{
  "Client": {
    "TermColorsEnable": true,
    "TermColors": {
      "Remote": {
        "DelimiterAttr": "Dim",
        "DelimiterBg": "Blue",
        "DelimiterFg": "Cyan",
        "RemoteAttr": "Dim",
        "RemoteBg": "Blue",
        "RemoteFg": "White",
        "CountAttr": "Dim",
        "CountBg": "Blue",
        "CountFg": "White",
        "HostnameAttr": "Bold",
        "HostnameBg": "Blue",
        "HostnameFg": "White",
        "IDAttr": "Dim",
        "IDBg": "Blue",
        "IDFg": "White",
        "StatsOkAttr": "None",
        "StatsOkBg": "Green",
        "StatsOkFg": "Black",
        "StatsWarnAttr": "None",
        "StatsWarnBg": "Red",
        "StatsWarnFg": "White",
        "TextAttr": "None",
        "TextBg": "Black",
        "TextFg": "White"
      },
      "Client": {
        "DelimiterAttr": "Dim",
        "DelimiterBg": "Yellow",
        "DelimiterFg": "Black",
        "ClientAttr": "Dim",
        "ClientBg": "Yellow",
        "ClientFg": "Black",
        "HostnameAttr": "Dim",
        "HostnameBg": "Yellow",
        "HostnameFg": "Black",
        "TextAttr": "None",
        "TextBg": "Black",
        "TextFg": "White"
      },
      "Server": {
        "DelimiterAttr": "AttrDim",
        "DelimiterBg": "BgCyan",
        "DelimiterFg": "FgBlack",
        "ServerAttr": "AttrDim",
        "ServerBg": "BgCyan",
        "ServerFg": "FgBlack",
        "HostnameAttr": "AttrBold",
        "HostnameBg": "BgCyan",
        "HostnameFg": "FgBlack",
        "TextAttr": "AttrNone",
        "TextBg": "BgBlack",
        "TextFg": "FgWhite"
      },
      "Common": {
        "SeverityErrorAttr": "AttrBold",
        "SeverityErrorBg": "BgRed",
        "SeverityErrorFg": "FgWhite",
        "SeverityFatalAttr": "AttrBold",
        "SeverityFatalBg": "BgMagenta",
        "SeverityFatalFg": "FgWhite",
        "SeverityWarnAttr": "AttrBold",
        "SeverityWarnBg": "BgBlack",
        "SeverityWarnFg": "FgWhite"
      },
      "MaprTable": {
        "DataAttr": "AttrNone",
        "DataBg": "BgBlue",
        "DataFg": "FgWhite",
        "DelimiterAttr": "AttrDim",
        "DelimiterBg": "BgBlue",
        "DelimiterFg": "FgWhite",
        "HeaderAttr": "AttrBold",
        "HeaderBg": "BgBlue",
        "HeaderFg": "FgWhite",
        "HeaderDelimiterAttr": "AttrDim",
        "HeaderDelimiterBg": "BgBlue",
        "HeaderDelimiterFg": "FgWhite",
        "HeaderSortKeyAttr": "AttrUnderline",
        "HeaderGroupKeyAttr": "AttrReverse",
        "RawQueryAttr": "AttrDim",
        "RawQueryBg": "BgBlack",
        "RawQueryFg": "FgCyan"
      }
    }
  },
  ...
}

You notice the different sections - these are different contexts:

Remote: Color configuration for all log lines sent remotely from the server to the client.
Client: Color configuration for all lines produced by a DTail client by itself (e.g. status information).
Server: Color configuration for all lines produced by the DTail server by itself and sent to the client (e.g. server warnings or errors)
MaprTable: Color configuration for the map-reduce table output.
Common: Common color configuration used in various places (e.g. when it's not clear what's the current context of a line).

When you do so, make sure that you check your "dtail.json" against the JSON schema file. This is to ensure that you don't configure an invalid color accidentally (requires "jsonschema" to be installed on your computer). Furthermore, the schema file is also a good reference for all possible colors available:

jsonschema -i dtail.json schemas/dtail.schema.json

Serverless mode

All DTail commands can now operate on log files (and other text files) directly without any DTail server running. So there isn't a need anymore to install a DTail server when you are on the target server already anyway, like the following example shows:

% dtail --files /var/log/foo.log

% dmap --files /var/log/foo.log --query 'from TABLE select .... outfile result.csv'

The way it works in Go code is that a connection to a server is managed through an interface and in serverless mode DTail calls through that interface to the server code directly without any TCP/IP and SSH connection made in the background. This means, that the binaries are a bit larger (also ship with the code which normally would be executed by the server) but the increase of binary size is not much.

Shorthand flags

The "--files" from the previous example is now redundant. As a shorthand, It is now possible to do the following instead:

% dtail /var/log/foo.log

Of course, this also works with all other DTail client commands (dgrep, dcat, ... etc).

Spartan (aka plain) mode

There's a plain mode, which makes DTail only print out the "plain" text of the files operated on (without any DTail specific enriched output). E.g.:

% dcat --plain /etc/passwd > /etc/test
% diff /etc/test /etc/passwd  # Same content, no diff

This might be useful if you wanted to post-process the output.

Standard input pipe

In serverless mode, you might want to process your data in a pipeline. You can do that now too through an input pipe:

% dgrep --plain --regex 'somethingspecial' /var/log/foo.log |
    dmap --query 'from TABLE select .... outfile result.csv'

Or, use any other "standard" tool:

% awk '.....' < /some/file | dtail ....

New command dtailhealth

Prior to DTail 4, there was a flag for the "dtail" command to check the health of a remote DTail server (for use with monitoring system such as Nagios). That has been moved out to a separate binary to reduce complexity of the "dtail" command. The following checks whether DTail is operational on the current machine (you could also check a remote instance of DTail server, just adjust the server address).

% cat check_dtail.sh
#!/bin/sh

exec /usr/local/bin/dtailhealth --server localhost:2222

Improved documentation

Some features, such as custom log formats and the map-reduce query language, are now documented. Also, the examples have been updated to reflect the new features added. This also includes the new animated example Gifs (plus documentation how they were created).

I must admit that not all features are documented yet:

Server side scheduled map-reduce queries
Server side continuous map-reduce queries
Some more docs about terminal color customization
Some more docs about log levels

That will be added in one of the future releases.

Integration testing suite

DTail comes already with some unit tests, but what's new is a full integration testing suite which covers all common use cases of all the commands (dtail, dcat, dgrep, dmap) with a server backend and also in serverless mode.

How are the tests implemented? All integration tests are simply unit tests in the "./integrationtests" folder. They must be explicitly activated with:

% export DTAIL_INTEGRATION_TEST_RUN_MODE=yes

Once done, first compile all commands, and then run the integration tests:

% make
.
.
.
% go clean -testcache
% go test -race -v ./integrationtests

Improved code

Not that the code quality of DTail has been bad (I have been using Go vet and Go lint for previous releases and will keep using these), but this time I had new tools (such as SonarQube and BlackDuck) in my arsenal to:

Reduce the complexity of a couple of functions (splitting code up into several smaller functions)
Avoid repeating code (this version of DTail doesn't use Go generics yet, though).

Other than that, a lot of other code has been refactored as I saw fit.

Use of memory pools

DTail makes excessive use of string builder and byte buffer objects. For performance reasons, those are now re-used from memory pools.

What's next

DTail 5 won't be released any time soon I guess, but some 4.x.y releases will follow this year fore sure. I can think of:

New (but backwards compatible) features which don't require a new major version bump (some features have been requested at work internally).
Even more improved documentation.
Dependency updates.

I use usually DTail at work, but I have recently installed it on my personal OpenBSD machines too. I might write a small tutorial here (and I might also add the rc scripts as examples to one of the next DTail releases).

I am a bit busy at the moment with two other pet projects of mine (one internal work-project, and one personal one, the latter you will read about in the next couple of months). If you have ideas (or even a patch), then please don't hesitate to contact me (either via E-Mail or a request at GitHub).

E-Mail your comments to paul@nospam.buetow.org :-)

Other related posts are:

2021-04-22 DTail - The distributed log tail program
2022-03-06 The release of DTail 4.0.0 (You are currently reading this)
2022-10-30 Installing DTail on OpenBSD
2023-09-25 DTail usage examples

Thanks!

Paul

Back to the main site