Overview

Events are only useful once they are recorded somewhere an administrator can read them. A tracer does that job: it takes Stalwart’s events and sends them to one destination, such as a log file or an OpenTelemetry collector. Each tracer is a Tracer object (found in the WebUI under Settings › Telemetry › Tracers), whose @type selects the destination:

@type	Destination	Page
Log	Text files in a directory, with a new file started at a set interval	Log file
Stdout	The console (standard error)	Console
Journal	The systemd journal	Journal
OtelHttp, OtelGrpc	An OpenTelemetry collector, over HTTP or gRPC	OpenTelemetry

Several tracers can be active at the same time, each with its own settings, so one destination can receive a detailed record while another receives only problems. A new installation has one Log tracer at level info.

Common settings

Whatever its destination, every tracer answers the same questions: how severe an event must be to be recorded, which events to record, and what to do under load. The following fields express these choices:

• level is the least severe level recorded: error, warn, info, debug or trace, with info by default. The levels are described in Events.
• events and eventsPolicy include or exclude events by name. With the default policy, exclude, and an empty list, every event at or above the level is recorded.
• lossy selects how the tracer behaves when it cannot keep up, as described below.
• enable turns the tracer off without deleting it.

Level and event selection combine. The following tracer, meant for a troubleshooting session, writes four SMTP and delivery events to the console. Its level is debug, because one of the events, delivery.attempt-start, is logged at that level:

{
  "@type": "Stdout",
  "level": "debug",
  "eventsPolicy": "include",
  "events": {
    "smtp.mail-from": true,
    "smtp.rcpt-to": true,
    "delivery.attempt-start": true,
    "delivery.delivered": true
  }
}

Event loss under load

The server passes events from its threads to the tracers through bounded queues, so that a slow destination cannot stall mail processing. The price of that design is that events can be lost. When a tracer falls behind, its backlog grows up to a limit, and past that limit events are dropped.

Which events are lost depends on lossy, which is disabled by default:

• A tracer with lossy set to false keeps a large backlog, and drops the newest events only when the backlog is exceeded.
• A tracer with lossy set to true keeps a smaller backlog and, once it is exceeded, records only error events.

Events can also be dropped before they reach any tracer, regardless of lossy, when the server produces them faster than it can distribute them.

For every 60 seconds in which events were dropped, the server logs a telemetry.events-dropped event at warn level, with the number of events lost in total. Its presence means that the logs of that period are incomplete. The usual cause is a high event volume, typically from a tracer at debug or trace.

Reading the logs

Line format

The Log and Stdout tracers write one line per event, in the following format:

<timestamp> <LEVEL> <description> (<event>) <key> = <value>, <key> = <value>, ...

The timestamp is in UTC, with a precision of one second. The following line illustrates the format, although the keys shown vary from event to event:

2026-01-14T09:12:44Z INFO Message delivered (delivery.delivered) queueId = 1834620912, from = "alice@example.org", to = "bob@example.net", hostname = "mx.example.net", code = 250, elapsed = 812ms

With the multiline option, each key is written on its own line instead. Each event also includes the keys of the session in which it occurred, such as the remote address and the listener, so a line can be understood without looking for the start of its session.

Related lines

A single message or connection produces many lines. Two keys connect them:

• queueId identifies a message in the outbound queue, from the moment it is queued to its final delivery or bounce. Searching the log for the queue identifier shows the life of one message.
• spanId identifies a session, such as one SMTP connection. The text format omits it, but the journal, OpenTelemetry and webhooks include it, and the stored delivery history uses it.

Missing events

Several frequently searched events are logged at debug, so they do not appear at the default level. Authentication failures and spam filter verdicts are among them. Events lists them and shows how to raise individual events to info. Database queries are logged at trace.

Parsing logs

The log format is designed for people reading logs, not for programs. Event names, keys, levels and the line layout can change between releases, so tools that parse the logs, such as fail2ban or CrowdSec filters, may need to be adjusted after an upgrade. Stalwart’s own automatic IP blocking covers authentication failures and similar abuse without log parsing.