Domain lists

Some knowledge about domains cannot be derived from a message: whether a domain belongs to a reputable service, whether its mail is unwanted on this server, whether it hides links behind redirections, or whether it hands out free or disposable addresses. Several checks of the spam filter take that knowledge from lists of domains. Most of these lists are filled by the rule set that Stalwart downloads on first start, and administrators add their own entries to them.

The entries of the static lists are MemoryLookupKey objects. The namespace of an entry names the list, and its key holds the domain. The WebUI shows these objects under Settings › Lookups › In-Memory Keys, Settings › Spam Filter › Lists › Blocked Domains, Settings › Spam Filter › Lists › Spam Traps, Settings › Spam Filter › Lists › Trusted Domains, Settings › Spam Filter › Lists › URL Redirectors. The following object adds example.com to the trusted domains:

{
  "namespace": "trusted-domains",
  "key": "example.com"
}

The lists are lookup lists like any other. Rules and Sieve scripts can therefore test them with key_exists, and use them for checks of their own.

Trusted domains

The trusted-domains list holds domains whose mail is considered legitimate when it is authenticated, such as large mail providers and well-known services. The rule set provides about 800 entries.

A domain in the list has two effects. The first concerns the score. A message whose From domain is in the list and passes DMARC receives the TRUSTED_DOMAIN tag, which scores -7.0. Because DMARC must pass, a forged From address in a trusted domain does not receive the tag.

The second effect saves work. Addresses, domains and links in a trusted domain are not looked up in the DNS blocklists, which saves queries. Domains hosted on the server are treated as trusted for this purpose too.

Blocked domains

The blocked-domains list holds domains whose mail is refused. A message whose From domain is in the list receives the BLOCKED_DOMAIN tag, which the rule set defines as a reject tag, so the message is refused at the end of DATA whatever its score. The rule set provides no entries for this list.

URL redirectors

The url-redirectors list holds the domains of link-shortening and redirection services, such as bit.ly. The rule set provides about 1000 entries.

A link whose domain is in the list hides its real destination, so the filter follows the redirection, and analyses the final link, with its host and domain, in the same way as the other links of the message. The final destination is also what the AI classifier is given. A link to a redirector receives the REDIRECTOR_URL tag.

Redirections can be chained, so the filter sets a limit. At most three redirections are followed for each link. A longer chain receives the URL_REDIRECTOR_NESTED tag (1.0), and is not followed further.

Following a link means contacting a server chosen by the sender of the message, so the requests are restricted:

• Only https:// links are followed. A redirector link, or a redirection step, that uses plain http:// is never fetched, and the message receives the URL_REDIRECTOR_INSECURE tag (2.0).
• Only servers with public addresses are contacted. A link whose host resolves to a private, loopback or link-local address is not followed.
• Each request reads the Location of the redirection without following it automatically, and validates the server’s TLS certificate.

Each request also takes time, and a message can contain many redirector links. The following fields of SpamSettings limit the work done per message:

Field	Default	Limit
maxRedirectUrls	10	Redirection requests made for one message. Results found in the cache do not count.
redirectTimeout	5 seconds	Time allowed for each request.
redirectCacheTtl	1 day	How long a resolved link is remembered.

Resolved links are cached. They are kept in the spam lookup cache, which also holds the DNS blocklist results, and whose size is spamLookup on the Cache singleton. A request that fails is remembered for negativeTtl, 1 hour by default, so that the same broken link is not requested for every message.

Free and disposable mail providers

Free and disposable addresses cost nothing to obtain, which makes them attractive to spammers. Two lists identify addresses at such providers: one for providers of free mailboxes, and one for providers of disposable, short-lived addresses.

These two are not static lists. They are HTTP lookups that the rule set defines and Stalwart downloads periodically. stwt_free_domains is refreshed every 30 days, and stwt_disposable_domains daily.

An address at one of these providers receives a tag named after the provider type and the location, such as FREEMAIL_FROM or DISPOSABLE_REPLY_TO. The locations checked are the envelope sender and the From, Reply-To, To, Cc, Bcc and Disposition-Notification-To headers.

On their own, the tags score 0 in the rule set, except DISPOSABLE_DNT, which scores 0.5. Their value lies in combination: rules combine them with other findings, for example a From address at one free provider with a Reply-To at another. To stop downloading a list, set enable to false on its HTTP lookup.