Data store

The data store is the database at the centre of Stalwart. It holds every structured record the server uses, and the rest of the server is built on top of it. The choice of its backend also decides whether the server can run as a cluster.

The records in the data store fall into a few groups. Accounts, domains and roles describe who uses the server. Mailboxes, message metadata, flags and threads describe the mail, and calendars, events, address books and contacts describe the collaboration data. The change history lets clients synchronise only what has changed since their last visit. The mail queue and the server’s configuration objects complete the list.

Message contents, attachments, Sieve scripts and file contents are not structured records. They are written to the blob store instead, unless the blob store is set to keep them in the data store.

Because the configuration itself lives in the data store, the server must know where the data store is before it can read any other setting. Every other part of the server depends on it, so its availability and performance determine those of the whole server.

Backends

The backend is selected by the @type of the DataStore object (found in the WebUI under Settings › Storage › Data Store):

@type	Backend	Suited to
RocksDb	RocksDB, an embedded key-value database	Single nodes; the default
Sqlite	SQLite, an embedded SQL database in one file	Small single nodes
PostgreSql	PostgreSQL	Single nodes and clusters
MySql	MySQL or MariaDB	Single nodes and clusters
FoundationDb	FoundationDB, a distributed transactional key-value store	Large clusters

The first decision is between an embedded database and a database server. RocksDB and SQLite are embedded: the Stalwart process opens the database files itself, on the local disk, and needs no other service. For the same reason, several nodes cannot share them. A cluster therefore requires PostgreSQL, MySQL or FoundationDB, whose data every node can reach over the network.

Each variant carries its own fields. The embedded databases need a path to their files, while the SQL servers need a host, a database, an authUsername and an authSecret. The backend pages describe these fields, and the DataStore reference lists every one of them.

DataStore and config.json

The DataStore object is the only configuration object that is not kept in the data store, since the server needs it to find the data store in the first place. It is written instead to the node’s config.json file, which the server reads at startup (Configuration).

The object can be edited in the WebUI, the CLI or the JMAP API like any other object, but a change to it behaves differently in three ways. The first concerns clusters. A change is written only to the config.json of the node that served the request, and every node of a cluster must carry the same DataStore definition, so the change has to be made on each node.

The second concerns timing: a change takes effect only when the node restarts.

The third concerns the data. A change selects where the server looks for its data, and it does not copy any data. A node pointed at an empty database therefore starts without any of the existing accounts, messages or settings. Moving existing data to another backend is a separate task, described in Database migration.

Maintenance and tuning

Expired records are purged from the data store every day, according to the schedules of the DataRetention object. Data retention and reclaiming disk space describes the purge, together with the tasks that each backend requires of the operator. Connection pools, cache sizes and the other performance settings of each backend are discussed in Tuning the data store.