Architecture

Written for someone about to change something and deciding where the change belongs.

The shape

                  provider seam                        sink seam
                       │                                   │
  ┌──────────┐         │    ┌───────┐    ┌────────┐        │   ┌──────────┐
  │  Gmail   │─────────┼───▶│ parse │───▶│ filter │────────┼──▶│ .eml     │
  │  (IMAP)  │         │    │       │    │ engine │        │   │ markdown │
  └──────────┘         │    └───────┘    └────────┘        │   └──────────┘
       ▲               │                      ▲                      │
       │               │                      │                      ▼
  ┌──────────┐         │                ┌───────────┐          files on disk
  │  state   │◀────────┴────────────────│domains.txt│
  │  store   │      cursor saved        │(owned by  │
  └──────────┘      after a cycle       │ another   │
                                        │ program)  │
                                        └───────────┘

Data flows left to right and never back. Nothing downstream of the provider seam knows what a provider is; nothing upstream of the sink seam knows what a file is.

Packages

Package	Owns	Depends on
`cmd/mail-muncher`	Cobra command tree, flags, exit codes, logger setup. No behavior.	`internal/config`, `internal/filter`, `internal/provider/gmail`, `internal/pipeline`
`internal/config`	YAML schema, defaults, path expansion, validation.	`gopkg.in/yaml.v3` only
`internal/model`	The canonical `Message` and RFC822 parsing.	`enmime`
`internal/provider`	The `Provider` interface, `RawMessage`, `SyncState`, retry/backoff, a `Fake`.	nothing internal
`internal/provider/gmail`	OAuth, token cache, list/history/RAW fetch, label resolution.	`internal/provider`, `internal/config`
`internal/provider/imap`	`password_cmd`, TLS dial and login, per-mailbox UID cursors, `BODY.PEEK[]` fetch.	`internal/provider`, `internal/config`, `go-imap/v2`
`internal/filter`	Match-tree compilation, the `Engine`, the domain-file cache.	`internal/config`, `internal/model`
`internal/sink`	On-disk layout, `.eml` and markdown writers, atomic writes.	`internal/config`, `internal/model`
`internal/state`	Per-account sync state files and the `flock`-based lockfile.	`internal/provider`
`internal/pipeline`	One cycle: fetch → parse → evaluate → sink → save state. The manifest, the exit-code grading, quarantine.	everything above
`internal/mcpserver`	The MCP tools, the archive index over the `dest` trees, and the path jail.	`internal/config`, `internal/model`, `internal/sink`, `internal/filter`, `internal/pipeline`

The dependency graph is acyclic and shallow on purpose. internal/config at the bottom knows about no other internal package; internal/pipeline and internal/mcpserver at the top are the only places that know about all of them.

internal/mcpserver sits beside the pipeline rather than above it: it reads the archive on disk directly, and reaches into the pipeline only through Runner.Cycle for the sync tool. It never touches a provider.

The two seams

The provider seam — pluggable input

internal/provider.Provider is the architectural invariant:

type Provider interface {
	Name() string
	Fetch(ctx context.Context, state SyncState, fn func(RawMessage) error) (SyncState, error)
}

A provider streams RawMessage values — a provider-scoped stable id, complete RFC822 bytes, an internal date, and label names — and returns an updated SyncState. Everything downstream sees those two types and never a provider SDK type. That is what makes the choice of Gmail-over-IMAP a reversible decision rather than a one-way door.

The test of the seam: adding a provider must require zero changes to internal/pipeline, internal/filter, or internal/sink. If your new provider needs a pipeline change, the seam is wrong and the fix belongs in the interface, not in a special case downstream.

Three details the interface deliberately pins down:

ID must be stable across runs. It is half the input to the sinks’ idempotency digest, so an unstable id means re-downloading and re-writing the same message under a new name forever. Gmail uses the message id; IMAP uses account:mailbox:uidvalidity:uid.
Raw must be byte-faithful. The .eml sink writes it verbatim and internal/model parses from it. Re-encoding anywhere upstream breaks DKIM verification for every archived message.
SyncState is carried through, not owned. A provider updates the fields it understands (HistoryID for Gmail, Extra keys for others) and leaves the rest alone. SeenIDs belongs to the pipeline.

Fetch streams rather than returning a slice so a large first scan does not have to fit in memory, and so the pipeline can store as it goes.

The sink seam — pluggable output

internal/sink.Sink is the other end:

type Sink interface {
	Format() config.Format
	Path(msg *model.Message, rule *config.Rule) string
	Plan(msg *model.Message, rule *config.Rule) (path string, skipped bool, err error)
	Store(msg *model.Message, rule *config.Rule) (path string, skipped bool, err error)
}

Path is pure and does no I/O. Plan adds exactly one Stat. Store writes. The split is what makes --dry-run honest: it reports the precise path a real run would produce without touching the destination tree.

Sinks hold no per-message state and are safe to reuse across messages and rules. sink.ForRule returns one sink per format on a rule, in the rule’s order, collapsing duplicates — that is the fan-out point.

A new output format is a new Sink implementation plus a config.Format constant plus a case in sink.New. Nothing else changes.

Reading back: the archive

internal/mcpserver is the only code that reads the archive rather than writing it, and it treats the on-disk layout as the schema: it parses the [:16] digest out of a basename as the message id, reads the .md frontmatter for metadata, and falls back to parsing the .eml when there is no markdown rendering. The index is lazy and keyed by size and mtime, so an unchanged file is never re-parsed.

Its one security-relevant invariant is the path jail. Every rule dest is resolved once at construction; a caller-supplied path is then checked lexically against those roots, resolved with filepath.EvalSymlinks, and checked again against the resolved roots. Two checks, because either alone is defeatable: lexical alone follows a planted symlink, and post-resolution alone can be raced. Only .eml and .md are readable, the walk never follows symlinks, and a refusal is deliberately indistinguishable from “no such file” so the jail cannot be used to probe the filesystem.

The invariants

Break any of these and something subtle breaks downstream.

Identity determines path. sink.Basename is <unix-seconds>-<sha256(account + ":" + id)[:16]>-<slug(subject)>, where the width comes from the exported sink.HashLen. Only the digest carries identity; the timestamp and slug are for humans. This is why:

Re-running is safe. The file being there means an earlier cycle wrote it.
The slug is ASCII-only. Non-ASCII filenames are subject to filesystem Unicode normalization (HFS+ stores NFD), which can make the name written differ from the name the next cycle stats for — and that existence check is the entire idempotency story.
Renaming an account or changing the digest input rewrites every future filename and orphans everything already archived.
The fragment is public layout, not an implementation detail: internal/mcpserver parses it back out of a filename as the message id it reports to agents. Changing HashLen changes an API.

The seen set is belt and braces, not the idempotency key. provider.SyncState.SeenIDs is capped at 2000, FIFO, so any scan larger than that evicts its earliest ids. This is harmless and must stay that way: the real idempotency key is the deterministic filename plus skip-on-exists. A re-run after eviction re-downloads the message and then skips it at the write, costing bandwidth and nothing else. Do not “fix” this by unbounding the set — an unbounded set grows without limit in a file that is rewritten every cycle, to buy a guarantee the sinks already provide.

Writes claim a name, they do not replace one. A message file goes to a temp file in its destination directory, is fsynced and chmodded, and is then hard-linked into place with os.Link. link(2) fails with EEXIST rather than clobbering — unlike rename(2), which cannot express “only if absent” — so the kernel decides whether the name is free at the instant it is claimed. That is the idempotency check, not a step after one: there is no window between deciding and acting. A filesystem without hard links falls back to O_CREAT|O_EXCL, trading the no-partial-file guarantee for the same no-clobber one.

Attachments and state files still use temp-and-rename, deliberately: their names are not idempotency markers, so replacing is correct for them.

Symlinks in the layout are refused, never followed. Nothing a sink writes is legitimately a link, so one means something else is placing them there. ErrSymlink covers a link at a message’s final path and at any directory component the sink itself creates (<YYYY>, <MM>, <basename>.attachments). The rule’s own dest: is exempt — the operator chose that path. Existence checks use Lstat, not Stat, so a link planted at a destination is an error rather than a silent “already stored”.

Message-level errors never abort a cycle by default. A message that will not parse, or one whose renderings failed to write, is handled by on_message_failure: quarantine (the default) parks the raw bytes under the quarantine directory, counts it, and lets the cursor advance; abort returns a *MessageError and the account’s state is not saved. Either way the message is never both undelivered and forgotten. Provider and auth errors always abort the account — they will not get better by trying the next message — but the other accounts still run.

Domain files are read once per cycle, and unreadable is a reportable event. filter.Engine.BeginCycle clears the cache and Preload re-reads every referenced list up front, so Engine.Degraded() is authoritative before the first message is fetched. That ordering is what makes on_degraded_filter: fail implementable at all: it can only promise to store nothing if it knows first. A missing file is still not an evaluation error — the predicate matches nothing — but it is reported, and by default (hold) it stops the cursor from advancing.

State advances past unmatched messages — unless the filter was degraded. A message evaluated once is normally never evaluated again, whether or not a rule claimed it. The exception is deliberate: under hold or abort the cursor is held, precisely so mail evaluated against an unreadable list is evaluated again. Changing rules still does not retroactively archive old mail; deleting a state file and re-scanning does.

There are two locks, and they answer different questions.

<state_dir>/mail-muncher.lock — flock-based, held for the duration of each cycle by run, by every daemon tick, and by the MCP sync tool. It answers “is a cycle running right now?” and is what stops a cron invocation racing a daemon on the same cursors.
<state_dir>/instance/mail-muncher.lock — held by a daemon for its whole process lifetime. It answers “is something else already polling this state directory?”, which the cycle lock cannot: that one is released between ticks, so a second daemon starting while the first sleeps would pass it and then poll forever alongside it. It lives in a sub-directory so it cannot be confused with the cycle lock.

Both return an error matching state.ErrLocked, which pipeline.ExitCode grades as exit 3.

There is one manifest shape. pipeline.Manifest is what run --json writes, what daemon --json streams one line at a time, and what the MCP sync tool returns. Its JSON tags are API surface, not debug output — see manifest.md. A new field must be additive and omitempty unless it is genuinely always present.

Where does my change go?

Change	Where
A new config key	`internal/config/config.go` (struct + default), `validate.go` (rule + severity), and `docs/configuration.md`. Add it to a test in `config_test.go`.
A new match predicate	`internal/filter/compile.go` (a `compileX` function, a case in `compileNode`, an entry in `predicates`), the package doc in `doc.go`, and `docs/filters.md`. If it needs message data that does not exist yet, add an accessor to `internal/model/message.go` first.
A new predicate whose value is a file path	Also add the key to `filePathPredicates` in `internal/config/expand.go`, or its `~`/`$VAR` will not expand and `validate` will not check it.
A new output format	A new `Sink` in `internal/sink`, a `config.Format` constant, a case in `sink.New`, and `validateFormats` prose.
A new provider	A package under `internal/provider/`, implementing `Provider`. A config block, a case in `validateAccounts`, and a case in `pipeline.DefaultProviderFactory` — that switch is the one line outside `internal/provider` that may change. Nothing else in pipeline, and nothing in filter or sink.
Anything about what a message exposes	`internal/model`. Parsing lives in `parse.go`, accessors in `message.go`.
A new CLI flag	`cmd/mail-muncher`. Keep behavior in an internal package and the command thin.
A new manifest field	`internal/pipeline/manifest.go`, and `docs/manifest.md`. Additive and `omitempty`; the JSON tags are API surface.
A new MCP tool, or an argument on one	`internal/mcpserver/tools.go` (the input/output structs — schemas are inferred from them, so they cannot drift), `server.go` (`registerTools`, `ToolNames`), and `docs/mcp.md`. Anything that reads a file must go through `Archive.Resolve`.
Retry or backoff behavior	`internal/provider/retry.go`, which is shared. Gmail’s judgment of which errors are retryable lives in `gmail.go` (`retryable`, `isRateLimited`) because it needs the error body.

Design decisions worth knowing

Gmail API first, IMAP second — and the seam held. The Gmail API gives OAuth, format=RAW (so storage and parsing stay provider-neutral anyway), labels as data, and users.history for cheap incremental sync, which made it the cheaper first implementation. IMAP was the test of the claim above, and it passed: adding it took a package under internal/provider/, a config block, and one case in DefaultProviderFactory. internal/pipeline, internal/filter and internal/sink were not touched.

The two things IMAP needed that Gmail did not, it got from the interface as it already stood. Per-mailbox UID bookkeeping lives in SyncState.Extra, which is free-form per-provider state. And an empty RawMessage.ThreadID is not a gap: IMAP has no conversation id, and model.Parse already synthesized one from the References chain for exactly this case.

A UIDVALIDITY change is a resync, not a hint. When an IMAP server’s UIDVALIDITY differs from the stored one, the mailbox’s UID cursor is discarded and the fetch falls back to initial_lookback. That is the protocol saying every UID you remember now names a different message, and honouring it half-heartedly loses or duplicates mail silently. The UIDVALIDITY is also part of the message id, so the resync re-archives its window under fresh filenames rather than colliding with what the mailbox’s previous incarnation left behind: duplicated mail is recoverable, skipped mail is not.

The history cursor is best-effort. Gmail keeps roughly a week of history. A 404 from users.history.list is not an error: the cursor is cleared and the same cycle falls through to a full scan. The full-scan path is not a legacy path — it is the permanent fallback, and it must keep working.

The mailbox cursor is read before listing, not after. In fullScan, users.getProfile is called first. Mail arriving during a scan then still shows up in the next incremental cycle, and the seen-set absorbs the overlap. Taking the cursor afterwards would silently skip those messages.

gmail.query is an optimization, not a filter, and it applies to the first-ever scan only. The gate is firstEver := state.HistoryID == 0 && state.LastSyncTime.IsZero(), computed from the incoming state before the expired-cursor branch clears anything — so a recovery scan is not a first run even though it ends up on the same code path, and it deliberately sends no query. History results are not query-filtered either, and the query is never re-applied locally, because the filter engine is the single authority on what is kept. Two places that both decide what is archived would eventually disagree.

Narrowing the query is therefore only ever a way to make one first run cheaper. It cannot bound steady-state behavior, and a query that excludes something your rules would keep makes the two scan modes inconsistent for no benefit.

Spam and Trash are excluded by default, on both paths. gmail.include_spam_trash defaults to false, and the default is a security decision: delivered mail ends up in an LLM’s context window, and Spam is the highest-density source of hostile, attacker-authored text a mailbox contains.

The interesting part is not the default but the invariant underneath it. The history path has no includeSpamTrash parameter and reports everything added anywhere in the mailbox, so whichever way the setting points, the two paths must be made to point the same way — a full scan that enumerated a different population than an incremental cycle over the same window would lose mail silently, because the expired-cursor fallback installs a fresh cursor on the way out and never revisits what it skipped. So:

The full scan passes the setting straight to IncludeSpamTrash, on first-ever and recovery scans alike.
The incremental path cannot ask the server, so it downloads and then drops messages carrying SPAM or TRASH before they reach the pipeline — one messages.get spent to learn what a message is and then discard it. The same predicate also runs on the full-scan path, where it is normally a no-op; one predicate on both routes is what makes the agreement structural.

A dropped message is not an error and not a parse failure. It is not marked seen, either: the seen-set records deliveries, and leaving excluded ids out of it is what keeps the two paths’ persisted state identical. Nothing loops as a result — the history cursor advances past a message whatever we decide about it, and a later full scan excludes it server-side.

Once fetched, rules remain the only authority on what is kept: with the key on, not: {label: SPAM} is how you take one folder and not the other.

Recovery scans overlap the watermark by 24 hours. RecoveryOverlap is subtracted from the stored LastSyncTime before it becomes the after: bound, so mail that arrived while the previous cycle was listing is not stepped over. The overlap re-fetches a day of mail and then skips all of it at the write — cheap insurance bought with the idempotency the sinks already guarantee.

Parsing never panics. model.Parse recovers panics from the MIME parser and returns them as errors. Mail is hostile input from strangers; one malformed message must not kill a cycle.

Markdown is a rendering, not a copy. It is lossy by design — most visibly, inline cid: images stay unresolved links. The .eml is the fidelity copy, and any feature request of the form “make markdown byte-accurate” is really a request to read the .eml.

Read-only is structural on both providers, by different means. Gmail requests only gmail.readonly: there is no write path to remove, because there is no write path. IMAP has no scopes to lean on, so it is enforced twice in the client — folders are opened with EXAMINE rather than SELECT, and bodies are fetched with BODY.PEEK[] rather than BODY[]. Either alone suffices on a compliant server; both are used because a bare BODY[] sets \Seen on mail a human is also reading, and nothing behind that one field would stop it. It has a test whose negative control proves the test can fail.

Testing the seams

internal/provider.Fake replays canned RawMessage values and can simulate a fetch dying partway through with partial state. It is how the pipeline gets tested end to end with no network:

p := provider.NewFake("gmail", msgs...)

The Gmail provider itself is tested against an httptest server holding canned JSON, via gmail.NewWithHTTPClient(ctx, account, srv.Client(), srv.URL, opts). The IMAP provider is tested against go-imap’s in-memory server on a loopback socket — a real server speaking the real protocol, because the behaviours that matter there (UIDVALIDITY, the n:* range quirk, \Seen on a non-PEEK fetch) are the server’s behaviours, and a hand-written fake would only assert what we already believed. Options.TLSConfig lets a test mint a throwaway certificate so the default tls: true path is exercised rather than assumed; there is no config key for it, and there should not be one.

No test in this repo touches the network. See CONTRIBUTING.md.