Regaining control of your mailbox (neomutt, notmuch, mhonarc)

I believe in the UNIX philosophy, it says that a computer program should do one thing and do it well, suggesting that we should have a dedicated tool for every small task, and then they can be combined to achieve a bigger goal with a great flexebility. In this post I’ll show how I apply this philosophy to my emails and what advantages this approach can open in the age of web-based mail clients.

The goals of my setup are:

1. Convenient key bindings and a text editor of my choice to work with e-mails
2. Shell-scripting for searching, sorting, tagging, slicing, dicing and doing stuff automatically with my e-mails
3. Control of what is kept on the mail server vs locally.
4. Ability to read (and write) mail offline
5. No bloated, slow and thus frustrating software

Mail folders: mbsync + neomutt

To regain control of our mailbox we need to have it synced to our local computer, then do the stuff using our powerful tools, and then sync the changes to the server. So to begin - we need to enable IMAP protocol on the server and additionally enable message deletion over IMAP (so we can delete or move emails locally and it will be reflected on the server).

Your mailbox on your mail server probably consists of a number of standard folders such as INCOMING, Trash, Spam, Drafts, Sent, etc. My goal is to keep my incoming folder empty after reading the new mail, so for the messages that I want to retain on the server I created an additional folder called ‘Retained’. It’s important to have these folders named using only Latin letters, otherwise isync will turn the names to unreadable garbage after the syncing. There is no any point in having additional folders to categorize mail contents (we will see a more robust approach for mail sorting and categorization soon).

For syncing mailboxes we will use mbsync(isync). For manual work with emails we will use neomutt mail reader.

In short, isync keeps mail in Maildir format on your local computer and takes care of syncing the state of your maildir with the server. neomutt - is a pretty minimalistic mail client and the main program in this setup, it works from a terminal, has customizable keybindings and its functionality can be extended through external scripts.

In addition to the folders synced with the server we need one local folder, which will store the messages that we want to keep only locally (so the evil hackers won’t find your embarassing photos, after hacking into your mail account). Here is an example from the .muttrc:

mailboxes "=INBOX" "=Retained" "=Drafts" "=Sent" "=Spam" "=Trash" "=../LocalStorage"
macro index,pager gr "<change-folder>=Retained<enter>" "go to Retained"
macro index,pager gl "<change-folder>=../LocalStorage<enter>" "go to LocalStorage"
macro index,pager Mr "<save-message>=Retained<return><return><check-stats>" "move mail to Retained"
macro index,pager Ml "<save-message>=../LocalStorage<return><return><check-stats>" "move mail to LocalStorage"

We add Local Storage folder (using a relative path) and some useful key bindings to work with our custom folders. Don’t add Local Storage to isync config cause it should not be synced. Local Storage can be shared between multiple email accounts if you want to.

Finally, its a good idea to keep local copies of the mails in an encrypted file system see my older post, and you probably should exclude synced folders from your backups, otherwise messages deleted from the server will pop up again after restoring the backup.

Massaging mails with notmuch and shell-scripts

Only one word - notmuch. Well, technically its two words joined together, but its a single program. This is an awesome piece of software that indexes your Maildirs , maintain tags, supports various search queries and works from a command line. In the .notmuch-config, point path to your root mail directory (which contains all of your mail in its subdirectories), notmuch will create .notmuch directory in this folder and will keep the index of all your mail there.

Now, lets write a simple script that processes all new mail in Maildir and assigns tags to the new messages:

notmuch new

notmuch tag +family -new tag:new and from:bro@something.org
notmuch tag +family -new tag:new and from:sis@something.org
notmuch tag +social_media -new tag:new and from:@twitter.com
notmuch tag +unsorted -new tag:new

‘notmuch new’ - is a command to incorporate new mail into the notmuch database. All other commands in this snippet assume that notmuch assigns ‘new’ tag to all new mail (this can be set in the notmuch config). The commands are self-explanatory, we assign tags: social_media, family and unsorted, and remove the ‘new’ tag from the messages found by the given search criteria. (Using of ‘tag:new’ in the searches makes it work faster, cause we scan only new mail).

Now since we have different tags for different mail, lets see them sorted in neomutt. If neomutt was compiled with notmuch support this should work:

set nm_query_type = "threads"
virtual-mailboxes "v_Family" "notmuch://?query=tag:family"
virtual-mailboxes "v_Soc-Media" "notmuch://?query=tag:social_media"
virtual-mailboxes "v_Unsorted" "notmuch://?query=tag:unsorted"

With the right nm\_default\_url value set in your config, you should see new mailboxes containing appropriate messages independently from their physical location in your maildir (LocalStorage, INBOX, Sent, Drafts etc.).

Now, lets add some manipulations. Since all mail in Maildir is just a set of files you can use bash and notmuch together to perform actions with your mail. For example lets move all social-media mails from INCOMING to trash.

INBOX_PATH="/home/<user/path_to_mail>/INBOX/cur"

notmuch search --output=files tag:social_media | grep ^${INBOX_PATH}| sed -e 's@$@ /home/<user/path_to_trash>@' |xargs -L1 mv

Using of ‘–output=files’ is the key. From here on, your imagination is the only limit. For example you can automatically extract receipts from your mail and save them to appropriate places or trigger some actions when you get a mail from a particular person. Or you just can spend a week cleaning and sorting mail in your Gmail account since 2005 and find some interesting and forgoten stuff.

Viewing html emails

HTML emails is an abomination of all time. Unfortunately it’s hardly possible to avoid this bloat nowadays, so here is my solution if you sometimes need to view them.

First, I don’t use any browser with JavaScript support to view html e-mails, it’s simply not safe. I need to render them in a lightweight browser.

Bind v key in neomutt to pipe the message contents to an external script

bind index,pager v  noop        
macro index,pager v "<pipe-message>view_html_mail<return>" "view in browser"

And here is my ‘view_html_mail’ script:

#!/bin/bash

TMP_DIR=`mktemp -d`

function cleanup {      
  rm -rf "$TMP_DIR"
}

trap cleanup EXIT

mhonarc --attachmentdir $TMP\_DIR -single\
| grep -v "Arc-authentication-results"\
| grep -v "Arc-message-signature"\
| grep -v "Arc-seal"\
| grep -v "Dkim-signature"\
> $TMP_DIR/message.html
w3m  "${TMP_DIR}/message.html"

It uses mhonarc to convert an email (MIME content) to a html file. Mhonarc extracts the attached images and converts image links to the paths in the file system there they can be viewed by the browser. By default Mhonarc strips every non “cid:” url from the links, in the elements like <img src="..">, so if the mail contains internet links to the images they will not render (it will render only images attached to the message). That is done for privacy and security reasons to avoid tracking and XSS-attacks. At your own risk you can force Mhonarc to allow “no cid” urls in the mails in a rc file, but its not recommended. I also remove some big mail headers to make the result more readable.

Conclusion

All five goals have been achieved by using only FOSS. You are still able to use a web client as a ‘normy’ when you need or want to, but now you have the power and ability to ramp up your mail infrastructure how you like it.