Using mutt for linux kernel mailing list

or: how I tried to get rid of Thunderbird and live happily

Suggestions, corrections, everything: jacopo [aT] jmondi [DoT] org
tags: mutt, linux kernel patches, imapfilter, offlineimap

Problem statement

Linux kernel development is all about sending and receiving patches by email.
Like it or not, this is by far the most efficient way to get the job done (most of the times) for a community at this scale.

Number of patches flowing day in day out is impressive, and the most difficult part is to not get overwhelmed by the huge amount of traffic.
There's a lot of number and statistics out there about the LKML traffic volume, to me it's enough to say I collected more than 55k emails in a 3 months time span. That's huge and un-manageable for a human being without a proper setup (and quite some time to spend reading emails).
The bare minimum requirements for a working setup (make sure you see what you're intended to see, do not get lost in email threads, keep track of relevant patches etc) doesn't sound that hard to be achieved, at least theoretically...
Well, it actually is and turns out that, to me at least, most of well-known email clients out there are not that well-suited for the task.

So, I said, why not give mutt a spin? I already use it for some personal email account, but I never used it for my daily job, for sure not for a mail-centric one as Linux kernel development is.
I had a small list of requirements to be respected to consider a solution satisfactory enough to finally get rid of Thunderbird, also because I spent quite some time tuning Thunderbird email filters, labeling and such to have it working properly.

* filter emails to redirect them in appropriate sub-folders

I keep mails directed to different mailing lists I follow in separate sub-folders. I would like to have them sorted out before I fetch them.

* see my replies in the same folder as the original email I replied to

This is somehow similar to Gmail's "conversation" mode I quite like

* be able to easily save/apply patches from the mailbox

Thunderbird sucked at this. I always found the "save as" method cumbersome, but maybe there are better ways of doing the same I'm not aware of.

Not that much to ask for, right? RIGHT???

My setup

First, I do not run my own mail server. Bad me, I know.
My ISP provides me an IMAP account where I fetch emails from, and of course an SMTP rely I use to send mails out.
So far I only used Thunderbird as mail retrieval agent (see definitions of the several components that play a role in sending/receiving emails here: Basic email concepts). I admit I was not aware of the complexity of such a system, as another undesirable side-effect of multi-purpose application like Thunderbird is hiding the complexity behind it. What Thunderbird did for me was in reality playing several distinct roles such as acting as mail transfer agent, retrieve email, filtering and synchronizing folders etc.. This is not bad per se, it simply makes harder getting to know what's going on behind the curtains once you finally want to have a look...

After some research, I found out several components that may replace what Thunderbird did for me:

* MUA: the mail client, where to read and send email from. That would be mutt, as you may have guessed already
* MRA: the retrieval agent. Mutt implements MRA functionality, but I prefer to use an external program: offlineimap.
* MTA: I could have used external transfer agents, such as msmtp but since mutt implements it already, I used mutt for connecting to the SMTP rely.
* Filtring/sorting: I want emails be filtered -before- I fetch them. As I do not run my own mail server, I could not have used postfix or similar tools. I have found imapfilter to be an optimal solution.
* Archiving: archivemail is an effective email archiving solution. It archives mail in tar.gz formats, fetching them directly from the remote IMAP folder.
* Address book: I have setup khal to synchronize with my DAV server, and neo-mutt understand auto-completion of email addresses!

Let's see how all this pieces fit together then...

Mutt

I won't spend that much time on this, as there are plenty of examples online, and I'm under the impression one could spend an entire life tuning it's own configuration files.
Generally speaking, I have instructed mutt that it will find emails in a local folder Maildir folder, synchronized my the MRA with the remote IMAP folder.
As mutt serves as MTA, I supplied it with my SMTP rely information, so it can send emails without using an external tool.

Some configuration snippets:


$cat ~/.mutt/muttrc

#setting for use with offlineimap: use mutt as MDA only
set mbox_type=Maildir
set folder=~/mail/jmondi
set spoolfile=+INBOX
set header_cache = ~/.cache/mutt

#setting for remote IMAP
set folder=imaps://mail.xxxx.xxx:xxx/

#use mutt as MTA with built-in smtp agent
set smtp_url=smtps://$my_user@mail.xxxx.xxx:465
set ssl_force_tls = yes
set copy=yes
set record=+Sent

set smart_wrap

set editor="/usr/bin/vim -c 'set tw=70 et' '+/^$'"
set send_charset="utf-8"

offlineimap

The retrieval agent: connects to IMAP server and locally synchronize folders (where mutt expects to find emails in Maildir format). A simple configuration for offlineimap:
$cat ~/.offlineimaprc

[general]
ui = ttyui
accounts = jmondi

[Account jmondi]
localrepository = jmondi-locale
remoterepository = jmondi-remote

[Repository jmondi-locale]
type = Maildir
localfolders = ~/mail/jmondi/

[Repository jmondi-remote]
type = IMAP
remotehost = mail.xxxx.xxx
remoteuser = jacopo@xxxx.xxx
remotepass = xxxxxxxxxxxxxxxxxxxxxx
realdelete = no
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
Pretty simple here: we instruct offlineimap where to save emails it has to fetch from the remote repository.
The local repository path shall match where mutt expects to find emails of course.

imapfilter

This is interesting: remember my requirements on filtering emails according to mailing list they where directed to? imapfilter can do that, in quite an effective way through Lua scripts that process emails directly on the remote IMAP folder.

You have been tempted to stop reading right away when you read Lua, weren't you? Well, I was, and even if I never wrote a single line of that yet-another-dynamically-typed-language I've been able to filter my emails quite nicely with some basic function (and some help from on-line resources I copied bits from :)

Here it is a hopefully enough commented Lua configuration script:


$cat .imapfilter/config.lua


options.timeout = 120
options.info = true

remoteimap = IMAP {
    server = "mail.xxxx.xxx",
    username = "jacopo@xxxx.xxx",
    password = "xxxxxxxxxxxxxxxxxxxxxxxxxx",
    ssl = 'tls'
}

remoteimap.INBOX:check_status()
newmesg = remoteimap.INBOX:is_unseen()
sentmesg = remoteimap.Sent:is_unseen()

-- Given a rule, apply it on msgs
runFilter = function (msgs, rule )
    for _,fields in pairs(rule) do
            local subres = {}
            local destination = fields["dest"][1]
            subres = msgs:contain_field(fields["field"], fields["value"])
            if subres:move_messages(fields["dest"]) == false then
                    print("Moved failed")
            end
    end
end

-- I could have done this better, I know...
runFilter_copy = function (msgs, rule )
    for _,fields in pairs(rule) do
            local subres = {}
            local destination = fields["dest"][1]
            subres = msgs:contain_field(fields["field"], fields["value"])
            if subres:copy_messages(fields["dest"]) == false then
                    print("Moved failed")
            end
    end
end

------
-- Mail Filters: sort by Mailing list ID
-----
ml_filters = {
        { field = "X-Mailing-List", value="linux-renesas-soc@vger.kernel.org", dest = remoteimap['INBOX/linux-renesas'] },
        { field = "X-Mailing-List", value="linux-gpio@vger.kernel.org", dest = remoteimap['INBOX/linux-gpio'] },
        { field = "X-Mailing-List", value="linux-media@vger.kernel.org", dest = remoteimap['INBOX/linux-media'] },
        { field = "List-Id", value="", dest = remoteimap['INBOX/linux-amlogic'] },
        { field = "List-Id", value="Greybus Development Mail List ", dest = remoteimap['INBOX/linux-greybus'] },
        { field = "List-Id", value="Peri Peri Discussions ", dest = remoteimap['INBOX/periperi'] },
        { field = "X-Mailing-List", value="linux-kernel@vger.kernel.org", dest = remoteimap['INBOX/linux-kernel'] },
}

-- Mark all messages directed to myself
-- '+' works as "OR"
local submsg = {}
submsg = newmesg:contain_to("jacopo@xxxx.xxx") +
         newmesg:contain_to("jacopo+xxxxxx@xxxx.xxx")
submsg:mark_flagged()

-- Now filter to specific mailing lists
runFilter(newmesg, ml_filters)

-- Copy single replies to inbox (like Gmail's "conversation" mode)
submsg = {}
submsg = sentmesg:contain_from("jacopo@xxxx.xxx") +
         sentmesg:contain_to("jacopo+xxxxx@xxxx.xxx")
submsg:copy_messages(remoteimap['INBOX'])

-- And put my replies to mailing lists in folders where they belong to
runFilter_copy(sentmesg, ml_filters)

Some more detailed examples of imap remote filters in no particular order:

https://thomaslevine.com/!/imapfilter/

https://raymii.org/s/blog/Filtering_IMAP_mail_with_imapfilter.html

https://www.npcglib.org/~stathis/blog/2012/07/09/linux-task-sorting-mail-with-imapfilter/

https://linux.die.net/man/5/imapfilter_config

I wish I could have done something like this with Thunderbird

other tools

Given the absurd dimension of LKML, synchronizing, fetching, labeling etc can take quite some time, the first time in particular.
I found archiving regularly with archivemail really useful.

searching and indexing: notmuch is great, and it comes integrated with neomutt or mutt-patched packets..

keyring: most of the tools I used can be integrated with gnome-keyring. Typing password every-time is annoying and I prefer not to keep them in plain text.

Get the job done

Where did we start from? Ah yes, patches, kernel, blah blah blah...

Now I have a setup that allows me to find emails in their right place, I see important stuff first, I do not miss patches, now I have no excuses for not working for real anymore... Too bad...

Now, let's say my friend "Joe Hacker" sent an interesting patch series I would like to test to prove him wrong and send him a bad review :).


 629 O   Mar 17 Joe Hacker  (3,4K) [PATCH 00/13] platform/twisted: Add console debugfs
 630 O   Mar 17 Joe Hacker  (2,1K) |->[PATCH 01/13] cros_ec: Add helper for event
 631 O   Mar 17 Joe Hacker  (2,0K) |->[PATCH 02/13] cros_ec: Add EC console read
 632 O   Mar 17 Joe Hacker  ( 14K) |->[PATCH 03/13] cros_ec: add debugfs, console
 633 O   Mar 17 Joe Hacker  (2,7K) |->[PATCH 04/13] cros_ec: Add support for
 634 O   Mar 17 Joe Hacker  ( 12K) |->[PATCH 05/13] platform/twisted: Add R/W helpers
 635 O   Mar 17 Joe Hacker  ( 13K) |->[PATCH 06/13] platform/twisted: Add support for
 636 O   Mar 17 Joe Hacker  (2,9K) |->[PATCH 07/13] platform/twisted: Add support for
 637 O   Mar 17 Joe Hacker  (1,4K) |->[PATCH 08/13] platform/twisted: Add power management ops
 638 O   Mar 17 Joe Hacker  (2,0K) |->[PATCH 09/13] platform/twisted: Add MKBP events
 639 O   Mar 17 Joe Hacker  (4,8K) |->[PATCH 10/13] platform/twisted: Add lightbar program
 640 O   Mar 17 Joe Hacker  (7,0K) |->[PATCH 11/13] platform/twisted: Control of suspend/resume
 641 O   Mar 17 Joe Hacker  (2,7K) |->[PATCH 12/13] platform/twisted: Add userspace lightbar
 642 O   Mar 17 Joe Hacker  (3,3K) |->[PATCH 13/13] platform/twisted: Avoid I2C xfer to EC
I can now walk to the series and "tag" the patches one by one (default mutt keybind "t").

Using the tag-prefix operator ";" I can now ask to mutt to copy/save patches to another mailbox (that would be ";"+"C").

Mutt will ask for a mailbox location, and I can give him a path where to save patches:

Copy tagged to mailbox: ~/project/linux/joe-is-wrong-patches.mbox

Create /home/jmondi/project/linux/joe-is-wrong-patches.mbox? ([yes]/no):

I can now git am the new created mailbox (that would be a Maildir directory, but still...) and apply the series!

I admit last time I was so excited by just setting up a tool has been ages ago when we were all tuning our conky scripts and competing on them... I should give it a spin again, sooner or later if I'm not too old and grumpy already..

Update

Copying patches as shown just above results in mangled file names, and you lose the patch order which is quite essential when you're applying patches interactively.

Here is great solution for copying patch series with much more saner names and apply them in order with git later:

https://flavioleitner.blogspot.it/2011/03/patch-workflow-with-mutt-and-git.html

Links and resources

https://jason.the-graham.com/2011/01/10/email_with_mutt_offlineimap_imapfilter_msmtp_archivemail/#preparation

https://pbrisbin.com/posts/mutt_gmail_offlineimap/

https://notmuchmail.org/notmuch-mutt/

https://wiki.archlinux.org/index.php/Mutt

https://flavioleitner.blogspot.it/2011/03/patch-workflow-with-mutt-and-git.html

https://warrenpost.wordpress.com/2010/04/07/archivemail/



[ Home ]