Using Procmail at Monash University

Procmail is a general purpose electronic mail filtering program available on most of Monash's Unix servers (eg. silas). Anybody receiving more than a couple of emails a day can greatly benefit from the features provided.

This is intended as a guide to helping people get started with procmail at monash. Comments and queries should be directed to Andrew.Cosgriff@cc.monash.edu.au, or posted on the monash.unix newsgroup.


  1. Setting up .forward
  2. A basic .procmailrc file
  3. Filing mail from friends
  4. Filing mail from mailing lists
  5. Mailing list digests
  6. Combining it all
  7. MH Folders
  8. Forwarding mail
  9. Using Biff/Comsat to tell you about new mail
  10. Using your shell
  11. Goofing yourself
  12. Avoiding Spam
  13. Avoiding duplicate emails
  14. Stripping MIME headers
  15. A personal PGP Key Server
  16. A personal file server
  17. Debugging
  18. Other resources
  19. Credits

Setting up .forward

First of all, put the following in a file in your home directory called .forward :

 "|IFS=' '&&exec /usr/local/bin/procmail -f-||exit 75 #ajc" 

Note that you should include the quotation marks in the file, and that the ajc at the end should be replaced with your own username (eg. jrsmi1).

Note : CC machines now share a common home directory, so you need to do two things :

  1. Create a .forward file saying something like the following :

    jrsmi1@silas.cc.monash.edu.au

    This will tell any machine that uses your CC home directory (currently ccds/ccas, the silas cluster, CC linux accounts and fawltyds.

  2. Then create a second file called .forward.silas-1 containing the procmail line above :

    "|IFS=' '&&exec	/usr/local/bin/procmail -f-||exit 75 #ajc"

    Then cp .forward.silas-1 .forward.silas-2 (since mail sent to silas could end up on either silas-1 or silas-2, and you want to process it either way).


A basic .procmailrc

The most basic .procmailrc file is a non-existent one - your mail will be delivered to the mail spool as normal. A slightly more useful example is one that files all incoming mail into a single "mbox"-style folder under your home directory:

MAILDIR=$HOME/Mail

:0:
incoming

Filing mail from friends

Perhaps you have a particular friend who sends you lots of mail. Their email address is friend@yoyo.cc.monash.edu.au. To file all mail from this person into a folder, you could use the following .procmailrc :

MAILDIR=$HOME/Mail

:0:
* ^From:.*friend@yoyo.cc.monash.edu.au
friend

Filing mail from mailing lists

"mailing lists" in this sense are ones that you send a "subscribe" message to some address, and get mail from it. With mailing lists, one must look at the "To:" header, not the "From:" one. One example is Monash's "downtime" mailing list :

MAILDIR=$HOME/Mail

:0:
* ^TOdowntime
downtime

Mailing list digests

Sometimes you might have subscribed to a "digest" version of a mailing list (you get an email, maybe once every couple of hours, containing all the messages sent to the list within that time). You can use formail (a companion program to procmail) to split the digest into separate emails like so :

:0:
* ^Subject:.*Blah.*Digest
| formail +1 -ds >>blah

Combining it all

If we add up what we've done so far, we end up with this :

MAILDIR=$HOME/Mail

:0:
* ^TOdowntime
downtime

:0:
* ^From:.*friend@yoyo.cc.monash.edu.au
friend

:0:
incoming

Note the ordering - if our friend happens to post to the downtime list, we want it to be filed in the downtime filder. Anything else falls through to the last recipe, which delivers the mail into the incoming folder. If we don't have a "catch-all" recipe like this, any leftover mail will simply be delivered into the mail spool (as if procmail was never there).


MH Folders

If you use MH, or any of its "front-end" programs like ExMH, MH-E and such, you'll want to file mail away in an MH-style folder. An MH folder is a directory (each email is a separate, numbered file). If we wanted downtime mail saved this way, we'd do the following :

MAILDIR=$HOME/Mail

:0:
* ^TOdowntime
downtime/.

Note that this doesn't update the "unseen" sequence that you may be using to determine which mail you haven't read yet. If you want that to happen, do this instead :

MAILDIR=$HOME/Mail
RCVSTORE=/usr/lib/mh/rcvstore -create

:0w:downtime.$LOCKEXT
* ^TOdowntime
| $RCVSTORE +downtime

Forwarding mail

You might find it advantageous to forward some mail to another site. You might want to forward certain emails (say, from a particular mailing list) to a friend (but also file it in a folder, of course) :

:0
* ^TOprocmail
{
  :0c
  ! friend@yoyo.cc.monash.edu.au

  :0w:
  procmail
}

Perhaps you have a permanent net feed at home, and any mail at home, you want forward at work, and vice-versa. This is a bit tricky, as we could end up with a mail loop if we're not careful. Here's how to do it :

:0c
*!^TO(ajc|Andrew.Cosgriff)@.*monash.edu.au
*!^(From|Sender):.*(ajc|Andrew.Cosgriff)@.*monash.edu.au
*!^X-Loop: ajc@cc.monash.edu.au
| (formail -A"X-Loop: ajc@bing.bofh.asn.au" | \
  /usr/sbin/sendmail -oi Andrew.Cosgriff@cc.monash.edu.au)

At uni, I'd then do :

:0c
*!^TOajc@bing.bofh.asn.au
*!^(From|Sender):.*ajc@bing.bofh.asn.au
*!^X-Loop: ajc@bing.bofh.asn.au
| (formail -A"X-Loop: ajc@cc.monash.edu.au" | \
  /usr/sbin/sendmail -oi ajc@bing.bofh.asn.au)

(making sure, of course, that sendmail lives in /usr/sbin). The presence of the X-Loop header prevents us from causing a mail loop by sending back mail that came from uni, say.


Using Biff/Comsat to tell you about new mail

comsat is the service used to notify you when new mail arrives - you get a message written to your screen telling you who the mesage was from, the subject, and a bit of the body of the message. Many people find that this interrupts their work if they get told about every piece of mail that arrives. By default, procmail will get comsat to notify you of new mail. To turn this off, add the following line to your .procmailrc :

COMSAT=no

Using your shell

Most shells (ksh, csh, zsh, bash and tcsh, for instance) will check for mail every so often, and tell you

You have new mail.

All of the above shells let you specify a list of mail folders to check for new mail. For instance, try putting these in your shell environment file (.cshrc, .profile, etc) :

Read the relevant shell manual page for a few other things you can do with these two variables (including customized messages and such). If you're using MH-style folders, specifying the directory (eg. $HOME/Mail/inbox ought to work.


Goofing yourself about email

You may find it useful to have procmail inform you via goofey if an important email arrives. Something like the following would work, placed near the top of your .procmailrc :

:0c
* ^Subject:\/.* urgent.*
| (echo "== mail received ==\n$MATCH" ) | /usr/local/bin/goofey --forget-fail -s cos

Avoiding Spam

If you've ever posted much to Usenet, or are on a few mailing lists, you've probably had your email address found by people who enjoy sending out unsolicited advertisements about various crud you never want to hear about. You can filter out any spam like so :

SPAM=$MAILDIR/lib/spam.txt

:0
* ? formail -ISubject | fgrep -iqsf $SPAM
/dev/null

$MAILDIR/lib/spam.txt contains a list of hostnames for sites that are known to send out spam. I used to have my list available here, but it's way out of date.

These days, I've found that the above technique isn't particularly effective by itself, but that, after filtering out all mailing list emails into other folders, any mail left that isn't personally addressed to me must therefore be spam, and gets filtered into a "possible spam" folder which I check every few days.

[2002] Nowadays, it's even worse. I'm having success by using SpamAssassin and Vipul's Razor, though.


Avoiding duplicate emails

Sometimes you'll receive multiple copies of the same email, sometimes for no apparent reason, apart from broken mail software somewhere out there, and other times because you got it by personal email as well as via a mailing list. In order to avoid this, you can keep a cache recently received Message-ID headers, and throw away any messages that have a message-id present in the cache. To do this, simply add the following to your ,procmailrc, above any recipes :

:0 Wh:dup
| formail -D 8192 msgid.cache

If you receive a lot of mail, still get duplicates (ie. with the same message-id's), you might want to increse the cache size.


Stripping MIME headers

Some people like to use old mailreaders that don't deal well with MIME headers. If you read a lot of postmaster mail, for instance, you'll end up reading a lot of "bounce" messages that contain multiple MIME parts, and since they're all plain text, you probably just want to read the damn thing without having to stuff around. Kevin Lentin suggests the following solution :

:0 fbw:
* ^TOpostmaster
| formail "-R Content-Type: X-Content-Type:" >>postmaster

A personal PGP Key Server

As an example of a "server"-like application you can implement with procmail, I used to make my PGP public key available via email at my previous workplace (we didn't have direct internet access, only email and news, batched up via UUCP) :

########################################
#
# tiny PGP KEY SERVER
#
# If e-mail message has subject of "get pgp key" or
# "send pgp key", then this will reply with my pgp key
# and log the request in $MAILDIR/pgpkey-request.log
#
# Orignally by me (Andrew Cosgriff), MIME niceties added by
# Richard Coleman (coleman@math.gatech.edu).
######################################################
MYADDR=ajc@unico.com.au
FROMSIG="Andrew's Mail Gremlin <$MYADDR>"
WHOAMI="Andrew J Cosgriff <$MYADDR>"
PGPKEYLOG=$MAILDIR/pgpkey-request.log

:0 c
* ^Subject:.*(send|get) *pgp *key
* !^X-Loop: $MYADDR
{
	FROM=`formail -rt -xTo:`
	SUBJECT=`formail -xSubject:`

	:0 ch: pgp$LOCKEXT
	| (formail -rI"From: $FROMSIG" \
		-I"Reply-To: $MYADDR" \
		-I"Subject: PGP public key" \
		-I"Mime-Version: 1.0" \
		-I"Content-Type: application/pgp; format=keys-only" \
		-I"Content-Description: PGP key of $WHOAMI" \
		-I"Content-Transfer-Encoding: 7bit" \
		-I"X-Loop: $MYADDR" ;\
		sh -c "pgp -kxaf $MYADDR 2>/dev/null") \
	| $SENDMAIL -oi -t

	:0 w: $PGPKEYLOG$LOCKEXT
	| echo "`date` : $FROM : $SUBJECT" >> $PGPKEYLOG
}

A personal file server

A second example is a file server, which can serve files from a directory (and its subdirectories) :

MYADDR=ajc@unico.com.au
FROMSIG="Andrew's Mail Gremlin <$MYADDR>"
WHOAMI="Andrew J Cosgriff <$MYADDR>"
FILESERVLOG=$MAILDIR/fileserv-request.log

########################################
#
# File Server
#
:0
* ^Subject: (get|send) file [0-9a-zA-Z]
* !^X-Loop: ajc@unico.com.au
* !^Subject:.*Re:
* !^FROM_DAEMON
* !^Subject: (get|send) file .*[/.]\.
{
	FROM=`formail -rt -xTo:`
	SUBJECT=`formail -xSubject:`

	:0ac:fileservlog.lock
	| egrep "^(Subject|From|Date):" >> $FILESERVLOG

	MAILDIR=$HOME/tmp/fileserv

	# reverse mailheader and extract name
	:0 fhw
	* ^Subject: (get|send) file \/[^ ]*
	| formail -rA "X-Loop: ajc@unico.com.au" \
		-rI"From: $FROMSIG" \
		-I"Reply-To: $MYADDR"

	# the requested filename (see procmailrc(5))
	FILE="$MATCH"

	:0 ah
	| cat - ./$FILE | $SENDMAIL -oi -t
}

Note that if someone requests a file like /something or ../something, the request won't match the initial conditions (we do this deliberately, so people can't request files outside the particular directory we're using).

To use it, one would send email with a subject of "send file blah" to be sent a file called blah - it's useful to have a file called index and/or help, which provide a list of files and instructions on retrieving them.


Debugging

If, after modifying your .procmailrc, you suddenly stop receiving any mail, you'll probably want to fix it rather quickly - to help, add the following to your .procmailrc, near the top ;

LOGFILE=procmail.log
VERBOSE=yes

which will log what procmail is trying to do. You should be able to get enough information from the logfile to fix your problem. If you still can't fix it, comment out the offending section, and seek help - try asking on the local monash newsgroup monash.unix. If you read the procmailex manual page, you'll see a scheme for caching the last few emails received, so if you're trying new stuff, you don't lose the email.

I've also installed proclint on silas, which you can use to sanity check your .procmailrc. It was written by Alan Stebbens.


Other Resources


Credits

Stephen van den Berg, the author of procmail, obviously deserves many thanks, as do the various people who provide useful help on the mailing list and/or have created useful online resources for procmail.

I'd also like to thank the following people for their input on this particular document :