Rutgers, The State University of New Jersey

Setting Up Imapd  

These notes apply to the University of Washington imapd, which is what we recommend for "normal" server at Rutgers. There are enough peculiarites about setting up imapd, and it's so badly documented, that it seems worth having a few notes.

The Simple Setup
A minimal imapd installation is easy enough: Put imapd in /usr/local/sbin. (You can get it from Track.) Edit /etc/inetd.conf to add an entry for it.
    imap  stream  tcp     nowait  root    /usr/local/sbin/imapd imapd
Make sure there is an entry in /etc/services:
    imap            143/tcp

If you do this, and don't tell your users anything special, you'll be able to have people read mail via Netscape and IE. Imapd will read mail from /var/mail/USERNAME, just as the normal Unix mail readers do. Thus there's good compatibility.

Problems with the Default Setup

So what's wrong with this? The big problem is that imapd has trouble with mail stored in the two most common mail formats: /var/mail/USER and mail.txt.

The format used by /var/mail/USER is referred to as "Unix format" or "mbox format". The problem with it is that imapd is unable to share the same file among two instances of imapd. When imapd checkpoints the file, it has to write a whole new copy. This makes sharing nearly impossible. Checkpointing is a normal operation, done to update the flags showing whether a message has been seen. The problem seems to be that Unix format doesn't preallocate space for the flags. There are also problems with the synchronization methods used for this format.

The net result for users is that they may see long pauses if they move quickly from message to message. Normally it will be only a few seconds, as a new copy of imapd starts up. But it can be 2 minutes if certain problems occur. It is also possible for flag updates to be lost. Similar problems may occur when copying a message into a separate file on the server.

A Possible Interim Solution: Mail.txt Format

Mail.txt, also called "Tenex format", is a format originated by mm. It is also supported by pine and imapd. It allows multiple connections to the same file. It would seem the ideal format for Rutgers, because it would allow a user to alternate among pine, mm, and IMAP. (This is not intended as a recommendation of mm. We are phasing it out. However support staff need to know that people who have been using mm will already have their mail in mail.txt.) If you have a large mail file, pine will be enormously faster when it's using mail.txt rather than Unix format.

Unfortunately there's one serious problem: support for mail.txt in imapd has been buggy. The bug has recently been fixed, but the copy of imapd that is on most of our systems does not yet have the fix. Until fixed versions are distributed, you should beware of using mail.txt.

The IMAP protocol requires that each message have a permanent identification number assigned to it. This number is called a "UID". The UID must persist from session to session, even if messages are deleted. Imapd implements this for Unix mail files by adding a message to the beginning that saves some state information. It then adds headers with the UID's to every individual message.

Unfortunately the stock imapd does not do this with mail.txt format. This causes big trouble with Netscape and IE. If you read mail with NS or IE, they cache information about your mail file locally. Suppose you do this at home. Then you go to work and read mail there. Suppose you delete a few messages. Now you go back home and read mail with Netscape again. Netscape won't know that those messages have been deleted. It will get confused about all messages after those that have been deleted, presenting the wrong message to you.

I have a version of imapd in testing that implements UID's properly. Once it is out, mail.txt should be a good format for use with mm, pine, and IMAP.

If you are currently using a normal Unix mailbox (/var/mail/USERNAME) and you want to start using mail.txt, it's easy. All you have to do is create an empty mail.txt file. You can do this with the command "touch mail.txt" in Unix. I believe you can also create a mail box called mail.txt using many of the IMAP clients. Once mail.txt exists, pine and imapd will use it. Every time they start, they will move any new mail from /var/mail/USERNAME into mail.txt. If you need to go back from mail.txt to normal Unix format, see the below.

BEWARE: If a user has been using mm, his mail will be in mail.txt. If he moves to pine or IMAP, his mail will continue to be in mail.txt. Both pine and imapd will move new mail from /var/mail/USERNAME to mail.txt and read mail.txt. Until the new version of imapd is out, this means that these users may see odd results from Netscape or IE.

A Longer-Term Solution: MBX Format

To deal with these problems imapd has its own mail format, referred to as mbx format. It has enough preallocated space that it is possible for multiple readers to share the same file. (You still can't expunge your file unless you're the only reader.) It has enough status information that Netscape and IE don't get confused if someone else deletes messages. The name of the default mail file for MBX format is INBOX, in your home directory.

The main problem with mbx format is that only IMAP understands it. It also has some binary data in it, so you can't as easily deal with it using normal Unix tools. (Even with mail.txt you shouldn't edit it with normnal Unix tools unless you understand all the details of the format. It's easy to break the file.)

Fortunately, the next major release of Pine will use the same file access code as imapd. Thus as of Pine 4.0, Pine will understand mbx format. At that point, mbx should be about as useful as mail.txt. There are supposedly some performance advantages to mbx.

So when Pine 4.0 comes out, we will at least consider using mbx for mail files. However this is future. For the moment I would only recommend it for adventuresome staff.

Converting to mail.txt or MBX Format

The easiest way to convert between formats is to use imapd itself.

To convert your mail into MBX or mail.txt, it's easy. All you have to do is create an empty file of the appropriate kind. Imapd will then move all your mail from /var/mail/USERNAME to that file the next time you read mail. It will continue to move new mail from /var/mail/USERNAME to your mail file every time you read mail.

Before doing this kind of conversion, make sure that the user has enough disk quota in his home directory. mail.txt and INBOX (the MBX format mail file) both go in the home directory. Do "ls -l /var/mail/USERNAME", look at the size, and then do "quota -v" to make sure there is enough space in the home directory. The sizes should be very similar, but make sure there's a slight amount of extra space available.

It's easy to create an empty mail.txt file. Just to "touch mail.txt" under Unix.

Creating an empty INBOX is slightly more challenging. I suggest using imapd:

     1 create #driver.mbx:INBOX
     2 logout
(Yes, you must type the line numbers.) It should be possible to do this using one of the IMAP clients, but I haven't been able to.
Going back to Unix format using IMAP

The next question is going to be: "What if I want to go back?" Nobody but imapd can read MBX format. Other programs can read mail.txt, but imapd may still be the most convenient way to convert from mail.txt.

Fortunately, you can get imapd to make you a copy in normal Unix format. There are two approaches: using IMAP itself, or using pine. It's easiest to use IMAP itself. However there's one caveat: When it creates files in Unix format, it uses a new version of the format. Pine and Sun's mailx can handle it. So can mm, in case anyone cares. But the copy of zmail we have cannot. Neither can /usr/local/bin/mail, but I have a fixed version that will go into Track shortly.

The rest of this section will talk about using Netscape (or another IMAP client) to move messages into a Unix-format file. The next section will talk about using Pine to do it.

This section will tell you how to put messages into a separate Unix-format file. It will not tell you how to put them back into your incoming Unix mailbox (/var/mail/USERNAME). That's hard to do with Netscape or IE, and if you're using pine, there's a better approach. (See the next section.)

If you need to convert from mail.txt or MBX to Unix format, I strongly suggest putting them into another file, and not back into /var/mail/USERNAME. People who use mail a lot should be saving their mail in folders anyway. So putting moving old mail from mail.txt or MBOX into a folder in Unix format is a sensible approach.

To move mail into a Unix-format mail file is easy. Use an IMAP client, create a new "folder", and copy all your mail to it. I've tested it with Netscape 4, and these instructions will apply in most detail to it.

You definitely want to create the folder from the IMAP client. In netscape you would use "File" and "New folder". Your IMAP client will issue the CREATE command to imapd. It will create an empty file in the default format, which is normal Unix format. Imapd will stick a special message at the beginning of the file that tells it the file is in Unix format. If you create an empty file in Unix (e.g. by "touch filename"), imapd will not know what format you intended, and it will copy your messages into the new file using mail.txt or MBX format. (Its default for a copy is to retain the existing format.)

In Netscape (at least on the PC) it is possible to select more than one message for copying by holding down control while you select with the mouse. It is possible to select all messages using control-A (or Edit, Select Message, All).

Before doing this, you should verify that you have enough disk quota to create the new copy. I've seen imapd simply crash if it runs out of disk space during a copy. Do "ls -l" of your existing mail.txt or INBOX, and then use "quota -v" to make sure you have enough space for the new copy.

Going back to Unix format using pine

If the user wants to go back to Unix mail format, and can't deal with having mail in a separate file, you can try putting the mail back into /var/mail/USERNAME. The safest way to do this is with Pine. You may also want to use Pine to put mail into a separate file.

It turns out that there are two slightly variants of Unix mail format. Imapd uses a new one. Pine and Sun's mailx understand it, but zmail does not. Thus we want pine to read the messages via imap but write into /var/mail locally.

First check to make sure you have enough quota in /var/mail. Do "ls -l" of mail.txt or INBOX, and look at "quota -v" to make sure your mail will fit in /var/mail/USERNAME.

Now rename mail.txt or INBOX to something else. Otherwise once you put the messges in /var/mail/USERNAME, imapd will see them as new mail and move them right back into mail.txt or INBOX. For this example, assume you've renamed mail.txt to mail.txt.hold

Make sure /var/mail/USERNAME exists. If it doesn't, send mail to yourself.

Make sure your pine has "aggregate commands" turned on. In the main screen, go into Setup, choose Config, and make sure enable-aggregate-command-set is checked.

Now use pine to move the messages from mail.txt.hold (or whatever you called it) into /var/mail/USERNAME or some other file. Point pine at the file using IMAP, e.g.

     pine -f '{mailserver}mail.txt.hold'
where mailserver is the name of the system running imapd for your cluster and mail.txt.hold is whatever you called the file having your messages.

Finally, tell pine to save all messages in /var/mail/USERNAME or another file.

     ;   - select
     a   - all messges
     a   - apply
     s   - save
Specify the filename /var/mail/USERNAME or some other file. Use your actual username of course.
About /etc/rimapd

If a user is going to use mbx, they'll only be able to access their mail through IMAP. If they also need to read mail using a text client, the obvious thing is to use pine. Pine understands IMAP. However when reading mail under Unix, you really don't want to have to login to IMAP each time. There's a way to avoid that: When you tell pine to connect to an IMAP server, before making an IMAP connection it will try to rsh a call to /etc/rimapd. If that works, it will get an IMAP connection without needing a password.

If you can set your system up so that all of your users can rsh into the IMAP server, this will make things easier for them. To set it up, all you have to do is make a symlink:

     ln -s /usr/local/sbin/imapd /etc/rimapd
About /etc/imapd.alert

IMAP has a facility for presenting "alerts" to users. When the user connects to an IMAP server, if it has an alert, the client is supposed to show it to the user in a way that it isn't likely to be misssed. IE and pine both put the message up in a box and make the user acknowledge it. NS ignores alerts, but this should be fixed RSN.

To create an alert using the U of Washington imapd, create the file /etc/imapd.alert. It's simply a text file. In theory you could make this a symlink to /etc/motd. However alerts are a pretty obtrusive mechanism. Your users are likely to complain if you use it for anything but the most critical messages.


For more information, contact
Last updated: Thursday, 14-Jan-1999 18:46:00 EST
© 1999 Rutgers, The State University of New Jersey. All rights reserved.