ref: 03d870e0283299404b0eb46689d13f8538e83a2f
dir: /sys/doc/nupas/nupas.ms/
.EQ delim $$ .EN .TL Scaling Upas .AU Erik Quanstrom quanstro@coraid.com .AB The Plan 9 email system, Upas, uses traditional methods of delivery to .UX mail boxes while using a user-level file system, Upas/fs, to translate mail boxes of various formats into a single, convenient format for access. Unfortunately, it does not do so efficiently. Upas/fs reads entire folders into core. When deleting email from mail boxes, the entire mail box is rewritten. I describe how Upas/fs has been adapted to use caching, indexing and a new mail box format (mdir) to limit I/O, reduce core size and eliminate the need to rewrite mail boxes. .AE .NH Introduction .LP .DS I Chained at his root two scion demons dwell .br – Erasmus Darwin, The Botanic Garden .DE .LP At Coraid, email is the largest resource user in the system by orders of magnitude. As of July, 2007, rewriting mail boxes was using 300MB/day on the WORM and several users required more than 400MB of core. As of July, 2008, rewriting mail boxes was using 800MB/day on the WORM and several users required more than 1.2GB of core to read email. Clearly these are difficult to sustain levels of growth, even without growth of the company. We needed to limit the amount of disk space used and, more urgently, reduce Upas/fs' core size. .LP The techniques employed are simple. Mail is now stored in a directory with one message per file. This eliminates the need to rewrite mail boxes. Upas/fs now maintains an index which allows it to present complete message summaries without reading indexed messages. Combining the two techniques allows Upas/fs to read only new or referenced messages. Finally, caching limits both the total number of in-core messages and their total size. .NH Mdir Format .LP In addition to meeting our urgent operational requirements of reducing memory and disk footprint, to meet the expectations of our users we require a solution that is able to handle folders up to ten thousand messages, open folders quickly, list the contents of folders quickly and support the current set of mail readers. .LP There are several potential styles of mail boxes. The maildir[1] format has some attractive properties. Mail can be delivered to or deleted from a mail box without locking. New mail or deleted mail may be detected with a directory scan. When used with WORM storage, the amount of storage required is no more than the size of new mail received. Mbox format can require that a new copy of the inbox be stored every day. Even with storage that coalesces duplicate blocks such as Venti, deleting a message will generally require new storage since messages are not disk-block aligned. Maildir does not reduce the cost of the common task of a summary listing of mail such as generated by acme Mail. .LP The mails[2] format proposes a directory per mail. A copy of the mail as delivered is stored and each mime part is decoded in such a way that a mail reader could display the file directly. Command line tools in the style of MH[3] are used to display and process mail. Upas/fs is not necessary for reading local mail. Mails has the potential to reduce memory footprint below that offered by mdirs for native email reading. However all of the largest mail boxes at our site are served exclusively through IMAP. The preformatting by mails would be unnecessary for such accounts. .LP Other mail servers such as Lotus Notes[4] store email in a custom database format which allows for fielded and full-text searching of mail folders. Such a format provides very quick mail listings and good search capabilities. Such a solution would not lend itself well to a tool-based environment, nor would it be simple. .LP Maildir format seemed the best basic format but its particulars are tied to the .UX environment; mdir is a descendant. A mdir folder is a directory with the name of the folder. Messages in the mdir folder are stored in a file named .I "utime.seq" . .I Utime is defined as the decimal .UX seconds when the message was added to the folder. For the inbox, this time will correspond to the .UX “From ” line. .I Seq is a two-digit sequence number starting with .CW "00." The lowest available sequence number is used to store the message. Thus, the first email possible would be named .CW "0.00." To prevent accidents, message files are stored with the append-only and exclusive-access bits turned on. The message is stored in the same format it would be in mbox format; each message is a valid mbox folder with a single message. .NH Indexing .LP When upas/fs finds an unindexed message, it is added to the index. The index is a file named .I "foldername" .idx and consists a signature and one line per MIME part. Each line contains the SHA1 checksum of the message (or a place holder for subparts), one field per entry in the .I "messageid/info" file, flags and the number of subparts. The flags are currently a superset of the standard IMAP flags. They provide the similar functionality to maildir's modified file names. Thus the `S' (answered) flag remains set between invocations of mail readers. Other mutable information about a message may be stored in a similar way. .LP Since the .I info file is read by all the mail readers to produce mail listings, mail boxes may be listed without opening any mail files when no new mail has arrived. Similarly, opening a new mail box requires reading the index and checking new mail. Index files are typically between 0.5% and 5% the size of the full mail box. Each time the index is generated, it is fully rewritten. .NH Caching .LP Upas/fs stores each message in a .CW "Message" structure. To enable caching, this structure was split into four parts: The .CW "Idx" (or index), message subparts, information on the cache state of the message and a set of pointers into the processed header and body. Only the pointers to the processed header and body are subject to caching. The available cache states are .CW "Cidx" , .CW "Cheader" and .CW "Cbody" . .LP When the header and body are not present, the average message with subparts takes roughly 3KB of memory. Thus a 10,000 message mail box would require roughly 30MB of core in addition to any cached messages. Reads of the .CW "info" or .CW "subject" files can be satisfied from the information in the .CW "Idx" structure. .LP Since there are a fair number of very large messages, requests that can be satisfied by reading the message headers do not result in the full message being read. Reads of the .CW "header" or .CW "rawheader" files of top-level messages are satisfied in this way. Reading the same files for subparts, however, results in the entire message being read. Caching the header results in the .CW "Cheader" cache state. .LP Similarly, reading the .CW "body" requires the body to be read, processed and results in the .CW "Cbody" cache state. Reading from MIME subparts also results in the .CW "Cbody" cache state. .LP The cache has a simple LRU replacement policy. Each time a cached member of a message is accessed, it is moved to the head of the list. The cache contains a maximum number of messages and a maximum size. While the maximum number of messages may not be exceeded, the maximum cache size may be exceeded if the sum of all the currently referenced messages is greater than the size of the cache. In this case all unreferenced messages will be freed. When removing a message from the cache all of the cacheable information is freed. .NH Collateral damage .LP .DS I Each new user of a new system uncovers a new class of bugs. .br — Brian Kernighan .DE .LP In addition to upas/fs, programs that have assumptions about how mail boxes are structured needed to be modified. Programs which deliver mail to mail boxes (deliver, marshal, ml, smtp) and append messages to folders were given a common (nedmail) function to call. Since this was done by modifying functions in the Upas common library, this presented a problem for programs not traditionally part of Upas such as acme Mail and imap4d. Rather than fold these programs into Upas, a new program, mbappend, was added to Upas. .LP Imap4d also requires the ability to rename and remove folders. While an external program would work for this as well, that approach has some drawbacks. Most importantly, IMAP folders can't be moved or renamed in the same way without reimplementing functionality that is already in upas/fs. It also emphasises the asymmetry between reading and deleting email and other folder actions. Folder renaming and removal were added to upas/fs. It is intended that mbappend will be removed soon and replaced with equivalent upas/fs functionality — at least for non-delivery programs. .LP Mdirs also expose an oddity about file permissions. An append-only file that is mode .CW 0622 may be appended to by anyone, but is readable only by the owner. With a directory, such a setup is not directly possible as write permission to a directory implies permission to remove. There are a number of solutions to this problem. Delivery could be made asymmetrical—incoming files could be written to a mbox. Or, following the example of the outbound mail queue, each user could deliver to a directory owned by that user. In many BSD-derived .UX systems, the “sticky bit” on directories is used to modify the meaning of the .CW w bit for users matching only the other bits. For them, the .CW w bit gives permission to create but not to remove. .LP While this is somewhat of a one-off situation, I chose to implement a version of the “sticky bit” using the existing append-only bit on our file server. This was implemented as an extra permission check when removing files. Fewer than 10 lines of code were required. .NH Performance .LP A representative local mail box was used to generate some rough performance numbers. The mail box is 110MB and contains 868 messages. These figures are shown in table 1. In the worse case—an unindexed mail box—the new upas/fs uses 18% of the memory of the original while using 13% more cpu. In the best case, it uses only 5% of the memory while using only 13% of the cpu. Clearly, a larger mail box will make these ratios more attractive. In the two months since the snapshot was taken, that same mail box has grown to 220MB and contains 1814 messages. .ps -2 .DS C .TS box, tab(:); c s s s s c | c | c | c | c a | n | n | n | n. Table 1 – Performance _ action:user:system:real:core size: :s:s:s:MB: _ old fs read:1.69:0.84:6.07:135 _ initial read:1.65:0.90:6.90:25 _ indexed read:0.64:0.03:0.77:6.5 .TE .DE .NL .NH Future Work .LP While Upas' memory usage has been drastically reduced, it is still a work-in-progress. Caching and indexing are adequate but primitive. Upas/fs is still inconsistently bypassed for appending messages to mail boxes. There are also some features which remain incomplete. Finally, the small increase in scale brings some new questions about the organization of email. .LP It may be useful for mail boxes with very large numbers of messages to divide the index into fixed-size chunks. Then messages could be read into a fixed-sized pool of structures as needed. However it is currently hard to see how clients could easily interface a mail box large enough for this technique to be useful. Currently, all clients assume that it is reasonable to allocate an in-core data structure for each message in a mail box. To take advantage of a chunked index, clients (or the server) would need a way of limiting the number of messages considered at a time. Also, for such large mail boxes, it would be important to separate the incoming messages from older messages to limit the work required to scan for new messages. .LP Caching is particularly unsatisfactory. Files should be read in fixed-sized buffers so maximum memory usage does not depend on the size of the largest file in the mail box. Unfortunately, current data structures do not readily support this. In practice, this limitation has not yet been noticeable. .LP There are also a few features that need to be completed. Tracking of references has been added to marshal and upas/fs. In addition, the index provides a place to store mutable information about a message. These capabilities should be built upon to provide general threading and tagging capabilities. .NH Speculation .LP Freed from the limitation that all messages in a mail box must be read and stored in memory before a single message may be accessed, it is interesting to speculate on a few further possibilites. .LP For example, it may be useful to replace separate mail boxes with a single collection of messages assigned to one or more virtual mail boxes. The association between a message and a mail box would be a “tag.” A message could be added to or removed from one or more mail boxes without modifying the mdir file. If threads were implemented by tagging each message with its references, it would be possible to follow threads across mail boxes, even to messages removed from all mail boxes, provided the underlying file were not also removed. If a facility for adding arbitrary, automatic tags were enabled, it would be possible to tag messages with the email address in the SMTP From line. .NH References .IP [1] D. Bernstein, “Using maildir format”, published online at .br http://cr.yp.to/proto/maildir.html .IP [2] F. Ballesteros .IR mails (1), published online at http://lsub.org/magic/man2html/1/mails .IP [3] MH Wikipedia entry, http://en.wikipedia.org/wiki/MH_Message_Handling_System .IP [4] Lotus Notes Wikipedia entry, http://en.wikipedia.org/wiki/Lotus_Notes .IP [5] D. Presotto, “Upas—a Simpler Approach to Network Mail”, Proceedings of the 10th Usenix conference, 1985.