IMAP Mail Issues, 9/25 to 10/9/2008

Updates are posted in chronological order with the newest first.

10/9, 4:08pm

The following was just sent to the department:

CS IMAP Status Update, Thursday Afternoon

All users are presently on imap.cs.wisc.edu.  The machine has been stable and
users have reported that mail performance seems better than on the old server.

There have been a few issues with clients and their compatibility with the new
server, but most of these have been resolved.

The lab will continue to monitor the server and its stability.

Any users experiencing issues with the new server or their mail clients should
try using webmail as a temporary measure and email lab@cs.wisc.edu with the
issue they are experiencing and any relevant messages or symptoms leading up
to it.  The lab can also be reached by phone at 262-2389.

In addition, this email was sent Wednesday at noon to the department:

CS IMAP Status Update, Wednesday Morning

Current Status:

  All users, including those who had been using imap2.cs.wisc.edu, have been 
  moved to the new imap.cs.wisc.edu.  The transition overnight went smoothly, 
  though there was a bump this morning around 8a that required a reboot on 
  the server. Few problems have surfaced since and the server appears to be 
  running well.

  The lab will be keeping a close eye on the machine's performance today
  and into the future.

Reminders for users moving back to imap:

  Your mail client needs to be reconfigured to use imap.cs.wisc.edu.
                                                                                
  Any dot (.) characters in mailbox names will be changed to underscores (_).   
                                                                                
  Pine users should keep in mind that pine has an odd user interface            
  while interacting with subfolders on the new server.  To open a "mail folder" 
  use <CR>, to open a "directory" (containing folders) use the '>' key.         
  If the folder has no messages, it seems as though the <CR> key will also work.

If you are experiencing problems with your email please try using web mail as
a temporary measure and email lab@cs.wisc.edu.  If web mail doesnt work or you
would like to contact the lab via phone, call 262-2389.

10/3, 3:55pm

The following email was just sent to faculty, staff, graduates, and undergrads.

CS IMAP mail server status update, Friday afternoon (10/3)

Current Status:

  Faculty and Staff are still using imap2.cs.wisc.edu, the temporary
  server that was deployed as an emergency work-around for the problem.
  This server is running the old software and the old configuration.

  All other IMAP users have been converted to a new IMAP server that
  uses new software with a better folder storage format and all folders
  on the IMAP server (not in AFS/home directories). This server is
  hosted on temporary hardware.

 imap.cs.wisc.edu (non-faculty/staff users):

  Service and performance on imap.cs.wisc.edu remains very good for 
  non-faculty/staff users.  The number of new issues reported to the CSL
  has decreased and the lab has worked through nearly all of the issues
  that have been reported to the present time.

  As mentioned in yesterday's status, the first full backup of the imap server
  has not yet been completed but is expected by the end of the weekend.

 imap2.cs.wisc.edu (faculty and staff):                             
                                                               
  Faculty and staff accounts remain on imap2.cs.wisc.edu.  

  Migration of these accounts to imap.cs.wisc.edu is still anticipated early 
  next week.  Affected users will recieve warning and instructions before 
  this occurs.

  Please note that as imap2 is a temporary solution, there is no web access
  set up and available for users presently residing on the server.

 pine users:

  One issue that has been reported is the disappearance of subfolders in
  a folder.  This issue is a user interface issue:

   <CR> opens a folder with messages
   ">" opens a directory with folders in it

  If <CR> is used on a folder with subfolders, only messages in the folder
  will appear and if there are no messages present, pine will report that 
  it is empty.

  ">" will view the folder and open a menu of the subfolders within.

Details regarding mail folder storage, folder naming, forwarding, and
caughtspam changes were emailed to all users Thursday afternoon and can
also be accessed in the CSL Docs at:
http://www.cs.wisc.edu/twiki/bin/view/CSDocs/IMapSep29

Please continue to report any problems to lab@cs.wisc.edu, or if email
is not functioning, by phone (262-0608).

Thank you for your patience.

10/2, 4:11pm

The following email was just sent to faculty, staff, graduates, and undergrads.

CS IMAP mail server status update, Thursday afternoon (10/2)
                                                                              
To recap:                                                            
                                                                        
1. Faculty and Staff are still using imap2.cs.wisc.edu, the temporary
   server that was deployed as an emergency work-around for the problem.   
   This server is running the old software and the old configuration.
               
               
2. All other IMAP users have been converted to a new IMAP server that         
   uses new software with a better folder storage format and all folders
   on the IMAP server (not in AFS/home directories). This server is
   hosted on temporary hardware.
                                                                            
Current status:                                                        
                                                           
imap2.cs.wisc.edu (faculty and staff):                             
                                                               
   This server has the same fundamental issues as the old CS IMAP server
   (high i/o overhead and whole-folder locking among others), but with
   only faculty and staff has not suffered catastrophic failure, and has
   delivered adequate (but not great) performance for the vast majority
   of the users.                                                        
                                                        
   Service is adequate but not great. When the load reaches high levels,  
   new mail is delayed.                                            
                                                            
   Migration to the new IMAP server is anticipated for early next week,   
   after we finish getting through the issues of an unplanned emergency
   conversion. The server is fully backed up.

imap.cs.wisc.edu (all other users):

    Service and performance is very good.

    We have worked through a number of issues with mail filtering and
    forwarding. There is an open issue for some users of pine, which we
    are working on. Documentation has not yet been updated.

    We have not yet completed the first full backup of the imap
    server. Completion of the backup initialization is expected by the
    end of the weekend.

    The extra load generated by the forced full backup initialization
    has lead to some small mail delivery delays. In order to avoid those
    delays, we have scaled back the backup rate.

A clarification about IMAP mail folder storage:

  The old IMAP server stored mail folders in ~/mail in your home
  directory. This was transparent to IMAP programs such as Thunderbird
  and the web mail interface, but directly visible to pine.

  The new IMAP server stores *all* IMAP folders on the IMAP server
  itself. The contents of ~/mail are no longer referenced by the IMAP
  server, and are not being updated. Unless you suspect a problem in the
  mailbox conversion (no problems have been reported to the lab) you can
  safely delete ~/mail if you are using imap.cs.wisc.edu

Folder naming:
                                                                        
  One visible change in the new IMAP server is that folder names may not
  contain the dot character ('.') as that is the folder hierarchy         
  separator. During the conversion, dots in folder names were changed to
  underscores (_). You may need to update filtering rules in your mail
  client.                                                                 
                                                                       
Pine folder access issues:                                 
                                                                  
  A few users have reported a problem accessing IMAP folders with
  Pine. We are continuing to work on this problem, which only seems to
  impact a few users. If we can not resolve this problem soon, we
  may have to recommend that users switch from pine to other IMAP
  clients.                                                           
                                                                       
Forwarding and caughtspam changes:                         

  There is a slight change in semantics of the mail filtering in the new
  IMAP server. We have fixed some problems, and believe that it is    
  working correctly (as expected) now. If you are having problems with
  mail forwarding, please let us know.
                                                                     
  The email management web form on www-auth will be updated soon with   
  slightly newer instructions.

  The "old" caughtspam folders -- caughtspam.1 2 and 3 -- are no longer
  accessible.

  There is a new caughtspam folder for all new caught spam on the new
  server. You may need to subscribe to it using the subscribe function of
  thunderbird or web mail.

Please continue to report any problems to lab@cs.wisc.edu or if email is
not functioning, by phone (262-0608).

10/1, 1:15pm

Since transitioning to the new IMAP server, load has stayed low and few problems have been reported. Users on imap2.cs.wisc.edu should continue to use that server.

The Lab continues to monitor the mail servers - any users that experience mail issues should contact the lab at 262-2389 or at lab@cs.wisc.edu.

9/30, 2:25pm

The following status email has been sent to faculty, staff, graduates and undergrads:

Subject: CS IMAP mail server status, Tuesday afternoon (9/30)
Date: Tue, 30 Sep 2008 14:17:24 -0500
From: David Parter <dparter@yfandes.cs.wisc.edu>

As most of you know, the CS IMAP mail server suffered a significant
failure on Thursday, and continued to suffer such that the server was
unusable.

To mitigate the impact, faculty and staff were moved to a temporary
alternate server, imap2.cs.wisc.edu, which did not suffer from the
catastrophic failure mode, but is less powerful hardware.

Over the weekend, we configured a new IMAP server, which uses different
software and a different configuration, and we expect to have
significantly improved performance. Unfortunately, converting to a new
server on an emergency basis means that we did not have the opportunity
to completely test the complete configuration, migration process,
or provide documentation. We also had to use existing hardware, which
may not be optimal for the server.

As of this morning, all users from imap.cs.wisc.edu (not including
faculty and staff on imap2.cs.wisc.edu) have been converted to the new
IMAP server.

This morning there were some related and un-related network problems
which contributed to various problems experienced by users. We have also
corrected a few configuration and tuning issues.

We will continue to monitor the server closely and be as responsive as
we can in correcting problems and tuning the configuration.

As soon as we are satisfied with the stability, reliability and
performance of the new servers, users on imap2.cs.wisc.edu will be
migrated to the new server (with advance notification).

We will also be purchasing and configuring new hardware to assure the
best possible service.

Please continue to report any mail problems to lab@cs.wisc.edu or call
262-2389.

Thank you for your patience,

      --the Computer Systems Lab

9/30, 11:01am

The lab continues to watch the IMAP servers. Most users are able to access their mail normally.

As the load has increased this morning some users continue to report problems and the lab is working through them as they are reported. Please contact the lab (262-2389 or lab@cs.wisc.edu) with any error messages and/or symptoms you are experiencing if mail doesn't seem to be working correctly - this will help us to find and resolve the outstanding issues.

Thank you for your patience.

9/30, 7:42am

The email copy and conversion from the old IMAP server finished and the new server was brought online this morning. All of a user's old mail folders have been left untouched in ~/mail.

This does not yet apply to users that were moved to imap2.cs.wisc.edu - those users have not been migrated to the new server and should continue to use imap2.

Lab staff are keeping an eye on both servers, which appear to be running normally.

9/29, 11:42am

As mentioned in an email swift has sent to the department, imap.cs.wisc.edu will be down most of the day and evening as we copy and convert existing mail for the new server.

There will be at least one user-visible change when the server returns: periods (.) in folder names will be replaced with underscores (_). The new server will be named imap.cs.wisc.edu, so no further client changes will be necessary for users that haven't been moved to imap2.

imap2 is working ok at present.

9/29, 10:30am

In an effort to alleviate the issues users are seeing with IMAP mail, the CSL has temporarily split the server into two. Users that have been moved to the new imap2.cs.wisc.edu will have an email with the subject "Your mail is on imap2.cs.wisc.edu" in their mailbox and will need to reconfigure their mail client(s) to use the new server. As this is only a temporary fix, there is no web mail interface for imap2 at this time.

Users that have not recieved this message should continue to use imap.cs.wisc.edu.

Outgoing mail is still functioning on the old IMAP server, though users may receive errors related to copying messages to their sent folder - this is caused by the load on the server and occurs after the message has been sent to its recipient.

The lab is working on a more permanent solution. A third IMAP server will be setup using different software, and users still on imap.cs.wisc.edu will be moved to this server. Once this third server has proven stable under load, users on imap2.cs.wisc.edu will then be moved over.

This page will carry status updates as we continue to work towards resolving this issue.

Please contact the lab at lab@cs.wisc.edu or at 262-2389 if you need further assistance.

-- JacobEla - 29 Sep 2008