One is about the third-party MsgFiler plug-in for Mail. In my previous post on this topic, I wrote:

Unlike LaunchBar, as far as I can tell, MsgFiler does not have a smart abbreviation engine that “learns” your preferred destinations based on abbreviated versions of their names. It just matches what you type to the actual names of the mailbox folders.

If, like me, you have lots of subfolders that start with the same word, then you need to type not just that whole word but the beginning of the next one (or hit the cursor keys multiple times to select the right item in the list of matches). For example, I have mailbox folders called “Betalogue – Admin,” “Betalogue – Mail,” “Betalogue – Feedback,” etc. In order to select one specific mailbox in that group, I have to type “Betalogue -” and then the first few letters of the next word. That’s a lot of typing.

The same issue applies to family members. I have a mailbox for each family member and, of course, many of them have the same last name, which I usually type first so that mailboxes are sorted alphabetically. Here again, in order to narrow things down to a specific mailbox, I have to type the whole last name and then the first few letters of the first name. Again, that’s a lot of typing.

A couple of readers wrote to indicate that you don’t necessarily have to type out the first word in the name of a mailbox. For example, to get to “Betalogue – Admin,” if I have multiple mailboxes starting with “Betalogue,” I could type “Admin” instead.

The problem with this, of course, is that I also have multiple mailboxes with the word “Admin” in their name. Similarly, when it comes to people’s names, if the last name is common to multiple mailboxes, I could type the first name. But here again, I have multiple acquaintances with the same last name, and I also often have multiple acquaintances with the same first name.

A more useful note with a view to reducing typing is that you don’t have to type out words in full. For example, to select “Betalogue – Admin” in MsgFiler, I just need to type out “bet ad.”

This is good to know, except, of course, that, in this particular case, in my mailbox drawer, “Betalogue – Admin” is not just a mailbox itself, but also a folder containing multiple subfolders. So I still get multiple results when typing “bet ad,” although the “Betalogue – Admin” mailbox itself is the first result (i.e. the one selected by default) and the other mailbox subfolders enclosed within the “Betalogue – Admin” mailbox folder are listed afterwards and can be selected directly by typing portions of the words in their own name.

It all depends on the names used for one’s various mailboxes and mailbox folders, of course. I have a long-established system that I use and I have no desire to change it after all these years. It means that, in my case, MsgFiler might require more typing than it would for other people with other naming systems.

It can still be argued, however, that, even in my situation, MsgFiler is still more convenient and efficient than the default message filing options available in Mail. I will have to give it another try over a number of days and see if I can really get used to it and find it more efficient than Mail’s options and consider it worth the expense and the additional burden of having to manage yet another third-party plug-in. (I am not giving up Mail Act-On for the single keystroke shortcuts that I have defined for my most common mailbox destinations.)

Speaking of Mail Act-On, another reader wrote to point out that Mail Act-On has its own feature for filing messages in any mailbox, aside from its rule-based keyboard shortcuts that can be defined by the user and used for specific mailboxes.

Under “Menus” in the Mail Act-On preference pane, there is a shortcut for a command named “Move Messages.” The default shortcut is F3.

However, as I wrote in my reply to this reader, I have to report that, on my system, that command is unusable. On my system, the window that pops up for message filing is totally unresponsive, or responds to key strokes (or mouse actions) after a delay of many seconds. It is unacceptable.

It was already like this a couple of years back when I first tried the feature, and it is still like this on my 2009 Mac Pro.

The reason might be that I have a large number of mailboxes (nearly 2,000). The reader who wrote to me and who uses this feature all the time only has 40 mailboxes. He has tens of thousands of messages, but only a limited number of mailboxes, and mostly relies on Mail’s search feature to locate individual messages.

I obviously have a different approach. And it looks like Mail Act-On’s “Move Messages” command is useless when the number of mailboxes is large. (Other Mail Act-On features still work fine.)

In closing, I think it’s fair to say that, for people frustrated with Mail’s built-in options for moving messages around, there are third-party alternatives, but that these third-party alternatives are an extra expense and might not work for everyone. My wish is still for Apple to include more efficient built-in solutions, but until that happens, I will definitely give MsgFiler another try.

Unlike AppleScript scripts or Mail-Act On rules, MsgFiler can be used to move messages to any mailbox in your mailbox drawer. The idea is that you define an application-wide keyboard shortcut for the “Move with MsgFiler” command that gets added to the “Message” menu in Mail after you’ve installed the plug-in.

With this keyboard shortcut, you invoke a small window where you can start typing the name of the intended destination, i.e. of the mailbox folder to which you want to move the selected message(s). MsgFiler finds and lists all mailboxes matching your typing, much like a utility like LaunchBar, which I use all the time.

Once you have narrowed down the list in MsgFiler to your intended destination (either by typing more characters or by using the cursor keys to select the desired item in the list), you can just move the selection to the destination by pressing Return.

MsgFiler works as advertised and is reasonably priced. But I had tried it a while back and decided not to purchase it because, in my experience, it still required too many keystrokes to obtain the desired destination.

Unlike LaunchBar, as far as I can tell, MsgFiler does not have a smart abbreviation engine that “learns” your preferred destinations based on abbreviated versions of their names. It just matches what you type to the actual names of the mailbox folders.

If, like me, you have lots of subfolders that start with the same word, then you need to type not just that whole word but the beginning of the next one (or hit the cursor keys multiple times to select the right item in the list of matches). For example, I have mailbox folders called “Betalogue – Admin,” “Betalogue – Mail,” “Betalogue – Feedback,” etc. In order to select one specific mailbox in that group, I have to type “Betalogue -” and then the first few letters of the next word. That’s a lot of typing.

The same issue applies to family members. I have a mailbox for each family member and, of course, many of them have the same last name, which I usually type first so that mailboxes are sorted alphabetically. Here again, in order to narrow things down to a specific mailbox, I have to type the whole last name and then the first few letters of the first name. Again, that’s a lot of typing.

That said, I will definitely give MsgFiler another try and see if I can live with its limitations and still improve the overall experience of moving stuff around in Mail.

(I am also reluctant to spend additional money and of course increase the risk of incompatibilities and instability in Mail by adding yet another plug-in.)

Here’s the situation as I encounter it on a daily basis. By default, I keep all my main mailbox folders in the “ON MY MAC” section of my mailbox drawer collapsed (i.e. closed) so that I can see them all. The last one in the list, as illustrated in the picture below, is a folder called “Work”:

As you can see, when all my folders are collapsed, the last one is still visible above the bottom edge of my mailbox drawer. But of course, if I expand (open) any of these main folders, the list of enclosed folders extends far beyond the bottom edge of the drawer and I have to scroll down to see them all:

Now let’s go back to the initial situation, i.e. when all the folders are collapsed:

If I now drag a selection of messages from my message list to the “Work” folder and wait for a fraction of a second, the automatic spring-loading mechanism causes Mail to expand the folder to reveal its contents:

So far so good. But let’s say the enclosed folder to which I want to drag my selection is lower down in the list. In order to reach it, I will have to scroll down the list of enclosed folder. But I have my index finger on my primary mouse button, because I am dragging stuff. So I cannot use the scroll wheel for this.

The expected standard behaviour in such a case is that, when I drag my selection near the bottom edge of the drawer and wait for a fraction of a second, Mail will start scrolling down the list to reveal the rest of the enclosed folders.

But that’s where there is a problem. The “near the bottom edge of the drawer” area that I am supposed to target with my dragging is not clearly defined and, more important, it overlaps with the area where one of the enclosed subfolders appears (in this case, the folder called “Graphic Designers”).

So what’s going to happen when I try to drag my selection near the bottom edge of the drawer and wait for a fraction of a second? Instead of simply scrolling down the list, Mail will also activate… the automatic spring-loading mechanism for the “Graphic Designers” folder:

But that’s not what I want! I do not want to drag my selection to a sub-subfolder inside the “Graphic Designers” folder! I want to drag my selection to a subfolder after the “Graphic Designers” folder.

Unfortunately, because of the overlap between the area that triggers scrolling and the area that triggers spring-loading, I cannot have one without the other. And the end result is that I have to scroll down through the list of sub-subfolders first before I can reach the rest of the list of subfolders.

Fortunately, Mail doesn’t continue this silly behaviour ad infinitum. Otherwise, it would be a nightmare. It only confuses scrolling and spring-loading once, presumably because after that, since the drawer is scrolling down, I am not longer lingering on any given position long enough to trigger spring-loading, until I actually choose to stop the scrolling by lifting my selection back up a bit.

And once I’ve actually dropped what I was dragging in the desired location and released the mouse button, of course Mail collapses everything back up and I am back to normal.

Still, it’s quite frustrating that, almost every time I want to archive something somewhere inside my hierarchy of mailbox folders, I am forced to deal with this confusion between scrolling and spring-loading and scroll through more things than I should have to.

Given that this is a problem that only affects people with a long list of folders in their mailbox drawer, and not the majority of users who never archive their mail outside their inbox and let it balloon up to thousands of messages, I wonder whether Apple is even aware that there is a usability issue here and, if so, whether they can be bothered to think about a way to fix it.

Admittedly, it’s a small problem, and one that I have learned to live with. But it still grates that I always have to do more scrolling than should be necessary and deal with unwanted spring-loading which, as far as I can tell, cannot be turned off (and one wouldn’t want to turn it off anyway, since it is necessary; the only option would be to turn it off only in this particular area of the drawer).

(And yes, I know that I can avoid all this by using right-click to browse my mailbox folder hierarchy via the “Move To” contextual menu command, but that comes with its own set of problems, because the menu font is bigger, and hierarchical menus require greater mouse movements with a greater risk of overshooting and having to start all over again.)

I have a few AppleScript scripts and keyboard-activated Mail Act-On rules for moving stuff to my most frequently used destinations, which helps alleviate the problem, but I cannot have a rule or script for every possible destination, so I still have to deal with this problem on a daily basis. And that, in my book, makes it more than a “minor” annoyance.

For a long period in the 80s and 90s, for his personal computing needs at home, he was a Windows guy, and that was the source of many passionate, yet light-hearted Mac vs. PC arguments between him and me (and my brother, another long-time Mac user).

Then, when Apple came up with Mac OS X and its Unix underpinnings, my dad made the switch and, as far as I know, he hasn’t regretted it one bit. He takes care of his own tech support needs. But from time to time, since I am the “Mac expert” in the family, I get an e-mail from him about something that puzzles him.

And so today I got this e-mail about Mail. The gist of it was that he wondered why, even though all his settings were correct, sometimes Mail didn’t check for new mail for certain accounts (he has several) when it was supposed to do so, i.e. during the automatic mail checking procedure that takes place at regular intervals. And he wondered whether that was linked to the fact that, “quite frequently” (his words), Mail would ask him for the password for this or that account… a dialog box that he usually got out of by clicking on the “Cancel” button.

I thought our e-mail exchange would be a good opportunity to revisit the long-standing and pretty serious problem with Mac OS X’s Mail and less-than-ideal network conditions.

Now, you need to know that, unlike me, my dad lives in an urban environment and is subscribed to a fast DSL Internet service with plenty of bandwidth. But of course, he uses a MacBook Pro laptop with its wireless connection to his router. That means that, even with more than enough bandwidth for his needs, he can still encounter occasional problems with his network connection, either because of a temporary failure in his wireless connection due to interference, physical obstacles or software or hardware flakiness beyond his control, or because of occasional problems with the provider itself.

This, in my experience providing tech support services over the years to a variety of clients with a variety of means to connect to the Internet, is pretty much the norm, and not the exception. (The exception is my own situation, i.e. a lousy satellite-based Internet connection that, in addition to all of the factors mentioned above, can also be affected by weather conditions and bandwidth throttling during peak hours.)

The single most scandalous thing about Mac OS X’s Mail is that its behaviour when checking for new mail appears to have been “designed” (if you can call it that) based on the assumption that network failures never occur. I don’t know what kind of weird bubble Apple’s Mail engineers live and work in, but one thing is for sure: it is completely disconnected from reality. I don’t know a single person who has never experienced network outages. Even with the best equipment, the most up-to-date software, and the most ideal technical environment, there will be occasional failures. It’s just a fact of life.

That means that the job of an Apple engineer working on Mail includes ensuring that the application’s behaviour when such a failure occurs will be reasonably user-friendly.

But that obviously is not part of the job description. Because Mail’s behaviour in such conditions is appallingly bad, and it has been that way for years now. How any self-respecting software engineer working on Mail at Apple can sleep at night knowing that thousands, if not tens or hundreds of thousands, of Mac users are currently experiencing the idiotic behaviour that I am about to describe once again is beyond me. These people should be ashamed of themselves.

If Mail tries to check for new Mail while a network/connection failure of some kind is occurring, here’s what happens.

If you are lucky, Mail takes your e-mail account off-line without telling you.

If you are not so lucky, Mail’s Dock icon starts bouncing endlessly, because the application is displaying a modal dialog box asking you to… re-enter your e-mail account’s password.

If you are wise enough to know better, you do like my dad and you click on the “Cancel” button in that modal dialog box. It gets rid of the dialog box, but it also takes the account off-line.

If you don’t know better, you actually follow Mail’s request and re-enter your password. But of course, like most people, you have set Mail to remember your password for you, so you are not sure you actually remember it correctly. “Is it the name of my dog or the name of my grandmother or something else entirely—maybe something weird that Pierre Igot asked me to use the last time he visited me because he was concerned about the safety of my e-mail account and didn’t want me to just use as a password the name of my dog or my grandmother, which dozens of people who know me directly or indirectly know very well themselves?”

“Oh well, let’s try the dog’s name.”

And then all hell breaks loose. Unless you are really lucky and you remembered the correct password, and then, if the network failure has not miraculously stopped during the interval (and even then, I am not sure that’s enough for Mail to recover at this point), the same thing will happen again and you’ll get the dialog box again. And this time you’ll hit “Cancel” and Mail will stop trying to check for new mail and take the account off-line.

If you’re not really lucky, you’ve typed the wrong password, and now you’re really screwed. Because even if you hit “Cancel” after getting the same dialog box five times and entering five different variations on that password and giving up—and then wait for the network failure to stop and try to check for new mail manually, the mail checking will fail, of course, because the password that Mail now remembers and uses is wrong, and you’ll be caught in an endless loop of the dialog asking for your password and Mail taking your account off-line.

Time to call Pierre, and hope that he can remember your password. If you’re lucky, this has happened before, and he’s stored your password in his encrypted password database. Otherwise, you’ll have to call your ISP’s tech support line and wait for half an hour until you can speak to a live individual and then identify yourself with personal questions and confidential information and so on and then the ISP might give you your actual password or reset it for you. (By that time, one hopes that the network failure will be resolved and you’ll be able to verify that it is the right password indeed.)

And all this, because Apple’s engineers apparently have not noticed this idiotic behaviour and how aggravating it is for so many people using Mac OS X’s Mail in the real world, where network failures do occur.

Now, let’s analyse this behaviour for a second here. First of all, why does Mail ask for your password when your password is perfectly correct, simply because there is a network failure? Is it really impossible for Mail to tell the difference between a failed connection and a wrong password? Is it really reasonable to assume that, if the connection to the server fails, the most likely explanation is that the user went behind Mail’s back and changed his e-mail account’s password using his ISP’s web interface or by calling the ISP on the phone?

It is not. It is not just unreasonable. It’s completely stupid. How often does the average user change his e-mail account’s password or have it changed by someone else?

And then, there is that thing with taking accounts off-line. Who on earth got that idea that e-mail accounts should ever be taken off-line when the user does not explicitly ask Mail to take them off-line? Why on earth does cancelling this idiotic modal dialog box asking for a password that Mail already knows and is perfectly correct cause Mail to take the account off-line even without the user’s permission?

It’s bad enough that this dialog box appears in the first place. But on top of it, whichever way you choose to respond to the dialog box, you will probably screw things up. The least disastrous option is indeed to hit the “Cancel” button, as my dad correctly figured out by himself, possibly in part because he has 40 years of computing with various user-hostile systems behind him. But even that least disastrous option is still a major pain in the neck, because it means that you have to know and remember that you have to take your account back on-line manually, either by forcing Mail to manually check for new mail or by clicking on the warning icon that appears next to the account’s Inbox in the mailbox drawer.

Of course, you cannot take your account back on-line until the network failure is gone, so there’s no point in trying to do it right after you’ve clicked on that “Cancel” button. No, you have to actually wait until the network failure is resolved, and then remember to take the account back on-line manually.

And then, if you were lucky or wise enough not to screw up your account’s password, things will work properly again.

But even a seasoned computer user like my dad couldn’t bring himself to think that Apple’s engineers would be so clueless that they didn’t even design Mail to recover automatically once the network failure is resolved. So, while he correctly figured out that he had to click on that “Cancel” button and that this particular weirdness probably had something to do with the fact that Mail didn’t seem to reliably check for new mail in all his accounts automatically all the time, he didn’t go as far as to think that he actually had to manually bring these accounts back on-line after the end of the network failure, even though he had never taken them off-line himself in the first place.

And if my dad with his 40 years of experience couldn’t quite figure it out, you just know that there must be hundreds of thousands of Mac OS X users enduring that very same problem and not knowing how to deal with it “properly,” i.e. by doing manually what Mail is utterly unable to do by itself automatically.

Who knows how these users finally get out of the situation that Mail has put them in. Maybe quitting and relaunching Mail actually forces Mail to take the accounts back on-line. I don’t know. There are so many different situations in Mail that it is impossible to even tell when an account is actually in an off-line status that requires the user to manually take the account back on-line or just in a greyed-out status that will somehow resolve itself without user intervention.

I have seen greyed out Inboxes with the warning icon (exclamation mark inside triangle). I have seen greyed out Inboxes with the danger icon (lightning bolt). I have seen Inboxes not greyed, but still with the warning icon, itself greyed out or not. I have seen it all. I have never understood what the difference is, and how to tell whether Mail will actually try to check for new mail in the affected account automatically or not.

The whole thing is just so stupid, and so incredibly frustrating. And it’s not just me and my lousy Internet connection. I have talked about this problem before, and I have received feedback indicating that it happens to all kinds of people, in all kinds of situations, with all kinds of Internet connections.

There clearly is something very wrong here. It’s been like this for years, and it does not look like Apple is interested in doing anything about it. What do we need to do? Do we need to start a petition? A boycott? Do we need to sabotage the Internet connections at 1 Infinite Loop so that Apple’s own engineers get a taste of the action? What exactly will it take?

This has been the situation for years now with Mac OS X’s Mail. Usually, you have to dismiss the dialog box by hitting “Cancel.” This usually takes the account off-line (another gem from Apple’s interface experts) but at least you can restore the normal situation by bringing the account back on-line and checking for new mail once the bandwidth situation has improved.

Much to the dismay of every Mac OS X user with occasional bandwidth unreliability, Apple has actually managed to make the situation worse in Snow Leopard—much, much worse.

Now, when checking mail in a low-bandwidth situation and getting a lack of response from a mail server, Mail 4.x still throws the modal dialog box in your face. But then when you dismiss it, which takes the account off-line, for some reason the idiotic Mail actually really forgets the account’s password.

Even if you bring the account back on-line, when you try to check for new mail and the bandwidth situation is back to normal, Mail still asks for the account’s password—even though it has not changed and should now be properly recognized and processed by the mail server. You have no choice but to manually enter the password again and check the “Remember this password” option to make sure Mail actually remembers it again (until next time).

And even when you’ve re-entered the password, Mail still takes the account off-line yet again, instead of initiating a proper connection with the server. You then have to take it on-line once more, and then finally things get back to normal and Mail manages to retrieve the new mail for the account from the server.

This is completely ridiculous. It’s quite clear that no one at Apple ever bothers to check Mail’s behaviour in situations with compromised bandwidth. I know that artificially reproducing such situations takes a bit of effort, but obviously even that little bit effort is too much for Apple’s poor overworked engineers. Maybe they could just try unplugging their Ethernet cable from time to time and see what happens. It might be enlightening.

But then, I don’t think they care one bit or pay any attention to this. All my bug reports about problems with low bandwidth in Mail and Safari over the years have been gleefully ignored, with a few unexpected exceptions. (Apple actually fixed this old problem with Safari and slow-loading pages in tabs in Safari 4.)

No, in Apple’s world, everything is always rosy, your Internet connection is always on and superfast, so you never ever have to deal with any low-bandwidth situations. Got that? Now shut up and smile.

It’s rather annoying, because that means that there is no easy way to copy just the e-mail address, which is what you’d want to do, for example, if you wanted to manually add the e-mail address to an Address Book card. Actually, there is no way at all to copy just the e-mail address, since, as soon as you hover over the “From:” field with your mouse button, the whole field becomes highlighted as a single blue widget and you cannot click within it to select part of it. (And that is with the “Use Smart Addresses” option off in Mail’s preferences. With the option on, you cannot even see the e-mail address.)

In addition, the command’s label reads “Copy Address” and not “Copy Sender.” Since when is someone’s name part of his/her “address”?

As per usual, when I posed this blog item, I also submitted a bug report to Apple. Unfortunately, I have to report that, three days later, this morning, I received the following response:

Engineering has determined that this issue behaves as intended based on the following information:

This is the correct behavior in Snow Leopard 10.6.

We consider this issue closed.  If you have any questions or concern regarding this issue, please update your report directly (http://bugreport.apple.com).

So that is that. As far as I can tell, the only way that now remains to select and copy the e-mail address only is to switch to the “Raw Source” message view via the “View” menu and then manually select the address.

Good grief. I guess that, with Apple’s engineers, it’s their way or the highway. Am I really so wrong to want to manage my Address Book entries without resorting to the flawed “Add to Address Book” menu command? I guess so.

As a reader pointed out to me, however, it appears that this behaviour already existed in versions of Mail prior to Snow Leopard. (I don’t know how far back it goes.)

The fact remains that I never noticed this behaviour until I migrated to Snow Leopard a couple of weeks ago. I don’t believe that, all of a sudden, I would have changed my dragging habits, causing Mail to display a behaviour that I never caused it to display before.

What is going on here then? Well, I do not have hard evidence to support this (it’s too complicated to obtain), but I strongly believe that what happened is that, while the direction-dependent behaviour did indeed exist in previous versions of Mail, Apple’s engineers tweaked it in Snow Leopard and made it so that it is now easier to trigger the vertical dragging behaviour (extending the selection) than it used to be, i.e. the angle within which Mail interprets the dragging as a vertical movement is now wider than it used to be.

I have two reasons to believe this. One is that, in my fairly informal and limited testing on my wife’s MacBook Pro (still running Leopard), I can definitely sense a difference in the spread of the angle within which Mail interprets the dragging as a vertical movement. It seems much narrower to me in Leopard than in Snow Leopard.

The other reason is that there is another area of the Mail interface where Apple has most definitely tweaked something too in Snow Leopard (i.e. in Mail 4.x), and that is the mailbox drawer, and more specifically what happens when you drag a message onto a enclosing mailbox folder in the mailbox drawer, causing the mailbox folder to expand and reveal the mailboxes it contains, and that list of mailboxes extends beyond the visible portion of the mailbox list in the mailbox drawer.

In such a situation, you have to keep holding your mouse button down near the bottom of the mailbox drawer to cause it to start scrolling down to reveal the rest of the list of mailboxes. And I have definitely noticed that, in Mail 4.x, this behaviour is not as smooth/quick as it used to be in previous versions of Mail. I often have to spend more time down there waiting for Mail to reveal the mailbox that I am looking for, because Mail doesn’t scroll down as readily as it used to do.

I don’t know exactly what they have changed, whether it’s the delay before the list starts scrolling or something else, but they have definitely changed something, and it’s obviously a very minor tweak with not so minor consequences (enough that I notice it, and it annoys me).

So if they have tweaked one area of the Mail interface in such a subtle, but significant way, there is no reason why they wouldn’t have tweaked another area, and that is what I believe is going on with the vertical dragging in the message list—hence my initial reaction that there was a new behaviour in Mail. There is simply no way that I changed my own behaviour when it comes to dragging messages, so something has changed. It is not a new behaviour, as I initially thought, but it’s definitely different from what it used to be—and not for the better.

I should also mention, in closing, that another reader pointed out that there is at least one other Apple application where the dragging behaviour is the same as in Mail, and that is the Address Book application. Since, in my limited testing, the Address Book appears to be as sensitive as Mail to the “verticality” of my dragging, there is reason to believe that this behaviour is not actually Mail-specific but an underlying routine in Mac OS X that is used by both Mail and Address Book but not, for whatever reason, iTunes and the Finder.

Since I upgraded to Snow Leopard, I have noticed, on more than one occasion, that when I attempted to drag a selected message in Mail’s message list in order to file it in a specific mailbox, instead of responding to my mouse movement by actually dragging the selected message, Mail would for some reason interpret my gesture as meaning that I wanted to extend my selection of messages by adding more contiguous messages to the current selection.

This never used to happen in Mail prior to version 4.x (i.e. the Snow Leopard version). It was quite irritating.

Today, I did more testing and was able to determine that, apparently, Apple has introduced a new behaviour where the meaning of clicking and dragging in Mail’s message list changes depending on the initial direction of the dragging.

If the dragging is more or less vertical, then Mail interprets it as meaning that the user wants to create a contiguous selection of several mail messages.

If the dragging is horizontal, then Mail interprets it as meaning that the user wants to drag the selected message(s) to a destination in the mailbox drawer on the left.

Of course, since my mailbox drawer contains a fairly long list of mailboxes that extends all the way to the bottom of it, it happens quite often that, when I click and drag on a message in the message list near the top of the list in order to file it in a mailbox somewhere near the bottom of the mailbox drawer, my dragging movement is, at least initially, more or less vertical.

Now, with this new behaviour introduced by Apple’s engineers in Snow Leopard, I am forced to try and remember that, when I want to drag a message or a bunch of messages to my mailbox drawer, I need to make sure that the dragging movement immediately after my clicking is more or less horizontal.

This is ridiculous. Did Apple actually test this in the real world? Don’t they have big 30″ screens with mailbox drawers that contain long lists of nested mailboxes? Can’t they see that it happens far too often that the user starts with a vertical drag even when what he means to do is to drag something to the left towards the mailbox drawer?

What pisses me off even more is that we went through exactly the same idiotic direction-dependent discrimination with draggable tabs in Safari 3.

And finally in Safari 4 Apple fixed this and switched back to a behaviour that is the always same regardless of the initial direction of the dragging movement.

I suppose this means that we are going to have to live with this stupid direction-dependent behaviour until the next major revision of Mail (i.e. of Mac OS X), when enough Mail users have complained that this was a stupid idea to begin with.

Couldn’t the Mail guys have talked to the Safari guys to inquire about how well the direction-dependent behaviour in Safari 3 was received and why they switched back to a direction-independent behaviour in Safari 4?

Besides, what other Apple application is there that exhibits the same behaviour as Mail 4.x?

iTunes 9? Nope. If you want to extend a selection vertically in list view, you have to you shift-click. You can click-and-drag on an item in the list in any direction to add it to a playlist in the source list on the left-hand side.

Finder? Nope. If you want to extend a selection vertically in list view or column view, you have to use shift-click. You can click-and-drag on an item in the list in any direction to add it to the side bar on the left-hand side.

What kind of planet do the Mail 4.x guys live on?

It turns out that, somewhat inevitably, the changes have introduced new problems.

Take for example what happened to me this morning. I got an e-mail in Mail with a new version of a PDF file that was already sent to me last week and that I had saved on my local hard drive. Back when I got the first version, I saved it and then renamed it in the Finder using a more useful file name.

So when I got the new version today, I brought up the “Save As…” dialog box in Mail to save the attachment, I navigated to the folder where I had saved the previous version, and I option-clicked on the file name of the previous version in the file list.

Mail correctly inserted the existing file name into the new file name field, and, in accordance with the new behaviour in Snow Leopard, it didn’t include the file name’s extension (.pdf) when copying the file name to the new file name field.

The problem is that, unlike an application such as Pages or TextEdit, Mail does not have a facility for handling file extensions in its “Save As…” dialog box for saving message attachments. There is no button to check to hide or reveal the file extension. So in the file name field I now had the existing file name, without the “.pdf” extension, and I didn’t know what would happen with the file extension.

What do you think happened when I hit the “Save” button? The first sign that something was wrong was that Mail didn’t warn me that there already was a file with the same name in the same location. I wondered whether that meant that Mail had saved the new attachment with the same name as the existing one, but with “-1” added to it or something like that.

So I went and opened the folder in the Finder. Much to my surprise, I saw that what Snow Leopard’s Mail had done is that it had saved the new attachment with the same file name as the existing one, but without the “.pdf” extension!

Somehow, the Finder still new that this file was a PDF file and displayed a preview of it correctly. But this was clearly not what I wanted. After years of forcing us old-school Mac users to use file extensions, for more or less valid reasons, surely Apple is not about to introduce behaviours that do away with them altogether?

I highly doubt it. I suspect that this is simply a bug introduced by the new behaviour and linked to the unique nature of the “Save As…” dialog box in Mail for attachments, which does not have the same features as a regular “Save As…” dialog box.

So back to Bug Reporter we go…

A number of bugs have been fixed, including this one, which I first reported in early 2008, about mangled quote levels when editing a reply in plain text.

It used to be that deleting a block of text with a certain quote level would cause the quote level of the first line of the next block of text to change. Not anymore. Here’s an example of a text selection that would cause problems in Mail 3.x:

In Mail 3.x, deleting this selection would cause Mail to change the quote level of the text that immediately followed. Not anymore:

It took a while to get Apple to fix this, but finally it is fixed, and plain text e-mail users can rejoice. (Not too much, though: Mail 4.0 does introduce new bugs. More on them later.)

At the time, I reported that I was unable to complete one of the main troubleshooting steps involved in trying to eliminate the application’s sluggishness, which was to force Mail to rebuild its “Envelope Index” file by moving the existing file out of the “Mail” folder inside my home library folder.

With some help and prompting from Betalogue readers, I then managed to fix this particular problem by narrowing its source down to a single e-mail message containing two attached Microsoft Excel files. Removing this message from the mail archive eliminated the crash and Mail was able to complete the rebuilding its “Envelope Index” file.

But did this fix my problems with sluggishness in Mail?

After a couple of weeks of regular use, I am pleased to report that it appears to have done the trick. I no longer see the “Loading…” message when opening a message in a separate window, even when there are other background processes going on at the same time in Mail.

At the time, I also indicated that I had submitted the issue with the crash during importing as a bug report to Apple. Since then, I have received a reply from Apple, stating that this is a “known issue”:

This is a follow up to Bug ID# 7028784.  After further investigation it has been determined that this is a known issue, which is currently being investigated by engineering.  This issue has been filed in our bug database under the original Bug ID# 5543122. The original bug number being used to track this duplicate issue can be found in the State column, in this format:  Duplicate/OrigBug#.

The original bug ID number (5543122) seems to indicate that Apple has known about this issue since the fall of 2007. Unfortunately, we are obviously still waiting for a fix. (But at least the problem is known and Apple was kind enough to respond to my bug report fairly quickly.)

It is fortunately possible to work around the bug “manually” by monitoring the importing process carefully. So if you ever need to rebuild your “Envelope Index” file and encounter a crash during the importing process, I suggest you follow the steps that I described in my last blog post in order to try and pinpoint the offending message(s) and remove them manually from your archive.

As for why rebuilding the “Envelope Index” file fixed the sluggishness in Mail, I obviously do not know. But it’s a known troubleshooting step and sometimes we have to use these steps even without knowing what it is exactly that they are meant to fix.

All I can say is that other troubleshooting steps didn’t help, and this appears to be the one that did it and restored my Mail application to its normal level of performance on my machine.

I have managed to solve the problem, with some prodding from a kind reader.

I carefully monitored the reimporting process (i.e. rebuilding of the “Envelope Index” file) and was able to notice that it would always crash when Mail reached a specific mailbox… the mailbox in which I archive my correspondence with my father, of all people.

I also noticed that both the system log and the crash log mentioned this:

Jun 29 09:47:14 Mac-Pro Mail[86791]: *** -[NSCFString stringByAppendingPathExtension:]: cannot append extension 'xls' to path ''
Jun 29 09:47:14 Mac-Pro Mail[86791]: Error: -[NSFileWrapper setPreferredFilename:] *** preferredFilename cannot be empty.

among other things… This led me to suspect that the mailbox with my dad’s archived messages contained a problem message that would cause this Mail crash every time.

I reopened Mail with the old “Envelope Index” file and went to check that particular mailbox. Again, I could see the list of messages, but when I tried to view any of them, I would get a blank window. I then tried to rebuild the mailbox (which, as indicated in the previous post, would normally fix this particular problem), and Mail crashed—with the same error messages in the crash log as above.

In other words, both the general reimport process and the simple mailbox rebuild process would cause Mail to crash when hitting that particular mailbox.

I then quit Mail, opened the “Mail” folder in my home library and located the mailbox folder for my dad’s correspondence. I manually removed it from its location and put it on the desktop.

I then relaunched Mail and tried to import that particular mailbox folder from its desktop location. I got the same crash again, but this time I could see that it was part way through the importing process, thereby confirming that it was occurring when Mail encountered a specific e-mail message.

I couldn’t tell which message from the Mail progress bar or the crash log, but given the mention of “xls” in the crash log, I started to suspect that it was an e-mail with Excel files attached to it. And sure enough, by sorting the e-mails by size in the Finder and using Quick Look to view the contents of the biggest ones, I was able to locate precisely one such message, with two Excel files in it:

I proceeded to remove the offending message manually from the mailbox folder, and tried to reimport the mailbox folder without it. And it worked!

Finally, I quit Mail again, trashed the “Envelope Index” file again, and relaunched Mail, once again triggering the reimport process, but this time with my dad’s mailbox not containing the offending message.

And it worked! Mail managed to go through the entire process of importing over 100,000 messages without crashing.

So that is the solution of this particular mystery. Apparently, there was something in that message with the two Excel files attached that Mail really didn’t like.

I will send the offending e-mail with a bug report to Apple, and maybe they can figure out what causes the problem in this particular case (although I doubt that an isolated problem such as this one will register on their radar…).

I can’t help but find it a bit ironic that Microsoft files are able to give me grief even when I am not using Microsoft products. But that might be a bit unfair; it might just be a coincidence.

Still, it confirms me in my habit of removing attachments from e-mail messages before archiving them. I didn’t remove them in this particular case because it was a couple of files that my dad had asked me to hold on to for a while, until he was sure he had safe copies at home on his own system. At the time, I filed the message away with the attachments and then of course forgot about it. I guess I’ll have to try and be even more religious about removing attachments in the future.

(It also does not help that Mail’s “Remove Attachments” feature is still so badly flawed, which this stupid behaviour of deselecting the message after its attachment has been removed and selecting the next one in the list—or nothing if it’s the last one in the list. I am sure they consider this a “feature,” but I find it a royal pain to have to reselect the message after removing its attachments just to be able to file it away…)

Long-time readers of this blog might remember that I am in fact quite familiar with this procedure, in particular because I was part of the AppleSeed program for the testing of prerelease builds of Mac OS X 10.4 (those were the days) and that I had to do a fair amount of troubleshooting in Mail during those early days of the Spotlight architecture.

So why didn’t I mention the procedure in my post about my recent troubles in Mail?

Well, in a nutshell, the answer is that I could not get it to work, and I was going to try and do more investigating before mentioning it again.

Quitting Mail, trashing the “Envelope Index” file in the “Mail” folder (in the home folder’s library) and relaunching Mail triggers a behaviour where Mail attempts to repeat the procedure that it first used when the user upgraded his system from Mac OS X 10.3 to Mac OS X 10.4, and thus from Mail 1.x to Mail 2.0.

That transition to the new Mail 2.0 required a complete conversion of all existing mailboxes to the new granular file system used in Mail since version 2.0, i.e. a system where each message is saved as a separate file, in order to be compatible with the Spotlight architecture.

When a user first launches Mail 2.0 (or a subsequent version) on a system where the “Mail” folder in his home library is still in the Mail 1.x format, Mail 2.0 gleefully announces that all the existing mailboxes have to be “imported” in order to work with its new file architecture. The user has no choice but to approve the procedure and sit back and watch while Mail 2.0 attempts to convert all existing mailboxes to the new file format.

The reason that I am mentioning all this is that trashing the “Envelope Index” file in an existing “Mail” folder that already uses the new file format triggers the same kind of behaviour. The next time you launch Mail, it tells you that it “needs to import your messages” and gives you no choice but to proceed or to quit the application:

If you click on “Continue,” Mail starts reimporting your existing message files and folders, effectively rebuilding the “Envelope Index” file in the process.

It’s not exactly the most intuitive interface, but this a troubleshooting procedure, so I guess we have to take what we get.

My problem with this procedure on my machine right now is that it systematically fails. Mail starts importing my thousands and thousands of messages, and completes approximately one half of the importing, and then it suddenly, unexpectedly quits.

In other words, Mail never manages to complete the reimporting procedure, including the rebuilding of the “Envelope Index” file, and I am unable to use Mail at all—until I restore the “Envelope Index” file backup that I have kept elsewhere on my hard drive, just in case.

To make matters worse, even after I restore the “Envelope Index” file from a backup and am able to use Mail again, there are some new problems that I have to fix first. For example, this aborted troubleshooting procedure tends to cause Mail to forget all my e-mail account passwords, for whatever reason. So I have to reenter them manually one by one, and even then, sometimes it takes several attempts before Mail actually agrees to remember these passwords properly without asking me to reenter them each and every time it checks for new mail.

The aborted troubleshooting procedure also tends to cause all kinds of problems in my mailboxes themselves. They seem intact and I can see individual messages is them, but when I double-click on the messages to open them, Mail displays a window whose contents are totally empty. I then have to select the affected mailbox, and use the “Rebuild” command. After that, things work normally and I am able to read my messages again. But I have hundreds of mailboxes and I don’t fancy having to rebuild each and every one of them.

Finally, the aborted troubleshooting procedure also seems to cause all kinds of problems with Spotlight. When I try searches on the “Entire message” in Mail, typically I only get an incomplete list of results. There are many items that do contain the searched string that are simply not there in the list of results. This problem seems somewhat connected to the previous one, in that, after I rebuild the mailbox containing the items missing from the list of search results, and repeat the search, then they finally appear in the list of results.

All this is to say that, unfortunately, at this point in time, the troubleshooting procedure consisting of trashing the “Envelope Index” file is not working too well for me. In fact, it’s not working at all, so I don’t know whether it would help fix the sudden sluggishness in Mail.

The only other thing that I have tried is to use a third-party utility tool (OnyX for Leopard) to rebuild Mail’s Envelope Index. OnyX has an option just for this, and I have tried it. But I am not sure exactly what it did and what it repaired, because it did not take very long (only a fraction of the time it would take for Mail itself to rebuild it) and the “Envelope Index” file does not seem to have changed much. It is still the same size and still has an old creation date.

(I am also not sure whether what OnyX did had anything to do with the other problems that I experienced in Mail while attempting to troubleshoot my sluggishness problem, i.e. the blank window messages, the password issue, and the lack of Spotlight search results.)

So until I find a reliable way to rebuild the “Envelope Index” file, I am afraid I won’t be able to fully verify whether that is what it takes to fix Mail’s sudden sluggishness and get rid of these pesky “Loading…” messages. For now, I have no choice but to continue to use my (possibly corrupted) existing “Envelope Index” file and hope for the best…

I am always careful to remove attachments before filing messages away (I save the attachments that I need to keep elsewhere before removing them), so that the total size of my “Mail” folder does not become unmanageable. But I do keep the messages themselves, and according to Mac OS X’s Finder I now have over 100,000 items in that folder, for a total size of over 1 GB.

Most of these messages are filed in specific mailboxes, so that my main account mailboxes (Inbox, Sent, Junk, Trash, and Drafts) are reasonably lightweight.

Until recently, this situation did not seem to be a problem for Mail, which was performing adequately on my three-year-old Mac Pro.

For some reason, however, a few weeks ago, Mail started displaying an unmistakable level of sluggishness. And when I say “displaying,” I mean it:

This is what I am now getting far more often than I would like when I am doing a very simple thing such as opening a message in a separate window to read it.

The message’s contents do appear eventually, but it can take several seconds, and there is absolutely no apparent reason for this delay.

It tends to happen more frequently when I try to open a message to read it while Mail is still in the process of sending a reply to another message. (My Internet connection is not very fast, so that can take a few seconds, even for sending a small message.) But it also happens when nothing else is going on in Mail. (I keep the Activity Viewer window open at all times in order to be able to monitor Mail’s various processes.)

This never used to be a problem in Mail, which, on my Mac Pro, was, until recently, perfectly able to handling multiple tasks simultaneously without any noticeable delay and was certainly able to open a message window and display the message pretty much instantly.

I haven’t changed anything to my settings or my hardware configuration, and there haven’t been any significant software updates that would explain such a change. And it’s also hard to imagine that, after several years of being able to handle thousands of messages without problems, Mail would all of a sudden have somehow reached a limit and lost its ability to perform at an acceptable pace.

I should stress that this is not a problem with the Mail UI itself, which remains responsive at all times. The problem is with the threads that the UI triggers.They all appear as separate processes in the Activity Viewer, with a progress bar and a “Stop” button for each (although some “Stop” buttons are greyed out and cannot be clicked on). Even the simple process of double-clicking on a message to open it in a window triggers a couple of threads that appear in the Activity Viewer.

Normally they just flash by and the message opens almost instantly. But now I see these threads stay there in the Activity Viewer for several seconds while Mail seemingly struggles to complete them (and displays the above-mentioned “Loading…” message in the message window instead of the actual message contents). I can still do other things in Mail or elsewhere in the meantime, but that is faint consolation. I don’t think I should have to wait several seconds for a message to open in a window so that I can read, especially when that message is just plain text with no pictures and no fancy “rich text” code.

I have tried various things, such as “rebuilding” my Inboxes (with the “Rebuild Mailbox…” command). But it has all been to no avail. Mail still is significantly more sluggish than it used to be when it comes to completing its various threads of activity. (Again, the UI itself is not sluggish. It’s only the threaded processes. But since pretty much everything that the user does in Mail triggers one such process, it very much affects the very usability of the application.)

Did Apple make some kind of change in the Mail code in a recent Mac OS X 10.5.x update that would have caused this problem to crop up? It’s quite possible. But it would be mighty hard to prove, especially since I don’t have a specific date for the moment this problem started happening to me on my Machine.

In all likelihood, what will happen is that I will have to live with this new problem for a few more months, and then Snow Leopard will come out with its many promised improvements in performance, and things will be better again. At least that is the best I can hope for at this stage, I am afraid.

Of course, Quick Look is first and foremost a Finder feature, so its use in other applications, even Apple’s own, is a bit of a hit-or-miss proposition. But it does work in other applications as well. For example, it works in Mail when you are viewing a message that someone sent you with an attachment. If you select the attachment’s icon in the message and press Space, Mail does display a preview of the attachment’s contents.

So it’s quite easy to get into the habit of using Space as a shortcut for previewing files in Mail as well.

Unfortunately, it does not work everywhere. For example, if you are in the process of composing a message in Mail and you insert a file attachment in your message, you cannot select the attachment’s icon and press Space to Quick Look it. If you try to do that, Mail actually interprets the Space keystroke as meaning that you want to insert a space character and writes the space over the selection, deleting the attachment!

Fortunately, there is still a way to use Quick Look to preview a file attachment even when composing messages. But you must refrain from using the keyboard shortcut for Quick Look. Instead, you can right-click on the attachment icon. “Quick Look Attachment” is one of the commands in that contextual menu, and it will let you peek inside the attachment.

I still find myself accidentally pressing the Space shortcut to Quick Look an attachment from time to time while composing a message, because it is simply impossible to remember that this only works when the attachment is in a non-editable message. But of course it’s easy to undo the accidental entry of a space character over the attachment, with command-Z, which restores the attachment. Then right-click to bring up the contextual menu works as an acceptable substitute.