Mac OS X 10.5: Broken Spotlight indexing for mail messages caused by presence of older Mail application
Posted by Pierre Igot in: MailMarch 28th, 2008 • 5:21 pm
I first wrote a post about a problem I had with broken Spotlight searches in Mac OS X 10.5’s Mail application.
I then wrote a post about the feedback I had received from Apple on this problem and how it had enabled me to fix the problem on my machine. In that post, I indicated that, based on Apple’s feedback and on the resolution, the cause of the problem on my machine was probably the presence of an older version of Mac OS X’s Mail application on one of my other (non-startup) partitions.
I have since received more feedback from Betalogue readers by e-mail, including the following message, which provides a pretty clear and convincing explanation of the problem. Since this reader sent it to me privately and agreed to let me share it with Betalogue readers, I thought I would post it here in order to further clarify the problem’s cause and solution:
BACKGROUND:
Launch Services maintains a list of file metadata and applications, to map files to concrete types and the applications that can handle them. It has a fairly decent mechanism for preferring specific applications for specific metadata (file extension, HFS type and creator codes, etc), using hints from the user if necessary (e.g. via Finder’s Get Info -> Open With). Unfortunately it’s not as robust about mapping metadata to concrete types, when more than one application is known to handle a specific set of metadata.
The Launch Services database is built by scanning standard locations (/Applications etc) on startup and login, and Finder registers applications on demand as it discovers them in new locations (e.g. when you mount a network drive, disk image, or browse to a folder containing an application bundle).
Spotlight uses Launch Services to find the concrete type for each file and folder it walks, and then maps that type to a registered Metadata Importer. If it encounters something there is no Importer for, that item is skipped.
In pre-Tiger versions of Mail.app, messages in a given mailbox were stored in single-file database format, contained in bundles using the file extension .mbox. The type of that bundle, “mbox”, was defined by Mail.app when Launch Services scanned it.
Tiger introduced a new mechanism within Launch Services for declaring concrete file types, in an effort to make things work better.
In Tiger and onward, the message storage format used by Mail.app was changed to use individual files, so that Spotlight could index them easily. The .mbox extension was given a new type (“com.apple.mail.box”) by Mail.app using the Tiger+ mechanism, with standard folder behavior instead of being an opaque bundle, and individual messages are stored as files in a Messages folder within it.
THE PROBLEM:
When I upgraded, the old /Applications folder was stored under /Previous Systems, and naturally I have been visiting there often while I collect all my previous applications, check for new versions, etc. Whenever I open it, Finder realizes it contains applications, and promptly registers them all with Launch Services. Including the old Mail.app, which maps the .mbox extension to type “mbox”.
This is where the unfortunate (and possibly “just plain stupid”) part of Launch Services comes in: the old way the Panther Mail.app mapped the extension .mbox is actually prioritized above the new way the Leopard Mail.app maps it. When Spotlight’s scanner gets a notice that a file changed, it walks each folder in the path to it, asking for the type of each. When it gets to a .mbox bundle, Launch Services tells it “mbox” instead of “com.apple.mail.mbox” like it should have. Spotlight has no Metadata Importer for “mbox”, and this type is not declared as behaving like a folder, so it stops without going inside it to find the actual messages.
This problem also shows up in Finder by having the .mbox things in ~/Library/Mail/Messages look like files with “house-front mail box” icons, instead of as folders with “office mail tray” icons.
Resetting the Launch Services database lets Spotlight properly import my Mail directory now, but whenever I browse to the old Applications folder, it will stop indexing any changes (like new email).
I’m still not done retrieving my old applications, but when I am, I plan to fix this problem by deleting all the old apps, reset the Launch Services database, and have Spotlight reindex the entire drive. Then it should have correct metadata for everything, and properly keep up when I get new Mail messages.
I’ve yet to find a way to fix Launch Services’ priorities, or have Finder stop telling it about those apps.
Hopefully something in this description will help you too. And hopefully I’ll get enough time to navigate my way around Apple’s bug reporting site to make sure they’re aware of the problem. I know people have filed bugs on these things individually, but I don’t know if anyone else has put it all together…
March 31st, 2008 at Mar 31, 08 | 12:15 pm
My Spotlght problems were similar to yours and the lsregister fix you mentioned (with a reboot) would get my system assigning the correct Leopard UTIs (“types”) to mboxes and mail messages. Attaching a drive containing an older copy of Mail would revert the fix, confirming your reader’s assertions.
Following the lsregister fix, I could then force Spotlight to index messages using the mdimport command, since the Launch Services db was in good shape. New mail, however, wasn’t indexed automatically until I added my account to the Staff group (something which the Leopard upgrade process should have done but didn’t – known bug), created a group named after my user account (ditto) and reset my home folder permissions using the Leopard install DVD. (The function to reset home folder perms and ACLs is confusingly accessed via the “Reset password” utility.) In Leopard, Spotlight is particularly careful about which users it shares its results with and this causes problems that persist after the LS database has been cleaned up.
I don’t know which of these steps was responsible for the post-lsregister fix but I have since repeated it on another machine with an identical problem and identical results. Sadly, it was on client time so I couldn’t investigate properly – he just needed a fixed machine.
For Apple to account for every variant of every possible upgrade to Leopard is tough and understandably, the upgrade procedure will never be perfect. However, I’m surprised that no one at Apple spotted that previous versions of Mail, with different UTIs listed in their Info.plist files, were polluting the Launch Services database. That seems like a fairly major goof. Combined with the other problems associated with permissions, it caused (and is causing) a lot of frustration. Perhaps there’s a bad date or version number in an older release of Mail.
There are some Leopard users who can search through their own messages when they run mdfind as root but who get no results via the GUI – proof that Spotlight is able to index messages correctly but is hiding their contents from the user when queried. :) The steps above should help that group.
Thanks for becoming a central point of knowledge on Spotlight. Very useful :)
March 31st, 2008 at Mar 31, 08 | 1:54 pm
Thanks for the additional information. :-)
April 22nd, 2008 at Apr 22, 08 | 11:38 am
Of all the spiel on the net about this problem, only your fixes worked for me (1. Fix LaunchServices 2. Re-do mdimport 3. Fix groups). Thanks for going to the trouble of putting this information in the public domain.
April 29th, 2008 at Apr 29, 08 | 7:21 am
Update: Okay, it worked for a day or so but now I’m back to not being able to search the ‘entire message’. I don’t need to tell you guys what a pain this is! Let’s hope 10.5.3 provides a fix, otherwise I’m going to be getting very familiar with mdutil and mdimport – or Entourage.
June 14th, 2008 at Jun 14, 08 | 2:54 am
Thanks very much for this set of posts, particularly this one with the detailed explanation.
The thing that was most important for me was one line of the explanation:
‘This problem also shows up in Finder by having the .mbox things in ~/Library/Mail/Messages look like files with “house-front mail box” icons, instead of as folders with “office mail tray” icons.’
In the Finder for me (in 10.5.2), there’s no ~/Library/Mail/Messages, but there is a ~/Library/Mail/Mailboxes, so I looked there. And all the .mbox files had *folder* icons.
I had run into this a couple days ago, while first trying to figure out what was up with the search-entire-message thing, and I couldn’t figure out why items that turned out to be bundles had folder icons. But I figured it wasn’t important.
Turns out it was a vital clue that the LS register was messed up.
I’m pretty sure I don’t have old versions of Mail.app anywhere around, so I don’t think that was the problem for me. But somehow the LS register had gotten confused about .mbox files. Possibly because I’m a longtime Eudora user? Dunno.
So I ran the lsregister -kill -r command that you mentioned in your other post. (And btw, macosxhints.com has a little more info about lsregister.) I got some worrying error messages, but it eventually completed. Then I did the mdimport to re-import ~/Library/Mail, just as you indicated in your other post, and voila! suddenly I could do Entire Message searches!
Also, I can do an Entire Message search on a single mailbox. And it seems to search both the subject line and the message body. Yay!
I haven’t yet determined whether it’s indexing new messages as they come in; I’ll figure that out soon. But if it isn’t, I can just set up a cron job to run mdimport once a day or so. That’ll be a minor nuisance, but it won’t make Mail.app unusable for me, as the non-searchable mail did.
So: thank you for saving Mail.app for me! This was the one biggest obstacle to my switching to it. There are still a few other issues for me (like the inability to set up an autoresponder that uses stationery), but I think I may be able to work around those.
Thanks again!
June 14th, 2008 at Jun 14, 08 | 4:11 am
Addendum: it does seem to be properly indexing new incoming messages as well. Cool.
June 28th, 2008 at Jun 28, 08 | 10:14 pm
Leopard Mail Entire Message Search – FINALLY FIXED!!!
After months of problems and no real fixes I finally found the solution here at Betalogue – Thank you!!! As with elysdir, it was the icon issue that eventually sent me in the right direction.
I have posted a simple version of your solution at Apple:
http://discussions.apple.com/thread.jspa?messageID=7462907#7462907
I know that you can easily fix this through Terminal Commands, and not all the following steps are necessary, but there are many people looking for a solution that only want to just push buttons…
== So here are my simple steps to fixing Apple Mail Search ==
1. Attach and turn on any external hard drives you have that may have a copy of Mail.app on them
2. Delete all OLD copies of Mail.app that you can find (don’t worry about Time Machine), but make sure you keep the latest version of Mail.app in your working Applications folder.
~ This is so that the LaunchServices database won’t easily become corrupt again
3. Install the latest version of Onyx – free to use and download from http://www.macupdate.com
~ I suggest Onyx, because some people don’t want to mess with the Terminal.
4. Quit all open applications and run Onyx (it will want to check your Hard Drive etc…that’s OK).
5. Go to the Maintenance pane and then the Rebuild tab and select only the LaunchServices box. Click Execute.
~ This should fix the primary problem with searches
6. When that process finishes, select only the Mail’s Envelope Index box. Click Execute.
~ This should fix related problems with other searches
7. When that process finishes, select only the Spotlight Index box. Click Execute. (This process of Rebuilding your Spotlight Index could take hours depending on how big your hard drive is).
~ This will give you a clean fast Spotlight database that Mail uses for Entire Message searches.
8. Quit Onyx
9. Open Mail.app and test your Entire Message Search!
====
Hope that helps someone – it certainly, finally, did for me!
Thanks again Betalogue!!