Mac OS X 10.5 Tip: Repairing broken Spotlight searches in Mail

Posted by Pierre Igot in: Mail
February 25th, 2008 • 3:29 pm

Based on the number of threads on Apple Discussions, this seems to be a recurring problem for Mac OS X users, and Apple’s own tech support people seem to be pretty clueless about what can be done to fix it.

It has just happened to me, and I was able to partially fix it, so here are my notes on the topic.

The symptoms are as follows: You try a Spotlight search within Mail, using the search field in the top-right corner of the toolbar.

Typically, Mail returns the expected results when the focus of the search is on the “From,” “To,” or “Subject” fields, but if you put the focus on “Entire Message,” Mail returns 0 results—even though you know very well that there should be plenty of results for your search.

The usual fixes for Spotlight problems don’t seem to work. Even if you add your “Mail” folder to the list of folders to exclude from Spotlight searches in System Preferences, and then remove it from the list, in order to force Spotlight to reindex its contents, the problem persists. This operation causes a brief spike in activity for the mdworker background process (which can be monitored using the Activity Monitor utility), but as far as I can tell this brief period of activity does not change anything to the problem.

There is also no point in trashing the “Envelope Index” file. As far as I can tell, it has nothing to do with Spotlight, which works at a granular level on individual message files and builds its own indexes in invisible files.

Eventually, I found the following solution, buried deep in an Apple Discussion thread on the topic. You need to enter the following two commands in Terminal:

/System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister -kill -r -domain local -domain system -domain user
mdimport ~/Library/Mail

The first command is executed in a few seconds. Don’t ask me exactly what it does. The “lsregister” seems to indicate that it has something to do with Launch Services, but that’s all I know. I just found the tip on-line and I am reproducing it here.

The second command actually initiates a complete Spotlight re-indexing of your “Mail” folder. Depending on how big your folder is, it can take a while. Mine is over 770 MB, with over 80,000 files, and it took about 10 minutes to index the whole thing.

The mdimport command also asked for my authorization to access my keychain at some point, which I did allow. And it produced a few lines of error messages:

2008-02-25 14:55:18.947 mdimport[98309:5b07] *** Assertion failure in +[PSClient clientForBundleIdentifier:], /SourceCache/PubSub/PubSub-59.3/Source/Bridges/Obj-C/
2008-02-25 14:59:39.651 mdimport[98309:10b] NSDataMessageStore: couldn't find body
2008-02-25 14:59:39.651 mdimport[98309:10b] NSDataMessageStore: couldn't find body
2008-02-25 14:59:39.652 mdimport[98309:10b] NSDataMessageStore: couldn't find body
2008-02-25 14:59:39.652 mdimport[98309:10b] NSDataMessageStore: couldn't find body
2008-02-25 14:59:39.653 mdimport[98309:10b] NSDataMessageStore: couldn't find body
2008-02-25 14:59:39.653 mdimport[98309:10b] NSDataMessageStore: couldn't find body
2008-02-25 15:00:41.449 mdimport[98309:10b] *** Failed to decode 7-bit data, treating as binary

But it appears to have worked, at least to the extent that Spotlight searches on “Entire Message” now return results again—when searches applies to “All Mailboxes,” that is.

However, I still get zero results when I try to apply a search on “Entire Message” to a specific mailbox rather than to “All Mailboxes.” So all is still not well. Spotlight searches in Mail are still partially broken. (System-wide Spotlight searches work, because they always apply to all mailboxes.)

It really is quite frustrating that such an essential feature (i.e. the ability to do searches in Mail) appears to be fundamentally broken in Mac OS X 10.5. But it has been a recurring problem in Mac OS X. I distinctly remember experiencing similar problems while testing early versions of Mac OS X 10.4. Eventually, I think Apple fixed the problem in Mac OS X 10.4, but now it looks like we have to go through the same thing all over again in Mac OS X 10.5.

12 Responses to “Mac OS X 10.5 Tip: Repairing broken Spotlight searches in Mail”

  1. apple4ever says:

    I just had this happen to me. Here is the solution.

    1. Enter the following into Terminal:

    sudo rm -R .Spotlight-V100/

    2. Go into Activity Monitor. Find the process “mds”. Kill it.

    3. Enter the follow into Terminal:

    sudo mdutil -E /

    That should fix it (after Spotlight reindexes the drive).

  2. Pierre Igot says:

    I am afraid it doesn’t help. Search for XXX in Entire Message still fails when the focus is on a specific mailbox rather than on “All Mailboxes.”

    And there is no such thing as “mds” in Leopard, as far as I know. I quit “mdworker” just in case. mdutil spawned a process called “Spotlight”, which proceeded to reindex everything (took about half an hour). But the Mail searches still don’t work right.

  3. ssp says:

    Interesting – a Spotlight problem I haven’t had yet. I’ll just toss in a few remarks.

    * lsregister can be an extremely useful tool for dealing with the Launch Services database. I wouldn’t use the kill option unless absolutely necessary though as that may also delete assignments of file types to applications which you may have made in the Finder. With a bit of luck something like lsregister -f /Applications/ might suffice for your purposes.

    * If you see the problem more frequently it might be interesting to run lsregister -dump before rebuilding the Launch Services database. It would also be interesting to see mdls output for elmx files in /Library/Mail and to see whether searching for terms in those messages with mdfind in the Terminal yields any results. Then try running mdimport -d2 on one of those files and look at the output. Try whether using the -r option of mdimport for the Mail importer solves the problem. Try whether restarting solves the problem. Plenty of things to try out, I guess.

    * You absolutely have an mds process on your system. If you don’t Spotlight won’t work, which would explain the problem as well.

    * I think that Mail’s subject-only searching uses a database separate from Spotlight’s. That can explain why those limited searches keep working.

  4. Pierre Igot says:

    Yes, lots of things to try, and it’s quite painful :-/.

    You are correct, I do have an mds process running. I was only looking at my user processes. mds is owned by root.

    And yes, the limited searches use a different index, possibly the Envelope Index. But a search on the entire message uses the Spotlight database, and I don’t understand why it works for All Mailboxes but not when the focus is on a specific mailbox.

  5. Pierre Igot says:

    ssp: What am I supposed to see when running lsregister -dump? I get a long list of various things, including information about the Mail application and about mail-related stuff. But I don’t really know what I am looking for.

    mdfind returns the expected results, since it’s equivalent to a system-wide Spotlight search, which works just fine.

    The output of the mdimport -d2 command on the message files is markedly different from the regular mdls output, but i don’t know if that means anything. In any case, the search still fails after all this.

    It looks to me like Spotlight is quite broken for Mail, really.

  6. ssp says:

    If rebuilding the launch services database helped (which I sort out doubt) I would expect some of the data in there after the rebuild being different from the data before the rebuild. The dumped database for example lets you find out which application claims which UTIs and which spotlight importers can import them – which might be relevant for your situation.

    If mdfind does its job properly it suggests that your Mail is properly indexed (so probably no launch services related problems which keep the indexing from happening) and just the Mail application may be bad at running the Spotlight query. Personally I haven’t seen that one yet. Does it persist across restarts? Are there any log messages when running the queries?

    It’s hard to tell what you mean by ‘markedly different’. If things are all right, the output will look quite different but the data inside it will be essentially the same (with mdls not containing the content and mdimport listing things like localised file type names). Things tend to be bad when big chunks of information are missing. But it sounds like that isn’t the case for you.

    Luckily things are still working fine for me. But now I’ll expect them to break any moment…

  7. Pierre Igot says:

    Well, I discovered that my “Spotlight” folder in my main library still contained an importer for MailTags (even though I uninstalled that software long ago), so I trashed that and decided to force Spotlight to reindex from scratch. For good measure, I also trashed the Launch Services cache altogether first. I did all this with Cocktail.

    After rebuilding the entire Spotlight index, I went back to Mail and, sure enough, Spotlight search was completely broken again, with searches on “Entire Message” not even working for “All Mailboxes” any longer. So I used the tip above to force Spotlight to reimport Mail’s folder, and now searches on “Entire Message” are working for “All Mailboxes”, but still are not working when the focus is on a specific mailbox.

    So this leads me to think that the MailTags importer had nothing to do with it, and that it is indeed a problem with Spotlight itself. Unfortunately, problems do persist across restarts. And I don’t see any relevant messages in the system log.

    By “markedly different,” I did indeed mean the content and localized stuff. Apart from that, there doesn’t appear to be anything significant missing. So I don’t think the problem is there. The problem is with Mail’s ability to tap into the Spotlight index for its searches.

  8. ssp says:

    Crap! Good luck solving that problem then. Seems exactly the kind that Apple love to leave in their software for ages.

  9. Paul Ingraham says:

    I’ve encountered this bug many times, more times than I care to think about, more often than I change my socks.

    I trust Spotlight searches in Mail so little that I virtually assume that it’s going to go badly, and quickly and reflexively resort to other means of finding what I’m looking for as soon as it’s becoming clear that Spotlight isn’t helping. But then again my has always be, er, “special,” what with completely non-functional junk mail filtering, and the way it displays about half of all my mail as being “to” a woman from my Address Book who is not at all me. And so on.

    Anyway, I have had some luck with semi-regularly re-indexing from scratch using the handy little Spotless utility. Usually that helps. A bit. For a while. relaunches also often work, and it’s a lot faster than re-indexing. An interesting workaround is to do a spotlight search of your messages from the Finder: this often succeeds where Mail fails.

    I also swear a lot at it, but I’m not sure if that helps or just makes me feel better.

  10. jharkin says:

    I’ve been trying to fix this problem since I first installed 10.5. I’ve looked at various solutions for months to no avail. Your suggestion with lsregister and mdimport worked! Thank you!!!

  11. Pierre Igot says:

    You’re welcome :).

    Unfortunately, as far as I can tell, it only restores the Spotlight functionality partially. Searches on specific mailboxes still don’t work, and Spotlight fails to index messages added after the mdimport. So you have to redo it regularly.

  12. pothe says:

    Hi had the same issue about 2 months ago.
    This fixed it for me:

    see Sidha33 – near the bottom.

    It worked a treat, all was well until today! Over the weekend I had a sudden increase in GB. Somehow I lost around 35 GB on my 160 GB macbook pro. I used Whatsize to show that spotlight v100 was 35 GB – so I then used MainMenu to rebuild the spotlight index. Great news! I got my 35 GB back…But

    This morning I now have the same issue as before – spotlight has stop working.

    Despite repeating the above, and all the other methods I have come across, mds simply won’t rebuild. It stays active for a few minutes and then stops.

    I hope the above helps you guys – it did for me, at least for a while…back to the drawing board

Leave a Reply

Comments are closed.