Mac OS X 10.4.3: Still choking on simple Spotlight searches
Posted by Pierre Igot in: MacintoshNovember 14th, 2005 • 11:49 am
As far as I can tell, the most important complaint regarding Spotlight in Mac OS X 10.4 (Tiger) is the utterly unresponsive behaviour of Spotlight when you start typing keywords to search for.
If this sounds like a pretty basic flaw, it’s because it is one.
The problem is that Apple decided that all Spotlight-related features should work based on a “live search” model, where the system start searching for matches for what you are typing as soon as you start typing—as opposed to waiting until you press the Return key or click on a button before starting the search.
I suppose that it’s a design choice that makes sense in theory. Why ask for the user to complete an extra step (press the Return key or click on a button) when you can start the search as soon as he starts typing?
In the real world, however, this is an utterly dumb decision. It only makes sense to cut this “extra step” if the extra step is only and always a waste of time. But if cutting the extra step causes all kinds of new problems that lead to significant time wasting, then it makes no sense whatsoever.
Yet that’s exactly what we have here. Because Spotlight starts searching as soon as you start typing, and because it searches for partial words as well as whole words, it frequently starts searching for things that you don’t want it to search for. If you start typing the keyword “article,” for example, it might interrupt you after you’ve typed “art” and start searching for documents that contain the partial word “art” throughout your hard drive.
And when it starts doing this, there’s nothing you can do to stop it! If your Spotlight search is in a Finder window, you immediately get the spinning pizza of death, and then you have no choice but to wait until the spinning pizza of death disappears. If you have already typed the rest of the “article” word, Mac OS X will only register the rest of your keystrokes (“icle”) once the spinning pizza of death is gone.
If your Spotlight search is in the Spotlight menu itself (in the top-right corner), things are apparently better, but it’s an illusion. You never get the spinning pizza of death when you type your keywords up there, but Mac OS X stills starts searching as soon as you’ve typed a few letters, regardless of whether the partial word search that it starts is actually something that you intended. And there is still a delay before it registers your additional keystrokes and actually starts searching for the keyword you want. You can see that by keeping an eye on the list of search results that it builds, because this list frequently contains references to items that match a portion of the word that you’ve just typed instead of the whole word.
Spotlight supposedly updates this list of search results in a “live” fashion, but even on a fast machine, this live updating is far from being immediate, and each time this happens it is yet more precious seconds that you are wasting waiting for Mac OS X to actually register what you want it to register and start searching for what you want it to search for.
The simple truth is that today’s computers contain hundreds of thousands of files, most of which are indexed by Spotlight, and searches for small, partial keywords will always yield hundreds of results that are utterly irrelevant and will cause your computer to slow down and force you to wait before you regain control of your machine.
And these waits occur each and every time you try to use Spotlight. In other words, they have a cumulative effect and amount of lots of wasted time in the daily use of Mac OS X.
Now, I just have one question for Apple: Is all this wasted time really worth the supposed advantages of a “live search” feature? Think about it: The only real advantage of a live search feature is that it saves you the “extra step” of having to hit Return or click on a button.
How many Mac users do you know who would rather waste time again and again waiting for Spotlight to finish searching for irrelevant matches and catch up with them instead of just taking the extra step of hitting Return or clicking on a button?
Apple cannot pretend that this design decision is in the name of consistency. There are numerous other search tools, including the many on-line search tools that people use, who do require the user to hit Return or click on a button. And even Apple’s own applications are not consistent. The search field in the Finder, Mail, iTunes, and System Preferences is a live search field that doesn’t require the user to hit Return or click on a button, but the search field in the Help Viewer application, for example, is not a live search field and Help Viewer does not start searching until the user hits Return after typing his keyword(s) (and of course there is no visual indication that Help Viewer actually requires this).
This problem with Spotlight choking on partial word searches has plagued every version of Tiger, including pre-release builds. I complained about it during the early stages of testing AppleSeed builds of Tiger, and nothing was ever done about it. Clearly someone at Apple made the decision to make Spotlight a “live search” feature, and no one was going to be able to change their minds—especially not a lowly volunteer AppleSeed tester.
I was hoping that Apple would come to its senses after the release of 10.4.0, based on worldwide user feedback on the “finished” product, which is why I waited for improvements in 10.4.1, 10.4.2, and finally 10.4.3, which was released last week. While some people allege that things have improved somewhat in 10.4.3, I just can’t see it myself. In my experience, Spotlight searches in the Finder still cause the spinning pizza of death all the time with completely irrelevant searches for partial keywords, and the system-wide Spotlight menu still lists irrelevant matches for several seconds before finally catching up with my typing and listing relevant results only.
I frequently find myself trying to type as fast as possible in order to avoid Spotlight starting to search for irrelevant partial words. How ridiculous is this? It should be the computer adapting to my typing speed, and not the other way around! Anyway, I am a fairly fast typist, but even I cannot prevent Spotlight from interrupting me with irrelevant behaviours while I am typing my keywords.
It also does not help that people with a bigger voice in the Mac world can’t tell it like it is. Take Rob Griffiths. He’s the mind behind the highly successful Mac OS X Hints web site, has written several books about the Mac and articles for Mac publications, and now has a Mac OS X Hints weblog on the Macworld web site.
What does he have to say about this rather obvious problem with Spotlight?
While you can’t totally avoid the extraneous results and the spinning ball, here’s one way to greatly minimize them. Instead of typing your search term in the Finder’s search box, first type it into any open application. For example, you could type in Safari’s address bar or Google search box, a TextEdit document, a Sticky note, or a widget. Now select the word with the mouse, and then hit Command-C to copy it to the clipboard. Switch back to the Finder, click in the Spotlight search box (or just hit Command-F), and then hit Command-V to paste your search term. You don’t even have to hit Return, since the Finder will still try to do a live search.
Can’t he see how ridiculous this is? Mac OS X is supposed to be an advanced, user-friendly operating system, yet for a simple thing such as a keyword search, we have to use a ridiculous workaround involving a third-party application and cutting and pasting?
Where is Rob Griffiths’ indignation? Where are the headlines on major Mac web sites about this major, crucial flaw in Spotlight’s implementation?
I really do think that Spotlight has great potential as a searching infrastructure in Mac OS X. But its currently implementation is so flawed that it really makes you wonder what goes through the minds of Apple’s engineers and decision-makers. Do they even use the technology themselves in real world situations?
This is the kind of problem that is typical of a user-hostile software developer such as Microsoft. But the fact that it can occur at Apple as well, as this particular issue with Spotlight demonstrates, does not bode well for the future of computing in general. If even Apple cannot avoid such user-hostile design decisions, then who can?
Maybe in 10 or 20 years, we’ll have computers that are so fast that live searches really are possible without any kind of performance hit and user interface hiccups. But right now, even with the fastest processors and the fastest hard drives, Spotlight is still a constant source of user frustration. I realize that finding a balance between making design decisions for the future and maintaining optimum usability today can be difficult at times. But for this particular problem, the balance is so obviously wrong that it’s a daily insult to Mac users worldwide.
November 14th, 2005 at Nov 14, 05 | 12:58 pm
I guess OS X could be a big deal better if Apple only fixed the problems you make the effort to describe :)
I slightly disagree with some of your points, though. I think the live search idea is great and should be done. But it’s not a trivial thing to do and Apple didn’t implement it well. I’d rather like them to do it properly :)
I also doubt your statement that the majority of files on the computer is indexed. Of the ~800000 files on my system, I believe at most 100000 to be indexed. All the other files can’t even be found by file name in OS X.4. This means that unless you’re prepared to use the command line and deal with the Unixy problems like case-sensitivity exhibited by the locate tool, you can’t find the vast majority of files on the system. As my personal files are quite well organised, I might add that I find it easy to locate the files in Spotlight’s index manually while I could often do with some help when it comes to the others.
(To add another really annoying problem with OS X, let me mention how hard it is to determine the number of files in a folder and its subfolders – information which, IIRC, used to be in the Finder’s info windows in the Classic Mac OS.)
November 14th, 2005 at Nov 14, 05 | 3:12 pm
Could you elaborate on the reasons why you feel that the live search idea is great? Are there any significant benefits other than removing one step (pressing Return or clicking on a button)?
As for the number of files that are indexed, I guess I should have made it clearer that I was referring to files that are not system-files. Obviously Mac OS X itself has lots of files that are not indexed by Spotlight (and should not be). But even beyond that, people have tons of files these days. Mail creates a file for each and every message. People have thousands of songs, and thousands of digital pictures. Maybe “hundreds of thousands,” but still many thousands at the very least.
(I was just thinking the same thing about determining the number of files. I wanted to check the total number of files on my “Documents” partition. No obvious way to do so.)
November 14th, 2005 at Nov 14, 05 | 7:13 pm
I’m lazy and like not having to type return (too much LaunchBar usage, I guess)
And I don’t think that it’s the live search idea is bad. It’s just the implementation that seems to be hard. Too hard for the people at Apple, judging from Spotlight’s performance.
I really don’t understand how can accept that fact that Spotlight ignores most of the files on your drive. It can’t even find them by name! So here you are with what pretends to be hottest finding technology around but it will find less of your files than any other computer system. It doesn’t even reverse back to the good old trusted method of just looking through all the folders. I do consider that a problem.
Also keep in mind that a part of the many files problem is Apple’s choice to implement Spotlight on a Unixy per-file basis for the time being. This is probably quite robust, but once you peek at what iCal, AddressBook or our very own UnicodeChecker do, it’s pretty absurd.
I tried to do an Automator thingy to do a file count but it just ended up being slow and returning 0 as a result as soon as non-trivial amounts of files were used. To count the number of files on a whole volume, you could just open disk utility and look it up there.
November 14th, 2005 at Nov 14, 05 | 9:01 pm
The live search idea is great, but only with good implementation. I am also so accustomed to Launch Bar’s powerful responsiveness that I tend to be prejudiced in favour of a (well-implemented) live search. As long as it works, it is lovely.
I agree with you strongly in general though, Pierre, and you are certainly not alone in your exasperation. My imagination is also strained in trying to understand how such basic limitations can be tolerated by Apple engineers… to say nothing of Apple’s celebrity. Hardly a day in my life goes by when I don’t think, “Doesn’t Steve have this problem, too? Why doesn’t he get it fixed?” :-)
November 15th, 2005 at Nov 15, 05 | 9:15 am
I’ve started using a little app called EasyFind that I downloaded in desperation a few days ago after a Finder search refused to list some fonts that I could see right there in a Finder window and that had been on my computer for years.
I haven’t put it through the paces yet but so far even if it beach balls on me it at least has the common decency to leave the rest of the computer responsive!
November 15th, 2005 at Nov 15, 05 | 10:55 am
ssp: I use LaunchBar too, but obviously its scope as a search tool is significantly more limited than Spotlight. (It doesn’t index file contents, only file names and metadata.) So it’s probably easier to achieve a live search behaviour that works with LB than with Spotlight.
I agree that there should be some tool for finding system files by name. The problem here is that, instead of adding Spotlight to Mac OS X’s existing search features, Apple replaced the existing features with Spotlight. Which means that you can no longer search for a file by file name as easily as you could in 10.3. And that you can no longer search for system files that are excluded from Spotlight indexing.
I guess Apple figures that if you are a geek that wants to look for system files, then you know how to use Unix tools that can search through all files on your system. I don’t agree with this. (A Mac geek still wants a proper UI for his tools.) But that appears to be Apple’s reasoning.
I don’t see the per-file basis as particularly problematic, as long as it stays behind the scenes for the end user. Ultimately, the user doesn’t care which approach is used as long as the tool works.
And according to Disk Utility, my “Documents” partition has over 150,000 files. And that partition does not even contain either my music or my digital pictures :-).
Paul: I suspect Steve Jobs doesn’t have 150,000 user-created files in his “Documents” folder. Maybe Spotlight works acceptably with a much smaller number of files? Still, that’s no excuse…
Brian: Yes, because of the Spotlight integration in the Finder, a locked-up Spotlight search locks up the entire Finder. But you can still switch to other applications while the Finder is locked up.
November 15th, 2005 at Nov 15, 05 | 3:11 pm
It turns out that you can actually find files using the UI!
It’s just so well hidden that I neither managed to do it so far nor saw any hints on how to do it.
I’m trying to write this up and make some screenshots to put it on my site as it could be quite useful.
November 16th, 2005 at Nov 16, 05 | 11:37 am
Totally agree spotlight is irritating on MANY levels. You didn’t even get into the horrible results window. ;)
I was also dismayed when I found out Spotlight didn’t index ALL files and that although you can disallow locations via System Preferences->Spotlight->Privacy, there is no way to ADD other folders on your drive to search.
I never want to search by content anyway. I always just hit command-f and do a search by name in a window. And every time I have to select “Name” from one of the pop up menus that default to Kind/Last Opened. Sigh.
Since Spotlight is THE headline feature of Tiger (along with Dashboard, pfft), this is quite disappointing.
November 16th, 2005 at Nov 16, 05 | 11:58 am
soosy: If you want a shortcut to do file name searches, use a smart folder with predefined settings. It’s not perfect, but it’s better than having to select “Name” each and every time:
Mac OS X 10.4 (Tiger) Tip: Create a smart folder for file name searches
And I am keeping the results window problems for another post—or rather bunch of posts :).
November 18th, 2005 at Nov 18, 05 | 2:11 am
Okay, I’ve re-thought my position on this. I’ve decided that I think the live search is gratuitous in the context of Spotlight. Sure, the live search works great in LaunchBar, but the situation is quite different there… and it’s just too easy to say that something is a good idea “if it’s well-implemented.” Duh. Kind of like saying a money tree is a nice idea “as long as it works.” But live search has no realistic chance of being well-implemented for this purpose, and meanwhile it seems to be a significant hazard to the usability of Spotlight.
November 18th, 2005 at Nov 18, 05 | 11:00 am
I think that live search is not realistic for Spotlight as long as we use hard drives for storage. Hard drives are simply too slow to provide a responsive architecture for a tool such as Spotlight in live search mode.
LaunchBar is very different because, as indicated, it only works with limited meta data (file names and a few other bits of information, but not the actual file contents) and also has its own abbreviation algorithm. It works well, but sometimes when I have lots of hard drive activity it too can become unresponsive and sluggish. Fortunately, it only happens occasionally, but it further prooves that LaunchBar in itself already stretches things near the limit of what’s doable in terms of live search on hard drives.
Maybe one day our computers will use holographic cubes for data storage and data retrieval will be much faster than with hard drives. But until then, live search in Spotlight is not realistic.
November 29th, 2005 at Nov 29, 05 | 10:59 am
I don’t think hard drives are the limiting factor here. The complete Spotlight index on my disk is around 300MB. I suspect that’s more than the average user has.
But even for that amount of data it’s not completely unrealistic to cache it in RAM completely these days. So I’d say it’s more a problem with the UI or the algorithm.
November 29th, 2005 at Nov 29, 05 | 11:22 am
When the global Spotlight search struggles to keep up with me, I can definitely hear lots of disk activity. On the other hand, in the Finder, the spinning beach ball of death appears to indicate a complete software lock-up, not just a delay due to excessive hard drive activity. But it’s hard to tell… In any case, whatever the actual cause, it’s inexcusable.
November 29th, 2005 at Nov 29, 05 | 5:24 pm
I don’t say that Spotlight isn’t accessing the hard drive. I’m just saying that hard drive speed issues can easily be worked around by investing in a bit of RAM.
November 29th, 2005 at Nov 29, 05 | 6:46 pm
I think you can reduce HD activity by just running cat on all files in the Spotlight index folder. If you have enough free memory before that, you should have all the data of your Spotlight indices in RAM afterwards. I did see some performance improvements after that (around 25% on queries with a few hundred results and around 50% on queries with many thousand results).