But even for that amount of data it’s not completely unrealistic to cache it in RAM completely these days. So I’d say it’s more a problem with the UI or the algorithm.

LaunchBar is very different because, as indicated, it only works with limited meta data (file names and a few other bits of information, but not the actual file contents) and also has its own abbreviation algorithm. It works well, but sometimes when I have lots of hard drive activity it too can become unresponsive and sluggish. Fortunately, it only happens occasionally, but it further prooves that LaunchBar in itself already stretches things near the limit of what’s doable in terms of live search on hard drives.

Maybe one day our computers will use holographic cubes for data storage and data retrieval will be much faster than with hard drives. But until then, live search in Spotlight is not realistic.

Mac OS X 10.4 (Tiger) Tip: Create a smart folder for file name searches

And I am keeping the results window problems for another post—or rather bunch of posts :).

I was also dismayed when I found out Spotlight didn’t index ALL files and that although you can disallow locations via System Preferences->Spotlight->Privacy, there is no way to ADD other folders on your drive to search.

I never want to search by content anyway. I always just hit command-f and do a search by name in a window. And every time I have to select “Name” from one of the pop up menus that default to Kind/Last Opened. Sigh.

Since Spotlight is THE headline feature of Tiger (along with Dashboard, pfft), this is quite disappointing.

It’s just so well hidden that I neither managed to do it so far nor saw any hints on how to do it.

I’m trying to write this up and make some screenshots to put it on my site as it could be quite useful.

I agree that there should be some tool for finding system files by name. The problem here is that, instead of adding Spotlight to Mac OS X’s existing search features, Apple replaced the existing features with Spotlight. Which means that you can no longer search for a file by file name as easily as you could in 10.3. And that you can no longer search for system files that are excluded from Spotlight indexing.

I guess Apple figures that if you are a geek that wants to look for system files, then you know how to use Unix tools that can search through all files on your system. I don’t agree with this. (A Mac geek still wants a proper UI for his tools.) But that appears to be Apple’s reasoning.

I don’t see the per-file basis as particularly problematic, as long as it stays behind the scenes for the end user. Ultimately, the user doesn’t care which approach is used as long as the tool works.

And according to Disk Utility, my “Documents” partition has over 150,000 files. And that partition does not even contain either my music or my digital pictures :-).

Paul: I suspect Steve Jobs doesn’t have 150,000 user-created files in his “Documents” folder. Maybe Spotlight works acceptably with a much smaller number of files? Still, that’s no excuse…

Brian: Yes, because of the Spotlight integration in the Finder, a locked-up Spotlight search locks up the entire Finder. But you can still switch to other applications while the Finder is locked up.

I haven’t put it through the paces yet but so far even if it beach balls on me it at least has the common decency to leave the rest of the computer responsive!

And I don’t think that it’s the live search idea is bad. It’s just the implementation that seems to be hard. Too hard for the people at Apple, judging from Spotlight’s performance.

I really don’t understand how can accept that fact that Spotlight ignores most of the files on your drive. It can’t even find them by name! So here you are with what pretends to be hottest finding technology around but it will find less of your files than any other computer system. It doesn’t even reverse back to the good old trusted method of just looking through all the folders. I do consider that a problem.

Also keep in mind that a part of the many files problem is Apple’s choice to implement Spotlight on a Unixy per-file basis for the time being. This is probably quite robust, but once you peek at what iCal, AddressBook or our very own UnicodeChecker do, it’s pretty absurd.

I tried to do an Automator thingy to do a file count but it just ended up being slow and returning 0 as a result as soon as non-trivial amounts of files were used. To count the number of files on a whole volume, you could just open disk utility and look it up there.

As for the number of files that are indexed, I guess I should have made it clearer that I was referring to files that are not system-files. Obviously Mac OS X itself has lots of files that are not indexed by Spotlight (and should not be). But even beyond that, people have tons of files these days. Mail creates a file for each and every message. People have thousands of songs, and thousands of digital pictures. Maybe “hundreds of thousands,” but still many thousands at the very least.

(I was just thinking the same thing about determining the number of files. I wanted to check the total number of files on my “Documents” partition. No obvious way to do so.)

I slightly disagree with some of your points, though. I think the live search idea is great and should be done. But it’s not a trivial thing to do and Apple didn’t implement it well. I’d rather like them to do it properly :)

I also doubt your statement that the majority of files on the computer is indexed. Of the ~800000 files on my system, I believe at most 100000 to be indexed. All the other files can’t even be found by file name in OS X.4. This means that unless you’re prepared to use the command line and deal with the Unixy problems like case-sensitivity exhibited by the locate tool, you can’t find the vast majority of files on the system. As my personal files are quite well organised, I might add that I find it easy to locate the files in Spotlight’s index manually while I could often do with some help when it comes to the others.

(To add another really annoying problem with OS X, let me mention how hard it is to determine the number of files in a folder and its subfolders – information which, IIRC, used to be in the Finder’s info windows in the Classic Mac OS.)