Mac OS X 10.5’s Finder: Still unable to cope elegantly with large number of files

Posted by Pierre Igot in: Macintosh
April 7th, 2008 • 2:04 pm

This morning, I had a folder containing approximately 27,000 MP3 files that I wanted to move from one volume to another, due to space constraints.

Because this folder has been holding a large number of files for a long time, I am quite familiar with the Finder’s lack of responsiveness when I try to view that folder’s contents in a Finder window in Mac OS X 10.5 (Leopard).

Typically, if I find an MP3 file that is located inside that folder via a Spotlight search and I then use the “Open Enclosing Folder” command to view that particular file within the context of its enclosing folder in a Finder window, I usually get a new Finder window that stays empty for 10-20 seconds, with a status bar that says “0 items” at the bottom, and then finally the Finder populates the window with the thousands of files, updates the status bar to indicate the actual number of items in the folder, and actually manages to highlight the specific file I wanted to reveal within its context and scroll down the list to where that file appears.

It is much better than what would happen in previous versions of Mac OS X, where the Finder would simply lock up completely, with the Spinning Beach Ball of Death, and then eventually display the contents of the enclosing folder, albeit often without highlighting the appropriate file or scroll down to where that file was located.

But it is still pretty poor. I realize that there are hardware constraints that make it difficult for Mac OS X, even on a fast machine, to be perfectly responsive when dealing with large numbers of files stored on the hard drive. But it is still wrong to display a misleading status message in the status bar (the folder does not contain “0 items”!). If Apple knows that the Finder is not going to be able to display the list of files promptly, then there should be no status message at all.

In addition, I still find it hard to believe that Apple’s engineers are not able to do better when dealing with large numbers of files. We live in a world in which most people have hundreds of thousands of files on their hard drive. We have machines with tons of raw processing power, and hard drives are getting faster all the time.

Complaints about the Finder’s poor performance with large numbers of files are nothing new, and while Apple has made some progress with the Finder in Mac OS X 10.5, they still don’t seem to consider it a priority to make the Finder able to handle large numbers of files more elegantly.

To get back to my example above, I first tried to move the folder containing 27,000 MP3 files from one volume to another by simply dragging the folder icon. The Finder did start the copying process, but then after copying approximately 10 files, it displayed an error message saying something like “Unable to copy item ‘XXX’ because an item called ‘.DS_Store’ already exists.

I am not kidding. And the only option in the alert with the error message was to dismiss it, which would cause the Finder to cancel the copying operation altogether. I tried a second time, just in case it was just a glitch, but I got the same error, after 10 files or so, and of course the item whose name was quoted in the error message was not called “.DS_Store” and, as far as I know, the only entity responsible for these (invisible) “.DS_Store” files is the Finder itself, so it was a bit rich for it to blame me for the existence of a file with the same name in the destination!

In any case, it clearly meant that this approach for copying my 27,000 files wouldn’t work.

So I tried the next available option, which was to open the folder, select all the 27,000 files with the “Select All” command (which worked instantly), and then attempt to drag the selection to a new folder on the destination volume.

I said “attempt to,” because of course the Finder is not exactly responsive when trying to drag a selection consisting of 27,000 different items. But with patience (i.e. by clicking and holding the mouse button down and waiting a while, and then trying to control the choppy dragging movement to the destination), I was actually able to do it. (Mind you, the Finder already gets choppy when the selection includes about 100 items, so by those standards I thought that the behaviour with a selection of 27,000 items wasn’t actually much worse!)

The Finder then displayed the candy bar animation that is effectively a progress indicator without any indication of progress, and a message saying that it was preparing for the copying operation. Why this preparation cannot be made part of the regular progress indicator for the copying operation is beyond me. (Does the user really need to know that the Finder has to go through two different phases when copying files? Can’t it all be part of a single copying process, with a proper progress bar from the very beginning?)

But what really gets me is when this “preparation” phase actually takes an order of magnitude longer than the main process itself. It was not the case for the copying here, but when the copying process was complete, I went back to my original folder and attempted to move all the original 27,000 files to the trash (since I now had safely copied them to another volume, and needed the space).

This time I used the keyboard shortcut (command-Delete) rather than a mouse movement, and the Finder displayed its little candy bar animation right away. But then the preparation phase for the process of moving 27,000 files to the trash actually took several minutes, during which all I had was the useless candy bar animation. Since the original folder was still open in a Finder window, I could actually see the Finder updating the status line at the bottom of the window, with the number of items listed gradually decreasing.

Yet all this, apparently, was still part of the “preparation” for the actual process of moving the files to the trash. Then after several minutes the status line at the bottom of the original Finder window came down to “0 items,” the candy bar animation changed to an actual progress bar, the message above the progress bar changed to something about actually moving the files and—boom! In two seconds it was over.

In other words, the process of moving 27,000 items to the trash included a “preparation” phase that lasted several minutes and an actual moving process that lasted about 1 second. This is really ridiculous. It makes the display of a progress bar for the second phase only utterly useless, since most of the waiting is caused by the first phase, for which there is no progress indicator.

And the fact that I could see the files gradually disappearing from the original Finder window during the “preparation” phase also seems to indicate that this “preparation” phase actually is an integral part of the process of actually moving the files.

So again, why does the Finder even make a distinction between a preparation phase and an action phase in its interaction with the end user? Is Apple really unable to predict how long this preparation phase is going to take? Why can’t they display a single progress bar animation for the entire process, and give the user a real sense of who long he’s going to have to wait?

I am afraid this experience confirms that the Finder still has a long way to go before it can really be considered a reliable, quality software product that can handle simple operations involving only a handful of items and larger operations involving thousands of files with equal ease and user-friendliness.


13 Responses to “Mac OS X 10.5’s Finder: Still unable to cope elegantly with large number of files”

  1. sol says:

    I rarely move such big amounts of files, but I understand your frustration. Why can’t Apple predict how long such a task may take ? Where this lack of proper estimation really shines on my MacBook Pro (and to a lesser extent, my iBook) is the estimation of time it takes to recharge the battery. Sometimes, it takes more than double the time indicated to recharge. I changed the menu bar display to indicate the percentage, and it is more useful. It is sad that a big company like Apple cannot predict the time to complete such a simple task with more accuracy.

    And Pierre, maybe Apple compares the copying of thousands of files to a fireworks display : it takes more time to prepare the explosives than it does to explode in the sky !

  2. Pierre Igot says:

    Yeah, but at least fireworks are spectacular. There is definitely nothing spectacular about the copying process itself! It clearly is not “worth the wait.”

  3. danridley says:

    Perre: Clearly the indeterminate progress bar for the “preparation” phase and the actual progress meter for the file operation phase are a general rule, and the fact that they’re out of balance for one particular operation doesn’t make them ridiculous in their conception, just insufficient for this particular case and ones like it.

    Most of the time, when dealing with smaller numbers of files and often transferring them from disk to disk, the preparation phase is momentary and the actual file operation takes a while. Even when dealing with thousands of files, like your original copy operation, the preparation undoubtedly took a couple of minutes, but most of the operation time took place during the copy, which had a progress meter.

    The indeterminate progress meter is far more useful than no indication at all (or a beachball), and I consider it progress (while still hoping 10.6 will continue to advance this area).

    (Windows Explorer, Nautilus and Konqueror, and basically every GUI file manager I’ve ever used choke on thousand-plus file operations. Indeed, Finder is above average in this regard, though that’s damning it with faint praise.

    The good news is that while DOS also choked on thousands of files, Unix command line interfaces handle them with aplomb, including OS X.)

    sol: estimating battery charge time (and battery drain time, for that matter) isn’t actually very simple at all. Equipment makers have to build in all types of estimations about how the batteries are accepting and will accept charge, and they can vary based on temperature, age and usage profile of the battery, and many other factors. Sometimes the battery simply isn’t going to accept a charge as fast as the equipment expects, or as fast as similar batteries do, if they’ve had a different usage history.

  4. Pierre Igot says:

    Dan: My observation in this particular case was that “most of the operation” took place during the preparation. The copy itself (with the progress meter) only lasted a fraction of a second!

    I suspect that the only reason that Apple feels the need to show the user two distinct phases (preparation and action) is because they are unable (or unwilling) to better estimate the time required for the preparation, whereas they believe that they are better able to estimate the time required for the action itself, which has a progress bar. However this particular case seems to show that the distinction between the two phases does not actually correspond to two very distinct stages in the process.

    And in fact my experience in Mac OS X 10.5, even with far smaller number of files, is that quite often the preparation time is much longer than the action time. It often looks like the switch from one phase to the other in the progress window is out of sync with the actual process.

    The example above is an extreme case. But this apparent lack of synchronization seems to have become more prevalent in Mac OS X 10.5, often making the progress window pretty much useless.

    And I also believe that it is precisely because UNIX is so good at handling large numbers of files that people are disappointed with Apple’s apparent inability or unwillingness to make the Finder benefit from the underlying strengh of the UNIX flavour on which Mac OS X is based.

  5. sol says:

    Pierre : I wasn’t implying that this waiting game is spectacular. I understand your frustration. However, I try to schedule my intensive file copying for times when I can drag the items to the new location and do something else and leave my computer alone for the time it takes to copy. The disadvantage of this is that you cannot schedule every copying task this way. Like danridley said, we can always hope 10.6 will adress this issue.

    danridley : I realize now that the charging battery is a more complex task than copying files. Luckily, I can deactivate this time estimating feature (as I did) and it is not always staring at me. Thank you for the insight on this matter.

  6. danridley says:

    When I said “like your original copy operation,” I was referring to the copying of the files that took place first, rather than the deletion. Moving thousands of files to Trash does only take a fraction of a second, but copying them takes longer, and I assume that indeed that copy took longer than the “preparing” phase did.

    The fact that the result was a bit silly for your second operation — the delete, on which you never got a meaningful progress meter — doesn’t invalidate the design, and indeed the progress meter was quite meaningful on your first operation, the copy.

    I’d argue that Finder *does* benefit from the underlying strength of Unix. Back when I was a Windows user, in Windows 95, Explorer could gracefully handle up to a couple of hundred files; the command prompt would gracefully handle a thousand or so. Both of those figures are orders of magnitude better on OS X (and both remain better on OS X than on Vista), and both will continue to get better as both the OSes and the hardware improve.

    But the gap between command line and GUI remains. A GUI file manager’s copy operation is fundamentally hampered by users’ expectations that the file manager will protect them from mistakes and inconveniences. Finder checks permissions, free disk space, open files, etc. before it commences the copy, and this takes time. (One reason using the command line is so fast is that it doesn’t check these things; it just stops if it runs into trouble along the way. If this leaves you with a half-moved folder, it lets you clean up; Finder attempts to pick up after itself. Not always successfully, of course.)

    Moving 27,000 files to another folder on the same disk (the Trash) is no big deal — that part really *is* a split-second operation. Building a list in memory of 27,000 files, checking their permissions and the destination folder to make sure the move will complete successfully, estimating how long the move will take, etc., is harder.

    Now I’d argue that it should take less time than it does — I’d be happier if you had 20 seconds of “preparing” instead of several minutes — but as far as the display side of it goes, I don’t see a better path than the one they’ve taken. If they ignore the preparation and show a meter from the beginning, then you get what happens on Windows in cases like this, where the progress bar says 0% for five minutes and then jumps to 100% right before the dialog box closes. If they show a separate progress bar for the preparation, then they have the problem of reaching 100% and people thinking the operation will be done, then starting over at 0 (and that’s assuming it’s even practical to estimate the preparation phase, which it may not be).

    (The fact that its preparation is sometimes buggy, and that when it finds errors it’s often poor at describing them, adds to the frustration of course; the .DS_Store weirdness you ran into in your copy operation is obviously bonkers. I’m arguing with your criticism of the progress meters, but I want to make it clear that I’m not arguing that Finder’s perfect in all matters or anything :-) )

  7. ssp says:

    So what exactly are the hardware constraints you realise to be there. The last time I checked the Finder couldn’t fit more than a few hundred files into a view at the same time. I seriously doubt that displaying the name and at least generic icons for those will tax current hardware in any significant way.

  8. Andrew Aitken says:

    Leopard is a step backwards for me when working with directories full of large files in the GUI. In Tiger I could use the little search box to “filter” when I see in the window, Leopard runs the silly spotlight search, which takes ages.

    If you try and select 27k files, the finder will cough and splutter no end. Heaven forbid you try and drag them anywhere…

    I just use the terminal if I have to work with that many files… I shouldn’t have to, but it gets the job done.

  9. Pierre Igot says:

    Dan: Sorry I confused the copy operation with the move-to-trash operation. It was indeed for the latter that the discrepancy between the preparation phase and the action phase was so great.

    That said, even for the copy, the preparation took a long time, during which it was impossible to tell when the copying process would actually start. (Worse still: If there are any potential errors, such as the presence of a file with the same name in the destination, then the warning only appears after the preparation, which requires your attention and forces you to interrupt what you are doing. These warnings should be displayed right away.)

    I am still not entirely convinced that the two-phase process is the best possible solution, but maybe it would be more tolerable if, as you said, it didn’t take as long. All the preparation checks you describe are steps that should be near instantaneous because they don’t actually involve moving large chunks of data. Even with thousands of files, I’d expect better performance. Like ssp suggests, I tend to think that the hardware constraints cannot be all that great and that the Finder is simply not very efficient. But this raises engineering issues that are way beyond my field of expertise. All I can say as an end user is that this preparation phase that the user has to endure is usually annoying long and that, in the case of the move-to-trash action at least, absurdly longer than the action phase, to the point that it makes the UI itself absurd.

    Andrew: I don’t see how Leopard differs from Tiger for Spotlight searches. If anything, Spotlight searches are significantly faster in Leopard, which is an improvement, not a step backwards. Did you mean to refer to the search box in the pre-Spotlight days (i.e. Panther)?

    Like I said in my post, dragging and dropping 27,000 files in Leopard’s Finder was actually possible, even though it was pretty choppy and I wouldn’t recommend it to less experienced users. It wasn’t actually that much worse than dragging and dropping 100 files, as a matter of fact.

    I cannot bring myself to using the Terminal, in part because I do not have enough experience with it and don’t want to take the risk of effecting very destructive actions. So in that sense I agree with Dan’s analysis of the difference between the command line and the GUI. I just think that the GUI side of things could be improved. Just because it is not as bad as Windows does not mean that it is great :).

  10. Andrew Aitken says:

    Oh dear, I am showing my age… :( – Yes, I did mean it was better way back in the Panther days when you could “filter” a directory of results…

    I don’t see a good reason why it should be choppy to drag some icons around a screen – regardless of whether it’s 5, or 5,000 icons. After you drop them, sure, I can live with a delay, for all the reasons Dan states. But before I even do anything with them?…

  11. Untitled says:

    27k files in one folder?! It’s hardly a surprise Finder choked. Organise them into folders; that’d help you and the system.

  12. Pierre Igot says:

    Thousands of files in this day and age is nothing unusual. People have thousands of music files, digital pictures, e-mails, etc. Whether they are organized in folders or not should be irrelevant. We live in a world that produces high numbers of files.

    Our computers should be able to handle them, regardless of the way that they are organized. Indeed, our computers produce generate thousands of files themselves, sometimes in flat lists in folders without any structure either. (Check out your main e-mail account’s Inbox or Sent Messages box in the Finder, and look inside the “Messages” folder. Or Address Book’s “Metadata” folder.)

    FWIW, I do plan to go through the 27,000 files and organize them better. But I haven’t got around to it yet. It shouldn’t matter. If the Finder allows me to accumulate thousands of files in a single folder, it should be able to handle such a number of files.

  13. icondaemon says:

    I always reach for the ditto command in Terminal when I must transfer more than just a few hundred items. A different tool for a different task. Compare moving 27k files to another volume to moving 27k sewing needles in a large box into another large box. You wouldn’t serially grab handfuls of needles to move them from one box to another until you were done: you’d just tip one box into another. Apple should invest a bit more engineering development into having the Finder recognize when a user wants to move such large numbers of files and bypass the slow, griding method in favor of a ‘ditto daemon’ which takes over.

    +- icondaemon -+

Leave a Reply

Comments are closed.