I had been getting very annoyed with my Firefox. Every time I downloaded a file, it took 30 seconds or so for the "Downloads" window to come up, during which time Firefox was unresponsive to keyboard and mouse inputs. I discovered the solution a couple of days ago, and I'm blogging it so that others may find it if they do a search.
The problem was that the file that stores the download history had gotten very big, and Firefox was pausing to parse that file into the display in the "Downloads" window. All I needed to do to fix it was to press the "Clean Up" button at the bottom of the "Downloads" window (available on the "Tools" menu).
There's an alternative. In Edit/Preferences..., on the "Download History" tab of the "Privacy" panel, there's a combo-box to select when to "Remove files from the Download Manager". It defaults to "Manually", which is what I like, but you can also select "Upon successful download" or "When Firefox exits".
This is in Firefox 1.5 on Linux. It might work differently in other versions or operating systems.