What is fragmentation and defragmentation?
When data is written to the hard drive, it is done so based on the space available. If a space is spotted, it will be filled by the data from the file. If the space was not large enough to accommodate the entire file, the next available space will be filled with the remaining portion of the file. This continues until the entire file has been stored on the hard drive. Thus, a single file may be broken up into dozens of chunks and spread across the hard drive. As time goes on the chunks will grow smaller in size and greater in number, which is referred to as fragmentation.
Fragmentation is not a good thing, and can decrease the performance of your hard drive. The reason is that in order to open a file, it has to find the first part of the file, then the second, then the third, and so on until it has found all of the chunks to form the file as a single entity. This process takes time, resulting in delays when searching for, opening, modifying, and saving files.
A good analogy would be if you write a 20 page report. Page 1 you place on your desk, page 2 in your briefcase, page three on your keyboard, page 4 on your nightstand, etc. If you ever want to read the report, you have to run throughout your house collecting all of these pages. Wouldn't it be faster if they were all together, in order, in a single place? Of course, and that's what defragmenting does.
By defragmenting your hard drive, you are reorganizing all of those chunks of data, combining those that go together and putting them in an order where it's easier to find what you need, when you need it. By doing so you speed up the overall production of your hard drive, at least to a certain degree. (The degree of improvement corresponds to the level of fragmentation.)
Does this vary by OS, or over time?
Now, those that say this isn't as important now are correct in some ways. The NTFS filing system, standard for those running Windows 2000/XP, is more efficient than the old FAT32, and doesn't become as fragmented as fast. However, it still happens, so you still need to defragment on occasion.
Another thing to consider is the size of the hard drive, the number of files, and the size of the files. The larger the hard drive the larger it will take, percentage wise, to become heavily fragmented. However, as the amount of stored files increases, the number of fragments also increases.
So, take two people who use their hard drives the same amount for the same types of activities. The one with a large hard drive will have to defragment less often because it takes longer for the fragmentation percentage to rise to a point where defragmentation is necessary. However, when he gets around to defragmenting it'll have to do more work because there are a greater number of fragments present. That's the difference between raw numbers and percentages.
Finally, Unix-based OSes seem almost invulnerable to fragmentation due to their design, so they don't have to worry about this sort of thing. For all of you Windows users, though, this should be a part of your regular system maintenance.
Now, when should you defragment?
Well, if you launch the defragmenter, there is an Analyze option. It will survey your drive, determine the level of fragmentation, and advise you whether or not you should defragment at that time. This is a much more efficient way of deciding as opposed to the 'It's the first Sunday of the month' strategy. Remember, fragmentation varies based on the amount and type of usage, not necessarily how many days it's been.
Which defragmenter should I use?
The standard Windows defragmenter should be fine. Others, such as Diskeeper, are more powerful, typically defragmenting faster, and at times with a slightly greater percentage of efficiency. They can also defragment in the background so that the level of fragmentation never rises above a certain point. It's easy to argue either side of this point, so I'll leave that decision up to you.
Also, as an FYI, the Windows defragmenter is actually an older, modified 'lite' version of Diskeeper. (A result of the Microsoft <-> Executive Software partnership.) Thus, by paying the $30+ you're simply getting the updated 'enhanced' version.
Is this good for my system?
Yes and no. Every time you use your hard drive, no matter for what task, you're taking a little bit of life away from it...every bit of action wears it down to a very small degree. Thus, the defragmentation process does shorten the life span to a small extent. However, you need to compare this to the how much wear and tear would be put on by having to constantly search the drive for chunks of files.
Depending on your level of fragmentation, the defragmentation may or may not be of greater harm than good. That's why it's best to consult the Analyze function of the defragmenter. If it determines that your drive is heavily fragmented, then it's in your best interest to defragment. Otherwise, leave it alone.
Basically, it's like taking medication...it'll cure your ailments, but taking too much, or taking it when your not sick can cause more harm than good.
Well, that was quite a mouthful! Congrats on reading to the end, and I hope that answers your questions. While some will take issue with various points, disputing the benefits of defragmentation, the above is the generally-held stance, and the one I believe in.
Hope this helps,