Geek question - defragging raid volumes?

Now I do consider myself something of a nerd but this one got me a bit puzzled. Being a proper geek my main PC "drive" is a 4 disk raid 10 affair, over lunch I casually hit the defrag button thinking I'd not done it in ages, like a year or so.

But then I got to thinking - would defragging actually make any difference given that sequential reads are going to be taken from physically seperate disks anyway and by the time a second read is required from the first disk again it'll have spun around a few times and the head is unlikely to be in the optimum position anyway? Or is the whole process aimed at reducing arm movement rather than waiting for the disk to spin round again?

Then I started getting a bit worried, I'm perhaps a little long in the tooth and remembering older Windoze OSs but how much does the OS actually know about the hardware? I'm used to quite a distinct hardware/software seperation - the BIOS presents an illusion to the OS - "I've got a C: volume, you don't need to know the rest". Isn't there the potential for a defrag to seriously screw up a raided volume? If you're moving bits of data around on a "drive" without knowing that the actual files are chopped into fine chunks and put in different places on different drives couldn't that result in a seriously fubar'd stripe (regardless of whether it's raid 0, 5 or 10)? Oh well, guess not really, seeing as it would have died pretty much immediately if it would cause a problem and it's still chugging away quite happily now.

So I'm just left kinda curious as to what exactly it's doing in there, how does it know how to re-arrange all the little pieces across multiple drives? Or does the OS just pass a "defrag" command to the SCSI/IDE/SATA bios and the bios just gets on with it saying "yeah ok, I'll let you know when I'm done, you go display a pretty progress bar or something"?

[Luckypants]

Dunno about PC gubbins, but on a real computer the RAID devices appear as conceptual volumes (3390 mod 4 IIRC) and management of the RAID device is down to a utility program that the storage techs run when required.

So I would suspect that on a PC the RAID device appears as a NTFS disk and management of data placement, cache and index is down to whatever software (or firmware?) runs the device.

[the_lone_wolf]

Originally Posted by MiniMatt

But then I got to thinking - would defragging actually make any difference given that sequential reads are going to be taken from physically seperate disks anyway and by the time a second read is required from the first disk again it'll have spun around a few times and the head is unlikely to be in the optimum position anyway?

[SoulKiss]

Originally Posted by MiniMatt

Now I do consider myself something of a nerd but this one got me a bit puzzled. Being a proper geek my main PC "drive" is a 4 disk raid 10 affair, over lunch I casually hit the defrag button thinking I'd not done it in ages, like a year or so.

But then I got to thinking - would defragging actually make any difference given that sequential reads are going to be taken from physically seperate disks anyway and by the time a second read is required from the first disk again it'll have spun around a few times and the head is unlikely to be in the optimum position anyway? Or is the whole process aimed at reducing arm movement rather than waiting for the disk to spin round again?

Then I started getting a bit worried, I'm perhaps a little long in the tooth and remembering older Windoze OSs but how much does the OS actually know about the hardware? I'm used to quite a distinct hardware/software seperation - the BIOS presents an illusion to the OS - "I've got a C: volume, you don't need to know the rest". Isn't there the potential for a defrag to seriously screw up a raided volume? If you're moving bits of data around on a "drive" without knowing that the actual files are chopped into fine chunks and put in different places on different drives couldn't that result in a seriously fubar'd stripe (regardless of whether it's raid 0, 5 or 10)? Oh well, guess not really, seeing as it would have died pretty much immediately if it would cause a problem and it's still chugging away quite happily now.

So I'm just left kinda curious as to what exactly it's doing in there, how does it know how to re-arrange all the little pieces across multiple drives? Or does the OS just pass a "defrag" command to the SCSI/IDE/SATA bios and the bios just gets on with it saying "yeah ok, I'll let you know when I'm done, you go display a pretty progress bar or something"?

Sorry cant help

Only used Proper RAID (5) on a real OS (Linux) with a filesystem that never needs Defragging

OK, a couple of questions to answer it seems:

The point of defragging isn't to deal with the disc spinning (that's inevitable due to how they work). The aim of the game is to get as much commonly accessed data near the spindle as possible (as this reduces arm movement and therefore seek time. The arm landing zone is near the spindle, which is really the only consideration as to inside or outside of the disc platter.

Defragmentation (particularly in NTFS volumes) also re-arranges the map of clusters (maintained by the OS). This eliminates any empty space near the start of the map, to increase speed accessing data from the map.

Now, in the case of a single disc, there's multiple platters. The OS doesn't care which platter has what really. But it will know, for example, that there are 8 inner tracks, each with a given number of cluters.

In the case of a RAID volume, you may be using RAID1, RAID2, etc etc (mirroring, striping etc). The OS has no real way to know this, as this is dealt with by the RAID drivers. The RAID drivers job is to translate addresses into physical areas of the disc by present a 'virtual disc.' So defragging works on this 'virtual disc.'

So the question is how accurately the translation happens. The answer is, it depends. Take the above example, The OS knows you have, say 500Gb on a single drive. It also knows this is a RAID volume. It doesn't know that you actually have 5 100Gb discs arranged in RAID0.

The way it does this trick, is to tell the OS that you have say, 128 clusters per track. In fact, each disc may only have 8 clusters. Note that I'm picking numbers out of the air to illustrate the point.

So the OS says it wants a commonly accessed cluster to move from the outside of the platter, to cluster 9 on the RAID device. The RAID drivers translate that from Disc1Track0Cluster9, to Disck2Track0Cluster0, and everyone is happy.

Now, that's degraffing a stiped volume. What about mirrored?

Mirrored is a different. In the above example, Disc1Track0Cluster9 on a mirrored RAID device may well be Disc1Track1Cluster0 AND Disc2Track1Cluster0. Not a problem so long as there's no bottlenecks.

Now, if you have something more complex, such as RAID5 (striped and partially mirrored) then the translation is a little more complex, but just as possible. In this case, it will apply the same translation as used for RAID0, but then conditionally mirror it. This will result in the defrag process taking roughly 2 eon's!!

RAID6 is worse, as that always mirrors the writing process, whereas RAID5 only does it conditionally.

SoulKiss, a couple of things to mention:
e2defrag (works with ext3 too if you downgrade to ext2 temporarily - and is beneficial), vxfs fsadm, defragfs for JFS, xfs_fsr. SFS can be defragged stateless (though I can't remember the utility to do it) and only HFS+ attempts to defrag itself whilst you're reading/writing to it with anything close to success.

Stay away from most of the above, and you're right, you don't need to defrag.

[TSM]

what i never get is why they did not do more tools for ext3 like e3defrag or e3resize etc... windows bods have it easy on resizing/creating/formatting partitions, i dont fancy recreating my 4TB array anytime soon.

Originally Posted by TSM

what i never get is why they did not do more tools for ext3 like e3defrag or e3resize etc... windows bods have it easy on resizing/creating/formatting partitions, i dont fancy recreating my 4TB array anytime soon.

ext3 was designed not to fragment. Which it doesn't really (compared to FAT16 drives).

But fragmentation is only half the problem with access times, as we both know. Even with a fully defragmented drive, you still need the cache to be able to read/write all your data, otherwise you're waiting for the next spin anyway.

[TSM]

Originally Posted by Baph

ext3 was designed not to fragment. Which it doesn't really (compared to FAT16 drives).

But fragmentation is only half the problem with access times, as we both know. Even with a fully defragmented drive, you still need the cache to be able to read/write all your data, otherwise you're waiting for the next spin anyway.

Having nice raid cards with 512Mb ram help with the transfer of images from our servers along with 10 spindles. The best transfer ive seen to desktop computers is about 500Mbps, which i think is limited by the raid6 being made up of SATA drives, mabey ile extend the amount of spindles some time soon.

Even though ext3 was designed not to fragment, it still will eventualy, but generaly its speedy. I just await ext4 to go stable.

I have looked at various other FSs but all too confusing and e3 just works fine with Centos.