As you may probably know, Linux supports various filesystems such as ext2, ext3, ext4, xfs, reiserfs, jfs among others. Few users really consider this part of a system, selecting default options of their distribution’s installer. In this article, I will give some reasons for a better consideration of the file-system and of its layout. I will suggest a top-bottom process for the design of a “smart” layout that remains as stable as possible over time for a given computer usage.
Note: this article has been published in HTML format at www.linuxconfig.org
7 Comments
Putting multiple raid partitions over the same set of drives will increase block spindle contention and you will end up with a lot of seeks. I realize that this targets a desktop configuration, but on an enterprise system you’d want separate spindles for each RAID array.
Sorry, that should be spindle contention, not “block spindle contention”
Of course, and I thought it was clear that following a top-bottom approach, if you need performance at all cost, the best choice is to map different physical volumes (RAID) to physically different hard disks. But this also means that you can afford the price of your requirements which is not always true, even in the business context!
Following that idea of performance at all cost, up to the extreme — and stupid! — case, having one file/directory per hard disks would also be the best from a strict performance point of view. Obviously, this is not a reasonable option. Therefore, you are always facing a deal between price versus performance.
By the way, RAID has been specifically set up to address this compromise: RAID historically meant Redundant Array of *Inexpensive* Disks.
Finally, I would be confident in the linux kernel to deal with the spindle contention problem. I mean, in the face of contention, linux may order requests to hard drives in a “smart” way, reducing contention to a minimum (well I hope it can do that actually, I do not have any evidence that it actually does it!
).
In an enterprise, cost is often not the primary consideration. Service level agreements will often dictate performance objectives/constraints and you will require that your I/O be deterministic (or as deterministic as possible) this becomes more difficult to predict when you have different raid volumes over different spindles, esp when they have different access patterns (and IOPS vs Streaming.)
There are some enterprise storage systems that do this in order to make use of the performance characteristics of hard disk drives, but you require pretty intelligent controllers and they typically spread these out over a large number of spindles in order to minimize contention. HP’s EVA/HSV series does volume virtualization over large pools of disks and their performance best practice guide is to have as many disks in the same disk group as possible. This buys you more spindles and less chance of contention.
Linux (IIRC) and HD command queues will themselves attempt to reorder actual requests to the drive in order to minimize seeks and rotational latency, but it can only do so much.
Additionally, it should be mentioned that LVM technology adds flexibility, but at the cost of performance. e.g. LVM on Linux can come with a cost around 30%.
Doh, I guess I should preview before posting…
“this becomes more difficult to predict when you have different raid volumes over different spindles”
should read:
“this becomes more difficult to predict when you have different raid volumes over the same spindles”
You are definitely right. I will just add that in the case where you have such deterministic requirements, the top-bottom approach is already used since needs have already been “contractualized”. Therefore, you will have to design your filesystem layout according to those requirements. Fortunately, you will end-up with the number of disks required, and you may not have different RAID arrays backed by the same hard drive because it may not fulfill your requirements.
This post is more targeted towards end-users, small companies, schools, small labs, where requirements are often not written down and where the filesystem layout is often not taken into consideration.
About the LVM overhead, do you have any reference(s)? I am pretty much interested in this area.
Thanks for your valuable input.
I have come across it on forums discussing filesystem benchmarks and have seen it first hand many times on my own personal systems and on enterprise systems, but I don’t have a URL to point you to. If you want to see the effects for yourself run some bonnie++ benchmarks.
Cheers.