Raid Levels

The basics
RAID combines two or more physical hard disks into a single logical unit by using either special hardware or software. Hardware solutions often are designed to present themselves to the attached system as a single hard drive, and the operating system is unaware of the technical workings. Software solutions are typically implemented in the operating system, and again would present the RAID drive as a single drive to applications.

There are three key concepts in RAID: mirroring, the copying of data to more than one disk; striping, the splitting of data across more than one disk; and error correction, where redundant data is stored to allow problems to be detected and possibly fixed (known as fault tolerance). Different RAID levels use one or more of these techniques, depending on the system requirements. The main aims of using RAID are to improve reliability, important for protecting information that is critical to a business, for example a database of customer orders; or to improve speed, for example a system that delivers video on demand TV programs to many viewers.

The configuration affects reliability and performance in different ways. The problem with using more disks is that it is more likely that one will go wrong, but by using error checking the total system can be made more reliable by being able to survive and repair the failure. Basic mirroring can speed up reading data as a system can read different data from both the disks, but it may be slow for writing if the configuration requires that both disks must confirm that the data is correctly written. Striping is often used for performance, where it allows sequences of data to be read from multiple disks at the same time. Error checking typically will slow the system down as data needs to be read from several places and compared. The design of RAID systems is therefore a compromise and understanding the requirements of a system is important. Modern disk arrays typically provide the facility to select the appropriate RAID configuration. PC Format Magazine claims that “in all our real-world tests, the difference between the single drive performance and the dual-drive RAID 0 striped setup was virtually non-existent. And in fact, the single drive was ever-so-slightly faster than the other setups, including the RAID 5 system that we’d hoped would offer the perfect combination of performance and data redundancy”.

OpenSource Storage Management

I came across OpenFiler a while ago and was intriged by it. Now I’ve taken the liberty to testing it in my lab, and I must say that I’m impressed by the features. It’s something every sysadmin should check out to see if it isn’t a viable solution for their overpriced storage solution…

.

Openfiler is a powerful, intuitive browser-based network storage software distribution. Openfiler delivers file-based Network Attached Storage and block-based Storage Area Networking in a single framework. Its uses the rPath Linux metadistribution and is distributed as a stand-alone Linux distribution. The entire software stack interfaces with third-party software that is all open source.

File-based networking protocols supported by Openfiler include: NFS, SMB/CIFS, HTTP/WebDAV and FTP. Network directories supported by Openfiler include NIS, LDAP (with support for SMB/CIFS encrypted passwords), Active Directory (in native and mixed modes) and Hesiod. Authentication protocols include Kerberos 5.

Openfiler includes support for volume-based partitioning, iSCSI (target and initiator), scheduled snapshots, resource quota, and a single unified interface for share management which makes allocating shares for various network file-system protocols a breeze.

Which is the greenest; disks or flash?

In his latest blog post, Anything But a Flash in the Pan, Jonathan Schwartz. touches a good point… We have RAM memory for “on the fly” memory and hard drivers to have persistent storage. Yet flash memory could (let me stand corrected; WILL) be used as a middle way option. Put apart the commercial notes (links towards Sun & ZFS), and you’ll probably find it an interesting read like me.

There are only two kinds of storage devices – those that have failed, and those that are about to fail. That’s the view most data centers have about the traditionally mechanical devices pejoratively referred to as “spinning rust.” All disk drives fail, cheap drives fail faster.

The architecture behind microsoft.com

Jeff Alexander blogged about the setup behind microsoft.com. It got the slashdot effect, and his blog is apparently (temporarily?) suspended. You can find the blog post below, as I dug it from out of the google caches.

Microsoft.com: What’s the story?
If you’ve ever wondered how microsoft.com uses our technology then read on. I recently came across some good information from the folks over at the Operations team at Microsoft.com. The thread basically talks about how we use IIS, Firewalls and Windows Server 2008. I think as we come up to launch next year it’s a really good and quick insight into what they do and how they do it. So enjoy the reading and let me know what you think..Pretend I’ve asked about how they protect our sites…

At this point we still don’t use firewalls for MS.COM sites and don’t have any plans on the books to put them in place. Here is the short answer as to why:

What we often forget when implementing virtualization solutions

“The beginning is the half of every action”
Someone once told me “There is nothing more permanent than temporarily.” (roughly translated), and it’s something you often see in the IT world. A server goes down, let’s do the quick fix now and do the in depth analysis/coding later. The last step is often moved the the refrigerator called “on hold”-, “TODO” or “when we have time” boxes.

The following situation might ring some bells:

X : How de we save on infrastructure costs?
Y : Maybe by virtualizing our infrastructure?
X : Sounds good, how do we do this?
Y : Let’s first try our lab/development/staging environment?
X : And if that works move all servers to it!

Help!!! My virtual servers are breeding like rabbits
Most companies who’ve started with virtualizations, like for example VmWare of Xen, have found themselves rushing (or stumbling) way too fast in this new enviroment. The virtual infrastructure needs the same amount of thought as your physical infrastructure. It’s not because a virtual server is created at a fraction of the time it would take a physical one, that one shouldn’t follow the same steps.

Perhaps the sexiest aspect of virtualization is its speed: You can create VMs in minutes, move them around easily, and deliver new computing power to the business side in a day instead of weeks. It’s fun to drive fast. But slow down long enough to think about making virtualization part of your existing IT processes

It’s not because it’s virtual that it doesn’t need to be managed

Tuning Disks I/O for best performance

Read up on this article.

Learn more about direct I/O, concurrent I/O, asynchronous I/O, and best practices for each method of I/O implementation. This three-part series on the AIX® disk and I/O subsystem focuses on the challenges of optimizing disk I/O performance. While disk tuning is arguably less exciting than CPU or memory tuning, it is a crucial component in optimizing server performance. In fact, partly because disk I/O is your weakest subsystem link, you can do more to improve disk I/O performance than on any other subsystem.

Credits go to “Ken Milberg” on writing the article. I enjoyed reading it, as it was very interesting.

Grinder : Load Testing Framework

What is Grinder?

The Grinder is a Java load-testing framework. It is freely available under a BSD-style open-source license. The Grinder makes it easy to orchestrate the activities of a test script in many processes across many machines, using a graphical console application. Test scripts make use of client code embodied in Java plug-ins. Most users of The Grinder do not write plug-ins themselves, instead they use one of the supplied plug-ins. The Grinder comes with a mature plug-in for testing HTTP services, as well as a tool which allows HTTP scripts to be automatically recorded.

How can I use it? A small tutorial…

Performance impact of the VmWare Virtual Switch

Let’s start out with the basics. Vmware has several products that can be used for virtualization. The most commonly know products are “vmware workstation”, “vmware server” & “vmware player”. They should actually be classed under “emulation” rather than device sharing. In my “hobby environment” I used the VmWare server; It’s free, and it’s solid.

Yet for the enterprise needs, esx is the way to go. Esx is a kernel on it’s own, and enables the virtual machines to really share the resources. This gives esx an extreme advantage over the other products, yet be aware that it also implies technical restrictions/difficulties.

As you can probably guess, adding an extra “emulation layer” will result in some performance loss. Those products will most likely suffice for function test/development environments. Yet a bit more performance and resource sharing is required for servers that need an enterprise production level.

Another thing you need to consider is infrastructure architecture you’re going to build. Here is where the article comes down to… The network sharing part in ALL vmware products is done thru a kind of “virtual switch”. This program is software, and is bound to cpu usage. When several servers share an environment within a vmware product, and one server starts to do a lot of bandwith. Then all servers will notice this as the virtual switch will need cpu power for this.

Don’t get me wrong here… I don’t want to bash the product, but I want to make you aware of this situation so that you can design your server farms for this.
For example: organise your farm so that the intensive servers share their environment with some “light” servers
Also make sure your system architectures know this limitation! This gives them the opportunity to design a system that suits a shared hosting environment. It’s just awfull if everybodies hard work goes down the drain, due to a design issue that could have been tackled.