Azure : Setting up a high available SQL cluster with standard edition

Introduction

It is important to know that you will only get an SLA (99,95%) with Azure when you have two machines deployed (within one availability set) that do the same thing. If this is not the case, then Microsoft will not guarantee anything. Why is that? Because during service windows, a machine can go down. Those service windows are quite broad in terms of time where you will not be able to negotiate or know the exact downtime.

That being said… Setting up your own high available SQL database is not that easy. There are several options, though it basically bears down to the following ;

• an AlwaysOn Availability Groups setup
• a Failover Cluster backed by SIOS datakeeper

Where I really like AlwaysOn, there are two downsides to that approach ;

• to really enjoy it, you need the enterprise edition (which isn’t exactly cheap)
• not all applications support AlwaysOn with their implementations

So a lot of organisations were stranded in terms of SQL and moving to Azure. Though, thank god, a third party tool introduced itself ; SIOS Datakeeper ! Now we can build our traditional Failover Cluster on Azure.

 

Design

Before we start, let’s delve into the design for our setup ;

Docker : Storage Patterns for Persistence

Introduction

One of the sensitive areas when it comes to docker is persistent storage… A typical service upgrade involves shutting down the “V1” container and pulling/starting the “V2” container. If no actions are taken, then all your data will be wiped… This is not really the scenario we want off course!

So today we’ll go over several variants when it comes down to data persistence ;

• Default : No Data Persistence
• Data Volumes : Container Persistence
• Data Only Container : Container Persistence
• Host Mapped Volume : Container Persistence
• Host Mapped Volume, backed by Shared Storage : Host Persistence
• Convoy Volume Plugin : Host Persistence

What do I mean with the different (self invented) persistence level ;

• Container : An upgrade of the container will not scratch the data
• Host : A host failure will not result in data loss

So let’s go through the different variants, shall we?

Azure & Docker : Shared storage anyone?

Introduction

Persistent storage is a though cookie for Docker… We’ve seen things like Flocker, Convoy, … Today, we’ll do a very rough (and experimental!) setup with an Azure storage account (via Azure File Share) as shared storage. How would such a design look?

RTO, RPO, … What the O?

Ever heard about the terms RTO (Restore Time Objective> and RPO (Recovery Point Objective)?

To explain it, let us take a look at this mockup…

In the middle, the crash indicates the time disaster struck. When we go back to the latest point we took our backup off-site (in regards to the affected crash site), where this point should be less than the maximum tolerable period in which data might be lost from an IT service due to a major incident. So the arrow towards the left indicates the RPO. Be aware that storing these backups on the same risk site does not fulfil your RPO!

The arrow to the left indicates the time needed to restore the service. Be aware that this is from a non-technical / business perspective. So merely starting up the system is not enough. Users of the service need to be able to use it again!

In terms of costs, be aware that strict objectives will imply more expensive solutions. The closer you want the objectives towards your crash zone, the more expensive it will become. An RPO of 7 days will still allow tape backups to be taken offsite once a week, where 1 day will still allow a nightly replication, yet where shorten time constraints implies near online replication.

Raid Levels

The basics
RAID combines two or more physical hard disks into a single logical unit by using either special hardware or software. Hardware solutions often are designed to present themselves to the attached system as a single hard drive, and the operating system is unaware of the technical workings. Software solutions are typically implemented in the operating system, and again would present the RAID drive as a single drive to applications.

There are three key concepts in RAID: mirroring, the copying of data to more than one disk; striping, the splitting of data across more than one disk; and error correction, where redundant data is stored to allow problems to be detected and possibly fixed (known as fault tolerance). Different RAID levels use one or more of these techniques, depending on the system requirements. The main aims of using RAID are to improve reliability, important for protecting information that is critical to a business, for example a database of customer orders; or to improve speed, for example a system that delivers video on demand TV programs to many viewers.

The configuration affects reliability and performance in different ways. The problem with using more disks is that it is more likely that one will go wrong, but by using error checking the total system can be made more reliable by being able to survive and repair the failure. Basic mirroring can speed up reading data as a system can read different data from both the disks, but it may be slow for writing if the configuration requires that both disks must confirm that the data is correctly written. Striping is often used for performance, where it allows sequences of data to be read from multiple disks at the same time. Error checking typically will slow the system down as data needs to be read from several places and compared. The design of RAID systems is therefore a compromise and understanding the requirements of a system is important. Modern disk arrays typically provide the facility to select the appropriate RAID configuration. PC Format Magazine claims that “in all our real-world tests, the difference between the single drive performance and the dual-drive RAID 0 striped setup was virtually non-existent. And in fact, the single drive was ever-so-slightly faster than the other setups, including the RAID 5 system that we’d hoped would offer the perfect combination of performance and data redundancy”.

OpenSource Storage Management

I came across OpenFiler a while ago and was intriged by it. Now I’ve taken the liberty to testing it in my lab, and I must say that I’m impressed by the features. It’s something every sysadmin should check out to see if it isn’t a viable solution for their overpriced storage solution… 😉

.

Openfiler is a powerful, intuitive browser-based network storage software distribution. Openfiler delivers file-based Network Attached Storage and block-based Storage Area Networking in a single framework. Its uses the rPath Linux metadistribution and is distributed as a stand-alone Linux distribution. The entire software stack interfaces with third-party software that is all open source.

File-based networking protocols supported by Openfiler include: NFS, SMB/CIFS, HTTP/WebDAV and FTP. Network directories supported by Openfiler include NIS, LDAP (with support for SMB/CIFS encrypted passwords), Active Directory (in native and mixed modes) and Hesiod. Authentication protocols include Kerberos 5.

Openfiler includes support for volume-based partitioning, iSCSI (target and initiator), scheduled snapshots, resource quota, and a single unified interface for share management which makes allocating shares for various network file-system protocols a breeze.

The architecture behind microsoft.com

Jeff Alexander blogged about the setup behind microsoft.com. It got the slashdot effect, and his blog is apparently (temporarily?) suspended. You can find the blog post below, as I dug it from out of the google caches.

Microsoft.com: What’s the story?
If you’ve ever wondered how microsoft.com uses our technology then read on. I recently came across some good information from the folks over at the Operations team at Microsoft.com. The thread basically talks about how we use IIS, Firewalls and Windows Server 2008. I think as we come up to launch next year it’s a really good and quick insight into what they do and how they do it. So enjoy the reading and let me know what you think..Pretend I’ve asked about how they protect our sites…

At this point we still don’t use firewalls for MS.COM sites and don’t have any plans on the books to put them in place. Here is the short answer as to why:

Understanding the basics concept of fast flux dns

One of the most active threats we face today on the Internet is cyber-crime. It’s a profitable business where IT capable criminals try to control the computers of the naive. They do this by infecting them with malware. There are various ways of introducing these malicious codes into the target systems, but that part is out of scope for this post.

The goal of fast-flux is decrease the detection chances/rate when doing a malicous action. This can fluctuate from a spam mailing to a denial-of-service attack. The basic concept of the “fast flux” is to have a fully qualified domain name with multiple IP addresses assigned to it. These IP addresses are swapped in and out of flux with extreme frequency, using a combination of round-robin IP addresses and a very short Time-To-Live. The hostname of a certain website may change as often as every three minutes.

The ip address used here are those of infected machines. So a browser connecting to the same website every 3 minutes would actually be connecting to a different infected computer each time. So the computer of your grandparents may simply be used to provide resources to the latest phishing attack.

It’s a common practice to build in load-distribution schemes which also do a health-check. This is useful so that the nodes (the computer from for example the grandparents) that are offline can be taken out of the flux. Enabling an always maintained content availability. A second layer is often added for security and fail-over: blind proxy redirection. Actually… a lot of techniques used in the world of legitimate webserver operations are used by these criminal computer networks.

The fast-flux it’s controlling element is often refered to as “mothership”. It’s simpilar to the control mechanisms used in the older botnets. But it has more features compared to the more conventional botnets. These mostly provide the basic backend infrastructure for the botnet (dns & http) as it serves the content towards the infected nodes.

Interesting readups:
Know Your Enemy: Fast-Flux Service Networks @ Honeynet.org (main reference)
the future of botnets : fast flux bot nets (tip!)
SpamTrackers.eu
Darkreading: here & here
Securityfocus article

Bigip : multiple virtual services running on different ports connected to one pool

Scenario
– Two application servers
– Each application server hosts different 3 services (different ports) which depend on eachother

Objective
When one of the services on a node goes down (checked by a monitor), then all services should be marked as down.

Possible Solutions

• Solution “divided” : A seperate pool for each service
  The way you normally do this, yet it’s not that clean as you make the situation a bit more bloated.
• Solution “combined” : One pool for all services
  Use the “translate service disable” option when creating a virtual server. This will disable port translation for the specific virtual server.

Example
If the virtual port is 65001, and the port used for the poolmembers is 65101, then when a request is send to the virtual ip on port 65001, then it will be rerouted to the pool member’s port 65101.
If the “translate service” option is set to “disable”, then the request will be sent to the pool member’s port 65001.
In this case you can setup one pool, with different checks for all services the nodes should provide. And create virtual servers pointing to one single pool.

man virtual

translate service
The options are enable or disable. You can turn port translation
off for a virtual server if you want to use the virtual server to
load balance connections to any service.

Bigip : connection mirroring

Scenario
There are two loadbalancers which are setup as a redudant pair which provides a default simple virtual server (with pool).
Let’s say you’d setup a connection towards this virtual server, and afterwards there’dd be a failover.

What happens to the connection that was setup to the load balancer that was setup?

• The connection is being migrated to the other load balancer.
• The connection remains as it was, directed to the “failed” load balancer.
• The connection is terminated/reset (by the BigIP).

It might be a surprise to some, that the correct answer is the second one. The connection remains “as it was”. Off course it won’t be functional; Yet there will be no fail over of this connection by default, or will the connection be terminated/reset by the BigIP
When (and how) the connection will be reset depends solely on the client!

So one might ask: “Why doesn’t the BigIP send a fin/rst to the client?”
Another question might answer this: “How would the BigIP be able to send the fin/rst packet as it failed?” A failover occurs when the unit isn’t accessible anymore (If it got disconnected from the network, crashed… etc). It wouldn’t be able to send this packet.

There is however a mechanism that does a fail over to the other BigIP. But there is a (performance) trade off involved. This mechanism is called “connection mirroring”. So the state of all the connections made to the active BigIP are also kept on the standby BigIP.

HOWTO
You can enable the “connection mirroring” thru :
– the command line (“man virtual”) : “b virtual *name* mirror conn enable”
– the GUI (virtual server -> advanced) : GUI Screenshot

So if you REALLY need it, you can use the option, yet be aware of the performance degradation that’ll cause.