When talking to customers about DevOps, I often get the two following questions ;
- Does this mean I have to get rid of ; ITIL / COBIT / … ?
- Do I have to start moving people around and creating new units?
The quick answer is ; No.
A typical parabel in any project methodology is ;
How do you eat an elephant? Take snack sized bites and work your way through it.
And the same goes for DevOps!
For those who have met me before, you must surely have heard me say the following…
I have a bit less than 10 pets at home… Ranging from cats, via chickens towards even a pot-bellied pig! They all receive great individual care. Actually, when it would be solely up to my wife, they would all be spoiled brats probably. 😉
I live in a rural part of Belgium, and my neighbour is a farmer. He has more than 100 cows. Does he care about these animals? Yes he does! He greatly does, and not only because it is his bread & butter.
Though at a given scale, it’s not possible to do individual care anymore… You need to manage these numbers in a different way. And the same goes for an IT organization!
An IT organization who has works in the “individual care” methodology, will not be able to exceed four releases per year. System administrators/engineers in such an organization can only manage up to about forty systems. Do they manage more? Probably. And this costs them dearly, because they are forced to neglect certain tasks. This is mainly in the “invisible” area of proactive care ; updates, upgrades, event monitoring, etc.
So you need to take a look at things in a different manner? Start automating things! And that is were DevOps comes into play. And I know, the term DevOps has been burned by a lot of (marketing) people in the industry to sell products. *saddened*
Though I always tell anyone who wants to hear it ;
“DevOps, in essence, is about bringing people together and getting the job done.”
So what does DevOps mean towards shops who are have invested a great deal into ITIL/COBIT/…whichever. Do you need to throw away all of your processes? No! Is there going to be change? YES!
What do I mean by those statements?
A typical example, when looking towards ITIL, is the “CAB” (change advisory board). To those who love ITIL the holy grail in order to control changes. To anyone else, the master of evil who reduces the time to market….
Am I saying the CAB is a bad thing? No. Am I saying the current way of doing it is good? NO! But what is the objective of a CAB? You want to protect your organization from introducing high risk changes. So we put a control element in there that could detect them. How is that working out for you? Yeah, I know…
Though the objective as such is a very good thing! The problem lies in the fact that the majority of changes are just crap. Let’s face it and be honest… In most cases, nobody has a clue what is going to happen. Why is this? Because we have systems built on the premise of “individual care”, but the landscape as such has grown too big.
Imagine maintaining your lawn with a manual lawn mower. Great if you have a rather modest yard. But if that one grows… You will not be able to manage it.
The same goes for IT systems. We need to automate stuff, so that changes will become standard changes. See what I did here? We are still using ITIL! A standard change is a change that was already pre approved, because the change was already vetted and approved. So it will not need approval any time you need to do this.
Will your CAB disappear? No. A lot of changes, maybe even changes introduced by release management, will become standard change. And you will focus on the ones who are triggered by having high risk characteristics.
Still not convinced? I can understand you. 😉 Why do you want to control those changes… Because you know the effects of doing risky changes in your organization can be a disaster. So it’s all about the MTTD & MTTR, or “mean time to detect” & “mean time to recover”, or… “How long does it take before I notice it is going wrong?” & “How long does it take to fix?”
When you are automating stuff, these metrics will decrease dramatically! One of the first things you start doing is implementing automated testing at the source. That way you’ll even detect errors before anything else is being triggered! So the change will not even have gone “live”.
When looking at automated deployments… this puts you in a position (and you should do this!) to facilitate rollback scenario’s. Because you want to be able to decrease that “MTTR”.
So do you have to throw away ITIL/COBIT/… processes? No, but there will be change in the content of those processes.
What will DevOps do to my organization?
This is a question I also get A LOT. 🙂 And with due reason! I see a lot of companies who create a “DevOps-team”, or bring “DevOps Engineers” functions to life. And often the first question I’m asked is ;
“Do I need to put everyone into one team?”
If the size of your organization allows it… then that would be great. Though in a lot of cases, you’ll (eventually) run into the physical limits of housing people.
The second most asked question is ;
Will everyone do dev and ops?”
The only correct answer here is no. Though… everyone should be aware of the nuances that each role/specialization has. When looking towards development, you see a “Full Stack Developer“-trend that’s been going on for a while.
To be honest… You are going to need a mix of both “generalists” (who cover the width) and of “specialists” who know the in and outs at low-level. In my mind, the “generalists” will become the “sweepers” (or “libero’s”) or your organization. They’ll need to jump into the holes / grey zones left between the specialists. They are “the starch” between technologies. Anyhow… I’m drifting off here. 🙂
Despite your organizational structure, your first main objective is to tear down possible invisible walls between departments!
This is the thing you should be focussing on! If this requires you to create a separate team… Then think again! That team will become isolated as they need the other guys/girls too. In some cases, having a separate team may be handy for agility purposes, though be aware that communication between the entire (IT) organization is critical.
So will I need to change the structure of my organization? No. But you’ll have to change the way communication & responsibility flows!
You do not want islands… You want “dev” and “ops” to be part of the entire life-cycle. So when you see the symptom that the dev & test environment are the responsibility of the “development”-team, and acceptance & production are solely in the hands of “operations / infrastructure”, then you failed!
Ops should be involved from the start… from genesis of anything. And dev should be involved beyond genesis, and also endure the pains of having to operate a possible Frankenstein monster.
Does that mean that there won’t be any friction? No. But when everyone get acquainted with the entire flow, then everyone can take the entire flow into consideration.
One of my mantra’s has always been ;
“The monkey walks into the direction of the carrot!”
So if you say to someone who they get a carrot when they maintain environment X, then they will not care about environment Y & Z. The carrot should be on the entire flow for everyone!
So how to shape DevOps in a large organization?
The basic starting point ;
Change “the carrot” of your “leads” to one that covers the entire lifecycle.
This will create a shift in their behaviour… Now ensure that the ones with an “architect”-role communicate with each other on a continuous manner. There should be no wall between them, they should be low latency peers! Architect A should not be thinking of X without sparring with architect B. And so on…
It’s about doing IT together!
But we are an organization with high security requirements!
Great! You should have already been doing DevOps…
Now you’ll probably will think ; “He’s lost it… I do not want to give my developers the power to annihilate my production systems!”.
And you are correct. Nobody should be able to annihilate your production systems. Nobody… Not even your operations people.
With test & deployment automation, you’ll go into a modus where you ;
- have an audit trail of ALL the changes
- where operations people will not need elevated rights on production to make changes
In addition, you can introduce approval flows to prevent unwarranted releases to environment X/Y/Z.
So, do you agree with me that we actually increased security on this one? I guess we can. 😉
What about DevOps and operational support?
DevOps is about automating stuff… and if we are Facebook (with 600 releases per day) or Github (with 400 releases per week), we are faced with a situation where operations will have to maintain an environment they know jack about?
The knowledge of the environment has suddenly become documented. All changes are done by code, so you have a full audit trail on it. And you can always check the state at given points in time. Though, I must admit… the way of doing things at heart will change.
Will you go directly from 0 to 100? No. We’ll be eating up that elephant in small chunks.
So at first you’ll probably introduce an approval flow before going to production, where you will say that certain requirements will have to be met. A practical example I can give your from my early days in a DevOps environment ; Before going into production, one of the requirements was that the operational team needed to sign of on a release. This was mainly focussed towards delivering a given set of documentation and having operational requirements (like monitoring, high availability tests, performance tests, etc) validated.
As a bit of an odd anecdote on that one, there was one “joker” into that process. We all know the “emergency” changes because it could not get stopped. By operations, it was allowed to get a joker on the requirements, though under one condition. The escalation flow for the systems that were referenced by the change would be switched to the mobile phone of the ones requesting the joker.
TL;DR or “Closing Thoughts”
- DevOps does not mean “throw away your current processes”. It’s about automating and remove “(time) waste” in processes.
- Technology can greatly help you on your journey. But it starts with people… and it keeps revolving around people.
- Break down the barriers between teams. If they are kept, then all else will fail. You should all be working towards the same goal. Not towards personal goals that are not aligned on macro level.
- Do your transition in steps… You do not need to go from 0 to 100 in 5 seconds. Take your time, but do each step properly!
- The best time to plant a tree was 20 years ago. The second best time is now… Do not dwell about it, set your goal and start doing (small) changes.