I've got to hand it to the Agile development guys — they were really good at liberating money out of organizations that all had trouble with something inherently difficult. The geniuses who developed Scrum and Extreme Programming executed masterfully; selling books and training; and they made some serious bank doing it. If you hang around Silicon Valley long enough, you know to applaud the hustle. It's the classic Rainmaker scam. You pay a man to make it rain on your crops, and when it rains, he takes the credit. If it doesn't rain, he comes up with an excuse that involves you paying more money.
So, given that, I'm befuddled by the Devops movement. It's got the potential to make a handful of people a lot of money in the same way that Agile did, but nobody is really executing on it. It's proper snake oil with all the trimmings: prescription of "culture change", few and vague concrete steps for implementation, and most of all, the promise to solve an age old problem.
How do you implement Devops?
Point one. Nobody seems to know. At least with Scrum, you could buy the book and take the course. From what I have gathered by reading blogs, if you want to apply Devops to organization, you do any or all of these things:
- Automate configuration with Puppet or whatever. You should be doing this anyway. Not earth shattering.
- One-step build and deploy. I'm still waiting for you tell me how these steps will solve my problems.
- Culture of respect & trust, good attitude toward failure. How about "culture of stop fucking up"? This is one of those obvious happy-horseshit type statements that makes you believe the salesman is benevolent. A developer who consistently ships broken code or a sysadmin who consistently pushes out broken configuration aren't going to get any better with respect or trust.
- Have cross-functional team members to facilitate communication Every technical person you hire should be cross functional enough that you don't need this.
One of the things I read frequently is that Devops is about building bridges and communication. What the shit does that even mean? Cute, but not useful. Clearly everybody deserves to be treated with respect in the workplace, but you can't make two different groups work together just by telling them to, or even by having cross-functional team members to coordinate. If you've hired people explicitly as peacemakers between development and ops, you fucked up somewhere in your hiring process; it's fixing a self-inflicted problem.
If you are going to pimp this stuff as "the new way of doing things", at least try to sell me a book.
What is the problem you want to solve?
The main issue with the Devops movement is that it treats symptoms, not problems. Yes, everybody wants to ship new code frequently and keep it stable, but the dev vs. ops feud is as old as the phrase "it's 98% done, I just have to test it". The symptoms of the problem are these:
- Developers write code on their workstations and it doesn't work in production.
- Systems administrators are slow and reluctant to change production configurations.
The underlying problem, however, is that dev and ops have different goals, and each group's problem solving skills is a product of those goals. The Devops movement does try to cultivate some kind of understanding that developers and systems administrators are both working toward the same end, to put food on the table, but you will never be able to effect cultural change just by saying so. Surly's only looking out for one person, and that's Surly.
What condition your condition is in
You will always have problems between development and operations if the two groups think so differently about technical problems. So, I offer a test. One technical question that will show you how different development and operations really are:
Devise a caching infrastructure for responses from the Google Maps Geocoder API.The end. Gather dev in one room, and ops in the other, and have them each come up with an answer. If they come up with the same answer, there is hope for your organization. If they don't, put them in one room and have them work it out until there is a unanimous solution, and everyone agrees that it is the best. If they can't agree on a solution, you have problems that no methodology can fix. (For bonus points, make this a universal interview question.)
After that exercise, development and operations should reasonably be on the same page. Next, you need to implement policy that will force convergence between development and operations. This is my prescription:
- Developers are in the on-call rotation If you ship a feature, you help support it. This one is first because it's the most important. If you architect a system, you write the Nagios alerts for it, and they page your phone. Believe me, you will get a crystal clear understanding of why ops throws up so many roadblocks after doing this for a few months.
- Developers develop in the same environment production runs in If you deploy to Linux, you develop on Linux. No more of this coding on your Macbook Pro and deploying to Ubuntu: that is why you can't have nice things.
- Downtime never happens twice After problems are fixed short-term, you make it first priority to ensure that the same failure does not happen again.
Since no methodology peddler ever wants to say this, I will: there's a point where you're simply fucked. Meaning, you can't solve the problem with the tools available. Sometimes, you have to fire people who aren't working out. Sometimes, you're too deep in technical debt and too pressed for time to do it the "right way". And sometimes, projects fail. It is what it is. This isn't defeatist, it's realist.
I am not trying to sell you a book, I am just being honest about the problems you face. None of this amounts to a methodology, as the Devops people would have you believe. If your developers and your sys admins are so culturally different that they can't agree on a solution to a simple technical problem, then your organization will not be fixed by some sunshine-up-your-ass methodology you read about in a blog or hear about at a conference. You need to change the culture the hard way, or replace people as necessary until the culture works.
The Devops movement smells of a scam in the making, not that I have any problem with that, after all, don't knock the hustle. However, I'd rather not see people with real problems get roped in, thinking that there's a magical 12-step program that will solve deep rooted problems. It just doesn't work that way. Even so, the Devops people have a bit of traction, and they're failing to capitalize on it. You've got a good thing going here, profit from that shit. Books, training, conferences, the whole bit. Get down to it.