Refine requirements with user stories, break up the ops team, and get full self-service automation and testing.
- We'd come through phase two of our lean and agile dev ops implementation at Red 30 Tech. Remember, while this specific story is fictional, it's composed of real experiences that Karthik and I have had during similar implementations at a variety of companies. - We had proven our approach on one project, and then other projects organically adopted our approach and platform. It was time for us to take this wider to the rest of our department. - So that's what we did. We had integrated devs and ops onto our initial Greenfield team, but otherwise, we still had a big, separate ops team.
- While my dev teams appreciated all the hard work that the ops team did, our analysis showed that they were still the bottleneck for many of our projects. One ops team trying to serve 10 dev teams wasn't working, and our value stream map reflected the wait time we were experiencing. - This was efficient from a local optimization point of view, less spend on ops. But much less efficient from an overall throughput point of view. Saving money doesn't save you money if it stops you from making money. - Hey, I'm pretty sure Snoop Dog said that. - True dat.
So we made the somewhat shocking declaration that we were discontinuing the operations team. Instead, we embedded several ops engineers onto each agile team. We had to hire a couple more to get the ratio right. - Everyone supported this. Since we were sharing sprint backlogs, everyone had visibility onto the huge number of tickets that ops had, required for every project that had previously been completely invisible. Devs even pitched in, learning on how to take on security, backups, reliability, performance, and deployment tasks that in the past, they just assumed someone else would handle.
- Also, we created a new team to create ops platforms. This team would provide the platforms for build, deployment, monitoring and other operations needs, but with an important difference. They would provide self-service platforms or services, but they didn't do work for people. - [Karthik] One of the worst practices we uncovered was the use of manual tickets. We thought that we had gotten rid of most of them with QA automation, but then we found that the ops team squandered most of their time performing trivial tasks like getting logs for developers, restarting services, and so on.
- We didn't need anyone performing low value work in our organization. Believe it or not, there are automated solutions for getting logs and deploying apps. So there were some ops engineers on each team, working in their sprint. And there was an ops platform team consisting of developers and operations experts, creating solution platforms. - This drove everyone to fix the problems the right way, instead of just dumping tasks on another team. - We made one other major improvement in this phase.
We had a backlog full of requirements, but those requirements were often pretty technical in scope, and it wasn't always clear how they should work. They'd passed testing, but were they the right thing? Were they adding value to the customer? - One of the things we learned was to change the way we were logging requirements, and started to state them as user stories on both the dev and the ops tickets. - [Ernest] A user story is usually stated in the form of what kind of user it is, what they want to be able to do, and why.
We'd started to have tickets implemented in a way that seemed valid to the engineer, but since they lacked context, when it got to the users, it didn't really fulfill their need. - [Karthik] That's right. So the PM's developed user personas. The normal person printing, the IT person administering the print solutions, and the procurement agent, and so on and so forth. And then they started writing requirements that way. - This helped a lot. Instead of just, add this field to this page, we instead heard, this kind of user needs to see this information to perform this task.
This greatly enhanced collaboration between product and the engineers on how best to accomplish those goals in the product. - It also provided a new role for the QA folks. They aligned more with the PM's and made it their first priority to understand the customer, so their testing would be more than just, is this button where the ticket said it should be? Instead, it became internal expert user feedback. - Next up, we'll talk about how we started to address metrics and measurement as we spread our new scheme department-wide.
- What is agile?
- What is lean?
- Measuring success
- Learning and adapting
- Building a culture of metrics
- Continuous learning
- Advanced concepts