Chris Cooney - Modern state of agile

Tim Butara: Hello everyone. Thanks for tuning in. We're joined today by returning guest Chris Cooney, the Head of Developer Advocacy for SaaS Observability Platform Coralogix. In our original episode with Chris, we talked about data observability and how it can help or must enable digital transformation. And today we'll be actually talking about the modern state Agile.

And this is a really exciting topic that I'm really excited to get in with you, Chris. So welcome back to the show. Very happy to have you back.

Chris Cooney: Thank you very much for having me back. This is fantastic. I really, really enjoyed the first episode we did, and hopefully this is the second of many, many more episodes, but yeah, this is brilliant and it's a fantastic topic, extremely relevant right now.

And as the whole industry is kind of taking a breather and looking at itself and deciding whether agile quote unquote agile is the way to go. So yeah, this is a brilliant, a very timely episode.

Tim Butara: So let's first get one thing out of the way. Are we talking about lowercase agile or agile with a capital A what's the difference and why does the difference matter?

Chris Cooney: Yeah, it's, it's, it's a fun one. It's one of those things where we could have just come up with two different names for it. But that would have been too easy. So we've decided to have a convention where it's either capitalized or not, generally speaking. And I'll try and phrase this cause I am extremely biased on this.

So I'll try and phrase this as fairly and as objectively as possible. Lowercase agile refers to the philosophy is set out by the agile manifesto. So the agile manifesto was a document that was written some time ago, decades Review and assess the assumptions that we've made in the software engineering industry.

So, if I was going to say lowercase agile, I would say that it's something like this. It's the manifesto goes individuals and interactions over processes and tools. Working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan.

And it does go out to say as well that the things on the left, the individual's interactions, working software, they are valued more than the things on the right, but it's not that the things on the right are valueless. So processes and tools still have their place. Comprehensive documentation is still very important.

And so on. So that anything that's tightly related to that, I would call lowercase ijl. That's about like the It's more the philosophy and the, the, the, the general approach. Then there is uppercase Agile. Uppercase Agile would be things like frameworks, certifications, general approaches, so things like scrum, Nexus, less, scrum of scrums safe, or all of the many different things.

These are all more formalized approaches to Agile. sometimes disparagingly called the Agile Industrial Complex, I think is the is the thing that comes up a lot. But they are all, I would say, more formalized approaches to Agile and often catches quite a bit of criticism that side of things. There's some justified, some maybe unfair, but, but yes, that's the distinction.

It's one is the core philosophy and the other is the whole world of frameworks and certifications and roles and all sorts of things that sit around that philosophy.

Tim Butara: So, which one are we going to talk about today, or is it going to be both a little bit?

Chris Cooney: I'd say that it's going to be both, because they are often confused, one for the other.

So, if I give you an example of that kind of confusion, I was the Principal Engineer in a previous role, I think I mentioned this in the first episode we did, but when I was a Principal Engineer, I was regularly going to teams and assessing how they were working. and how they were collaborating and how they were breaking down stories and so on.

And what I found was the teams that came to me said, we hate agile. I said, okay, what is it about agility that you don't like? We hate story points. And I thought, well, just don't use them then. It's fine. Don't worry. That's a classic. Mixing of lowercase and uppercase because uppercase agile is the story points.

It's the process and the approach that the formalized approach and lowercase agile says, no, no, no. There's an outcome you're trying to drive, you know, which is conversation and collaboration. And the story points are supposed to be a vehicle for that. So that was a really classic example where people mix it up.

And that's why I think we're going to end up talking about both because people often. Mistranslate the two and they mis misallocate the two depending on what their prior experience is and so on.

Tim Butara: Would you say that this mistranslation is an important factor in, you know, companies maybe not having success with adopting agility?

Chris Cooney: Yeah, I think there are a few things. So there's a recent study that was released. I say recent, maybe about six months ago or something. It's the first time I saw it, which was that there's a 268 percent higher failure rate for companies that adopt agile processes versus the ones that don't. And the study finished up and found that the two biggest factors that drive project success are a thorough assessment of requirements upfront, which would be like completely, completely wrong.

you know, anathema in many sort of Agile circles, I suppose, but the, the other thing was psychological safety, which is defined as the ability to raise concerns, speak your mind without the fear of reprisal. In other words, it's surface your mistakes is often the thing that Agile that psychological safety is often touted for.

It's the ability to say, Here is a problem that I have caused and I wished we need to fix it as a team. Those are the two factors that drive project success, product success that this study found. And it did not find that poker cards and, and story points and scrum and all this kind of thing. It did not find that that drove the outcomes that is expected.

However, the success, if you like, of. There are some justified critiques of this study, by the way. One is the definition of success. The other is the definition of agile, you know? So organizations, the, the, the, the, the phrase that I hated the most, I despise this phrase was, but that's not agile. And it bothered me a lot because that in and of itself, ironically, is not an agile mindset because it's like, you're focusing on the.

process and not, not on the point. So it really bothered me that phrase. And I, I, I made a massive effort to scrub it from my vocabulary. I say it's the kind of thing I used to say when I was like a junior engineer and I was like dogmatic about things. And as time went on, I became more experienced. I realized that reality requires some flexibility.

And so the dogmatic approach to which people have adopted Agile in the past has hurt them. And it's because they're coming from a. world of things like Prince2, or, you know, very, very, quote unquote, waterfall methodologies that require strict adherence to process, you know, and they promised that the trade off is basically adhere to Whatever formalized process has been put in place that will increase predictability and consistency.

And that will mean that projects finish on time. That's the very high level gist of some of these waterfall things. And the problem of course, is you can't predict the future. And Agile's response to that was, well, let's do less planning upfront. Then we'll kind of respond to change over following a plan to quote one of the lines in the Agile manifesto.

The problem is, is that. That mindset shift never really happened in a lot of companies. And they basically said, okay, what are the new rules we're following? And it was like, Oh, don't plan anything up front. Okay. You know, and, and so I think that's a really legitimate critique of this study and this, this general massive back class against agility is that.

The amount of people who were very resistant to thorough upfront analysis of requirements that I met under the flag of agile because they, they thought that, Oh, no, no, no, we should do the bare minimum analysis just to get moving and then we'll worry about everything else empirically. And I thought, should we not like spend a bit more time thoroughly understanding what we're doing?

And then it's fine. We can learn as we go as well, but. And I'm not saying that we're going to predict the future, but just better understand what's going on. So yeah, the definition of agile is incredibly important when we're deciding if it has or has not been successful. And that's why the big AI agile, the little AI agile thing is important, but also the, the missing pieces across many, many companies that have driven some of the more negative outcomes that companies have experienced.

It's pretty interesting as well.

Tim Butara: And how does all this relate to the two conflicts, one of process and framework and the other of dogma versus pragmatism?

Chris Cooney: Yeah. So I would say that it links to what I suggested before about. These companies have gone from very strict frameworks. So I'll give you an example.

Safe. Have you seen the safe framework?

Tim Butara: Yeah, I think I've discussed it even, even here on the podcast in one episode a while back.

Chris Cooney: So the scaled agile framework is the, is, is. I am not a fan personally. It is, it is, it's so complex. It's so rigorous. It reminds me of the Prince two handbook that I was forced to read several years ago, Prince two would be a much more, let's say waterfall approach to software engineering, but it's also.

Much more complex and scaled Agile framework is in that kind of category. It's extremely complex, lots of roles, lots of processes as a release train. There's architects, there's this, there's that, there's the other. And I, I always thought, why, why are things this complicated? What's going on? And it's the only, by the way, the only Agile framework that you 10 minutes.

Like, like any other Agile framework that exists, you can understand very quickly, except safe. Safe is the one that takes a really long time to. To master and understand, I always saw safe as the perfect example of the, of people going into this with like a waterfall Prince to mindset and going, okay, I'm going to just follow the agile principles and just, just don't blindly go after them and things are going to be okay.

Yeah, it's like

Tim Butara: Agile for the people who don't want to do Agile.

Chris Cooney: Yeah. It's so they can go to their investors and say, we're doing Agile. But really they're just doing, you know, the really important thing that I always thought was fascinating was the, the every time I would critique safe in public, I would always get, Oh, we use safe.

And it was amazing. It completely changed everything for us. And then when you dug, It was, there was always almost always a core of like 10 people who just worked really hard and they just like, they, they, they were really creative and they were really talented. And I always, one of those things that really disillusioned me with the whole agile movement was actually a lot of it just comes down to having talented people.

Like if you put talented people in a waterfall environment. You'll still probably get okay software. And if you put talented people in an agile environment, you'll still probably get okay software. And if you get talentless people and you put them in either environment, things aren't going to go well for you.

So I started to see agile as maybe less of a, of the driving force and the sort of revolutionary drumbeat that it was supposed to be, and, and that, that I think has been driven massively by organizations just following the cargo cult, just bringing in some consultant from somewhere, some agile consultant expert who's come in and gone, Oh.

You need to have a quarterly product backlog roadmap, and here is our quarterly backlog roadmap package. It costs 25, 000 a month. We'll make sure your backlog is amazing. That's why they call it the Agile industrial complex, by the way. This is the aspect of big A Agile that engenders a lot of critique.

And in this particular case, extremely justifiably, I have met so many snake oil salesmen in this space. It's unbelievable. And the amounts of like Agile meetups I've been to that were just Onboarding ramps for people's training courses and things. And that was pretty much all they were. So, yeah, I, I think this mindset and this almost desire to maintain the comfort and the fake certainty of waterfall organizationally has driven this industry of, you know, needlessly complex, fundamentally insane frameworks that don't really drive the outcomes they're supposed to drive.

And that has massively derailed what was originally referred to by the way, as corporate anarchy. That was what Agile, they would sometimes call themselves, the original signatories of the Agile manifesto called themselves corporate anarchists. And it was, it was, it was an act of rebellion and like an almost a revolutionary mindset in this kind of thing, you know, anyway.

It's kind of a different way of doing things, really exciting, really radical, you know, all these ideas. And for it to then just be adopted by the very group of people against which it was supposed to be a revolt is really, I think, where the failure and the decline started to happen.

Tim Butara: Yeah. I mean, we could go really deep on that one, but I'm interested.

So how can we improve? How can we, you know, resolve this, these conflicts, I guess? Yeah,

Chris Cooney: I think so. So whenever I was doing the coaching, I found that the best thing I could do was quickly get to the problems that the customer was trying to solve, the person was trying to solve. So in this case, I was working internally and I would go to teams and they would tell me, Oh, we're doing planning poker.

We're doing scrum. We're doing this. I'd say, great. If scrum works for you, scrum works for you. You're brilliant. It's, it's, you know, it gets a lot of bad press. It's actually pretty good. Awesome. And then they would say, Oh, but we were following scrum, but we just don't seem to be having the conversation.

You know, we don't be our backlog is a mess, you know, we're doing three hours of backlog refinement every week. And it's just not, nothing's happening for us and blah, blah, blah. And I would say to them, okay. And they go off, but, but scrum demands just one. I'll answer no, it doesn't it says you should allocate some time to refining improving your backlog It doesn't say how much time it doesn't give you a ceiling of how much time it just says look Here's a rough idea of the things you should be doing And I would always say that let's start with the outcome you're trying to drive and then work backwards So the outcome you're trying to drive is a nice clean backlog.

What are the kinds of things that you can do? that are going to drive that outcome. And they would go, Oh, it's possibly the, well, actually it was spending lots of time on the backlog. It's not working. Maybe there are too many items that are hot. So maybe we need a session where we just start deleting things, you know, always start archiving things.

How would that work? Maybe we need to be strict about what makes it onto the backlog. And what I found as soon as I gave them the first. first step, they would then follow that journey really well. And they would, they would, they would stop the, the word agile, which is evaporate into nothing. And all that would be the outcome of trying to drive.

And then I would always link it back and say, well, okay, now you have a nice clean backlog. What is that going to do for you? Like, what, what do you, what do you think is like, well, it's gonna make it really easy for us to see. the direction that we're going and give us that, like, view. Ah mate, so you, do you need a roadmap then?

That's, you know, and then it would link to the next thing, the next thing, the next thing. And what would generally happen was we, from first principles, would end up deriving many of the Agile principles anyway. But it would give teams a clear reason as to why they should do those things. It would break the cargo cult.

If there was one thing that I would say, As if anybody whose job it is to go and coach on agile practices, or just to be honest, just good software engineering practice, as we understand it today, it would be to break the cargo cult as thoroughly as possible, stop people from doing things just for the sake of them, stop people from doing things for busy work, or just on faith that they're going to do something.

Everybody has to understand why the team is doing the ceremonies that they're doing. I think that that would be the. Most painful thing that you have to do, but it's also the most valuable thing. And it breaks down that dogmatic attitude towards framework and practices and that kind of thing and gets people thinking about the team as an organism as something that needs to drive outcomes.

Tim Butara: On one episode, a really interesting one, we discussed, we mentioned two terms and they were agile theater and agile coldness, right? And agile theater is kind of you know, waterfall disguised as agile, while agile cultness is this adherence to these processes and ceremonies over actual agility, right? So, so you, you just demonstrated how basically you can get out of those two ruts and into like actual agility.

Chris Cooney: Oh yeah. I'll give you an example as well. I, when I was, I was actually in this team, we were building a solution on a website, a customer needed to understand all of the ad space they had on this, on the page, which was like designated HTML elements on the web page. They needed to basically lift all of the data about those ad spaces, dimensions, sizes across different screens and so on, and then make it available for people to bid.

And say, oh, I want to pay 1, 000 and I want to have the headline for 10 minutes here, you know. And then the solution obviously had to swap out the ads and things as well. And I remember at one point waking up one day and realizing we were doing sprints for two weeks sprints. It was basically sprung. With some Kanban principles in there, we're managing flow and of the work and that kind of thing.

And let's say about 25 percent of the way through my, I realized, I was like, all we've done here is take a waterfall plan, slice it up into two week segments. And then, you know, and that's it, that's all we've done. And, and, and they were, and it dawned on me because every, Month, we were producing these estimates.

We will be done in 10 sprints. We will be done in, you know, and I realized like at that point, all we were doing was just doing waterfall in segments. And I, and I, and I realized that the reason why I believe the cargo call is to blame for that is because at no point did we question, okay, why are we bothering with any of these ceremonies?

We are, we know where we want to get to. And we've been, the business has been really clear that anything less than this is pointless. So. Are we, do we actually think that these reflection points are helping us, or do we just need to get on with it and have reflection points as a team to improve the way we work?

In other words, the retrospective without the review. Because every time we did the sprint review, we would show the stakeholders and they would go, great, but it's not what we asked for yet. You know, yeah, yeah, we know, we're getting there, you know? And so, you know, And it's because I didn't understand why we're doing the ceremonies.

If we'd stopped and questioned that, I would have gone, Oh yeah, this is pointless. And that would have started a great conversation about growing into the solution, negotiating with the stakeholders. Oh, maybe we can drop feature X, feature Y, feature Z. And it would have, it would have changed how we built the software actually completely.

But yeah, so it's just that, that, that experience to me really solidified. how easy it is to sleepwalk into quote unquote, waterfall engineering by the back door and to just, just perform agile theater because it makes you feel good. It's like, Oh, how agile we're being, we're doing sprint reviews and everyone's got cards to estimate things.

And it feels nice. This is the, the the sort of subtle allure of Big A Agile is like the, It feels nice to do it because you're doing all these things and it all feels very, very new and very cutting edge and very fun. And none of it is driving value. And I think that's the thing that was the big message that was lost over the past few years was that the whole point of Agile is to bring the engineers closest to the value and have them iterating on the value as frequently and as vigorously as possible.

And instead they've just, it's just now like, It's just dev saying, well, that's not agile. Like what needs to go into the backlog. We need to follow the process and so on and so on and so on. So not great, but yeah.

Tim Butara: So based on everything that you've discussed, how do you expect or maybe how do you hope things to change or move forward or evolve in the whole Agile industry?

Chris Cooney: Yeah. So I think that what tends to happen in almost any. Domain of study is that there are trends and those trends come and go and they leave their mark behind. So like, just to give you an example, like architecture, there's classical architecture and then there's neoclassical architecture and.

Neoclassical architecture is an attempt to revive classical architecture, but it's clear that classical architecture had its time and then it also massively impacted the subsequent, you know, like the Renaissance is a massive, just point to classical architecture and so on. So. In the same way software engineering was very much about planning upfront.

And then the Agile thing comes in and we all start planning in, in, in increments and we start being more empirical and learning as we go and so on. That isn't going to go away. So while Agile itself may become a dirty word, many of the underlying principles are here to stay because they are, they have helped us to accelerate the, the sort of, the, scale and pace of software engineering in order to meet the demands of the modern economy.

It has worked many of the things that has worked. So this idea that agile drives higher failure rates, I think it's, maybe it's It's, you know, it's, it goes back to that thing of defining what actually is it for a company to take an Agile approach. I think that the most important thing that we can recognize over the next few years is that while Agile will become a dirty word, the underlying mechanisms have become part of the bedrock of software engineering.

So engineers now are coming out from universities and they are ready to ask difficult questions constantly. They're ready to constantly pivot. Phrases like scope creep are no way near as big as they were when I first started like 10 years ago. We talked about scope creep constantly. It was like the thing.

Oh, that's good. No, no, that's scope creep. That wasn't in the typical and now engineers are just a bit less sensitive to that sort of thing like oh, yeah stuff changes things happen, you know. This attitude to uncertainty has been built into us By the agile movement. And so I, I, I believe that those benefits are sticking around.

They are here to stay. It's gonna be a very long time before we see those things begin to erode that the driver for that will be economic rather than anything that comes from within software engineering. However, the frameworks and things. They are, that's going to be the real battleground. I think those are in real trouble.

Scrum, especially like scaled Agile framework. I think these frameworks have had, this is what I think the article defines as Agile, by the way, someone applying safe and then their project fails. I think. Because of people's inability to decouple the process from the value, and because of the tendency it would seem for companies to be dogmatic about these frameworks and processes, I think that they are going to be a major casualty.

I think they're going to be rejected. Almost summarily over the next few years as organizations move towards more of a, I imagine I can't swear on this podcast, so I'll just say a GFDI process, which is just do it. It would be the PG version and that model of just get on with it is becoming a lot more popular now.

And we can do that. We couldn't do that a few years ago, like as an industry, it didn't seem that we were like. Responsible enough to have just, just have JFDI processes and everyone just gets on with things. The impact of agile has been, I think, to unlock the ability for engineers to just get on with things.

And this was the case in the eighties and the nineties when I, when engineering was like, Very much in the back offices. Now it's at the front of a company. Engineers are working with every member of an organization, including customers. And I think that agile has enabled that it's accelerated that process towards being able to just get on with things.

So I think that's the, that's the mark that agile leaves on the industry. It's, it's unlocking the ability for people to just get on with it. We're moving from being consciously agile to consciously process obsessed slash Unconsciously agile and we're consciously we're just getting on with things. I think that's the transition we're experiencing now and that's what this back question is all about.

People don't feel like they need the frameworks anymore. It's like a kid has learned to ride a bike and you're putting the stabilizers back onto the bike and the kid will hate it. No, no, I can't go fast anymore. What are we doing? You know, you see they want the stabilizers off and I think that's what we're experiencing as an industry.

Tim Butara: I think those were some excellent observations and some great points to finish this particular conversation on Chris, another excellent one. You need to send me a link to that article or the study that we talked about so that we can add it to the show notes along with our first episode. And of course, if anybody wants to contact you, get in touch with you or learn more about your company, where can they do all that?

Chris Cooney: So I'm, I'm on LinkedIn. If you look for Chris Cooney Coralogix, I'm there. My email is chris.cooney at coralogix. com. Feel free to email there. I'm terrible for applying to emails and or just reading them in general. So I promise I'll get back to you as soon as I can. But yeah, it's, I've really, really enjoyed this conversation.

And I think that the topic is a fantastic one. It's extremely fascinating. And I, I am very excited to see where it goes over the next few years for sure.

Tim Butara: Yeah, it will definitely be interesting. I really like your predictions and, and yeah, some great points, great insights. Thank you again, Chris, for your time, for the great talk, your unique insights and unique perspective and till next time.

Thank you very much. Well, to our listeners, that's all for this episode. Have a great day, everyone. And stay safe.

Outro:

Thanks for tuning in. If you'd like to check out our other episodes, you can find all of them at agiledrop.com/podcast, as well as on all the most popular podcasting platforms, make sure to subscribe. So you don't miss any new episodes and don't forget to share the podcast with your friends and colleagues.