The Fog of Development – Why the Beginning of Software Projects is When We Know the Least

As the host of the Pulse by Intuji podcast, I’ve had the privilege of speaking with multiple distinguished industry experts about the challenges and triumphs of modern software development. Recently, I was joined by Maarten Dalmijn, an experienced agile and product management consultant, for a conversation that left me with numerous insights I’d like to share some more thoughts on.

Embracing the Unknown ❓

One of the concepts Maarten introduced was the idea of the “fog of beforehand” – a poetic way to describe the uncertainty that shrouds the beginning of any complex software development project. As someone who’s been on both sides of the fence, both as a service provider and a client, this resonated deeply with me.

Maarten explained, “Before we start, we simply don’t know enough. But as we do the work, this fog is going to slowly be removed.” This perspective challenges the traditional approach of extensive upfront planning, which Maarten aptly termed the “fog of speculation.”

The crux of the matter is this: at the outset of a software project, we’re at our point of least knowledge. We don’t just lack information – we often don’t even know what information we’re missing. It’s a classic case of “unknown unknowns.”

Maarten drew an interesting parallel to weather forecasting. Just as meteorologists become more accurate the closer we get to the date in question, our understanding of a software project becomes clearer as we progress. The challenge is in accepting and working with this uncertainty, rather than trying to eliminate it entirely.

This concept flies in the face of traditional project management approaches, which often demand detailed plans and timelines upfront.

As Maarten pointed out, “Our natural response when we lack information is going into meeting rooms, talking and talking, writing everything down. And that’s the worst thing you can do because all you’re doing is you’re writing all your understanding down that you have before starting, which is actually a point in time where you have the least understanding and the worst information.”

In essence, by trying to plan everything in detail at the start, we’re not reducing uncertainty – we’re simply codifying our ignorance. We’re creating what Maarten calls “the fog of speculation,” which can be even more dangerous than acknowledging our lack of knowledge.

Instead, Maarten advocates for an approach that embraces this initial uncertainty. He suggests starting with smaller, manageable pieces of work, and using the insights gained from these to inform future decisions. It’s about creating a feedback loop that allows us to learn and adapt as we go.

In my experience, and whilst it is always our intention to get up to speed as quickly as possible, this acceptance of initial uncertainty is crucial. I’ve seen projects falter because we tried to plan every detail from the outset, only to find our carefully laid plans crumbling in the face of reality. Maarten’s approach of embracing the fog and adapting as we go aligns perfectly with the agile principles we practice.

This approach requires a shift in mindset, not just for development teams, but for stakeholders as well. It means being comfortable with saying “we don’t know yet, but we have a plan to find out.” It means valuing adaptability over rigid adherence to initial plans. And most importantly, it means recognising that our understanding of the project will evolve over time, and building our processes to accommodate and even leverage this evolution.

As Maarten put it, “The further along the way we are, the more we will know, the better understanding and the better decisions we can make.” By embracing the fog at the start, we set ourselves up for a clearer vision as the project progresses.

The Trust Equation 🤝

Another key theme of our conversation was the importance of building and maintaining trust with stakeholders. Maarten emphasised the value of delivering tangible results frequently, saying, “Seeing tangible results, I think that builds trust. In moments where it’s all underwater, nobody sees anything… that destroys trust because they feel like they’re being played.”

This struck a chord with me. I’ve seen projects where we’ve struggled to engage key stakeholders, often because we weren’t demonstrating value in a way they could easily understand. However, I’ve also realised that this is a two-way street. While it’s crucial for us to communicate our progress effectively, it’s equally important for stakeholders to commit to understanding the complexities involved in these projects.

In my experience, the most successful projects are those where stakeholders invest time and effort to engage with the process. They need to be willing to learn about the unknowns and uncertainties inherent in complex software development. Without this commitment, our efforts to educate and inform can fall on deaf ears.

I’ve learned that it’s vital to set clear expectations from the outset. We need to communicate that our job isn’t just to write code or build systems, but to guide stakeholders through a journey of discovery and adaptation. At the same time, we need to make it clear that we expect stakeholders to be active participants in this journey, ready to invest their time and attention to understand the nuances of the project.

This mutual commitment creates a foundation of trust and respect. When stakeholders are truly engaged, they’re more likely to appreciate the value we’re delivering, even when it’s not immediately tangible. They’re also more likely to understand and accept the need for flexibility and adaptation as we navigate the “fog” together.

The Fixed-Price Conundrum 🏷️

Our discussion on fixed-price contracts versus more flexible approaches was particularly enlightening. Maarten likened fixed-price contracts for complex projects to “gambling,” and I couldn’t agree more.

At Intuji, we’ve learned the hard way that fixed-price contracts for large, complex projects often lead to lose-lose situations. Either we end up absorbing significant costs to deliver on our promises, or the client ends up with a product that meets the letter of the contract but falls short of their actual needs.

Maarten’s suggestion to “keep it as simple as possible” and start with smaller, more manageable pieces resonates with our current approach. It’s about finding that balance between providing certainty for the client and maintaining the flexibility to adapt as we learn more about the project.

Agile – A Mindset, Not a Rulebook 🧠

One of the most valuable takeaways from our conversation was the reminder that agile is a mindset, not a set of rules to be blindly followed. Maarten pointed out the proliferation of terms for “Scrum that doesn’t work,” which I found both amusing and insightful.

It’s a trap I’ve seen many teams fall into – following the ceremonies of Scrum or another agile framework without really understanding the principles behind them. As Maarten put it, “Very often in reality, scrum doesn’t work. You know, it’s what people are just trying to like follow all these steps and then they’re like, okay, we’re doing scrum, we’re being agile.”

This reinforced my belief that we need to be more flexible in our approach, adapting our processes to the specific needs of each project and client, rather than trying to force every project into the same agile mould.

Looking Ahead 🔭

When I asked Maarten about what businesses should be paying more attention to, his response was thought-provoking. He emphasised the need for new ways of working that preserve the agility of small teams even as organisations grow.

This aligns with challenges we’re facing at Intuji as we grow. How do we maintain the nimbleness and responsiveness that made us successful as a smaller team? It’s a question I’ll be pondering and discussing with our team in the weeks to come.

Final Thoughts 🤔

My conversation with Maarten was a reminder of why I really enjoy hosting these podcast episodes. It’s not just about sharing knowledge, but about challenging our assumptions and pushing us to think differently about how we approach our work.

Maarten’s parting advice to “take more risks” and “try more stuff out” is something I’ll be taking to heart. In the complex and constantly moving world of software development, our ability to adapt, learn, and grow is our most valuable asset.

As we continue to navigate the fog of uncertainty in our projects, I’m excited to put these insights into practice. I hope you’ve found them as valuable as I have, and I look forward to continuing this conversation with our team of developers, project managers, and leaders.

Remember, if you want to dive deeper into these topics, you can find more resources in our learning center, or check out our full range of podcast episodes on Spotify and other platforms as below;

➕ YouTube – https://www.youtube.com/@pulsebyintuji
➕ Spotify – https://open.spotify.com/show/34jThbSZ1iVcEF93Tey6dy
➕ Apple Podcasts – https://podcasts.apple.com/us/podcast/pulse
➕ Deezer – https://www.deezer.com/en/show/1000801452

Also, give us a follow on the platforms below to keep up with the latest from Pulse by Intuji:

👉 LinkedIn – https://www.linkedin.com/company/pulse-by-intuji
👉 YouTube – https://www.youtube.com/@pulsebyintuji
👉 TikTok – https://www.tiktok.com/@pulse.by.intuji
👉 Instagram – https://www.instagram.com/pulsebyintuji
👉 X/Twitter – https://twitter.com/pulsebyintuji
👉 Website – https://pages.intuji.com/pulse

And as always, we’d love to hear your thoughts and experiences. After all, it’s through sharing and discussion that we all grow and improve.

Julian