A pleasant walk through computing

Comment for me? Send an email. I might even update the post!

Basic Debugging and Problem Solving Questions

Photo by Elisa Ventur on Unsplash

While a short post, I hope this will help others as it did me. These questions came out of an experience trying to troubleshoot a deployment issue I was called into some time ago. The problem got solved, but the approach to identifying the problem wasn't . . . ideal. Among the challenges were pressure from above and the group dynamics of a thirty-person Zoom meeting.

Afterward, I performed a personal post-mortem to identify heuristics that would help with this general class of problems, the "It's not working and we don't know why but it needs to be fixed right now we're watching you" situation.

Actually, these are good questions for any debugging/troubleshooting, not just the under-pressure variety.

Questions

  1. What's the symptom? Get clear on this from the start and respectfully question what you're first told. "The prices are all wrong!" "Would you show me exactly what a where you're seeing that?" "Sure. See? All (and only) the shirt prices are multiplied by ten on this screen."
  2. What changed? In my situation, the exact same code was deployed to a different environment. The code worked in the other three environments. So, the problem wasn't the code. We needed to review the environment, not dive into the debugger. (It was a wrong deployment environment variable.)
  3. What's the dumbest thing it could be? Did something get misspelled? Did you push the wrong version?
  4. Working backward, what controls that behavior? It's so easy to assume you know what "must be" causing the symptom. Stop. Don't think "must be," but instead "might be" and especially "shouldn't be but let's check." What class method displays that text? What service calls that class?

Other Principles

  • Ask, "What's the real problem?" For example, the symptom might be "we're getting a 404 on this page." But the real problem might be, "Our main customer is blocked from completing a critical report." Maybe the customer problem can be addressed without immediately fixing the symptom.
  • Don't rush. Pressure is the enemy, and often an illusion. Solving the problem fast isn't as important as solving it right. Wanting to be the hero and feeling the pressure of being the expert inevitably causes delays and failure. David Marquet calls this "control the clock, don't obey the clock."
  • Change one thing at a time. This is a hard-won skill. Change one thing, and confirm other variables are "known good." Decades ago, I called my boss for help with a non-working printer. He asked, "Are you using a known-good cable?" "Uh, no." "You have to make sure the cable works!" He was right.
  • Always know what you changed. If you must change multiple variables, keep track.
  • Fix the problem, not the blame. Now isn't the time for a retrospective, or questioning why the code is written the way it is. That just derails the conversation and is evidence of a pathological organization. But do have a post-mortem when heads are clear.

Of all these, "what's the symptom?", "what changed?", and "don't rush" will take you very far.

"It's important to think when things are going crazy, if you want to take the smartest action to get them sane again." --Harry Dresden, Battle Ground

WIP Metaphor: Not Burgers. Books!

Developers Don't Cook Hamburgers

Metaphors1 are strong ways of explaining or clarifying difficult concepts. However, the wrong metaphor can weaken one's case.

In his article How finishing what you start makes teams more productive and predictable, Lucas F. Costa uses making hamburgers to illustrate work-in-progress. He's trying to show the importance and efficiency of finishing a task at a time and uses the following illustrations.

Serial

Concurrent

Here's the problem. His metaphor is dead wrong.

Let me be clear from the outset. Mr Costa is right about WIP and task switching in software development, and I assume his math equations are right, too. In fact, I encourage you to read the entire piece because it's good stuff.

However, I'm going to argue he shouldn't use the hamburger metaphor at all. It's too easy to poke holes in, and therefore not take seriously. If he tried to "educate the cook" in any of millions of restaurant kitchens on how to cook burgers, he'd rightfully be told he doesn't know what he's talking about.

I've worked as a cook. I do know what I'm talking about.

This point is really important. When trying to persuade people outside our field by using a metaphor, we must know that other subject matter. We can't say, "I know it's not the same, but do you see my point?"

The answer from senior management: "No, because you've made your point using an invalid example. Now I don't trust you. Get back to work on those features and stop complaining."

When you use a bad metaphor, the other person will focus on the flaws in the metaphor, not the strengths of your argument.

If we used this metaphor to try to persuade management or execs who don't understand software development why we should limit work-in-progress, we'd fail. Let's see why.

Computers Cook Hamburgers

How does short order cooking of burgers actually work? (My drawings aren't as good as Mr Costa's!)

Now, what happens if two people order burgers simultaneously?

Why doesn't it take 30 seconds each to assemble and plate the burgers? Because I can combine like tasks. Open both buns, slather both with mustard, place lettuce on both, then tomato on both. Then put the patties on the buns.

Finally, what happens if three customers order burgers one minute apart? Let's make the visual clearer.

Each burger has fixed grill times, with a flip in between. It only takes a couple of seconds to flip a burger.

At the two minute mark, I need to flip the first burger, then start grilling the third. That adds a couple of seconds to the third order.

Similarly, at the four minute mark, I flip the third, then assemble and plate the first.

I then assemble and plate the remaining burgers with almost no pause. My customers each wait about five minutes. This isn't theory; I've done it. If I cook my burgers one after the other, the first customer gets his burger in under five minutes, and the third gets hers in fifteen minutes. She won't be happy, and I'll go out of business.

Why is flipping burgers so different than coding software?

Cooking has significant wait time.

When I put a patty on the grill, I'm not stuck grilling the patty the entire two minutes. The patty is, essentially, grilling itself. I'm just waiting. This means I can start working on something else, and come back to flip the patty when it's ready.

Cooking has lots of async-await. It's highly asynchronous.2

In fact, as a cook I can take advantage of my wait times and prepare even more. Assembling a burger involves pre-assembly (buns and condiments), and final assembly (putting the patty on the burger). I can pre-assemble early and plate the burger sooner.

This is very much what a computer CPU is doing. A process is spawned that only uses a portion of memory, and so the CPU spawns another process. It's task switching, but that's what computers are good at, not people.

Does this mean WIP doesn't apply to cooking? No, it does apply. In fact, in a restaurant kitchen, there's a real danger of starting too many orders. This leads to cognitive overload. "Which patties are ready to flip?" It also leads to too many overlapping tasks. "I need to flip these four patties, assemble those five, plate these other two, and get back to these three patties before they burn."

The kitchen, though, has its own WIP limiter: grill space.3 You can only start so many burgers. This is why you have to wait so long in a crowded diner.

While cooking is highly asynchronous, it is also highly limited. It's a just-in-time system where you only start orders when you have capacity.

But we're still left with a dilemma. We want to explain to upper management why trying to code multiple features at once is severely counter-productive. What metaphor can we use that they'll immediately grasp?

Features Are Books

Software engineering is a creative endeavour. It's not kitchen work, and it's not assembly line work. We need to use a creativity-based metaphor.

If you were tasked to write three books in two years, how would you do it? Would you

  1. Write page one of each book, then page two, then page three, and so on?
  2. Write chapter one of each book, then chapter two, etc?
  3. Write book one, then book two, then book three?

Option one is patently ridiculous. And yet it's how management and developers often think when they're asked to add a new project or feature to an existing workload. "OK, sure, I can fit that in." Or, "Sure, I can do that at the same time."

No. No you can't. This is where Mr Costa's article and math are so helpful, and is the point he's trying to make.

Coding has no wait times. You can't start coding a feature, and then switch to another feature while the first one magically continues coding itself. Coding is highly synchronous.4

What are the consequences of trying to write three books asynchronously? Will they be finished sooner, the same, or later than if you write them one after the other?5

All three books will be late, and each will take longer to write. Why? Because of task switching. You don't need to be much of a writer to imagine trying to switch between three different books, plots, characters, locations, every page. You'd never read three books that way, right? The reason's the same. The writer needs to become immersed in the story. Switching to another story requires "getting back into it." In software development, this is often called "ramp up time" or "reloading the code into my head" and generally takes ten to twenty minutes after an interruption.

If you want all three books on time, write them one after the other.

There's another important business consideration here, and one that should appeal to all executives. In our metaphor of the writer, I want to finish the first book before starting the second. Why? Because I need the money. That book can get published and I can get royalties while writing my second book. I can also use the reviews of my first book to help improve my next one.

This is critical to understand in software development. We need to work on one feature at a time with minimal distraction so we can finish it more quickly and get feedback to both fix the existing feature and inform future features. We also want to get products to users quickly because that's where the company's revenue comes from.

Metaphors Are, Metaphorically, the Grail Choice in Indiana Jones

As the knight reminds in Indiana Jones and the Last Crusade, you must choose wisely. Like all metaphors, the writing metaphor is incomplete. For example, unlike writers finishing a book, programmers are constantly fixing previously released code. Imagine if writers were forced to revise and republish all their books throughout the year! Also, writers do sketch ideas for future books, and they'll edit one book while writing another. But would you have said to J.K. Rowling, "I know we have an August deadline for the next Harry Potter, but we need you to start another book, too, and have it finished in September without missing the August deadline. Plus these five magazine articles."

Of course not.


  1. The difference between metaphors and similes is a metaphor substitutes, a simile compares. "I watched the flaming chariot cross the sky" is a metaphor for the sun. "The sun crossing the sky is like a flaming chariot" is a simile. When you see "is like," it's a simile. In this article, I'm using "metaphor" throughout because it's easier to understand.

  2. Mr Costa does touch on batching, but his example is, again, inaccurate compared to kitchen cooking reality.

  3. Mr Costa notes this as well.

  4. Of course, there is technically some wait time, for example while code is building. But compared to restaurant cooking, there's essentially none.

  5. We're setting aside for the moment the danger of writer's block. We're assuming we can write continuously.

.NET MAUI Progressing From a Default Project Part 7 - Deploying, the Other People's Links Edition

The Series

Level: Moderate

This series assumes the developer is familiar with .NET MAUI, the Model-View-ViewModel pattern, and has worked through one or more tutorials such as the excellent James Montemagno's Workshop.

Source: https://github.com/bladewolf55/net-maui-progression

Checking My Ego

I kind of promised this part in the series, where I was going to take my progressively modified default app and show deploying it. But you know what? Other people have already covered this. So, here are their links and I don't think I have anything to add or clarify. Read them. They'll point you the right way.

I may still write a future post showing .NET MAUI CI/CD in Azure DevOps Pipelines. That would be interesting because it would ideally cover:

  • The local build and package script
  • Remote (integrated) build
  • Package once, deploy to many environments
  • How to manage testing versions before publishing to the platform stores

The thing is, I haven't done this with .NET MAUI, and I have some other topics I'm more interested in right now. But I'll eventually learn and write about it when I--hopefully--publish my first .NET MAUI application!

Wrap Up

This series has used the out-of-the-box .NET MAUI project template as a starting point for learning many key concepts and techniques. Some apply only to .NET MAUI, while most apply to general enterprise development.

I hope it's been helpful. You rock!