THE CAPABILITY EDGE — OCT 2026

Structure, pace, and opportunity — What thirty years of design reviews and one software paper agree on

A paper has been sitting on my desk for a few weeks, written by Kemerer and Paulk (IEEE Transactions on Software Engineering, 2009), about a large study of what makes technical reviews effective. They concluded that well-structured, well-paced code reviews found substantially more defects than unstructured, poorly paced reviews. The authors stated the underlying point this way:

Quality control mechanisms are an integral part of the process, not simply inspections layered on top afterward. To be clear, the IEEE paper was about software and software code reviews. I am not interested in the specific code review rates from the paper, or how fast was too fast to get good code. Nor am I going to pretend the study transfers to reviewing drawings, specifications, and test protocols. But I have spent three decades sitting in design reviews, and I would tell you the overarching concepts hold true. The reviews I have seen catch real problems had structure, pace, and opportunity. The ones that produced a signature and nothing else had none of the three. So here is what those words actually mean to a medical device design review.

Structure

A structured design review knows what it is doing before it starts.

• Entry criteria are met before the review is booked. A review of an incomplete package reviews the gaps you already knew about and misses the ones you didn’t.
• The scope is stated, along with what’s out of scope. Reviews that try to cover everything cover nothing.
• There is a question list, not a walkthrough. If the designer narrates and the room follows, you are testing the presentation, not the design. The inspection literature has been consistent for decades: structured reading finds more than unstructured reading.
• Someone in the room can say no. Independent enough that disagreeing is not a career killer.
• Decisions are recorded with their reasoning. The engineer inheriting the design later needs to know what was considered and rejected, not just what was approved.

Pace

A well-paced review gives people enough time with the material to actually see it.

• The volume of material is matched to the time available. Eighty pages of drawings, specifications, and protocols do not become reviewable because ninety minutes is what the calendar set.
• The package is issued against a defined window, not “before the meeting.” If the window is missed, the review moves. That single rule changes behavior faster than anything else here.
• Reviewers come having read it. If the first time an issue is spoken aloud is in the room, the meeting is the first read, and the meeting is the worst possible place to do your reading.

The design review is usually required, auditable, and leaves a record. Preparation is none of those things. So, when a project runs tight, nobody cancels the design review. They cut the reading. However, the record still gets signed, and nothing shows what was lost. Which is why opportunity matters, and why it is the issue nobody writes into a procedure. Structure and pace are things a team can impact. Opportunity is something a team has to be given.

Opportunity

Opportunity means the design team has cover, time, and visible leadership support to conduct a thorough review. Not permission in a procedure, but permission in practice. The IEEE paper’s authors made a similar observation, and it is the sentence I keep returning to:

That is software engineering from 2009, describing one reason why project teams get formed to fix on-market devices. Here is what it looks like when the opportunity is real:

• “Not ready” is an available outcome. If every review can only end in approval, it is not a review. It is a signature ceremony. A gate that cannot be failed is not a gate.
• Preparation is in the schedule as work, with hours against it and an owner. Not an assumption about what happens between tasks. What isn’t in the plan is what gets cut first.
• Leadership asks what the review found, not whether it happened. Those two questions produce entirely different behavior in the room. The first one makes finding a problem a success. The second one makes it an inconvenience.
• The engineer who raises the expensive problem is treated as having done the job. Once it has gone the other way in front of the team even once, your next several reviews are quiet, and you will believe they went well.
• The milestone is evidence of readiness, not a date to be reached. When the milestone itself becomes the objective, the work reshapes itself to produce the artifact instead of the result.

Organizational culture is not a soft factor here. If time to market, or worse, time to an arbitrary milestone is the only thing that is measured, then quality goes out the window, and no amount of structure in your procedure will fix it.

Why it is worth protecting

A design review is the cheapest place in a program to find a problem. Nothing is built, no tooling is cut, nothing has been verified. The same issue found at verification costs a protocol rerun. Found after submission, it costs a deficiency response and a launch date.

Structure, pace, and opportunity are not quality overhead. They are the difference between paying once and paying twice. I’ve written separately about what that difference adds up to across a program.

Design Review (DR) scorecard

The DR scorecard measures structure and pace. Twenty questions across five dimensions: (1) preparation and planning, (2) reviewer composition and independence, (3) content coverage and traceability, (4) conduct and rigor, (5) follow-through and closure, scored out of 100. Use it on a DR you have already held. It takes about fifteen minutes. Tell me what you found.

Opportunity is not in it, deliberately. You cannot score your leadership’s appetite for bad news from a DR, and you wouldn’t want that written down and kept in a design file. Opportunity shows up a different way: run the scorecard across your last five reviews and look at the trend. One weak review may lead to a poor design. The same weak scorecard dimension five times is not a DR problem. It is what the organization is actually asking for.

Next month: The Capability Edge Live

A free three-part series, 10–12 November, 45 minutes a day at noon Eastern. Day one picks up exactly where this issue stops: the gaps an experienced reviewer sees in a design file. Day two is what those gaps cost. Day three shows you how to build the capability that closes them. Register, and the scorecard and the Training ROI Calculator come with it.

Register: design-quality-services.kit.com/capability-edge-live​

Also in November: MEDevice Silicon Valley

We’ll be at booth 940 in Santa Clara on 18–19 November. If your design reviews are the thing you keep meaning to fix, bring a review record, and we’ll go through it at the booth. That conversation works better in person than by email.

Source

Kemerer CF, Paulk MC. “The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data.” IEEE Transactions on Software Engineering, Vol. 35, 2009. Both quotations are verbatim.

The edge isn’t knowing the regulation. It’s implementing it.

John Salvato, PhD - Founder & President, Design Quality Services