changing one thing at a time is the slow way to be wrong

Why one factor at a time cannot work

Suppose you are baking something and you want to know what makes it come out right. You suspect four things matter: temperature, time, how much yeast, and how long it rests.

The natural approach is to change one and hold the rest still. Temperature up, everything else fixed — better or worse? Then time. Then yeast. Then rest.

Two problems. First, you are spending all your runs learning about one factor at a time, so you need a lot of them.

Second, and much worse: you can only ever see each factor at whatever settings the others happened to be on. If more yeast only helps when the dough rests longer, and you tested yeast at a short rest, your experiment says yeast does not matter. That conclusion is wrong, it is confident, and nothing in your data will tell you so.

A factorial design tests every combination. Every factor is measured across the full range of every other factor, so an interaction shows up as an interaction instead of hiding as noise.

Does it actually recover the truth?

This is testable in a way most spreadsheets never are. The sample data is generated from a known model, so you can check whether the arithmetic recovers the effects that were deliberately put in.

Note the last row. Factor B genuinely does nothing, the workbook recovers it as approximately nothing, and Lenth's method correctly declines to flag it. A method that finds everything is as useless as one that finds nothing — five out of five real effects and zero out of ten false positives is what a working method looks like.

Note also that two of the five real effects are interactions (AC and AD), and AC is negative. One-factor-at-a-time cannot find either.

Lenth's method, and why unreplicated designs need it

An unreplicated 2⁴ has 16 runs and 15 effects to estimate. That uses up every degree of freedom, so there is nothing left over to estimate the noise with — which means no standard significance test works.

Lenth's method solves it by assuming most effects are inert, and using the size of the small effects to estimate the noise. It runs in two passes so that large real effects cannot inflate the noise estimate and hide themselves:

s0  = 1.5 x median|effect|
PSE = 1.5 x median{ |effect| : |effect| < 2.5 x s0 }

It is the correct tool for this design and it is almost completely absent from spreadsheet templates, which is a large part of why this workbook exists. Both the margin of error and the simultaneous margin of error are reported — 1.446 and 2.935 on the shipped data.

The four centre points do a second job: they test for curvature. If the middle of your design space sits off the plane the corners define, a straight-line model is the wrong shape and you need a response surface design instead. On the shipped data curvature is not significant (F = 0.013), which is correct, because the data was generated from a linear-plus- interaction model.

How to run a 2^4 factorial experiment

What is inside the file

Seven tabs. The response data is generated from a known model, which is what makes the recovery table above possible — you can verify the workbook finds the right answer because the right answer is known.

Opening it in Excel, Google Sheets or Numbers

It is one .xlsx file. There are no macros, no add-ins and nothing to install, which is what makes it portable — a macro-driven template would be Excel-only.

This one leans on FDIST, MEDIAN, NORMSINV, RANK and TINV — the functions people most often worry about losing in another app. All of them exist in Excel, Google Sheets, Apple Numbers and LibreOffice Calc, so the file works the same in all four.

The arithmetic, written out

contrast    = sum of (sign x response) across all 16 runs
effect      = contrast / 8            # for a 2^4
coefficient = effect / 2
SS          = contrast^2 / 16

s0          = 1.5 x median|effect|
PSE         = 1.5 x median{ |effect| : |effect| < 2.5 x s0 }
ME          = t(0.975, d) x PSE
SME         = t(gamma, d) x PSE

curvature F = (mean of corners - mean of centre points)^2 scaled / pure error

The two-pass structure in Lenth's method is the point of it. The first pass gets a rough scale from all the effects; the second recomputes it using only effects small enough to plausibly be noise. Without that second pass, a few large real effects inflate the noise estimate enough to hide themselves.

How I know the numbers are right

The checker does two different things, because a workbook can be perfectly self-consistent and still wrong. It reimplements every calculation independently in Python and diffs. And because the response data is generated from a known model, it also checks that the arithmetic recovers that model.

Last run: 0 numeric mismatches, 0 property failures, 0 formula errors.

• effect = 2 × coefficient, for all 15 terms
• The 15 sums of squares add exactly to the factorial total (16 runs = 1 df mean + 15 df effects)
• The design is orthogonal — all 105 column pairs have zero dot product
• Lenth's method flags all 5 real effects and zero of the 10 inert ones (ME 1.446, SME 2.935)
• Curvature is not significant (F = 0.013) on data generated from a linear-plus- interaction model — a curvature test that fires on linear data is broken

Compared with the alternatives

Questions people ask before buying

Related spreadsheets