Home / DOE / Analyze / Evaluate Design / Evaluate Design How-To

Evaluate Design How-To¶

This walkthrough measures the design the rest of the DOE section uses, so every number below is one you can reproduce.

Every chart in this family is drawn from a fitted model, so the walkthrough starts by building one. The same experiment is used on all ten chart walkthroughs, so once you have the regression sheet you can work through any of the others without setting the data up again.

The data is on this page rather than in a file to download. Press Copy for Excel, then paste it into a blank worksheet.

The data¶

A three factor experiment on a bonding process, run twice through. Temp is in degrees C, Press in psi and Time in seconds, and the response Strength is the peel strength of the finished bond. Each of the eight factor combinations was run twice, giving 16 runs.

Temp	Press	Time	Strength
180	40	10	49.5
220	40	10	65.4
180	60	10	61.8
220	60	10	66.8
180	40	20	52.3
220	40	20	70.7
180	60	20	60.7
220	60	20	70.0
180	40	10	50.6
220	40	10	66.4
180	60	10	61.2
220	60	10	67.2
180	40	20	51.4
220	40	20	67.1
180	60	20	61.9
220	60	20	71.6

Each factor takes exactly two values, so this is a full factorial of the three at two levels each, replicated once. That matters for more than one chart: a factor with exactly two observed values is the only kind that can carry a Cube Plot axis, and a design with centre points would have failed that test.

Set the design up¶

Run all five steps below. Evaluate Design reads the design sheet rather than the regression sheet, so the Regression worksheet step 5 produces is not what this walkthrough reads. Run it anyway: it is what puts the model's three interactions into the design sheet's stored model, and this walkthrough's numbers are the numbers of that seven column model.

Put the data in Excel

Press Copy for Excel above the table. In Excel, open a blank worksheet, click cell A1, and press Ctrl+V. You should have headers in row 1 and the 16 runs in rows 2 through 17.
Make a design sheet to hold it

From the Excel ribbon: QXL DOE > Create Design > Special > Create Custom Design.

On the first page set the number of factors to 3 and the number of runs to 16. On the second page set the number of outputs to 1. That gives you an empty design sheet of the right shape, which is how data collected outside Quantum XL gets analysed.
Fill in the design sheet

Copy the four columns of data into the three factor columns and the one output column. Name the factors Temp, Press and Time, and the output Strength, so the names on your charts match the ones quoted below.
Answer the two questions a custom design asks

A custom design does not record how it was built, so the first time you run a regression, run Optimize, or open Modify Design on it, Quantum XL asks for what the sheet cannot say, and stores your answers so it never asks again.

On the first page leave Type: at Quantitative for all three factors. On the second, the interaction editor, move AB, AC and BC from Interactions available into Interactions in model: three main effects plus those three interactions is the model every walkthrough below assumes. The remaining degrees of freedom fall as you add each one, because the sixteen runs are fixed and every term spends one.

Afterwards the same editor is reachable at any time from QXL DOE > Modify Design > Inputs > Add/Remove Interactions.
Run the regression

QXL DOE > Analyze Design > Run Regression. A worksheet called Regression is added after the design sheet, and it is the sheet the walkthroughs below read.

Evaluate it¶

Click any cell on the design sheet.
Press QXL DOE > Analyze Design > Design Diagnostics > Evaluate Design.
The dialog opens with your design sheet selected in Select the design sheet to analyze.
Leave Include power analysis ticked, and leave the three boxes at their defaults: Anticipated coefficient 1, Standard deviation 1, Significance level 0.05.
Press OK.

A worksheet called Evaluate Design is added. Your design sheet is unchanged.

Read the summary¶


Design sheet	the name of your design sheet
Runs	`16`
Model columns	`7`

Seven columns for a model of six terms, because the constant is a column too. There is no Blocks line, because this design is not blocked, and no Engine status line, because nothing went wrong.

Read the design measures¶

Measure	Value
Log determinant of X'X	19.4081
D-efficiency (%)	100.00
G-efficiency (%)	100.00
A-efficiency (%)	100.00
A value	0.4375
I (average prediction variance)	0.145833
Maximum prediction variance over the design region	0.4375
Max VIF	1.000
Average VIF	1

All three efficiencies are exactly 100.00, and that is the point of this design. A full factorial run twice through has orthogonal coded columns, so its information matrix is 16 times the identity. That is precisely the reference the efficiencies are scaled against, so a balanced full factorial scores 100 on every one of them. A design that scores less has columns that overlap.

Two numbers follow from the same fact and are worth checking against each other. The A value is \(7/16 = 0.4375\), the sum of the seven diagonal entries of the inverse, each of which is \(1/16\). The maximum prediction variance is also \(0.4375\): it sits at a corner of the design region, where the model row is seven entries of \(\pm 1\), and the corners are exactly where the runs are. That the two coincide is why G-efficiency lands on 100 as well.

Average VIF reads 1 and Max VIF reads 1.000. They are the same number written in two number formats, which is deliberate: the sheet uses each measure's own format rather than one format throughout.

Read the variance inflation factors¶

Model column	VIF
Temp	1.000
Press	1.000
Time	1.000
Temp*Press	1.000
Temp*Time	1.000
Press*Time	1.000

These are the model matrix column labels, not the regression sheet's term names. The constant column is called Const, a main effect is the factor's own name with no coded letter after it, and an interaction is its factors' names joined with an asterisk. The regression report writes the same terms as Temp (A) and AB; this sheet does not.

Six rows for seven model columns. The constant is not in the table at all: a variance inflation factor measures how much a predictor's variance is inflated by its overlap with the other predictors, and the constant is not a predictor, so there is no row rather than a row reading Not Avail.

A variance inflation factor of exactly 1 means no overlap whatsoever, which is what orthogonal columns give. Add a centre point or drop a run and these climb above 1.

Read the power table¶

Model column	1 replicate	2 replicates	3 replicates	4 replicates	5 replicates
Const	0.944	1.000	1.000	1.000	1.000
Temp	0.944	1.000	1.000	1.000	1.000
Press	0.944	1.000	1.000	1.000	1.000
Time	0.944	1.000	1.000	1.000	1.000
Temp*Press	0.944	1.000	1.000	1.000	1.000
Temp*Time	0.944	1.000	1.000	1.000	1.000
Press*Time	0.944	1.000	1.000	1.000	1.000

Above the table sit the three values it was computed from, written as you typed them:


Anticipated coefficient	1
Standard deviation	1
Significance level	0.05

The Const row does have a power figure, unlike the variance inflation table. Power is a test on one coefficient, and the constant has a coefficient like any other column.

Every row is the same number, again because the columns are orthogonal: each one is estimated with the same precision, so each one is equally easy to detect. On a design whose columns overlap these rows part company, and the column with the largest variance inflation factor is the one with the lowest power.

"1 replicate" means the design as it stands, all sixteen rows, not one pass through the eight combinations. Two replicates means running the whole sixteen-row design twice, for thirty-two runs.

Ask for a harder effect¶

The default anticipated coefficient of 1 against a standard deviation of 1 is a very large effect, which is why the table is nearly all ones. Ask for something more demanding.

Press Evaluate Design again.
Change Anticipated coefficient to 0.5. Leave the other two boxes alone.
Press OK.

A second worksheet is written, called Evaluate Design (2), and the first is left where it is. Its power table reads, for every column:

1 replicate	2 replicates	3 replicates	4 replicates	5 replicates
0.431	0.776	0.922	0.976	0.993

That is the table doing its job: a coefficient of 0.5 against a standard deviation of 1 has a 43 percent chance of being flagged by the design as it stands, and needs three passes to pass 90 percent. Nothing else on the sheet changed, because nothing else depends on those three values.

Only the ratio matters. Anticipated coefficient 0.5 with standard deviation 1 gives exactly the same table as 1 with 2, because the two numbers enter the power as one ratio. They are asked for separately so you type the two you actually know.

Read the alias matrix block¶

There is no table. The block carries this sentence instead:

The alias matrix is not shown because the model already holds every two-way interaction and every eligible quadratic, so there are no alias terms left.

That sentence is accurate and it is worth understanding. The alias-term set is every two-way interaction plus every eligible quadratic, and this model already holds all three of them. A quadratic is eligible only for a continuous factor taking more than two distinct values, and each factor here takes exactly two, so no quadratic is eligible. Nothing is left over to be aliased with.

The three-factor interaction ABC is not an alias term. The set stops at order two, so ABC never appears in this block whatever the model holds.

To see a populated alias matrix, take AB, AC and BC back out of the model with QXL DOE > Modify Design > Inputs > Add/Remove Interactions and evaluate again. The three interactions are then alias terms, and the matrix shows how each biases the three main effects.

Read the correlations¶

Model column	Temp	Press	Time	Temp*Press	Temp*Time	Press*Time
Temp	1.000	0.000	0.000	0.000	0.000	0.000
Press	0.000	1.000	0.000	0.000	0.000	0.000
Time	0.000	0.000	1.000	0.000	0.000	0.000
Temp*Press	0.000	0.000	0.000	1.000	0.000	0.000
Temp*Time	0.000	0.000	0.000	0.000	1.000	0.000
Press*Time	0.000	0.000	0.000	0.000	0.000	1.000

Six by six, the constant excluded, because a column that never varies has no correlation with anything. Every off-diagonal entry is zero, which is the same orthogonality the efficiencies and the variance inflation factors reported, seen pair by pair.

The signs are reported as computed. No absolute values are taken and no colour is applied.

Read the leverage¶

Sixteen rows, numbered 1 through 16, and every one reads 0.4375.

They sum to \(7\), which is the number of model columns, and that is a property of leverage rather than a coincidence: the leverages of a design always add up to the number of terms in the model, so the average leverage is \(7/16 = 0.4375\). On a balanced design every run sits exactly at the average. On an unbalanced one the runs with the most influence on the fit stand out here.

Turn the power analysis off¶

Press Evaluate Design once more.
Clear Include power analysis. The three boxes below grey out.
Press OK.

The worksheet has five blocks instead of six. The power block is gone entirely, and the three values it was computed from are gone with it. Nothing takes their place: there is no heading, no note and no empty table, because nothing on this sheet was computed from those numbers.