ProTest — Covering Array Generator

Generate minimal test matrices that cover all t-way parameter interactions. A covering array guarantees every pair (or triple, etc.) of parameter values appears in at least one test case, using far fewer cases than exhaustive testing. Research shows nearly all software failures are caused by interactions between 2-6 parameters.

When to Use

• Testing combinations of configuration options, feature flags, or input parameters
• 3+ parameters with multiple values each where exhaustive testing is impractical
• You want pairwise (2-way) or higher-order coverage guarantees

How to Run

protest generate -i model.cahtt -o results.csv

Workflow

1. Identify parameters and their possible values from code or context
2. Identify constraints — invalid combinations (e.g., Safari only on Mac)
3. Create a .cahtt model file (see format below)
4. Run: protest generate -i model.cahtt -o results.csv
5. Read results.csv — each row is a test case

Model File Format (.cahtt)

{
  "Version": 1,
  "GlobalOrder": 2,
  "Parameters": [
    {
      "Name": "OS",
      "Values": [
        { "Name": "Windows", "Weight": 1, "IsNegative": false },
        { "Name": "Linux", "Weight": 1, "IsNegative": false }
      ]
    },
    {
      "Name": "Browser",
      "Values": [
        { "Name": "Chrome", "Weight": 2, "IsNegative": false },
        { "Name": "Firefox", "Weight": 1, "IsNegative": false }
      ]
    }
  ],
  "ConstraintText": "IF [OS] = \"Windows\" THEN [Browser] <> \"Safari\";",
  "Seeds": [
    { "OS": "Windows", "Browser": "Chrome" }
  ],
  "SubModels": []
}

Weight > 1 biases toward that value. IsNegative marks excluded values. Seeds are must-include rows. SubModels group parameters for higher-order coverage ("Order": 3).

CLI Quick Reference

Flag	Default	Purpose
-i, --input	required	Model file (.cahtt or .pict)
-o, --output	required	Output file (.csv, .md, .cahtt, .pict); repeatable
--engine	from model	pict or sipo
--strength	from model (2)	T-way coverage strength
--trials	1000/10000	Parallel trials (best result kept)
--max-time	no limit	Seconds before stopping
--seed	auto	Deterministic random seed
--cores	~75%	Parallel cores

Run with generate --help for all options.

Example

protest generate -i model.cahtt -o results.csv --engine sipo --max-time 30
protest generate -i model.cahtt -o results.csv -o results.md