Home / Concepts / Engines (PICT vs SIPO)
Engines: PICT vs SIPO¶
ProTest includes two generation engines. Both produce valid covering arrays with guaranteed t-way coverage, but they use different algorithms and often produce different-sized results.
PICT (Pairwise Independent Combinatorial Testing)¶
PICT is Microsoft's open-source combinatorial test generation engine. It is the default engine in ProTest.
Strengths:
- Fast generation, especially for small to medium models
- Mature, well-tested codebase
- Full constraint language support
- Often generates fewer test cases than SIPO when t=3 or greater
How it works: PICT uses a greedy algorithm that builds the covering array one row at a time, selecting values that cover the most uncovered tuples.
SIPO (Simulated Annealing)¶
SIPO is ProTest's custom generation engine based on simulated annealing — a probabilistic optimization technique inspired by the annealing process in metallurgy.
Strengths:
- Typically produces 10-30% fewer test cases than PICT when using pairwise coverage (t=2)
- Can escape local minima through its temperature-based exploration
- Tunable parameters for balancing speed vs quality
Results Vary
The reduction in test case count compared to PICT depends on the model's parameters, values, constraints, and interaction strength. In some scenarios the improvement is significant; in others it may be minimal. Always compare both engines for your specific model if minimizing test count is important.
SIPO Parameters:
| Parameter | Default | Description |
|---|---|---|
| Base iterations | 10,000 | Number of optimization iterations |
| Initial temperature | 5.0 | Starting temperature (higher = more exploration) |
| Final temperature | 0.1 | Ending temperature (lower = more refinement) |
Higher iteration counts and wider temperature ranges produce better results but take longer.
Choosing an Engine¶
| Consideration | PICT | SIPO |
|---|---|---|
| Speed | Faster | Slower |
| Pairwise (t=2) | Good | Often better (10-30% fewer test cases) |
| Higher-order (t=3+) | Often better than SIPO | Good |
| Constraints | Full support | Full support |
| Reproducibility | Deterministic with seed | Deterministic with seed |
| Best for | Higher-order coverage, quick generation, CI/CD | Pairwise coverage, minimizing test count when time permits |
Parallel Trials¶
Both engines support parallel trials — running multiple generation attempts simultaneously and keeping the best result. This is controlled by:
--trialson the CLI- Automatic in the UI
More trials increase the chance of finding a smaller covering array, at the cost of more CPU time.
Using in the UI¶
Select the engine in the generation settings on the left sidebar, or use Advanced Mode for SIPO tuning parameters.
Using on the CLI¶
# Use PICT (default)
protest generate -i model.cahtt -o results.csv --engine pict
# Use SIPO with defaults
protest generate -i model.cahtt -o results.csv --engine sipo
# Use SIPO with custom tuning
protest generate -i model.cahtt -o results.csv --engine sipo --sipo-base 50000 --max-time 120