Home / Monte Carlo / Model Building / Correlations / Math Details
Math Details¶
What each family means for the numbers you get, what each parameter is allowed to be, why a set of correlations can be refused, and what the nearest valid matrix guarantees. What the tool is for is on Correlations, and the controls are on Options.
What this page does not cover
How Quantum XL constructs the correlated values, and the procedure by which it computes the nearest valid matrix, are not documented. The published methods those rest on are named and cited in the references below, so the mathematics can be looked up; what is not written down here is how Quantum XL carries them out. This page states what each control means, what each number you enter has to satisfy, and what the results are guaranteed to have, which is what a reader needs in order to read the dialog and trust its output.
Notation¶
| Term | Description |
|---|---|
| \(k\) | the number of inputs in the correlation |
| \(\rho\) | one off-diagonal entry of a correlation matrix |
| \(\theta\) | the strength parameter of the Clayton or Gumbel family |
| \(\nu\) | the Student-t degrees of freedom |
| \(R\) | the \(k\) by \(k\) correlation matrix |
What a correlation changes and what it leaves alone¶
A correlation changes which values of the inputs turn up together. It does not change what values any one input can take, or how often it takes them.
That is a guarantee rather than an approximation. Each input's own distribution is untouched, so truncation, custom distributions and empirical data all behave exactly as they do on an uncorrelated input, and the marked distribution you see in Mark Input is the distribution the input still has.
Two consequences worth knowing:
- A correlation consumes no extra sampling. The same number of values is drawn per trial as without it, so run sizes and sampling settings mean the same thing.
- A rule that amounts to no dependence reproduces an uncorrelated run exactly, value for value. Two settings do that: an identity matrix, meaning every off-diagonal entry is 0, and a Gumbel theta of exactly 1.
The four families¶
The families differ in where in the range the dependence is concentrated, which is what decides whether extreme values of two inputs tend to occur on the same trial.
| Family | Where the dependence sits | Both inputs extreme together |
|---|---|---|
| Gaussian | spread evenly over the whole range | no more likely than the middle of the range suggests |
| Student-t | spread evenly, with both tails strengthened | more likely in both tails, and more so as \(\nu\) falls |
| Clayton | at the low end | low values occur together more than high values do |
| Gumbel | at the high end | high values occur together more than low values do |
This matters most for a tolerance or reliability question, where the answer is decided by how often two inputs are both near a limit at the same time. Two models with the same pairwise correlation strength and different families can disagree on that and agree on almost everything else.
The families are the Gaussian, Student-t, Clayton and Gumbel-Hougaard copulas of the literature; see the references.
What each parameter is allowed to be¶
Every rule below is enforced when Save correlation is pressed, and the sentence quoted is the refusal you get.
A correlation matrix, Gaussian and Student-t¶
The diagonal is fixed at 1 by the grid, so only a matrix read from a worksheet can break the first rule, and the grid mirrors what you type, so only a worksheet can break the second.
Correlation group 1: The correlation between inputs 1 and 2 must be a number between -1 and 1.
A correlation of exactly \(+1\) or \(-1\) is allowed. It means the two inputs always land at the same percentile, or always at opposite percentiles. That is a perfectly good joint distribution, and it is not the same thing as duplicating an input: an input tied at \(+1\) to another with a different distribution keeps its own distribution, matched percentile for percentile.
Degrees of freedom, Student-t only¶
Degrees of freedom is not a whole number.
Correlation group 1: Degrees of freedom must be a whole number of at least 1 (got 0).
Smaller \(\nu\) strengthens both tails. Large \(\nu\) makes the family behave much like the Gaussian.
Theta, Clayton¶
Correlation group 1: Clayton theta must be greater than zero.
A larger \(\theta\) ties the inputs more tightly. As \(\theta\) approaches 0 they approach independence, but 0 itself is refused rather than treated as independence.
Theta, Gumbel¶
Correlation group 1: Gumbel theta must be a finite number of at least 1 (1 means no correlation).
A larger \(\theta\) ties the inputs more tightly. \(\theta = 1\) is exactly independence and is accepted.
Reflected, Clayton and Gumbel only¶
Reflecting an input turns each of its percentiles into the opposite percentile, so its relationship with the other inputs runs the other way and, for these two tail-weighted families, the tightly dependent end moves with it. Reflect one input of a Clayton pair and that input's high values are the ones that occur with the other's low values.
This is how a theta family expresses a negative relationship. It has no matrix to carry a minus sign per pair, so the direction is carried per input instead.
The number you type is Spearman's, not Pearson's¶
The grid holds Spearman's rank correlation coefficient \(\rho_S\), the correlation of the two inputs' percentile positions. It does not hold Pearson's product-moment correlation, and there is no setting anywhere in the dialog that makes it hold one.
Rank is the right quantity to ask a modeller for. Turning a uniform percentile into a value of the input's own distribution is a monotone map, and a monotone map leaves percentile positions alone. A rank correlation therefore survives that step unchanged, and a Pearson correlation does not: the same dependence between two lognormal inputs and between two normal inputs gives the same Spearman figure and two different Pearson figures.
The scale the engine works on, and the exact relation¶
Quantum XL does not use \(\rho_S\) as it stands. Each entry is converted to the correlation in normal space that produces it, by the relation of Iman and Conover (1982):
Entries of exactly \(\pm 1\) are carried across unchanged.
For the Gaussian family this relation is exact, and it is the classical relation between Spearman's rho and the correlation parameter of a bivariate normal (Kruskal, 1958). So:
If the family is Gaussian, the number you type is, in theory, the Spearman rank correlation the simulated inputs converge to as the number of trials grows without limit. It is the target, and the engine solves for the parameter that hits it.
Two things about that statement are worth being precise about, because they are easy to get the wrong way round.
It does not require the inputs to be normal. Spearman's rho depends only on the copula and is unchanged by any monotone transformation of the marginals, so the guarantee holds whatever distributions the inputs carry: lognormal, Weibull, triangular, custom. Normal marginals are not a condition of it.
With normal marginals the Pearson figure is a different number. If the inputs happen to be normal and the family is Gaussian, the pair is jointly normal and its Pearson correlation is \(\rho\), the converted value, not the \(\rho_S\) you typed. One checkable case:
| You type \(\rho_S\) | The engine uses \(\rho\) | Spearman of the run | Pearson of the run, normal inputs only |
|---|---|---|---|
| 0.600000 | 0.618034 | 0.600000 | 0.618034 |
\(2 \sin(\pi \times 0.6 / 6) = 2 \sin 18^{\circ} = 0.618034\), and \((6/\pi)\arcsin(0.618034/2) = 0.600000\) exactly, which is the round trip.
The two scales agree only at \(0\) and at exactly \(\pm 1\), and \(\rho\) is always at least as large in size as \(\rho_S\).
The Student-t family is close, not exact¶
The same conversion is used for the Student-t family, where it is not exact: the rank correlation of a t copula depends slightly on the degrees of freedom, which this relation does not carry. The error was measured across the family's whole usable range and its largest value is \(7.07 \times 10^{-4}\). A typed 0.600 therefore lands near 0.6007 rather than on 0.6000.
This was accepted deliberately, so that rank entry could be offered for every matrix family rather than only for the Gaussian.
Why a set of correlations can be impossible¶
A correlation matrix cannot hold arbitrary numbers, because the entries constrain each other. If A and B move together strongly and B and C move together strongly, then A and C cannot move strongly in opposition, whatever number you type for that pair.
The exact condition is that \(R\) must be positive semi-definite: every eigenvalue at or above zero, equivalently
A matrix that fails this describes relationships no data could have, and it is refused:
Correlation group 1: The correlations are mathematically inconsistent in that the matrix is not positive semi-definite.
Semi-definite, not definite. A matrix with a zero eigenvalue is accepted. That is what a correlation of exactly \(\pm 1\) produces, and so does a genuinely real case such as three inputs all pairwise at exactly \(-0.5\).
What the nearest valid matrix guarantees¶
When a matrix family's entries fail that condition and a consistent replacement can be found, the refusal is followed by the offer described on Options. What comes back is guaranteed to have three properties.
It is consistent. It satisfies the positive semi-definite condition above, with a small margin rather than sitting exactly on the boundary, so it is safely inside the set of usable matrices rather than on its edge.
It is a correlation matrix. Every diagonal entry is exactly 1 and every off-diagonal entry lies within \([-1, 1]\).
It is the nearest such matrix, where nearness is the total squared change over all the entries, the Frobenius distance:
So the replacement is the smallest total change that reaches a consistent matrix, and not, for instance, a change confined to the one pair that looks most suspect. Every entry can move, and a set that is far from consistent comes back changed a long way, while one that is only just inconsistent comes back barely changed.
The method has a name, and it is a published one. The replacement is the nearest correlation matrix in the sense of Higham (2002), computed by an alternating-projections method with the Dykstra correction (Dykstra, 1983) that makes the limit the nearest point of the intersection rather than merely a point in it. The conversion between the rank scale and the normal-space scale, applied before and after the repair in rank entry, is the Iman and Conover (1982) relation given earlier. Later methods for the same problem exist and are cited in the references. How Quantum XL performs the computation is not documented here; the papers are named so the mathematics can be looked up.
Three further points:
- In rank entry the numbers you get back are in rank units, the same units you typed.
- The replacement is checked by the same rule that refused the original. If the check does not pass, or no replacement can be found, you are given the original refusal and nothing else, so you never see a matrix that would fail on the next press.
- Nothing is saved by accepting. The numbers go into the grid for you to look at, and Save correlation is left for you to press.
Why a discrete input falls short of the requested correlation¶
A discrete input takes a small set of values, so however finely its percentile is placed, what comes out is one of a handful of values. The distinct percentile positions the input can occupy are therefore few and heavily tied, and a rank correlation computed over tied positions cannot reach an arbitrary target.
The dependence is applied in full. The achieved rank correlation is smaller in size than the requested one, and the shortfall grows as the input's set of values gets coarser. This is a property of discrete data rather than a limit of the software, and it is why the dialog states it rather than refusing.
See Also¶
- Correlations
- Options
- How-To
- Supported Distributions, which says which distributions are discrete
- Percent Contribution, whose LMG method is the one whose attribution stays well defined when inputs are correlated
References¶
Copulas, and the separation of dependence from the marginals
- Sklar, A. (1959). Fonctions de repartition a n dimensions et leurs marges. Publications de l'Institut de Statistique de l'Universite de Paris, 8, 229-231.
- Nelsen, R. B. (2006). An Introduction to Copulas, 2nd edition. Springer.
- Joe, H. (2014). Dependence Modeling with Copulas. CRC Press.
- Genest, C., & MacKay, J. (1986). The joy of copulas: bivariate distributions with uniform marginals. The American Statistician, 40(4), 280-283.
Rank correlation, and why it is the quantity to ask for
- Kruskal, W. H. (1958). Ordinal measures of association. Journal of the American Statistical Association, 53(284), 814-861.
- Embrechts, P., McNeil, A. J., & Straumann, D. (2002). Correlation and dependence in risk management: properties and pitfalls. In M. A. H. Dempster (ed.), Risk Management: Value at Risk and Beyond, 176-223. Cambridge University Press.
- McNeil, A. J., Frey, R., & Embrechts, P. (2015). Quantitative Risk Management: Concepts, Techniques and Tools, revised edition. Princeton University Press.
Inducing a requested rank correlation
- Iman, R. L., & Conover, W. J. (1982). A distribution-free approach to inducing rank correlation among input variables. Communications in Statistics - Simulation and Computation, 11(3), 311-334.
The nearest correlation matrix
- Higham, N. J. (2002). Computing the nearest correlation matrix: a problem from finance. IMA Journal of Numerical Analysis, 22(3), 329-343.
- Dykstra, R. L. (1983). An algorithm for restricted least squares regression. Journal of the American Statistical Association, 78(384), 837-842.
- Qi, H., & Sun, D. (2006). A quadratically convergent Newton method for computing the nearest correlation matrix. SIAM Journal on Matrix Analysis and Applications, 28(2), 360-385.
- Borsdorf, R., & Higham, N. J. (2010). A preconditioned Newton algorithm for the nearest correlation matrix. IMA Journal of Numerical Analysis, 30(1), 94-107.
The individual families
- Demarta, S., & McNeil, A. J. (2005). The t copula and related copulas. International Statistical Review, 73(1), 111-129.
- Clayton, D. G. (1978). A model for association in bivariate life tables and its application in epidemiological studies of familial tendency in chronic disease incidence. Biometrika, 65(1), 141-151.
- Gumbel, E. J. (1960). Bivariate exponential distributions. Journal of the American Statistical Association, 55(292), 698-707.