XLSTAT-Method Validation

Tutorial
View tutorial
s
XLSTAT-Method Validation is an Excel add-in which has been developed to provide XLSTAT users with a powerful solution for comnparing and validating statistical methods.

This analytical software solution provides you with leading-edge methods such as Bland Altman comparison, Passing and Bablock regression and Deming Regression.

Features

You can find tutorials that explains how XLSTAT-Method Validation works here.

Demo version

A trial version of XLSTAT-Method Validation is included in the main XLSTAT download.

Prices and ordering

These analyses are included in the XLStat-Biomed and XLStat-Premium packages.

 

 

 

DETAILED DESCRIPTIONS

Method comparison

Tutorial
View a tutorial

When to use method comparison

When developing a new method you might want to be sure that the results that are similar to a reference or comparative method. If there is a difference, you should investigate the reasons. It may be due to a bias that depends on the position on the scale variation. In addition If a new measurement method is preferred for economic reasons, but do not performed as good as the reference method, you might take into account the bias while reporting the results.

XLSTAT tools to compare methods

XLSTAT provides a series of tools to evaluate the performance of a method compared to another.

Paired t-test for method comparison

Among the comparison methods, a paired t-test can be computed. The paired t-test allows to test the null hypothesis H0 that the mean of the differences between the results of the two methods is not different from 0, against an alternative hypothesis Ha that it is.

Scatter plots for comparing methods

First, you can draw a scatter plot to compare the reference or comparative method against the method being tested. If the data are on both sides of the identity line (bisector) and close to it, the two methods give close and consistent results. If the data are above the identify line, the new method overestimates the value of interest. If the data are under the line, the new method underestimates the value of interest, at least compared to the comparative or reference method. If the data are crossing the identify line, there is a bias that depends on where you are on the scale of variation. If the data are randomly scattered around the identity line with some observations that are far from it, the new method is not performing well.

Bias for method comparison

The bias is estimated as the mean of the differences between the two methods. If replicates are available, a first step computes the mean of the replicates. The standard deviation is computed as well as a confidence interval. Ideally, the confidence interval should contain 0.

Note: The bias is computed for the criterion that has been chosen for the Bland Altman analysis (difference, difference % or ratio).

Bland Altman and related comparison methods

Bland and Altman recommend plotting the difference (T-S) between the test (T) and comparative or reference method (S) against the average (T+S)/2 of the results obtained from the two methods. In the ideal case, there should not be any correlation between the difference and the average whether there is a bias or not. XLSTAT tests whether the correlation is significantly different from 0 or not. Alternative possibilities are available for the ordinates of the plot: you can choose between the difference (T-S), the difference as a % of the sum (T-S)/(T+S), and the ratio (T/S). On the Bland Altman plot, XLSTAT displays the bias line, the confidence lines around the bias, and the confidence lines around the difference (or the difference % or the ratio).

Histogram and box plot for method comparison

Histogram and box plots of the differences are plotted to validate the hypothesis that both are normally distributed, which is used to compute confidence intervals around the bias and the individual differences. When the size of the samples is small the histogram is of little interest and one should only consider the box plot. If the distribution does not seem to be normal, one might want to verify that point with a normality test, and one should consider with caution the confidence intervals.

Difference plot for comparing methods

The difference plot shows the difference between the two methods against the average of both methods, or against the reference method with an estimate of the bias, using the criterion that has been chosen (difference, difference in %, or ratio), the standard error and a confidence interval being as well displayed.

 

Passing and Bablok regression

Tutorial
View a tutorial
Passing and Bablok (1983) developed a regression method that allows comparing two measurement methods (for example, two techniques for measuring concentration of an analyte), which overcomes the assumptions of the classical single linear regression that are inappropriate for this application. As a reminder the assumptions of the OLS regression are

  • The explanatory variable, X in the model y(i)=a+b.x(i)+ε(i), is deterministic (no measurement error),
  • The dependent variable Y follows a normal distribution with expectation aX
  • The variance of the measurement error is constant.

Furthermore, extreme values can highly influence the model.

Passing and Bablok proposed a method which overcomes these assumptions: the two variables are assumed to have a random part (representing the measurement error and the distribution of the element being measured in medium) without needing to make assumption about their distribution, except that they both have the same distribution. We then define:

  • y(i) = a+b.x(i)+ ξ(i)
  • x(i) = A+B.Y+ η(i)

Where ξ and η follow the same distribution. The Passing and Bablok method allows calculating the a and b coefficients (from which we deduce A and B using B=1/b and A=-a/b) as well as a confidence interval around these values. The study of these values helps comparing the methods. If they are very close, b is close to 1 and a is close to 0.

Passing and Bablok also suggested a linearity test to verify that the relationship between the two measurement methods is stable over the interval of interest. This test is based on a CUSUM statistic that follows a Kolmogorov distribution. XLSTAT provides the statistic, the critical value for the significance level chosen by the user, and the p-value associated with the statistic.

 

Deming regression

Tutorial
View a tutorial
Deming (1943) developed a regression method, that allows comparing two measurement methods (for example, two techniques for measuring concentration of an analyte), which supposes that measurement error is present in both X and Y. It overcomes the assumptions of the classical linear regression that are inappropriate for this application. As a reminder the assumptions of the OLS regression are

  • The explanatory variable, X in the model y(i)=a+b.x(i)+e(i), is deterministic (no measurement error),
  • The dependent variable Y follows a normal distribution with expectation aX
  • The variance of the measurement error is constant.

Furthermore, extreme values can highly influence the model.

Deming proposed a method which overcomes these assumptions: the two variables are assumed to have a random part (representing the measurement). The distribution has to be normal. We then define:

  • y(i)=y(i)*+e(i)
  • x(i)=x(i)*+ η(i)

Assume that the available data (yi, xi) are mismeasured observations of the “true” values (y(i)*, x(i)*) where errors ε and η are independent. The ratio of their variances is assumed to be known:

  • d=s2(e)/s2(h)

In practice, the variance of the x and y is often unknown which complicates the estimate of d but when the measurement methods for x and y are the same they are likely to be equal so that d=1 for this case. XLSTAT-Survival Analysis allows you to define d.

We seek to find the line of “best fit” y* = a + b x*, such that the weighted sum of squared residuals of the model is minimized.

Where h and ε follow a normal distribution. The Deming method allows calculating the a and b coefficients as well as a confidence interval around these values. The study of these values helps comparing the methods. If they are very close, then b is close to 1 and a is close to 0.

The Deming regression has two forms:

  • Simple Deming regression: The error terms are constant and the ratio between variances has to be chosen (with default value being 1). The estimation is very simple using a direct formula (Deming, 1943).
  • Weighted Deming regression: In the case where replicates of the experiments are present, the weighted Deming regression supposes that the error terms are not constant but only proportional. Within each replication, you can take into account the mean or the first experiment to estimate the coefficients. In that case, a direct estimation is not possible. An iterative method is used (Linnet, 1990).

Confidence interval of the intercept and slope coefficient are complex to compute. XLSTAT-Survival Analysis uses a jackknife approach to compute confidence intervals, as stated in Linnet (1993).

About KCS

Kovach Computing Services (KCS) was founded in 1993 by Dr. Warren Kovach. The company specializes in the development and marketing of inexpensive and easy-to-use statistical software for scientists, as well as in data analysis consulting.

Mailing list Join our mailing list

Home | Order | MVSP | Oriana | XLStat
QDA Miner | Accent Composer | Stats Books
Stats Links | Anglesey

Share: FacebookFacebook TwitterTwitter RedditReddit
Del.icio.usDel.icio.us Stumble UponStumble Upon

 

Like us on Facebook

Get in Touch

  • Email:
    sales@kovcomp.com
  • Address:
    85 Nant y Felin
    Pentraeth, Isle of Anglesey
    LL75 8UY
    United Kingdom
  • Phone:
    (UK): 01248-450414
    (Intl.): +44-1248-450414