Software for Exploratory Data Analysis and Statistical Modelling » Statistical Modelling

Generalized Linear Models – Poisson Regression

Ralph — Sun, 26 Jun 2011 09:28:50 +0000

The Generalized Linear Model (GLM) allows us to model responses with distributions other than the Normal distribution, which is one of the assumptions underlying linear regression as used in many cases. When data is counts of events (or items) then a discrete distribution is more appropriate is usually more appropriate than approximating with a continuous distribution, especially as our counts should be bounded below at zero. Negative counts do not make sense.

Fast Tube by Casper

To investigate using Poisson regression via the GLM framework consider a small data set on failure modes (here).

> failure.df = read.table("twomodes.dat", header = TRUE)
> failure.df
  Mode1 Mode2 Failures
1  33.3  25.3       15
2  52.2  14.4        9
3  64.7  32.5       14
4 137.0  20.5       24
5 125.9  97.6       27
6 116.3  53.6       27
7 131.7  56.6       23
8  85.0  87.3       18
9  91.9  47.8       22

The machinery is run in two modes and the objective of the analysis is to determine whether the number of failures depends on how long the machine is run in mode 1 or mode 2 and whether there is an interaction between the time in each mode to increases or decreases the number of failures.

The response for this set of data is the number of failures (count) so a Poisson regression model is considered.

> fmod1 = glm(Failures ~ Mode1 * Mode2, data = failure.df, family = poisson)
> summary(fmod1)
 
Call:
glm(formula = Failures ~ Mode1 * Mode2, family = poisson, data = failure.df)
 
Deviance Residuals: 
       1         2         3         4         5         6         7         8         9  
 0.91003  -1.15601  -0.28328  -0.10398   0.03526   0.84825  -0.49211  -0.57298   0.64821  
 
Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept)  2.105e+00  4.481e-01   4.698 2.63e-06 ***
Mode1        7.687e-03  4.285e-03   1.794   0.0729 .  
Mode2        4.703e-03  1.163e-02   0.405   0.6858    
Mode1:Mode2 -1.978e-05  1.037e-04  -0.191   0.8487    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
(Dispersion parameter for poisson family taken to be 1)
 
    Null deviance: 16.996  on 8  degrees of freedom
Residual deviance:  3.967  on 5  degrees of freedom
AIC: 55.024
 
Number of Fisher Scoring iterations: 4

The model output does not provide any support for an interaction between the number of time spent in the two different modes of operation. If we remove the interaction term and re-fit the model, using the update function, we get:

> fmod2 = update(fmod1, . ~ . - Mode1:Mode2)
> summary(fmod2)
 
Call:
glm(formula = Failures ~ Mode1 + Mode2, family = poisson, data = failure.df)
 
Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.21984  -0.44735  -0.05893   0.68351   0.87510  
 
Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept) 2.175168   0.255456   8.515  < 2e-16 ***
Mode1       0.007015   0.002429   2.888  0.00387 ** 
Mode2       0.002549   0.002835   0.899  0.36852    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
(Dispersion parameter for poisson family taken to be 1)
 
    Null deviance: 16.9964  on 8  degrees of freedom
Residual deviance:  4.0033  on 6  degrees of freedom
AIC: 53.06
 
Number of Fisher Scoring iterations: 4

This output suggests that the time of operation in mode 1 is important for determining the number of faults but the time of operation in mode 2 is not important. One last step gives us:

> fmod3 = update(fmod2, . ~ . - Mode2)
> summary(fmod3)
 
Call:
glm(formula = Failures ~ Mode1, family = poisson, data = failure.df)
 
Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.43194  -0.56958  -0.00745   0.66742   0.82231  
 
Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept) 2.237196   0.243053   9.205  < 2e-16 ***
Mode1       0.007705   0.002264   3.403 0.000667 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
(Dispersion parameter for poisson family taken to be 1)
 
    Null deviance: 16.9964  on 8  degrees of freedom
Residual deviance:  4.8078  on 7  degrees of freedom
AIC: 51.865
 
Number of Fisher Scoring iterations: 4

The diagnostic plots are shown below which do not indicate any major problems with the final model, especially given the small number of data points.

Four diagnostic plots for a Poisson regression model based on total failures

Other useful resources are provided on the Supplementary Material page.

Classification Trees using the rpart function

Ralph — Tue, 21 Sep 2010 19:22:50 +0000

In a previous post on classification trees we considered using the tree package to fit a classification tree to data divided into known classes. In this post we will look at the alternative function rpart that is available within the base R distribution.

Fast Tube by Casper

A classification tree can be fitted using the rpart function using a similar syntax to the tree function. For the ecoli data set discussed in the previous post we would use:

> require(rpart)
> ecoli.df = read.csv("ecoli.txt")

followed by

> ecoli.rpart1 = rpart(class ~ mcv + gvh + lip + chg + aac + alm1 + alm2, 
  data = ecoli.df)

We would then consider whether the tree could be simplified by pruning and make use of the plotcp function:

> plotcp(ecoli.rpart1)

Once the amount of pruning has been determined from this graph or by looking at the output from the printcp function:

> printcp(ecoli.rpart1)
 
Classification tree:
rpart(formula = class ~ mcv + gvh + lip + chg + aac + alm1 + 
    alm2, data = ecoli.df)
 
Variables actually used in tree construction:
[1] aac  alm1 gvh  mcv 
 
Root node error: 193/336 = 0.5744
 
n= 336 
 
        CP nsplit rel error  xerror     xstd
1 0.388601      0   1.00000 1.00000 0.046959
2 0.207254      1   0.61140 0.61658 0.045423
3 0.062176      2   0.40415 0.45596 0.041758
4 0.051813      3   0.34197 0.38342 0.039359
5 0.031088      4   0.29016 0.36269 0.038571
6 0.015544      5   0.25907 0.30570 0.036136
7 0.010000      6   0.24352 0.31088 0.036375

The prune function is used to simplify the tree based on a cp identified from the graph or printed output threshold.

> ecoli.rpart2 = prune(ecoli.rpart1, cp = 0.02)

The classification tree can be visualised with the plot function and then the text function adds labels to the graph:

> plot(ecoli.rpart2, uniform = TRUE)
> text(ecoli.rpart2, use.n = TRUE, cex = 0.75)

Other useful resources are provided on the Supplementary Material page.

Classification Trees

Ralph — Sat, 18 Sep 2010 09:23:21 +0000

Decision trees are applied to situation where data is divided into groups rather than investigating a numerical response and its relationship to a set of descriptor variables. There are various implementations of classification trees in R and the some commonly used functions are rpart and tree.

Fast Tube by Casper

To illustrate the use of the tree function we will use a set of data from the UCI Machine Learning Repository where the objective of the study using this data was to predict the cellular localization sites of proteins.

The data provided on the website is shown here:

> ecoli.df = read.csv("ecoli.txt")
> head(ecoli.df)
    Sequence  mcv  gvh  lip chg  aac alm1 alm2 class
1  AAT_ECOLI 0.49 0.29 0.48 0.5 0.56 0.24 0.35    cp
2 ACEA_ECOLI 0.07 0.40 0.48 0.5 0.54 0.35 0.44    cp
3 ACEK_ECOLI 0.56 0.40 0.48 0.5 0.49 0.37 0.46    cp
4 ACKA_ECOLI 0.59 0.49 0.48 0.5 0.52 0.45 0.36    cp
5  ADI_ECOLI 0.23 0.32 0.48 0.5 0.55 0.25 0.35    cp
6 ALKH_ECOLI 0.67 0.39 0.48 0.5 0.36 0.38 0.46    cp

We can use the xtabs function to summarise the number of cases in each class.

> xtabs( ~ class, data = ecoli.df)
class
 cp  im imL imS imU  om omL  pp 
143  77   2   2  35  20   5  52

As noted in the comments the package that I used was the tree package:

> require(tree)

The complete classification tree using all variables is fitted to the data initially and then we will try to prune the tree to make it smaller.

> ecoli.tree1 = tree(class ~ mcv + gvh + lip + chg + aac + alm1 + alm2,
  data = ecoli.df)
> summary(ecoli.tree1)
 
Classification tree:
tree(formula = class ~ mcv + gvh + lip + chg + aac + alm1 + alm2, 
    data = ecoli.df)
Variables actually used in tree construction:
[1] "alm1" "mcv"  "gvh"  "aac"  "alm2"
Number of terminal nodes:  10 
Residual mean deviance:  0.7547 = 246 / 326 
Misclassification error rate: 0.122 = 41 / 336

The tree function is used in a similar way to other modelling functions in R. The misclassification rate is shown as part of the summary of the tree. This tree can be plotted and annotated with these commands:

> plot(ecoli.tree1)
> text(ecoli.tree1, all = T)

To prune the tree we use cross-validation to identify the point to prune.

> cv.tree(ecoli.tree1)
$size
 [1] 10  9  8  7  6  5  4  3  2  1
 
$dev
 [1]  463.6820  457.4463  447.9824  441.8617  455.8318  478.9234  533.5856  586.2820  713.2992 1040.3878
 
$k
 [1]      -Inf  12.16500  15.60004  19.21572  34.29868  41.10627  50.57044  64.05494 180.78800 355.67747
 
$method
[1] "deviance"
 
attr(,"class")
[1] "prune"         "tree.sequence"

This suggests a tree size of 6 and we can re-fit the tree:

> ecoli.tree2 = prune.misclass(ecoli.tree1, best = 6)
> summary(ecoli.tree2)
 
Classification tree:
snip.tree(tree = ecoli.tree1, nodes = c(4, 20, 7))
Variables actually used in tree construction:
[1] "alm1" "mcv"  "aac"  "gvh" 
Number of terminal nodes:  6 
Residual mean deviance:  0.9918 = 327.3 / 330 
Misclassification error rate: 0.1548 = 52 / 336

The misclassification rate has increased but not substantially with the pruning of the tree.

Other useful resources are provided on the Supplementary Material page.

Data used in this post: Ecoli Data Set.

Variable selection using automatic methods

Ralph — Sat, 22 May 2010 11:15:03 +0000

When we have a set of data with a small number of variables we can easily use a manual approach to identifying a good set of variables and the form they take in our statistical model. In other situations we may have a large number of potentially important variables and it soon becomes a time consuming effort to follow a manual variable selection process. In this case we may consider using automatic subset selection tools to remove some of the burden of the task.

It should be noted that there is some disagreement about whether it is desirable to use an automated method but this post will focus on the mechanics of doing it rather than the debate about whether to be doing it at all.

The R package leaps has a function regsubsets that can be used for best subsets, forward selection and backwards elimination depending on which approach is considered most appropriate for the application under consideration.

In previous post we considered using data on CPU performance to illustrate the variable selection process. We load the required packages:

> require(leaps)
> require(MASS)

First up we consider selecting the best subset of a particular size, say four variables for illustrative purposes (nvmax argument), and we specify the largest possible model which in this example has six variables:

regsubsets(perf ~ syct + mmin + mmax + cach + chmin + chmax,
  data = cpus, nvmax = 4)

A summary for the output from this function is shown here:

> summary(reg1)
Subset selection object
Call: regsubsets.formula(perf ~ syct + mmin + mmax + cach + chmin + 
    chmax, data = cpus, nvmax = 4)
6 Variables  (and intercept)
      Forced in Forced out
syct      FALSE      FALSE
mmin      FALSE      FALSE
mmax      FALSE      FALSE
cach      FALSE      FALSE
chmin     FALSE      FALSE
chmax     FALSE      FALSE
1 subsets of each size up to 4
Selection Algorithm: exhaustive
         syct mmin mmax cach chmin chmax
1  ( 1 ) " "  " "  "*"  " "  " "   " "  
2  ( 1 ) " "  " "  "*"  "*"  " "   " "  
3  ( 1 ) " "  "*"  "*"  " "  " "   "*"  
4  ( 1 ) " "  "*"  "*"  "*"  " "   "*"

The function regsubsets identifies the variables mmin, mmax, cach and chmax as the best four.

Alternatively we could perform a backwards elimination and the function will indicate the best subset of a particular size, from one to six variables in this example:

> reg2 = regsubsets(perf ~ syct + mmin + mmax + cach + chmin + chmax,
  data = cpus, method = "backward")
> summary(reg2)
Subset selection object
Call: regsubsets.formula(perf ~ syct + mmin + mmax + cach + chmin + 
    chmax, data = cpus, method = "backward")
6 Variables  (and intercept)
      Forced in Forced out
syct      FALSE      FALSE
mmin      FALSE      FALSE
mmax      FALSE      FALSE
cach      FALSE      FALSE
chmin     FALSE      FALSE
chmax     FALSE      FALSE
1 subsets of each size up to 6
Selection Algorithm: backward
         syct mmin mmax cach chmin chmax
1  ( 1 ) " "  "*"  " "  " "  " "   " "  
2  ( 1 ) " "  "*"  " "  " "  " "   "*"  
3  ( 1 ) " "  "*"  "*"  " "  " "   "*"  
4  ( 1 ) " "  "*"  "*"  "*"  " "   "*"  
5  ( 1 ) "*"  "*"  "*"  "*"  " "   "*"  
6  ( 1 ) "*"  "*"  "*"  "*"  "*"   "*"

The subset of four variables is the same for this example as the best subsets approach. The third approach if forward selection:

> reg3 = regsubsets(perf ~ syct + mmin + mmax + cach + chmin + chmax,
  data = cpus, method = "backward")
> summary(reg3)
Subset selection object
Call: regsubsets.formula(perf ~ syct + mmin + mmax + cach + chmin + 
    chmax, data = cpus, method = "backward")
6 Variables  (and intercept)
      Forced in Forced out
syct      FALSE      FALSE
mmin      FALSE      FALSE
mmax      FALSE      FALSE
cach      FALSE      FALSE
chmin     FALSE      FALSE
chmax     FALSE      FALSE
1 subsets of each size up to 6
Selection Algorithm: backward
         syct mmin mmax cach chmin chmax
1  ( 1 ) " "  "*"  " "  " "  " "   " "  
2  ( 1 ) " "  "*"  " "  " "  " "   "*"  
3  ( 1 ) " "  "*"  "*"  " "  " "   "*"  
4  ( 1 ) " "  "*"  "*"  "*"  " "   "*"  
5  ( 1 ) "*"  "*"  "*"  "*"  " "   "*"  
6  ( 1 ) "*"  "*"  "*"  "*"  "*"   "*"

For this data set, as there are only six variables, we do not see divergence between the subsets chosen by the different methods.

Linear regression models with robust parameter estimation

Ralph — Sat, 15 May 2010 10:54:51 +0000

There are situations in regression modelling where robust methods could be considered to handle unusual observations that do not follow the general trend of the data set. There are various packages in R that provide robust statistical methods which are summarised on the CRAN Robust Task View.

As an example of using robust statistical estimation in a linear regression framework consider the CPUs data that was used in previous posts on linear regression and variable selection. For this data we could fit a model with six variables using least squares and also with a fast MM-estimator from the robustbase package.

First step is to make the functions and data available for analysis:

require(MASS)
require(robustbase)

The linear model using least squares is fitted as follows:

> cpu.mod1 = lm(perf ~ syct + mmin + mmax + cach + chmin + chmax, data = cpus)

The summary for this model:

> summary(cpu.mod1)
 
Call:
lm(formula = perf ~ syct + mmin + mmax + cach + chmin + chmax, 
    data = cpus)
 
Residuals:
     Min       1Q   Median       3Q      Max 
-195.841  -25.169    5.409   26.528  385.749 
 
Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -5.590e+01  8.045e+00  -6.948 4.99e-11 ***
syct         4.886e-02  1.752e-02   2.789  0.00579 ** 
mmin         1.529e-02  1.827e-03   8.371 9.42e-15 ***
mmax         5.571e-03  6.418e-04   8.680 1.33e-15 ***
cach         6.412e-01  1.396e-01   4.594 7.64e-06 ***
chmin       -2.701e-01  8.557e-01  -0.316  0.75263    
chmax        1.483e+00  2.201e-01   6.738 1.64e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 59.99 on 202 degrees of freedom
Multiple R-squared: 0.8649,     Adjusted R-squared: 0.8609 
F-statistic: 215.5 on 6 and 202 DF,  p-value: < 2.2e-16

The linear model using an MM-estimator is fitted as follows:

cpu.robmod1 = lmrob(perf ~ syct + mmin + mmax + cach + chmin + chmax,
  data = cpus, control = lmrob.control(max.it = 100))

Note that we need to increase the default number of iterations (50) to allow the routine to converge to a solution. The summary for this model:

> summary(cpu.robmod1)
 
Call:
lmrob(formula = perf ~ syct + mmin + mmax + cach + chmin + chmax, 
    data = cpus, control = lmrob.control(max.it = 100))
 
Weighted Residuals:
      Min        1Q    Median        3Q       Max 
-144.0045   -9.4554    0.7691   13.3757  759.6953 
 
Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -3.6634297  6.1697645  -0.594 0.553329    
syct         0.0063112  0.0043877   1.438 0.151868    
mmin         0.0098798  0.0034726   2.845 0.004897 ** 
mmax         0.0024463  0.0004525   5.407 1.80e-07 ***
cach         0.8702102  0.2551245   3.411 0.000782 ***
chmin        2.4078436  1.3319413   1.808 0.072130 .  
chmax        0.1016861  0.1494902   0.680 0.497145    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Robust residual standard error: 19.27 
Convergence in 65 IRWLS iterations
 
Robustness weights: 
 15 observations c(1,8,9,10,31,32,96,97,98,153,154,156,169,199,200) are outliers with |weight| = 0 ( < 0.00048); 
 14 weights are ~= 1. The remaining 180 ones are summarized as
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
0.002472 0.831300 0.963700 0.849800 0.989500 0.998900 
Algorithmic parameters: 
tuning.chi         bb tuning.psi refine.tol    rel.tol 
 1.5476400  0.5000000  4.6850610  0.0000001  0.0000001 
 nResample     max.it     groups    n.group   best.r.s   k.fast.s      k.max  trace.lev compute.rd 
       500        100          5        400          2          1        200          0          0 
seed : int(0)

The two models differ in the variables that are considered important and the output from the lmrob function provides a summary of the weights that have been allocated to the data. A total of fifteen of the data points have been allocated very small weights by the fitting algorithm.

Related posts:

The update function for simplifying model selection.
Data analysis using simple linear regression models.
The dropterm function for simplifying models.

Manual variable selection using the dropterm function

Ralph — Wed, 12 May 2010 20:14:06 +0000

When fitting a multiple linear regression model to data a natural question is whether a model can be simplified by excluding variables from the model. There are automatic procedures for undertaking these tests but some people prefer to follow a more manual approach to variable selection rather than pressing a button and taking what comes out.

Fast Tube by Casper

When there are a large number of variables it is awkward to manually go through each one in turn to make a decision about simplification to a more parsimonious model. In R there is a function dropterm that removes some of this task by assuming that we are interested in considering the outcome of dropping each model term one at a time.

To illustrate this consider the cpus data set in the MASS package which contains information about a relative performance measure and characteristics of 209 CPUs. We load the package first to make the data available:

library(MASS)

We first fit a linear model with six explanatory variables:

cpu.mod1 = lm(perf ~ syct + mmin + mmax + cach + chmin + chmax, data = cpus)

The function dropterm requires a fitted model, which we saved in the last command, and optionally we could specify what test to use to compare the initial model and each of the possible alternative models with one less variable. We can choose to perform an F test:

> dropterm(cpu.mod1, test = "F")
Single term deletions
 
Model:
perf ~ syct + mmin + mmax + cach + chmin + chmax
       Df Sum of Sq    RSS    AIC F Value     Pr(F)    
              727002 1718.3                      
syct    1     27995 754997 1724.2   7.779  0.005793 ** 
mmin    1    252211 979213 1778.5  70.078 9.416e-15 ***
mmax    1    271147 998149 1782.5  75.339 1.326e-15 ***
cach    1     75962 802964 1737.0  21.106 7.640e-06 ***
chmin   1       358 727360 1716.4   0.100  0.752632    
chmax   1    163396 890398 1758.6  45.400 1.640e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The output from the function call indicates that we could excude the chmin variable then re-fit the model and continue again with the same checking process.

Update:

The dropterm function considers each variable individually and considers what the change in residual sum of squares would be if this variable was excluded from the model. There is a link between this F test and the t test that appears as part of the model summary – this is because of the link between these two distributions. For this model we would have:

> summary(cpu.mod1)
 
Call:
lm(formula = perf ~ syct + mmin + mmax + cach + chmin + chmax, 
    data = cpus)
 
Residuals:
     Min       1Q   Median       3Q      Max 
-195.841  -25.169    5.409   26.528  385.749 
 
Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -5.590e+01  8.045e+00  -6.948 4.99e-11 ***
syct         4.886e-02  1.752e-02   2.789  0.00579 ** 
mmin         1.529e-02  1.827e-03   8.371 9.42e-15 ***
mmax         5.571e-03  6.418e-04   8.680 1.33e-15 ***
cach         6.412e-01  1.396e-01   4.594 7.64e-06 ***
chmin       -2.701e-01  8.557e-01  -0.316  0.75263    
chmax        1.483e+00  2.201e-01   6.738 1.64e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 59.99 on 202 degrees of freedom
Multiple R-squared: 0.8649,     Adjusted R-squared: 0.8609 
F-statistic: 215.5 on 6 and 202 DF,  p-value: < 2.2e-16

Let us consider the syct variable. The t statistic in the model summary is 2.789 and if we square this value we get 7.779 which is the F statistic produced by the dropterm function.

Related posts:

The update function for simplifying model selection.
Data analysis using simple linear regression models.
Including factors in a regression model via analysis of covariance.

Using the update function during variable selection

Ralph — Sun, 09 May 2010 18:31:13 +0000

When fitting statistical models to data where there are multiple variables we are often interested in adding or removing terms from our model and in cases where there are a large number of terms it can be quicker to use the update function to start with a formula from a model that we have already fitted and to specify the terms that we want to add or remove as opposed to a copy and paste and manually editing the formula to our needs.

Consider the oil-bearing rocks data set that is available with the R software which is used extensively as an example by many authors. One model that can be used as a starting point is a linear model with additive terms for the three variables:

> rock.mod1 = lm(log(perm) ~ area + peri + shape, data = rock)
> summary(rock.mod1)
 
Call:
lm(formula = log(perm) ~ area + peri + shape, data = rock)
 
Residuals:
    Min      1Q  Median      3Q     Max 
-1.8092 -0.5413  0.1735  0.6493  1.4788 
 
Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)  5.333e+00  5.487e-01   9.720 1.59e-12 ***
area         4.850e-04  8.657e-05   5.602 1.29e-06 ***
peri        -1.527e-03  1.770e-04  -8.623 5.24e-11 ***
shape        1.757e+00  1.756e+00   1.000    0.323    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 0.8521 on 44 degrees of freedom
Multiple R-squared: 0.7483,     Adjusted R-squared: 0.7311 
F-statistic:  43.6 on 3 and 44 DF,  p-value: 3.094e-13

Given this model, saved as an object rock.mod1, we might be interested in considering adding an interaction term between the area and perimeter measurements. The update function has various options and the simplest case is to specfiy a model object and a new formula. The new formula can use the period as short hand for keep everything on either the left or right hand side of the formula and the plus or minus sign used to add or remove terms to the model. In the case of adding an interaction term our call would be:

> rock.mod2 = update(rock.mod1, . ~ . + area:peri)

The first function argument is the name of the model we fitted previously and the periods indicate that we want to use the same response variable and to start with the whole formula but add an interaction term between area and perimeter – the colon is used to specify an interaction term by itself. This fitted model is now:

> summary(rock.mod2)
 
Call:
lm(formula = log(perm) ~ area + peri + shape + area:peri, data = rock)
 
Residuals:
    Min      1Q  Median      3Q     Max 
-1.7255 -0.4760  0.1256  0.6539  1.4269 
 
Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)  6.567e+00  8.533e-01   7.696 1.28e-09 ***
area         3.769e-04  1.025e-04   3.678  0.00065 ***
peri        -2.141e-03  3.734e-04  -5.733 8.94e-07 ***
shape        4.022e-01  1.859e+00   0.216  0.82974    
area:peri    6.641e-08  3.583e-08   1.854  0.07065 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 0.8295 on 43 degrees of freedom
Multiple R-squared: 0.7669,     Adjusted R-squared: 0.7452 
F-statistic: 35.37 on 4 and 43 DF,  p-value: 4.404e-13

The update function can also be used to change other aspects of the linear model or in fact many other types of model are set up to repsond sensibly to using this function.

Related posts:

Manual variable selection with the dropterm function.
Data analysis using simple linear regression models.
Including factors in a regression model via analysis of covariance.

Analysis of Covariance – Extending Simple Linear Regression

Ralph — Wed, 28 Apr 2010 19:25:58 +0000

The simple linear regression model considers the relationship between two variables and in many cases more information will be available that can be used to extend the model. For example, there might be a categorical variable (sometimes known as a covariate) that can be used to divide the data set to fit a separate linear regression to each of the subsets. We will consider how to handle this extension using one of the data sets available within the R software package.

There is a set of data relating trunk circumference (in mm) to the age of Orange trees where data was recorded for five trees. This data is available in the data frame Orange and we make a copy of this data set so that we can remove the ordering that is recorded for the Tree identifier variable. We create a new factor after converting the old factor to a numeric string:

orange.df = Orange
orange.df$Tree = factor(as.numeric(orange.df$Tree))

The purpose of this step is to set up the variable for use in the linear model. The simplest model assumes that the relationship between circumference and age is the same for all five trees and we fit this model as follows:

orange.mod1 = lm(circumference ~ age, data = orange.df)

The summary of the fitted model is shown here:

> summary(orange.mod1)
 
Call:
lm(formula = circumference ~ age, data = orange.df)
 
Residuals:
      Min        1Q    Median        3Q       Max 
-46.31030 -14.94610  -0.07649  19.69727  45.11146 
 
Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 17.399650   8.622660   2.018   0.0518 .  
age          0.106770   0.008277  12.900 1.93e-14 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 23.74 on 33 degrees of freedom
Multiple R-squared: 0.8345,     Adjusted R-squared: 0.8295 
F-statistic: 166.4 on 1 and 33 DF,  p-value: 1.931e-14

The test on the age parameter provides very strong evidence of an increase in circumference with age, as would be expected. The next stage is to consider how this model can be extended – one idea is to have a separate intercept for each of the five trees. This new model assumes that the increase in circumference is consistent between the trees but that the growth starts at different rates. We fit this model and get the summary as follows:

> orange.mod2 = lm(circumference ~ age + Tree, data = orange.df)
> summary(orange.mod2)
 
Call:
lm(formula = circumference ~ age + Tree, data = orange.df)
 
Residuals:
    Min      1Q  Median      3Q     Max 
-30.505  -8.790   3.738   7.650  21.859 
 
Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) -4.457493   7.572732  -0.589   0.5607    
age          0.106770   0.005321  20.066  < 2e-16 ***
Tree2        5.571429   8.157252   0.683   0.5000    
Tree3       17.142857   8.157252   2.102   0.0444 *  
Tree4       41.285714   8.157252   5.061 2.14e-05 ***
Tree5       45.285714   8.157252   5.552 5.48e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 15.26 on 29 degrees of freedom
Multiple R-squared: 0.9399,     Adjusted R-squared: 0.9295 
F-statistic:  90.7 on 5 and 29 DF,  p-value: < 2.2e-16

The additional term is appended to the simple model using the + in the formula part of the call to lm. The first tree is used as the baseline to compare the other four trees against and the model summary shows that tree 2 is similar to tree 1 (no real need for a different offset) but that there is evidence that the offset for the other three trees is significantly larger than tree 1 (and tree 2). We can compare the two models using an F-test for nested models using the anova function:

> anova(orange.mod1, orange.mod2)
Analysis of Variance Table
 
Model 1: circumference ~ age
Model 2: circumference ~ age + Tree
  Res.Df     RSS Df Sum of Sq      F    Pr(>F)    
1     33 18594.7                                  
2     29  6753.9  4     11841 12.711 4.289e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Here there are four degrees of freedom used up by the more complicated model (four parameters for the different trees) and the test comparing the two models is highly significant. There is very strong evidence of a difference in starting circumference (for the data that was collected) between the trees.

We can extended this model further by allowing the rate of increase in circumference to vary between the five trees. This additional term can be included in the linear model as an interaction term, assuming that tree 1 is the baseline. An interaction term is included in the model formula with a : between the name of two variables. For the Orange tree data the new model is fitted thus:

> orange.mod3 = lm(circumference ~ age + Tree + age:Tree, data = orange.df)
> summary(orange.mod3)
 
Call:
lm(formula = circumference ~ age + Tree + age:Tree, data = orange.df)
 
Residuals:
    Min      1Q  Median      3Q     Max 
-18.061  -6.639  -1.482   8.069  16.649 
 
Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)  1.920e+01  8.458e+00   2.270  0.03206 *  
age          8.111e-02  8.119e-03   9.991 3.27e-10 ***
Tree2        5.234e+00  1.196e+01   0.438  0.66544    
Tree3       -1.045e+01  1.196e+01  -0.873  0.39086    
Tree4        7.574e-01  1.196e+01   0.063  0.95002    
Tree5       -4.566e+00  1.196e+01  -0.382  0.70590    
age:Tree2    3.656e-04  1.148e-02   0.032  0.97485    
age:Tree3    2.992e-02  1.148e-02   2.606  0.01523 *  
age:Tree4    4.395e-02  1.148e-02   3.828  0.00077 ***
age:Tree5    5.406e-02  1.148e-02   4.708 7.93e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 10.41 on 25 degrees of freedom
Multiple R-squared: 0.9759,     Adjusted R-squared: 0.9672 
F-statistic: 112.4 on 9 and 25 DF,  p-value: < 2.2e-16

Interesting we see that there is strong evidence of a difference in the rate of change in circumference for the five trees. The previously observed difference in intercepts is now longer as strong but this parameter is kept in the model – there are plenty of books/websites that discuss this marginality restrictin on statistical models. The fitted model described above can be created using lattice graphics with a custom panel function making use of available panel functions for fitting and drawing a linear regression line for each panel of a Trellis display. The function call is shown below:

xyplot(circumference ~ age | Tree, data = orange.df,
  panel = function(x, y, ...)
  {
    panel.xyplot(x, y, ...)
    panel.lmline(x, y, ...)
  }
)

The panel.xyplot and panel.lmline functions are part of the lattice package along with many other panel functions and can be built up to create a display that differs from the standard. The graph that is produced:

Analysis of Covariance Model fitted to the Orange Tree data

This graph clearly shows the different relationships between circumference and age for the five trees. The residuals from the model can be plotted against fitted values, divided by tree, to investigate the model assumptions:

xyplot(resid(orange.mod3) ~ fitted(orange.mod3) | orange.df$Tree,
  xlab = "Fitted Values",
  ylab = "Residuals",
  main = "Residual Diagnostic Plot",
  panel = function(x, y, ...)
  {
    panel.grid(h = -1, v = -1)
    panel.abline(h = 0)
    panel.xyplot(x, y, ...)
  }
)

The residual diagnostic plot is:

Residual diagnostic plot for the analysis of covariance model fitted to the Orange Tree data

There are no obvious problematic patterns in this graph so we conclude that this model is a reasonable representation of the relationship between circumference and age.

Additional: The analysis of variance table comparing the second and third models shows an improvement by moving to the more complicated model with different slopes:

> anova(orange.mod2, orange.mod3)
Analysis of Variance Table
 
Model 1: circumference ~ age + Tree
Model 2: circumference ~ age + Tree + age:Tree
  Res.Df    RSS Df Sum of Sq      F    Pr(>F)    
1     29 6753.9                                  
2     25 2711.0  4    4042.9 9.3206 9.402e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Simple Linear Regression

Ralph — Fri, 23 Apr 2010 08:51:57 +0000

One of the most frequent used techniques in statistics is linear regression where we investigate the potential relationship between a variable of interest (often called the response variable but there are many other names in use) and a set of one of more variables (known as the independent variables or some other term). Unsurprisingly there are flexible facilities in R for fitting a range of linear models from the simple case of a single variable to more complex relationships.

In this post we will consider the case of simple linear regression with one response variable and a single independent variable. For this example we will use some data from the book Mathematical Statistics with Applications by Mendenhall, Wackerly and Scheaffer (Fourth Edition – Duxbury 1990). This data is for a study in central Florida where 15 alligators were captured and two measurements were made on each of the alligators. The weight (in pounds) was recorded with the snout vent length (in inches – this is the distance between the back of the head to the end of the nose).

The purpose of using this data is to determine whether there is a relationship, described by a simple linear regression model, between the weight and snout vent length. The authors analysed the data on the log scale (natural logarithms) and we will follow their approach for consistency. We first create a data frame for this study:

alligator = data.frame(
  lnLength = c(3.87, 3.61, 4.33, 3.43, 3.81, 3.83, 3.46, 3.76,
    3.50, 3.58, 4.19, 3.78, 3.71, 3.73, 3.78),
  lnWeight = c(4.87, 3.93, 6.46, 3.33, 4.38, 4.70, 3.50, 4.50,
    3.58, 3.64, 5.90, 4.43, 4.38, 4.42, 4.25)
)

As with most analysis the first step is to perform some exploratory data analysis to get a visual impression of whether there is a relationship between weight and snout vent length and what form it is likely to take. We create a scatter plot of the data as follows:

xyplot(lnWeight ~ lnLength, data = alligator,
  xlab = "Snout vent length (inches) on log scale",
  ylab = "Weight (pounds) on log scale",
  main = "Alligators in Central Florida"
)

The scatter plot is shown here:

Scatter plot of the weight and snout vent length for alligators caught in central Florida

The graph suggests that weight (on the log scale) increases linearly with snout vent length (again on the log scale) so we will fit a simple linear regression model to the data and save the fitted model to an object for further analysis:

alli.mod1 = lm(lnWeight ~ lnLength, data = alligator)

The function lm fits a linear model to data are we specify the model using a formula where the response variable is on the left hand side separated by a ~ from the explanatory variables. The formula provides a flexible way to specify various different functional forms for the relationship. The data argument is used to tell R where to look for the variables used in the formula.

Now that the model is saved as an object we can use some of the general purpose functions for extracting information from this object about the linear model, e.g. the parameters or residuals. The big plus with R is that there are functions defined for different types of model, using the same name such as summary, and the system works out what function we intended to use based on the type of object saved. To create a summary of the fitted model:

> summary(alli.mod1)
 
Call:
lm(formula = lnWeight ~ lnLength, data = alligator)
 
Residuals:
     Min       1Q   Median       3Q      Max 
-0.24348 -0.03186  0.03740  0.07727  0.12669 
 
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -8.4761     0.5007  -16.93 3.08e-10 ***
lnLength      3.4311     0.1330   25.80 1.49e-12 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
Residual standard error: 0.1229 on 13 degrees of freedom
Multiple R-squared: 0.9808,     Adjusted R-squared: 0.9794 
F-statistic: 665.8 on 1 and 13 DF,  p-value: 1.495e-12

We get a lot of useful information here without being too overwhelmed by pages of output.

The estimates for the model intercept is -8.4761 and the coefficient measuring the slope of the relationship with snout vent length is 3.4311 and information about standard errors of these estimates is also provided in the Coefficients table. We see that the test of significance of the model coefficients is also summarised in that table so we can see that there is strong evidence that the coefficient is significantly different to zero – as the snout vent length increases so does the weight.

Rather than stopping here we perform some investigations using residual diagnostics to determine whether the various assumptions that underpin linear regression are reasonable for our data or if there is evidence to suggest that additional variables are required in the model or some other alterations to identify a better description of the variables that determine how weight changes.

A plot of the residuals against fitted values is used to determine whether there are any systematic patterns, such as over estimation for most of the large values or increasing spread as the model fitted values increase. To create this plot we could use the following code:

xyplot(resid(alli.mod1) ~ fitted(alli.mod1),
  xlab = "Fitted Values",
  ylab = "Residuals",
  main = "Residual Diagnostic Plot",
  panel = function(x, y, ...)
  {
    panel.grid(h = -1, v = -1)
    panel.abline(h = 0)
    panel.xyplot(x, y, ...)
  }
)

We create our own custom panel function using the buliding blocks provided by the lattice package. We start by creating a set of grid lines as the base layer and the h=-1 and v=-1 tell lattice to align these with the labels on the axes. We then create a solid horizontal line to help distinguish between positive and negative residuals. Finally we get the points plotted on the top layer.

The residual diagnostic plot is shown below:

Residual Diagnostics Plot for the Linear Regression Model

The plot is probably ok but there are more cases of positive residuals and when we consider a normal probability plot we see that there are some deficiencies with the model:

qqmath( ~ resid(alli.mod1),
  xlab = "Theoretical Quantiles",
  ylab = "Residuals"
)

The function resid extracts the model residuals from the fitted model object. The plot is shown here:

Quantile-Quantile Plot for the Linear Regression Model

We would hope that this plot showed something approaching a straight line to support the model assumption about the distribution of the residuals. This and the other plots suggest that further tweaking to the model is required to improve the model or a decision would need to be made about whether to report the model as is with some caveats about its usage. I am interested in the thoughts/comments/suggestions from how other people would proceed when faced with this situation – feel free to add in the comments.

Related posts:

Manual variable selection with the dropterm function.
The update function for simplifying model selection.
Including factors in a regression model via analysis of covariance.

Two-way Analysis of Variance (ANOVA)

Ralph — Mon, 15 Feb 2010 21:45:02 +0000

The analysis of variance (ANOVA) model can be extended from making a comparison between multiple groups to take into account additional factors in an experiment. The simplest extension is from one-way to two-way ANOVA where a second factor is included in the model as well as a potential interaction between the two factors.

As an example consider a company that regularly has to ship parcels between its various (five for this example) sub-offices and has the option of using three competing parcel delivery services, all of which charge roughly similar amounts for each delivery. To determine which service to use, the company decides to run an experiment shipping three packages from its head office to each of the five sub-offices. The delivery time for each package is recorded and the data loaded into R:

delivery.df = data.frame(
  Service = c(rep("Carrier 1", 15), rep("Carrier 2", 15),
    rep("Carrier 3", 15)),
  Destination = c(rep(c("Office 1", "Office 2", "Office 3",
    "Office 4", "Office 5"), 9)),
  Time = c(15.23, 14.32, 14.77, 15.12, 14.05,
  15.48, 14.13, 14.46, 15.62, 14.23, 15.19, 14.67, 14.48, 15.34, 14.22,
  16.66, 16.27, 16.35, 16.93, 15.05, 16.98, 16.43, 15.95, 16.73, 15.62,
  16.53, 16.26, 15.69, 16.97, 15.37, 17.12, 16.65, 15.73, 17.77, 15.52,
  16.15, 16.86, 15.18, 17.96, 15.26, 16.36, 16.44, 14.82, 17.62, 15.04)
)

The data is then displayed using a dot plot for an initial visual investigation of any trends in delivery time between the three services and across the five sub-offices. The colour aesthetic is used to distinguish between the three services in the plot.

ggplot(delivery.df, aes(Time, Destination, colour = Service)) + geom_point()

This code produces the following graph:

Graph of the delivery time for different services and destintions

The graph shows a general pattern of service carrier 1 having shorter delivery times than the other two services. There is also an indication that the differences between the services varies for the five sub-offices and we might expect the interaction term to be significant in the two-way ANOVA model. To fit the two-way ANOVA model we use this code:

delivery.mod1 = aov(Time ~ Destination*Service, data = delivery.df)

The * symbol instructs R to create a formula that includes main effects for both Destination and Service as well as the two-way interaction between these two factors. We save the fitted model to an object which we can summarise as follows to test for importance of the various model terms:

> summary(delivery.mod1)
                    Df  Sum Sq Mean Sq  F value    Pr(>F)    
Destination          4 17.5415  4.3854  61.1553 5.408e-14 ***
Service              2 23.1706 11.5853 161.5599 < 2.2e-16 ***
Destination:Service  8  4.1888  0.5236   7.3018 2.360e-05 ***
Residuals           30  2.1513  0.0717                       
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

We have strong evidence here that there are differences between the three delivery services, between the five sub-office destinations and that there is an interaction between destination and service in line with what we saw in the original plot of the data. Now that we have fitted the model and identified the important factors we need to investigate the model diagnostics to ensure that the various assumptions are broadly valid.

We can plot the model residuals against fitted values to look for obvious trends that are not consistent with the model assumptions about independence and common variance. The first step is to create a data frame with the fitted values and residuals from the above model:

delivery.res = delivery.df
delivery.res$M1.Fit = fitted(delivery.mod1)
delivery.res$M1.Resid = resid(delivery.mod1)

Then a scatter plot is used to display the fitted values and residuals where the colour asthetic highlights which points correspond to the three competing delivery services:

ggplot(delivery.res, aes(M1.Fit, M1.Resid, colour = Service)) + geom_point() +
  xlab("Fitted Values") + ylab("Residuals")

The xlab() and ylab() are used to change the text on the axis labels. The residual diagnostic plot is:

Diagnostic Residual Plot for Delivery Time Model

There are no obvious patterns in this plot that suggest problems with the two-way ANOVA model that we fitted to the data.

As an alternative display we could separate the residuals into destination sub-offices, where the facet_wrap() function instructs ggplot to create a separate display (panel) for each of the destinations.

ggplot(delivery.res, aes(M1.Fit, M1.Resid, colour = Service)) +
  geom_point() + xlab("Fitted Values") + ylab("Residuals") +
  facet_wrap( ~ Destination)

To produce the following alternative residual plot:

Diagnostic Residual Plot for Delivery Time Model by Destination

No obvious problems in this diagnostic plot.

We could also consider dividing the data by delivery service to get a different view of the residuals:

ggplot(delivery.res, aes(M1.Fit, M1.Resid, colour = Destination)) +
  geom_point() + xlab("Fitted Values") + ylab("Residuals") +
  facet_wrap( ~ Service)

This creates the following graph:

Diagnostic Residual Plot for Delivery Time Model by Service

Again there is nothing substantial here to lead us to consider an alternative analysis.

Lastly we consider the normal probability plot of the model residuals, using the stat_qq() option:

ggplot(delivery.res, aes(sample = M1.Resid)) + stat_qq()

The quantile plot is:

Normal Probability Plot for Delivery Time Model

This plot is very close to the straight line we would expect to observe if the data was a close approximation to a normal distribution. To round off the analysis we look at the Tukey HSD multiple comparisons to confirm that the differences are between delivery service 1 and the other two competing services:

> TukeyHSD(delivery.mod1, which = "Service")
  Tukey multiple comparisons of means
    95% family-wise confidence level
 
Fit: aov(formula = Time ~ Destination * Service, data = delivery.df)
 
$Service
                        diff        lwr       upr     p adj
Carrier 2-Carrier 1 1.498667  1.2576092 1.7397241 0.0000000
Carrier 3-Carrier 1 1.544667  1.3036092 1.7857241 0.0000000
Carrier 3-Carrier 2 0.046000 -0.1950575 0.2870575 0.8856246

Even with the multiple comparison post-hoc adjustment there is very strong evidence for the differences that we have consistenly observed throughout the analysis.

We can use ggplot to visualise the difference in mean delivery time for the services and the 95% confidence intervals on these differences. We create a data frame from the TukeyHSD output by extracting the component relating to the delivery service comparison and add the text labels by extracting the row names from the data frame.

delivery.hsd = data.frame(TukeyHSD(delivery.mod1, which = "Service")$Service)
delivery.hsd$Comparison = row.names(delivery.hsd)

We then use the geom_pointrange() to specify lower, middle and upper values based on the three pairwise comparisons of interest.

ggplot(delivery.hsd, aes(Comparison, y = diff, ymin = lwr, ymax = upr)) +
  geom_pointrange() + ylab("Difference in Mean Delivery Time by Service") +
  coord_flip()

The coord_flip() is used to make the confidence intervals horizontal rather than vertical on the graph. This can be confusing for creating the axis labels as we specify the label where it would appear prior to the filp of coordinates. In the example above we add text to the y axis but this now appears on the x axis in the final graph:

Plot of Confidence Intervals for Mean Differences using Tukey HSD