Software for Exploratory Data Analysis and Statistical Modelling » Statistical Analysis

Generalized Linear Models – Poisson Regression

Ralph — Sun, 26 Jun 2011 09:28:50 +0000

The Generalized Linear Model (GLM) allows us to model responses with distributions other than the Normal distribution, which is one of the assumptions underlying linear regression as used in many cases. When data is counts of events (or items) then a discrete distribution is more appropriate is usually more appropriate than approximating with a continuous distribution, especially as our counts should be bounded below at zero. Negative counts do not make sense.

Fast Tube by Casper

To investigate using Poisson regression via the GLM framework consider a small data set on failure modes (here).

> failure.df = read.table("twomodes.dat", header = TRUE)
> failure.df
  Mode1 Mode2 Failures
1  33.3  25.3       15
2  52.2  14.4        9
3  64.7  32.5       14
4 137.0  20.5       24
5 125.9  97.6       27
6 116.3  53.6       27
7 131.7  56.6       23
8  85.0  87.3       18
9  91.9  47.8       22

The machinery is run in two modes and the objective of the analysis is to determine whether the number of failures depends on how long the machine is run in mode 1 or mode 2 and whether there is an interaction between the time in each mode to increases or decreases the number of failures.

The response for this set of data is the number of failures (count) so a Poisson regression model is considered.

> fmod1 = glm(Failures ~ Mode1 * Mode2, data = failure.df, family = poisson)
> summary(fmod1)
 
Call:
glm(formula = Failures ~ Mode1 * Mode2, family = poisson, data = failure.df)
 
Deviance Residuals: 
       1         2         3         4         5         6         7         8         9  
 0.91003  -1.15601  -0.28328  -0.10398   0.03526   0.84825  -0.49211  -0.57298   0.64821  
 
Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept)  2.105e+00  4.481e-01   4.698 2.63e-06 ***
Mode1        7.687e-03  4.285e-03   1.794   0.0729 .  
Mode2        4.703e-03  1.163e-02   0.405   0.6858    
Mode1:Mode2 -1.978e-05  1.037e-04  -0.191   0.8487    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
(Dispersion parameter for poisson family taken to be 1)
 
    Null deviance: 16.996  on 8  degrees of freedom
Residual deviance:  3.967  on 5  degrees of freedom
AIC: 55.024
 
Number of Fisher Scoring iterations: 4

The model output does not provide any support for an interaction between the number of time spent in the two different modes of operation. If we remove the interaction term and re-fit the model, using the update function, we get:

> fmod2 = update(fmod1, . ~ . - Mode1:Mode2)
> summary(fmod2)
 
Call:
glm(formula = Failures ~ Mode1 + Mode2, family = poisson, data = failure.df)
 
Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.21984  -0.44735  -0.05893   0.68351   0.87510  
 
Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept) 2.175168   0.255456   8.515  < 2e-16 ***
Mode1       0.007015   0.002429   2.888  0.00387 ** 
Mode2       0.002549   0.002835   0.899  0.36852    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
(Dispersion parameter for poisson family taken to be 1)
 
    Null deviance: 16.9964  on 8  degrees of freedom
Residual deviance:  4.0033  on 6  degrees of freedom
AIC: 53.06
 
Number of Fisher Scoring iterations: 4

This output suggests that the time of operation in mode 1 is important for determining the number of faults but the time of operation in mode 2 is not important. One last step gives us:

> fmod3 = update(fmod2, . ~ . - Mode2)
> summary(fmod3)
 
Call:
glm(formula = Failures ~ Mode1, family = poisson, data = failure.df)
 
Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.43194  -0.56958  -0.00745   0.66742   0.82231  
 
Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept) 2.237196   0.243053   9.205  < 2e-16 ***
Mode1       0.007705   0.002264   3.403 0.000667 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
 
(Dispersion parameter for poisson family taken to be 1)
 
    Null deviance: 16.9964  on 8  degrees of freedom
Residual deviance:  4.8078  on 7  degrees of freedom
AIC: 51.865
 
Number of Fisher Scoring iterations: 4

The diagnostic plots are shown below which do not indicate any major problems with the final model, especially given the small number of data points.

Four diagnostic plots for a Poisson regression model based on total failures

Other useful resources are provided on the Supplementary Material page.

Fractional Factorial Designs using FrF2

Ralph — Wed, 18 May 2011 18:17:18 +0000

The FrF2 package for R can be used to create regular and non-regular Fractional Factorial 2-level designs. It is reasonably straightforward to use.

First step is to install the package then make it available for use in the current session:

require(FrF2)

A basic call to the main functino FrF2 specifies the number of runs in the fractional factorial design (which needs to be a multiple of 2) and the number of factors. For example a three factor design would have a total of eight runs if it was a full factorial but if we wanted to go with four runs then we can generate the design like this:

> FrF2(4, 3)
   A  B  C
1  1 -1 -1
2 -1  1 -1
3 -1 -1  1
4  1  1  1
class=design, type= FrF2

The default output labels the factors A, B, C and so on and the factor levels are -1 and +1 for the two levels of each factor. We can change the level names to low and high using the default.levels function argument:

> FrF2(4, 3, default.levels = c("low", "high"))
     A    B    C
1 high high high
2  low high  low
3 high  low  low
4  low  low high
class=design, type= FrF2

The factors can be specified as a list of names rather than the number of factors via the factor.names argument:

> FrF2(4, factor.names = c("One", "Two", "Three"),
  default.levels = c("low", "high"))
   One  Two Three
1  low high   low
2 high high  high
3  low  low  high
4 high  low   low
class=design, type= FrF2

These are the basics and there are other features for greater control over the confounding between factors and their interactions that is introduced by using a fractional factorial design.

Data Mining with WEKA

Ralph — Sun, 30 Jan 2011 17:53:31 +0000

There are a number of good open source projects for statistics and data mining, for example the software WEKA developed at the University of Waikato.

The description on their website states that:

Weka is a collection of machine learning algorithms for data mining tasks.
The algorithms can either be applied directly to a dataset or called from
your own Java code. Weka contains tools for data pre-processing,
classification, regression, clustering, association rules, and visualization.
It is also well-suited for developing new machine learning schemes.

The software is written in Java and available under the GNU General Public Licence. The website also provides access to data sets from the UCI Machine Learning website for use with WEKA.

Gapminder

Ralph — Thu, 06 Jan 2011 18:53:41 +0000

As many people are aware Hans Rosling is an enthusiastic swedish academic with a passion for statistics who recently presented the program The Joy of Stats. One of the great things about Hans Rosling is his presentations and the interactive graphics that he uses to make his points.

Fast Tube by Casper

The gapminder software used in his presentations is available for experimentation and it is interesting to have a play with it as it can help with thinking about ways to effectively present data. The website also has information about the data sources.

Plotting Time Series data using ggplot2

Ralph — Thu, 30 Sep 2010 21:05:18 +0000

There are various ways to plot data that is represented by a time series in R. The ggplot2 package has scales that can handle dates reasonably easily.

Fast Tube by Casper

As an example consider a data set on the number of views of the you tube channel ramstatvid. A short snippet of the data is shown here:

> head(yt.views)
        Date Views
1 2010-05-17    13
2 2010-05-18    11
3 2010-05-19     4
4 2010-05-20     2
5 2010-05-21    23
6 2010-05-22    26

The ggplot function is used by specifying a data frame and the aes maps the Date to the x-axis and the number of Views to the y-axis.

ggplot(yt.views, aes(Date, Views)) + geom_line() +
  scale_x_date(format = "%b-%Y") + xlab("") + ylab("Daily Views")

The axis labels for the Date variable are created with the scale_x_date function where the format is specified as a Month/Year combination with the %b and %Y formatting strings. The graph that is produced is shown here:

Time Series Plot Example with ggplot2 package

Other useful resources are provided on the Supplementary Material page.

Classification Trees using the rpart function

Ralph — Tue, 21 Sep 2010 19:22:50 +0000

In a previous post on classification trees we considered using the tree package to fit a classification tree to data divided into known classes. In this post we will look at the alternative function rpart that is available within the base R distribution.

Fast Tube by Casper

A classification tree can be fitted using the rpart function using a similar syntax to the tree function. For the ecoli data set discussed in the previous post we would use:

> require(rpart)
> ecoli.df = read.csv("ecoli.txt")

followed by

> ecoli.rpart1 = rpart(class ~ mcv + gvh + lip + chg + aac + alm1 + alm2, 
  data = ecoli.df)

We would then consider whether the tree could be simplified by pruning and make use of the plotcp function:

> plotcp(ecoli.rpart1)

Once the amount of pruning has been determined from this graph or by looking at the output from the printcp function:

> printcp(ecoli.rpart1)
 
Classification tree:
rpart(formula = class ~ mcv + gvh + lip + chg + aac + alm1 + 
    alm2, data = ecoli.df)
 
Variables actually used in tree construction:
[1] aac  alm1 gvh  mcv 
 
Root node error: 193/336 = 0.5744
 
n= 336 
 
        CP nsplit rel error  xerror     xstd
1 0.388601      0   1.00000 1.00000 0.046959
2 0.207254      1   0.61140 0.61658 0.045423
3 0.062176      2   0.40415 0.45596 0.041758
4 0.051813      3   0.34197 0.38342 0.039359
5 0.031088      4   0.29016 0.36269 0.038571
6 0.015544      5   0.25907 0.30570 0.036136
7 0.010000      6   0.24352 0.31088 0.036375

The prune function is used to simplify the tree based on a cp identified from the graph or printed output threshold.

> ecoli.rpart2 = prune(ecoli.rpart1, cp = 0.02)

The classification tree can be visualised with the plot function and then the text function adds labels to the graph:

> plot(ecoli.rpart2, uniform = TRUE)
> text(ecoli.rpart2, use.n = TRUE, cex = 0.75)

Other useful resources are provided on the Supplementary Material page.

Classification Trees

Ralph — Sat, 18 Sep 2010 09:23:21 +0000

Decision trees are applied to situation where data is divided into groups rather than investigating a numerical response and its relationship to a set of descriptor variables. There are various implementations of classification trees in R and the some commonly used functions are rpart and tree.

Fast Tube by Casper

To illustrate the use of the tree function we will use a set of data from the UCI Machine Learning Repository where the objective of the study using this data was to predict the cellular localization sites of proteins.

The data provided on the website is shown here:

> ecoli.df = read.csv("ecoli.txt")
> head(ecoli.df)
    Sequence  mcv  gvh  lip chg  aac alm1 alm2 class
1  AAT_ECOLI 0.49 0.29 0.48 0.5 0.56 0.24 0.35    cp
2 ACEA_ECOLI 0.07 0.40 0.48 0.5 0.54 0.35 0.44    cp
3 ACEK_ECOLI 0.56 0.40 0.48 0.5 0.49 0.37 0.46    cp
4 ACKA_ECOLI 0.59 0.49 0.48 0.5 0.52 0.45 0.36    cp
5  ADI_ECOLI 0.23 0.32 0.48 0.5 0.55 0.25 0.35    cp
6 ALKH_ECOLI 0.67 0.39 0.48 0.5 0.36 0.38 0.46    cp

We can use the xtabs function to summarise the number of cases in each class.

> xtabs( ~ class, data = ecoli.df)
class
 cp  im imL imS imU  om omL  pp 
143  77   2   2  35  20   5  52

As noted in the comments the package that I used was the tree package:

> require(tree)

The complete classification tree using all variables is fitted to the data initially and then we will try to prune the tree to make it smaller.

> ecoli.tree1 = tree(class ~ mcv + gvh + lip + chg + aac + alm1 + alm2,
  data = ecoli.df)
> summary(ecoli.tree1)
 
Classification tree:
tree(formula = class ~ mcv + gvh + lip + chg + aac + alm1 + alm2, 
    data = ecoli.df)
Variables actually used in tree construction:
[1] "alm1" "mcv"  "gvh"  "aac"  "alm2"
Number of terminal nodes:  10 
Residual mean deviance:  0.7547 = 246 / 326 
Misclassification error rate: 0.122 = 41 / 336

The tree function is used in a similar way to other modelling functions in R. The misclassification rate is shown as part of the summary of the tree. This tree can be plotted and annotated with these commands:

> plot(ecoli.tree1)
> text(ecoli.tree1, all = T)

To prune the tree we use cross-validation to identify the point to prune.

> cv.tree(ecoli.tree1)
$size
 [1] 10  9  8  7  6  5  4  3  2  1
 
$dev
 [1]  463.6820  457.4463  447.9824  441.8617  455.8318  478.9234  533.5856  586.2820  713.2992 1040.3878
 
$k
 [1]      -Inf  12.16500  15.60004  19.21572  34.29868  41.10627  50.57044  64.05494 180.78800 355.67747
 
$method
[1] "deviance"
 
attr(,"class")
[1] "prune"         "tree.sequence"

This suggests a tree size of 6 and we can re-fit the tree:

> ecoli.tree2 = prune.misclass(ecoli.tree1, best = 6)
> summary(ecoli.tree2)
 
Classification tree:
snip.tree(tree = ecoli.tree1, nodes = c(4, 20, 7))
Variables actually used in tree construction:
[1] "alm1" "mcv"  "aac"  "gvh" 
Number of terminal nodes:  6 
Residual mean deviance:  0.9918 = 327.3 / 330 
Misclassification error rate: 0.1548 = 52 / 336

The misclassification rate has increased but not substantially with the pruning of the tree.

Other useful resources are provided on the Supplementary Material page.

Data used in this post: Ecoli Data Set.

Charting the performance of cricket all-rounders – IT Botham

Ralph — Mon, 16 Aug 2010 19:59:54 +0000

Cricket is a sport that generates a large volume of performance data and corresponding debate about the relative qualities of various players over their careers and in relation to their contemporaries. The cricinfo website has an extensive database of statistics for professional cricketers that can be searched to access the information in various formats.

As an initial example we will consider the English legend Sir Ian Botham who played 102 test matches for England between his debut in 1977 until his final game in 1992.

The first obvious breakdown is to consider how Botham performed against the six countries that he played against during his test career. A summary of his statistics are shown here:

 Opposition Matches Bat Inns Runs NO Bowl Inns Wicket Catch
  Australia      36       49 1673  2       66     148    57
      India      14       16 1201  0       23      59    14
New Zealand      15       22  846  2       28      64    14
   Pakistan      14       20  647  1       18      40    14
  Sri Lanka       3        3   41  0        6      11     2
West Indies      20       37  792  1       27      61    19

Botham only played three matches against Sri Lanka so it is difficult to properly assess his performance against them. If the above table is stored in a data frame itb.opp then we can create a histogram of the total runs (or wickets) by opposition country:

ggplot(itb.opp, aes(Opposition, Runs)) + geom_bar() + xlab("Country") +
  ylab("Total Runs")

This code produces the following graph:

IT Botham Total Runs by Opposition

The total wickes graph is produced by the next code:

ggplot(itb.opp, aes(Opposition, Wicket)) + geom_bar() + xlab("Country") +
  ylab("Total Wickets")

IT Botham Total Wickets by Opposition

We may now want to delve deeper into the performance against different nations to take into account the number of games or innings where Botham batted or bowled. The traditional way to assess performance is to calculate batting and bowling averages and we can do this by opposition which provides the following data frame:

> itb.opp.sum
 Opposition Discipline  Average
  Australia    Batting 29.35088
      India    Batting 70.64706
New Zealand    Batting 42.30000
   Pakistan    Batting 32.35000
  Sri Lanka    Batting 13.66667
West Indies    Batting 21.40541
  Australia    Bowling 27.65541
      India    Bowling 26.40678
New Zealand    Bowling 23.43750
   Pakistan    Bowling 31.77500
  Sri Lanka    Bowling 28.18182
West Indies    Bowling 35.18033

This can be converted into a dot plot so we can see whether Botham had a high batting average than bowling average, which is often taken to be one of the signs of an all-rounder.

ggplot(itb.opp.sum, aes(Average, Opposition, colour = Discipline)) +
  geom_point()+ xlab("Average") + ylab("")

The graph is shown here:

IT Botham Batting and Bowling Averages by Opposition

We can see the differences in performance based on the opposition. Botham’s performance against the West Indies, by far the strongest team during most of his international career, were worse than against the other countries. However, his averages were far from embarassing when compared to other players at the time. The graph also shows that Botham enjoyed batting and bowling against India.

We can divide this data further based on whether the matches were played in England or outside of England and this data is shown here:

> itb.opp.ha.sum
  Opposition Venue Discipline  Average
   Australia  Away    Batting 30.22581
       India  Away    Batting 61.55556
 New Zealand  Away    Batting 50.44444
    Pakistan  Away    Batting 16.00000
   Sri Lanka  Away    Batting 13.00000
 West Indies  Away    Batting 14.17647
   Australia  Home    Batting 28.30769
       India  Home    Batting 80.87500
 New Zealand  Home    Batting 35.63636
    Pakistan  Home    Batting 34.16667
   Sri Lanka  Home    Batting 14.00000
 West Indies  Home    Batting 27.55000
   Australia  Away    Bowling 28.44928
       India  Away    Bowling 25.53333
 New Zealand  Away    Bowling 27.44444
    Pakistan  Away    Bowling 45.00000
   Sri Lanka  Away    Bowling 21.66667
 West Indies  Away    Bowling 39.50000
   Australia  Home    Bowling 26.96203
       India  Home    Bowling 27.31034
 New Zealand  Home    Bowling 20.51351
    Pakistan  Home    Bowling 31.07895
   Sri Lanka  Home    Bowling 30.62500
 West Indies  Home    Bowling 31.97143

A dot plot is created from this data with a separate panel for each of the six opposition countries and the averages divided into batting and bowling performances. The coloured dots in the graph indicated whether the average is for matches at home or away.

ggplot(itb.opp.ha.sum, aes(Average, Discipline, colour = Venue)) +
  geom_point() + facet_wrap( ~ Opposition) +
  xlab("Batting Average") + ylab("")

This graph is shown below:

IT Botham Batting and Bowling Averages by Country and Home/Away

We can see that the difference between home and away peformance is, in general, not very large for bowling averages but in some cases there is a noticeable difference in batting averages. When looking at Botham’s performances against the West Indies his statistics at home are much better than his away performance, suggesting that his main struggles against the strong West Indies team were in the Caribbean. This might be due to his swing bowling being more suitable to English conditions compared to pitches in the West Indies.

To round off this brief look at the career of IT Botham let us consider some other important statistics, in particular games where he performed with the bat and ball.

Overall Botham scored 14 hundreds and 22 fifties out of 161 innings so he reached fifty runs every five innings or so.
He also took 27 five wicket hauls and 17 four wicket hauls so he took four or more wickets every four innings or so.
He took 120 catches.

Individual matches of excellence include five games with a century and at least five wickets:

Year  Opposition       Ground Venue Runs Wicket
1978 New Zealand Christchurch  Away  133      8
1978    Pakistan       Lord's  Home  108      8
1980       India       Mumbai  Away  114     13
1981   Australia        Leeds  Home  199      7
1984 New Zealand   Wellington  Away  138      6

These performances and others show why Botham was considered such a great player as he produced some sustained periods of excellent all-round cricket rather than having one discipline more dominant for a long period of time.

Generating Balanced Incomplete Block Designs (BIBD)

Ralph — Fri, 16 Jul 2010 12:04:50 +0000

The Balanced Incomplete Block Design (BIBD) is a well studied experimental design that has various desirable features from a statistical perspective. The crossdes package in R provides a way to generate a block design for some given parameters and test wheter this design satisfies the BIBD conditions.

For a BIBD there are v treatments repeated r times in b blocks of k observations. There is a fifth parameter lambda that records the number of blocks where every pair of treatment occurs in the design.

We first load the crossdes package in our sessions:

require(crossdes)

The function find.BIB is used to generate a block design with specific number of treatments, blocks (rows of the design) and elements per block (columns of the design).

Consider an example with five treatments in four blocks of three elements. We can create a block design via:

> find.BIB(5, 4, 3)
     [,1] [,2] [,3]
[1,]    1    3    4
[2,]    2    4    5
[3,]    2    3    5
[4,]    1    2    5

This design is not a BIBD because the treatments are not all repeated the same number of times in the design and we can check this with the isGYD function. For this example:

> isGYD(find.BIB(5, 4, 3))
 
[1] The design is neither balanced w.r.t. rows nor w.r.t. columns.

This confirms what we can see from the design.

Let us instead consider a design with seven treatments and seven blocks of three elements to see whether we can create a BIBD with these parameters:

> my.design = find.BIB(7, 7, 3)
> my.design
     [,1] [,2] [,3]
[1,]    1    2    5
[2,]    3    4    5
[3,]    1    3    6
[4,]    2    3    7
[5,]    2    4    6
[6,]    1    4    7
[7,]    5    6    7
> isGYD(my.design)
 
[1] The design is a balanced incomplete block design w.r.t. rows.

In this situation we are able to generate a valid BIBD experiment with the specified parameters.

R Commander – two-way analysis of variance

Ralph — Fri, 25 Jun 2010 09:42:16 +0000

Two way analysis of variance models can be fitted to data using the R Commander GUI. The general approach is similar to fitting the other types of model in R Commander described in previous posts.

Fast Tube by Casper

The “Statistics” menu provides access to some analysis of variance models via the “Means” sub-menu:

Multi-way ANOVA – the simplest two-way analysis of variance model that can be applied to a set of data.

The “Models” menu provides access to various diagnostics for analysis of variance models via the “Graphs” sub-menu including:

Basic diagnostic plots – four commonly used residual diagnostics including fitted values versus residuals and a normal probability plot for the residuals.

Other useful resources are provided on the Supplementary Material page.