Software for Exploratory Data Analysis and Statistical Modelling » Data Manipulation

Programming with R – Processing Football League Data Part II

Ralph — Fri, 03 Dec 2010 10:26:39 +0000

Following on from the previous post about creating a football result processing function for data from the football-data.co.uk website we will add code to the function to generate a league table based on the results to date.

To create the league table we need to count various things such as the number of games played, number of wins/draws/losses, goals scored etc. This information is available in the results object that is loaded from a csv file in the function as it stands.

To facilitate these calculations we create a data frame with a row for each team in the division and then calculate the statistics required – this was a reason for ordering the factors in the HomeTeam and AwayTeam columns of the results table. The data frame is created with the code below:

tmpTable = data.frame(Team = teams,
    Games = 0, Win = 0, Draw = 0, Loss = 0,
    HomeGames = 0, HomeWin = 0, HomeDraw = 0, HomeLoss = 0,
    AwayGames = 0, AwayWin = 0, AwayDraw = 0, AwayLoss = 0,
    Points = 0,
    HomeFor = 0, HomeAgainst = 0,
    AwayFor = 0, AwayAgainst = 0,
    For = 0, Against = 0, GoalDifference = 0)

There are a number of slots that are may be redundant in a league table but are used for intermediate calculations, such as HomeWin and AwayWin that are combined to find the total number of victories for a team.

The number of games played by each team home and away are counted using the table command for the two columns respectively.

tmpTable$HomeGames = as.numeric(table(tmpResults$HomeTeam))
tmpTable$AwayGames = as.numeric(table(tmpResults$AwayTeam))

The labels created by the table command are discarded using the as.numeric function to retain only the number of games. The table command is also used to count the number of wins, draws and losses at home and away for each team. The commands are shown here:

tmpTable$HomeWin =
    as.numeric(table(tmpResults$HomeTeam[tmpResults$FTR == "H"]))
tmpTable$HomeDraw =
    as.numeric(table(tmpResults$HomeTeam[tmpResults$FTR == "D"]))
tmpTable$HomeLoss =
    as.numeric(table(tmpResults$HomeTeam[tmpResults$FTR == "A"]))
 
tmpTable$AwayWin =
    as.numeric(table(tmpResults$AwayTeam[tmpResults$FTR == "A"]))
tmpTable$AwayDraw =
    as.numeric(table(tmpResults$AwayTeam[tmpResults$FTR == "D"]))
tmpTable$AwayLoss =
    as.numeric(table(tmpResults$AwayTeam[tmpResults$FTR == "H"]))

Note that we subset on the values in the FTR column, which is full-time result, and then count. The subsetting is reversed when looking at the away fixtures because a victory for the team is now an away win rather than a home win.

This information is then combined to get total games played, won etc.

tmpTable$Games = tmpTable$HomeGames + tmpTable$AwayGames
tmpTable$Win = tmpTable$HomeWin + tmpTable$AwayWin
tmpTable$Draw = tmpTable$HomeDraw + tmpTable$AwayDraw
tmpTable$Loss = tmpTable$HomeLoss + tmpTable$AwayLoss

The total points is calclated by multiplying the number of wins, draws and losses by the number of points awarded for each match outcome.

tmpTable$Points = winPoints * tmpTable$Win +
    drawPoints * tmpTable$Draw + lossPoints * tmpTable$Loss

The next set of calculations are to count the number of goals scored, goals conceeded and goal difference. The tapply function is used for these calculations.

tmpTable$HomeFor =
    as.numeric(tapply(tmpResults$FTHG, tmpResults$HomeTeam, sum, na.rm = TRUE))
tmpTable$HomeAgainst =
    as.numeric(tapply(tmpResults$FTAG, tmpResults$HomeTeam, sum, na.rm = TRUE))
 
tmpTable$AwayFor =
    as.numeric(tapply(tmpResults$FTAG, tmpResults$AwayTeam, sum, na.rm = TRUE))
tmpTable$AwayAgainst =
    as.numeric(tapply(tmpResults$FTHG, tmpResults$AwayTeam, sum, na.rm = TRUE))

The tapply function applies the sum to the number of goals scored at home or away, and the number of goals conceeded by each team in the division. These are then combined to create totals home and away:

tmpTable$For =
    ifelse(is.na(tmpTable$HomeFor), 0, tmpTable$HomeFor) +
    ifelse(is.na(tmpTable$AwayFor), 0, tmpTable$AwayFor)
tmpTable$Against =
    ifelse(is.na(tmpTable$HomeAgainst), 0, tmpTable$HomeAgainst) +
    ifelse(is.na(tmpTable$AwayAgainst), 0, tmpTable$AwayAgainst)

The ifelse statement is used to handle situations where a team hasn’t played a home and/or away fixture yet. The goal difference is easy to calculate:

tmpTable$GoalDifference = tmpTable$For - tmpTable$Against

Now that all of the statistics have been calculated we sort the table based on the number of points, goal difference and finally alphabetically. There might be different ways that we can order the teams but this is what we will use for the time being:

tmpTable =
  tmpTable[order(- tmpTable$Points, - tmpTable$GoalDifference, tmpTable$Team),]

The ordering might look odd but we want to ranking from highest to lowest points and goal difference but then in ascending alphabetical order for the teams.

The whole function is now:

football.process.v2 = function(datafile, country, divname, season, teams, winPoints = 3, drawPoints = 1, lossPoints = 0)
{
## Validation Function Arguments
 
if (missing(datafile))
{
stop("Results csv file not specified.")
}
 
if (missing(country))
{
warning("Country of league not specified.")
country = ""
}
 
if (missing(divname))
{
warning("Name of league division not specified.")
divname = ""
}
 
## Import Results
 
tmpResults = read.csv(datafile)[,c("Date","HomeTeam","AwayTeam","FTR","FTHG","FTAG")]
 
if (missing(teams))
{
warning("Team names not specified - extracted from results data.")
teams = sort(unique(c(as.character(tmpResults$HomeTeam), as.character(tmpResults$AwayTeam))))
}
 
tmpResults$HomeTeam = factor(tmpResults$HomeTeam, levels = teams)
tmpResults$AwayTeam = factor(tmpResults$AwayTeam, levels = teams)
 
## Create Empty League Table
 
tmpTable = data.frame(Team = teams,
Games = 0, Win = 0, Draw = 0, Loss = 0,
HomeGames = 0, HomeWin = 0, HomeDraw = 0, HomeLoss = 0,
AwayGames = 0, AwayWin = 0, AwayDraw = 0, AwayLoss = 0,
Points = 0,
HomeFor = 0, HomeAgainst = 0,
AwayFor = 0, AwayAgainst = 0,
For = 0, Against = 0, GoalDifference = 0)
 
## Count Number of Games Played
 
tmpTable$HomeGames = as.numeric(table(tmpResults$HomeTeam))
tmpTable$AwayGames = as.numeric(table(tmpResults$AwayTeam))
 
## Count Number of Wins/Draws/Losses
 
tmpTable$HomeWin = as.numeric(table(tmpResults$HomeTeam[tmpResults$FTR == "H"]))
tmpTable$HomeDraw = as.numeric(table(tmpResults$HomeTeam[tmpResults$FTR == "D"]))
tmpTable$HomeLoss = as.numeric(table(tmpResults$HomeTeam[tmpResults$FTR == "A"]))
 
tmpTable$AwayWin = as.numeric(table(tmpResults$AwayTeam[tmpResults$FTR == "A"]))
tmpTable$AwayDraw = as.numeric(table(tmpResults$AwayTeam[tmpResults$FTR == "D"]))
tmpTable$AwayLoss = as.numeric(table(tmpResults$AwayTeam[tmpResults$FTR == "H"]))
 
tmpTable$Games = tmpTable$HomeGames + tmpTable$AwayGames
tmpTable$Win = tmpTable$HomeWin + tmpTable$AwayWin
tmpTable$Draw = tmpTable$HomeDraw + tmpTable$AwayDraw
tmpTable$Loss = tmpTable$HomeLoss + tmpTable$AwayLoss
tmpTable$Points = winPoints * tmpTable$Win + drawPoints * tmpTable$Draw + lossPoints * tmpTable$Loss
 
## Count Goals Scored and Conceeded
 
tmpTable$HomeFor = as.numeric(tapply(tmpResults$FTHG, tmpResults$HomeTeam, sum, na.rm = TRUE))
tmpTable$HomeAgainst = as.numeric(tapply(tmpResults$FTAG, tmpResults$HomeTeam, sum, na.rm = TRUE))
 
tmpTable$AwayFor = as.numeric(tapply(tmpResults$FTAG, tmpResults$AwayTeam, sum, na.rm = TRUE))
tmpTable$AwayAgainst = as.numeric(tapply(tmpResults$FTHG, tmpResults$AwayTeam, sum, na.rm = TRUE))
 
tmpTable$For = ifelse(is.na(tmpTable$HomeFor), 0, tmpTable$HomeFor) +
ifelse(is.na(tmpTable$AwayFor), 0, tmpTable$AwayFor)
tmpTable$Against = ifelse(is.na(tmpTable$HomeAgainst), 0, tmpTable$HomeAgainst) +
ifelse(is.na(tmpTable$AwayAgainst), 0, tmpTable$AwayAgainst)
 
tmpTable$GoalDifference = tmpTable$For - tmpTable$Against
 
## Sort Table
## By Points
## By Goal Difference
## By Team Name (Alphabetical)
 
tmpTable = tmpTable[order(- tmpTable$Points, - tmpTable$GoalDifference, tmpTable$Team),]
 
tmpTable = tmpTable[,c("Team", "Games", "Win", "Draw", "Loss", "Points", "For", "Against", "GoalDifference")]
 
## Return Division Information
 
tmpSummary = list(Country = country, Division = divname, Season = season, Teams = teams,
Results = tmpResults, Table = tmpTable)
 
invisible(tmpSummary)
}

There are other functionality that we might want to add to the function.

Programming with R – Processing Football League Data Part I

Ralph — Tue, 23 Nov 2010 14:14:45 +0000

In this post we will make use of football results data from the football-data.co.uk website to demonstrate creating functions in R to automate a series of standard operations that would be required for results data from various leagues and divisions.

The first step is to consider what control options should be available as part of the function and here is a list of some arguments that will be used for this implementation of a football result data processing function:

The name of a csv data file from the football-data.co.uk website.
A text string to specify the country and division for the data.
A text string specifying the season.
A list of teams in the division (optional), which could be used to test for data entry errors in the data file.
The number of points for a win, draw or loss. This might seem a strange option initially but different leagues might award different points for the three outcomes.

Some of this information might appear optional but is included so that we can write a custom print function at a later date to display a meaningful summary of the object (list) that will be created by the function.

The first part of our function is concerned with checking the various values provided to the function arguments. Our skeleton function is as follows:

football.process.v1 = function(datafile, country, divname, season,
  teams, winPoints = 3, drawPoints = 1, lossPoints = 0)
{
 
}

Here we have specified default options for three of the arguments with the most likely number of points for each match outcome, i.e. 3 points for a win and 1 point for a draw.

To illustrate the working of the result processing function we will use a small exert from the start of the 2010/2011 English Premiership season which is shown below:

Div,Date,HomeTeam,AwayTeam,FTHG,FTAG,FTR,HTHG,HTAG,HTR,Referee
E0,14/8/2010,Aston Villa,West Ham,3,0,H,2,0,H,M Dean
E0,14/8/2010,Blackburn,Everton,1,0,H,1,0,H,P Dowd
E0,14/8/2010,Bolton,Fulham,0,0,D,0,0,D,S Attwell
E0,14/8/2010,Chelsea,West Brom,6,0,H,2,0,H,M Clattenburg
E0,14/8/2010,Sunderland,Birmingham,2,2,D,1,0,H,A Taylor
E0,14/8/2010,Tottenham,Man City,0,0,D,0,0,D,A Marriner
E0,14/8/2010,Wigan,Blackpool,0,4,A,0,3,A,M Halsey
E0,14/8/2010,Wolves,Stoke,2,1,H,2,0,H,L Probert
E0,15/8/2010,Liverpool,Arsenal,1,1,D,0,0,D,M Atkinson
E0,16/8/2010,Man United,Newcastle,3,0,H,2,0,H,C Foy
E0,21/8/2010,Arsenal,Blackpool,6,0,H,3,0,H,M Jones
E0,21/8/2010,Birmingham,Blackburn,2,1,H,0,0,D,M Oliver
E0,21/8/2010,Everton,Wolves,1,1,D,1,0,H,L Mason
E0,21/8/2010,Stoke,Tottenham,1,2,A,1,2,A,C Foy
E0,21/8/2010,West Brom,Sunderland,1,0,H,0,0,D,K Friend
E0,21/8/2010,West Ham,Bolton,1,3,A,0,0,D,A Marriner
E0,21/8/2010,Wigan,Chelsea,0,6,A,0,1,A,M Dean
E0,22/8/2010,Fulham,Man United,2,2,D,0,1,A,P Walton
E0,22/8/2010,Newcastle,Aston Villa,6,0,H,3,0,H,M Atkinson
E0,23/8/2010,Man City,Liverpool,3,0,H,1,0,H,P Dowd
E0,28/8/2010,Blackburn,Arsenal,1,2,A,1,1,D,C Foy
E0,28/8/2010,Blackpool,Fulham,2,2,D,0,1,A,M Oliver
E0,28/8/2010,Chelsea,Stoke,2,0,H,1,0,H,M Atkinson
E0,28/8/2010,Man United,West Ham,3,0,H,1,0,H,M Clattenburg
E0,28/8/2010,Tottenham,Wigan,0,1,A,0,0,D,P Dowd
E0,28/8/2010,Wolves,Newcastle,1,1,D,1,0,H,S Attwell
E0,29/8/2010,Aston Villa,Everton,1,0,H,1,0,H,M Jones
E0,29/8/2010,Bolton,Birmingham,2,2,D,0,1,A,K Friend
E0,29/8/2010,Liverpool,West Brom,1,0,H,0,0,D,L Probert
E0,29/8/2010,Sunderland,Man City,1,0,H,0,0,D,M Dean

This is stored in a file E0test.csv so that we can use the read.csv function to import the results data and then process it.

The first series of commands that we add to the function are for checking various function arguments specified by the user to ensure that they are sensible. First up we check whether a results data file has been specified as we cannot do any processing without any results. The simple test is for whether a file name has been specified:

if (missing(datafile))
{
    stop("Results csv file not specified.")
}

It might be sensible to check whether the object datafile is actually a character string specifying a file, but this hasn’t been done for now. We then check whether the country name and division have been specified and set them to blank strings if they haven’t been set by the user.

if (missing(country))
{
    warning("Country of league not specified.")
    country = ""
}
 
if (missing(divname))
{
    warning("Name of league division not specified.")
    divname = ""
}

Next up we import the data file and only save the columns of interest (at this point of the development of the function at least. There are many more columns of information that we need in the raw data from the website,

tmpResults =
    read.csv(datafile)[,c("Date","HomeTeam","AwayTeam","FTR","FTHG","FTAG")]

The square brackets are used to subset on a part set of columns and only save these. Then we check whether the team names have been specified by the user and if not extract them from the data provided:

if (missing(teams))
{
    warning("Team names not specified - extracted from results data.")
    teams = sort(unique(c(as.character(tmpResults$HomeTeam),
        as.character(tmpResults$AwayTeam))))
}

The sort function is used to order the team names alphabetically which is the order often used in league tables, especially when no games have been played. We then convert the columns HomeTeam and AwayTeam into factors, which allows teams that haven’t played a fixture yet to be included in the table.

tmpResults$HomeTeam = factor(tmpResults$HomeTeam, levels = teams)
tmpResults$AwayTeam = factor(tmpResults$AwayTeam, levels = teams)

To round off the first part of creating the result processing function we create a list object to return at the end of the function.

tmpSummary = list(Country = country, Division = divname,
    Season = season, Teams = teams, Results = tmpResults)

The function so far:

football.process.v1 = function(datafile, country, divname, season, teams, winPoints = 3, drawPoints = 1, lossPoints = 0)
{
## Validation Function Arguments
 
if (missing(datafile))
{
stop("Results csv file not specified.")
}
 
if (missing(country))
{
warning("Country of league not specified.")
country = ""
}
 
if (missing(divname))
{
warning("Name of league division not specified.")
divname = ""
}
 
## Import Results
 
tmpResults = read.csv(datafile)[,c("Date","HomeTeam","AwayTeam","FTR","FTHG","FTAG")]
 
if (missing(teams))
{
warning("Team names not specified - extracted from results data.")
teams = sort(unique(c(as.character(tmpResults$HomeTeam), as.character(tmpResults$AwayTeam))))
}
 
tmpResults$HomeTeam = factor(tmpResults$HomeTeam, levels = teams)
tmpResults$AwayTeam = factor(tmpResults$AwayTeam, levels = teams)
 
## Return Division Information
 
tmpSummary = list(Country = country, Division = divname, Season = season, Teams = teams,
Results = tmpResults)
 
invisible(tmpSummary)
}

We then test this function with the data file shown above. First up we create our own list of teams in the English Premiership for 2010/2011 and specify some of the other function arguments while using the defaults for points.

> E0teams.1011 = c("Arsenal", "Aston Villa", "Birmingham", "Blackburn",
+ "Blackpool", "Bolton", "Chelsea", "Everton", "Fulham", "Liverpool",
+ "Man City", "Man United", "Newcastle", "Stoke", "Sunderland",
+ "Tottenham", "West Brom", "West Ham", "Wigan", "Wolves")
> print(football.process.v1("E0test.csv", "England", "Premiership",
    "2010-2011", E0teams.1011))
$Country
[1] "England"
 
$Division
[1] "Premiership"
 
$Season
[1] "2010-2011"
 
$Teams
 [1] "Arsenal"     "Aston Villa" "Birmingham"  "Blackburn"   "Blackpool"  
 [6] "Bolton"      "Chelsea"     "Everton"     "Fulham"      "Liverpool"  
[11] "Man City"    "Man United"  "Newcastle"   "Stoke"       "Sunderland" 
[16] "Tottenham"   "West Brom"   "West Ham"    "Wigan"       "Wolves"     
 
$Results
        Date    HomeTeam    AwayTeam FTR FTHG FTAG
1  14/8/2010 Aston Villa    West Ham   H    3    0
2  14/8/2010   Blackburn     Everton   H    1    0
3  14/8/2010      Bolton      Fulham   D    0    0
4  14/8/2010     Chelsea   West Brom   H    6    0
5  14/8/2010  Sunderland  Birmingham   D    2    2
6  14/8/2010   Tottenham    Man City   D    0    0
7  14/8/2010       Wigan   Blackpool   A    0    4
8  14/8/2010      Wolves       Stoke   H    2    1
9  15/8/2010   Liverpool     Arsenal   D    1    1
10 16/8/2010  Man United   Newcastle   H    3    0
11 21/8/2010     Arsenal   Blackpool   H    6    0
12 21/8/2010  Birmingham   Blackburn   H    2    1
13 21/8/2010     Everton      Wolves   D    1    1
14 21/8/2010       Stoke   Tottenham   A    1    2
15 21/8/2010   West Brom  Sunderland   H    1    0
16 21/8/2010    West Ham      Bolton   A    1    3
17 21/8/2010       Wigan     Chelsea   A    0    6
18 22/8/2010      Fulham  Man United   D    2    2
19 22/8/2010   Newcastle Aston Villa   H    6    0
20 23/8/2010    Man City   Liverpool   H    3    0
21 28/8/2010   Blackburn     Arsenal   A    1    2
22 28/8/2010   Blackpool      Fulham   D    2    2
23 28/8/2010     Chelsea       Stoke   H    2    0
24 28/8/2010  Man United    West Ham   H    3    0
25 28/8/2010   Tottenham       Wigan   A    0    1
26 28/8/2010      Wolves   Newcastle   D    1    1
27 29/8/2010 Aston Villa     Everton   H    1    0
28 29/8/2010      Bolton  Birmingham   D    2    2
29 29/8/2010   Liverpool   West Brom   H    1    0
30 29/8/2010  Sunderland    Man City   H    1    0

Other useful resources are provided on the Supplementary Material page.

Useful functions for data frames

Ralph — Mon, 09 Aug 2010 20:38:15 +0000

The R software system is primarily command line based so when there are large sets of data it is not easy to browse the data frames. There are various useful functions for working with data frames.

For example, after loading data from a text file we might want to view the first few lines of a set of data. The functions head and tail return the first or last parts of a vector, matrix, table, data frame or function.

Consider the Orange data set that is available in R. We can view the first few lines

> head(Orange)
  Tree  age circumference
1    1  118            30
2    1  484            58
3    1  664            87
4    1 1004           115
5    1 1231           120
6    1 1372           142

or the last few lines:

> tail(Orange)
   Tree  age circumference
30    5  484            49
31    5  664            81
32    5 1004           125
33    5 1231           142
34    5 1372           174
35    5 1582           177

Another useful function is str, which compactly displays the internal structure of an R object. On this set of data we get:

> str(Orange)
Classes ‘nfnGroupedData’, ‘nfGroupedData’, ‘groupedData’ and 'data.frame':      35 obs. of  3 variables:
 $ Tree         : Ord.factor w/ 5 levels "3"<"1"<"5"<"2"<..: 2 2 2 2 2 2 2 4 4 4 ...
 $ age          : num  118 484 664 1004 1231 ...
 $ circumference: num  30 58 87 115 120 142 145 33 69 111 ...
 - attr(*, "formula")=Class 'formula' length 3 circumference ~ age | Tree
  .. ..- attr(*, ".Environment")= 
 - attr(*, "labels")=List of 2
  ..$ x: chr "Time since December 31, 1968"
  ..$ y: chr "Trunk circumference"
 - attr(*, "units")=List of 2
  ..$ x: chr "(days)"
  ..$ y: chr "(mm)"

There is quite a bit of additional information attached to this data frame, mainly due to it having more than one class.

Creating Date Objects using Character Strings

Ralph — Thu, 10 Sep 2009 19:03:34 +0000

The use of dates can frequently be problematic because there is such a wide range of format used to store data information. The R system has various facilities for defining and working with dates and can handle a wide range of formats that might be encountered in a set of data.

The function as.Date can convert a string into a Date object using a user specified format, which can take various forms. The first argument to the function is a character string with the date and the second argument is a second string that provides information about the specific format. The format specifiers use a percent symbol followed by a character, e.g. %d indicates that a day is being specified and %m corresponds to a month etc. If the parts of the date are separated by a symbol such as a slash or a dash then these are included in the format specifier.

If the date is in the form 01/04/2009 then the format string would be %d/%m/%Y. The upper case Y indicates that the year includes century information which is a safer format to use so that there is no ambiguity in any calculations. The following code will convert the date from a character string into a Date object:

> date1 = "01/04/2009"
> as.Date(date1, format = "%d/%m/%Y")
[1] "2009-04-01"

A simple alternative would be where a dash is used to separate the day, month and year:

> date2 = "07-04-2002"
> as.Date(date2, format = "%d-%m-%Y")
[1] "2002-04-07"

We can compare two dates to see whether one is before the other:

> date1 < date2
[1] TRUE

There are other operations that we can perform on dates.

Using R to Access Data in a MySQL database

Ralph — Fri, 24 Jul 2009 07:57:39 +0000

The R import/export manual discusses various approaches to handling data and mentions that R is not suitable for working with large data sets because data objects are stored in memory during a session. There are situations where using a database to hold the data and making use of one of the R libraries for database connectivity to access the data or to save the data.

MySQL is a popular database system and there is a library RMySQL that can be used to access this database – it is important to ensure that the version of MySQL matches with the R library. If there isn’t a match then the system might exhibit erratic behaviour.

The first step is to make the RMySQL library available in the working session:

library(RMySQL)

If this command runs without problems then we need to create a connection object for our session:

con = dbConnect(MySQL(), dbname = "test", user = "user1",
  password = "pwd1")

The first argument to this function creates and initializes a MySQL client. It returns an driver object that allows you to connect to one or several MySQL servers. The dbname argument is used to specify the name of the database and the user and password arguments should be self-explanatory.

After creating our connection successfully we can get R to list the tables that are stored in this database:

> dbListTables(con)
[1] "co2"

In this example there is only one table and its name is returned by the dbListTables function. To read the data from this table we use the dbReadTable function and specify the connection object as well as the name of the table in the MySQL database:

> dbReadTable(con, "co2")
   Plant        Type  Treatment conc uptake
1    Qn1      Quebec nonchilled   95   16.0
2    Qn1      Quebec nonchilled  175   30.4
3    Qn1      Quebec nonchilled  250   34.8
...

We can save the table to a data frame object rather than the default action of printing to the console.

After undertaken some analysis we might want to save a data set to the database and the dbWriteTable function is used:

> dbWriteTable(con, "CO2", data.frame(CO2),
  overwrite = TRUE)
[1] TRUE

The first argument is the connection object, the second is the name that the table will be referred to in the database and the third argument is the data to be saved. In this case we have used the overwrite argument to copy over any existing table of the same name.

We can delete a table using the dbRemoveTable function:

> dbRemoveTable(con, "CO2")
[1] TRUE

and when we reach the end of our need for the connection, the dbDisconnect function will remove the connection that we have been using:

> dbDisconnect(con)
[1] TRUE

Sequences and Other Regular Arrangements of Data

Ralph — Tue, 26 May 2009 19:08:28 +0000

In Statistical analysis there are frequently situations where regular structures occur, such as in designed experiments, and R has facilities for generating data frames in a simple way.

The function expand.grid can be used to create a design by specifying a series of factors and the levels for these factors. A data frame with all the combinations of the factors levels will be created. For example, if we had a two factor experiment where the first factor had four levels labelled A, B, C and D and the second factor had three levels labelled I, II, and III then we could create the data frame for the design using this code:

expand.grid(Factor1 = c("A", "B", "C", "D"), Factor2 = c("I", "II", "III"))

which would produce the following output:

   Factor1 Factor2
1        A       I
2        B       I
3        C       I
4        D       I
5        A      II
6        B      II
7        C      II
8        D      II
9        A     III
10       B     III
11       C     III
12       D     III

It is also possible to create various other sequences using the seq and rep commands. To create the numbers from 1 to 10 we could run this code:

> 1:10
 [1]  1  2  3  4  5  6  7  8  9 10

Alternatively the seq function provides greater control over start and end values and the step between each variable. A couple of examples are shown below:

> seq(1, 5)
[1] 1 2 3 4 5
> seq(10, 1, -2)
[1] 10  8  6  4  2

The negative step indicates that the sequence is decreasing.

Another common pattern is where we might want to repeat a number a given number of times. To get ten replicates of the number one we use the rep function:

> rep(1, 10)
 [1] 1 1 1 1 1 1 1 1 1 1

If we want to repeat a sequence multiple times then we provide the sequence as the first argument to the rep function. So to repeat the numbers one to ten twice we would write:

> rep(1:10, 2)
 [1]  1  2  3  4  5  6  7  8  9 10  1  2  3  4  5  6  7  8  9 10

So the 1:10 evaluates first to the numbers one to ten and the whole thing is repeated twice. A further arrangement where we might want to repeat each element of the sequence a given number of times is accessed by nesting a rep call inside a rep function. The second argument becomes a vector of the same length as the first argument. As an example:

> rep(1:5, rep(3, 5))
 [1] 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5

Transformations to Create New Variables

Ralph — Mon, 18 May 2009 20:21:05 +0000

There are many situations where we might be interested in creating a new variable by transforming one of the variables already in the data frame. The R programming language can be used for either simple transformations or more complicated mathematical expressions where necessary.

Fast Tube by Casper

There are many situations where the logarithmic scale is used for data and if we have data on its original scale then we can use the log function in R to create a new variable. The default base for the log function is the natural logarithm. Referring back to the olive oil data set used in previous posts if we wanted to create a new variable that is the logarithm of one of the numeric variables then we could use code such as:

log(olive.df$palmitic)

which produces the following output:

  [1] 6.980076 6.992096 6.814543 6.873164 6.957497 6.814543 6.826545 7.003065 6.986566 6.944087
 [11] 6.957497 6.943122 6.979145 6.774224 6.858565 7.051856 6.849066 7.153052 6.867974 6.858565
 [21] 6.979145 6.902743 6.962243 6.970730 6.970730 7.181592 7.186144 7.214504 7.228388 7.166266
...

If we had wanted to to take the square root of this variable instead we would use the sqrt function:

sqrt(olive.df$palmitic)

and the output would be:

  [1] 32.78719 32.98485 30.18278 31.08054 32.41913 30.18278 30.36445 33.16625 32.89377 32.20248
 [11] 32.41913 32.18695 32.77194 29.58040 30.85450 33.98529 30.70831 35.74913 31.00000 30.85450
 [21] 32.77194 31.54362 32.49615 32.63434 32.63434 36.26293 36.34556 36.86462 37.12142 35.98611
...

It is possible to create more complicated expressions based on different transformations using the R programming language.

For example we might want to centre a vector of data by removing the mean value for the data and scale to unit variance. Based on the palmitic variable used in the previous examples we could use the scale function to perform this operation:

> scale(olive.df$palmitic)
              [,1]
  [1,] -0.92970611
  [2,] -0.85259700
  [3,] -1.90246724
...
[571,] -1.43388109
[572,] -1.61182519
attr(,"scaled:center")
[1] 1231.741
attr(,"scaled:scale")
[1] 168.5923

The mean and standard deviations used for the scaling are saved as attributes by the function.

Cross-tabulation of Data

Ralph — Fri, 15 May 2009 19:50:13 +0000

The contingency table is used to summarise data when there are factors in the data set and we are interested in counting the number of occurrences of each combination of factor variables. In R there are different ways that these types of table can be produced and manipulated as required.

Fast Tube by Casper

The main two functions that are used to produce contingency tables are table and xtabs. We can use these two functions to get one, two or higher dimension tables that summarise the number of records that correspond to the combination of variables used to create the table.

The simplest case is where we are interested into a summary based on a single variable and the syntax is straightforward. The function table takes a single argument that corresponds to a vector of data. For example, if we are working with a data frame based on an unbalanced design and wanted to count the number of observations corresponding to each treatment we might run some code like:

table(temp.design3$Treatment)

which would produce a simple summary table:

A B C D 
7 7 3 7

If there was a second factor in the data set corresponding to different plots, labelled 1 to 4, then we could generate a two dimensional contingency table by adding a second argument to the function call like:

table(temp.design3$Treatment, temp.design3$Plot)

and the output would be of the form:

The function xtabs can be used to create the same contingency tables but the function works using a formula in a similar vein to the modelling functions. So to get the one dimensional table we would write code similar to this:

xtabs(~ Plot, data = temp.design3)

which would summarise the data by plot:

Plot
1 2 3 4 
7 7 7 3

Note that the output is slightly different to using the table function. The two dimensional table would be created like this:

xtabs(~ Treatment + Plot, data = temp.design3)

and the output would be:

         Plot
Treatment 1 2 3 4
        A 2 2 2 1
        B 2 2 2 1
        C 1 1 1 0
        D 2 2 2 1

These functions can be extended to higher dimensions and the output is based on 2×2 tables for each combination of the other variables.

Working with Subsets of Data

Ralph — Fri, 08 May 2009 18:30:45 +0000

There are often situations where we might be interested in a subset of our complete data and there are simple mechanisms for viewing and editing particular subsets of a data frame or other objects in R.

We might be interested in using one of the variables to select a particular subset. A square bracket notation is used after the name of an object to indicate that we are interested in specific rows or columns of the data and there are a large number of options that could be used. For example, if we consider the olive oil data set used in ggobi demonstrations, we could view the data for one of the regions using the following code:

olive.df[olive.df$Area == "North-Apulia",]

which would give the following output:

   Region         Area palmitic palmitoleic stearic oleic linoleic linolenic arachidic eicosenoic
1       1 North-Apulia     1075          75     226  7823      672        36        60         29
2       1 North-Apulia     1088          73     224  7709      781        31        61         29
3       1 North-Apulia      911          54     246  8113      549        31        63         29
...

Within the square brackets in this example we test a condition which runs a vector of TRUE and FALSE values and R interprets our intention as viewing only those rows where the condition returned TRUE. The comma is used to separate between rows and columns in this case as we have two dimensions in the data frame. All columns are included as there is no expression after the comma in the example above.

It is possible to work with multiple conditions so for the olive oil data we could select one of the other regions, South Apulia, and only data points where stearic variable is greater than 250 units. The could used in this case would be:

olive.df[olive.df$Area == "South-Apulia" & olive.df$stearic > 250,]

to give output:

    Region         Area palmitic palmitoleic stearic oleic linoleic linolenic arachidic eicosenoic
88       1 South-Apulia     1410         232     280  6715     1233        32        60         24
89       1 South-Apulia     1509         209     257  6647     1240        42        62         30
90       1 South-Apulia     1317         197     256  7036     1067        40        60         22
...

If we were interested in a particular column of data then we would specify the name of the column(s) after the comma in the square brackets. For example we could view the palmitic column only with this code:

olive.df[,"palmitic"]

Multiple columns could be selected by providing a vector of column names, such as:

olive.df[,c("palmitic","oleic")]

More complicated expressions are possible with a bit of imagination. For example if we wanted to view the even numbered rows only then we could use the condition seq(2,10,2) which would provide the numbers 2, 4, 6, 8 and 10.