When we have a set of data with a small number of variables we can easily use a manual approach to identifying a good set of variables and the form they take in our statistical model. In other situations we may have a large number of potentially important variables and it soon becomes a time consuming effort to follow a manual variable selection process. In this case we may consider using automatic subset selection tools to remove some of the burden of the task.

It should be noted that there is some disagreement about whether it is desirable to use an automated method but this post will focus on the mechanics of doing it rather than the debate about whether to be doing it at all.

The **R** package **leaps** has a function **regsubsets** that can be used for best subsets, forward selection and backwards elimination depending on which approach is considered most appropriate for the application under consideration.

In previous post we considered using data on CPU performance to illustrate the variable selection process. We load the required packages:

> require(leaps) > require(MASS) |

First up we consider selecting the *best* subset of a particular size, say four variables for illustrative purposes (**nvmax** argument), and we specify the largest possible model which in this example has six variables:

regsubsets(perf ~ syct + mmin + mmax + cach + chmin + chmax, data = cpus, nvmax = 4) |

A summary for the output from this function is shown here:

> summary(reg1) Subset selection object Call: regsubsets.formula(perf ~ syct + mmin + mmax + cach + chmin + chmax, data = cpus, nvmax = 4) 6 Variables (and intercept) Forced in Forced out syct FALSE FALSE mmin FALSE FALSE mmax FALSE FALSE cach FALSE FALSE chmin FALSE FALSE chmax FALSE FALSE 1 subsets of each size up to 4 Selection Algorithm: exhaustive syct mmin mmax cach chmin chmax 1 ( 1 ) " " " " "*" " " " " " " 2 ( 1 ) " " " " "*" "*" " " " " 3 ( 1 ) " " "*" "*" " " " " "*" 4 ( 1 ) " " "*" "*" "*" " " "*" |

The function **regsubsets** identifies the variables **mmin**, **mmax**, **cach** and **chmax** as the *best* four.

Alternatively we could perform a backwards elimination and the function will indicate the *best* subset of a particular size, from one to six variables in this example:

> reg2 = regsubsets(perf ~ syct + mmin + mmax + cach + chmin + chmax, data = cpus, method = "backward") > summary(reg2) Subset selection object Call: regsubsets.formula(perf ~ syct + mmin + mmax + cach + chmin + chmax, data = cpus, method = "backward") 6 Variables (and intercept) Forced in Forced out syct FALSE FALSE mmin FALSE FALSE mmax FALSE FALSE cach FALSE FALSE chmin FALSE FALSE chmax FALSE FALSE 1 subsets of each size up to 6 Selection Algorithm: backward syct mmin mmax cach chmin chmax 1 ( 1 ) " " "*" " " " " " " " " 2 ( 1 ) " " "*" " " " " " " "*" 3 ( 1 ) " " "*" "*" " " " " "*" 4 ( 1 ) " " "*" "*" "*" " " "*" 5 ( 1 ) "*" "*" "*" "*" " " "*" 6 ( 1 ) "*" "*" "*" "*" "*" "*" |

The subset of four variables is the same for this example as the *best* subsets approach. The third approach if forward selection is used:

> reg3 = regsubsets(perf ~ syct + mmin + mmax + cach + chmin + chmax, data = cpus, method = "forward") > summary(reg3) Subset selection object Call: regsubsets.formula(perf ~ syct + mmin + mmax + cach + chmin + chmax, data = cpus, method = "forward") 6 Variables (and intercept) Forced in Forced out syct FALSE FALSE mmin FALSE FALSE mmax FALSE FALSE cach FALSE FALSE chmin FALSE FALSE chmax FALSE FALSE 1 subsets of each size up to 6 Selection Algorithm: forward syct mmin mmax cach chmin chmax 1 ( 1 ) " " " " "*" " " " " " " 2 ( 1 ) " " " " "*" "*" " " " " 3 ( 1 ) " " "*" "*" "*" " " " " 4 ( 1 ) " " "*" "*" "*" " " "*" 5 ( 1 ) "*" "*" "*" "*" " " "*" 6 ( 1 ) "*" "*" "*" "*" "*" "*" |

For this data set, as there are only six variables, we do not see divergence between the subsets chosen by the different methods.

Second the comment about the danger of these methods, e.g.

http://www.stata.com/support/faqs/stat/stepwise.html

Can you point to any relatively modern sources that suggest that this is a *good* idea … ? (Although http://dx.doi.org/10.1111/j.1461-0248.2009.01361.x shows that for some ecological data sets it’s not too bad. If we’re just going to

count votes, though, the anti-selectionists are much louder than the

pro-selectionists …)

!

Opinion, as is often the case in technical discussions, seems to be polarised between manual and automatic variable selection methods. For me, one of the main issues with automatic selection methods is that they can be applied without too much thought and the answer taken as fact without due consideration of whether a good model has been identified.

Typo in Third Approach.

You are mentioning to use “Forward” but code is for “Backward” (Edit and thanks for the spot).