Model selection tool

Introduction

This tutorial provides information about Partek model selection tool, how to use this function and some common mistakes which we should avoid to do. The dataset used in the tutorial is a simulated human microarray intensity values in log space. The data is not used for diagnostic procedure, only to show how to use the function.

Select a classification model

Download the zip file from the link below. The download contains the following files:

training set data–28 samples (11 disease samples and 15 normal samples) on 9953 genes
test set data – 8 samples on 9953 genes
configuration of the model builder (.pcms file)
deployed model (.pbb file)

A classification model has two parts: variables and classifier. The model selection tool in Partek Genomics Suite uses cross-validation to choose the best classification model and gives the accuracy estimate of the best model.

1-level cross-validation is used to select the best model to deploy. There are two ways to report the unbiased accuracy estimate (or correct rate): 2-level cross validation on the same data set, or deploy the model on a independent test set. We will show both in this tutorial.

Open Partek Genomics Suite, choose File>Open.. from the main menu to open the trainingSet.fmt.
Select Tools > Predict > Model Selection from the Partek main menu
In Cross-Validation tab, choose to Predict onType, Positive Outcome is Disease, Selection Criterion is Normalized Correct Rate (Figure 1)
Choose 1-Level Cross-Validation option, and use Manually specify partition option as 5– use 1-level cross validation option is to select the best model to deploy

Figure 1. Model selection dialog -- cross validation configuration

Choose Variable Selection tab, to use ANOVA to select variables. The number of genes selected are based on the p-value generated from the 1-way ANOVA model which factor is Type. In each iteration of cross validation, we will use the training set to perform ANOVA, take the top N number of genes with the most significant p-values to build the classifier. The Configure button allow you to specify ANOVA model if you want to include multiple factors (Figure 2).
Since we don't know how many genes should be used to build the model, we will try to use 10, 20, 30, 40, 50 genes – the more options you try, the longer time it takes to run. In the How many groups of variables do you want to try, select Multiple groups with size from 10 to 50 step 10

Figure 2. Model selection dialog -- variable selection configuration

Click on Classification tab, select K-Nearest Neighbor, choose 1 and 3 neighbors using default Euclidean distance measure (Figure 3)

Figure 3. Model selection dialog -- KNN configuration

Select Discriminant Analysis option, use the default setting which has the Linear with equal prior probabilities option checked
Click on Summary tab, we have configured 15 models to choose from (Figure 3)

Figure 4. Model selection dialog -- Summary page

The more models configured, the long time it takes to run, in this example, in order to save time, we only specified 15 models and choose 5-fold cross-validation. You can also click on Load Spec button to load the above configuration from file tuturial.pcms

When click on Run, a dialog as (Figure 4) will display, some classifiers like discriminant analysis are not recommended to perform on dataset with more number of variables than that of samples.

Figure 5. Discriminant analysis model is not recommended to perform on data which has more variables than samples

Click Run without those models button to dismiss the dialog, there will be 12 models in this model space

Since we are doing 5-fold cross validation, there will be 6 samples held out as test set in each iteration, and the models are built on the rest 22 samples training set. After it is done, all the 12 models have been tested on the 28 samples, and the correct rate will reported, they are displayed in the summary page in descending order of the normalized correct rate, the top one is the best model among the 12 models (Figure 5).

Figure 6. 1-level cross-validation result: 20 variables 3 nearest neighbor with Euclidean distance measure is the best model among the 12 models in this dataset

Click on Deploy button to deploy the model using the whole dataset, save the file as 20var-3NN-Euclidean.ppb. It will run ANOVA on the 28 samples to generate the top 20 genes and build a model using 3 K-Nearest neighbor based on Euclidean distance measure.
Since the deployed model was from the whole 28 samples, in order to know the correct rate, we need a test set to run the model on.

Deploy a model

To get unbiased correct rate, the test set sample must be independent from the training set. Now we are going to load another dataset, it has 8 samples with logged intensity values on the set of genes as that of the training data set

Choose File>Open... to open to browse and open testSet.fmt
Choose 20var-3NN-Euclidean.ppb to open, click on Test button to run

Cross validation

Common mistakes

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

Your Rating:

Results:

3

rates

PGS Documentation

Page tree

Introduction

Select a classification model

Cross validation

Common mistakes

Additional Assistance