####################################################################################################

Electronic appendix to the BMC Medical Research Methodology article:

A measure of the impact of CV incompleteness on prediction error estimation with 
application to PCA and normalization

Roman Hornung*, Christoph Bernau, Caroline Truntzer, Rory Wilson, Thomas Stadler, and Anne-Laure Boulesteix

* for questions please contact:  hornung@ibe.med.uni-muenchen.de

####################################################################################################


Program and Platform:
######################

- Program: R, version 3.0.1

- Used R packages:

     'BatchExperiments', version: 1.4
     'BatchJobs', version: 1.5
     'corpcor', version: 1.6.6
     'docval' (*)
     'ggplot2', version: 1.0.0
     'gridExtra', version: 0.9.1
     'MASS', version: 7.3-26
     'mvtnorm', version: 0.9-9994
     'pamr', version: 1.54
     'pbdMPI', version: 0.2-0
     'plyr', version: 1.8.1
     'proto', version: 0.3-10
     'R.utils', version: 1.27.1
     'reshape', version: 0.8.5
     'reshape2', version: 1.4.1
     'snowfall', version: 1.84-4

  Used Bioconductor packages:  

     'affy', version: 1.38.1
     'ArrayExpress', version: 1.20.0
     'CMA', version: 1.18.0
     'genefilter', version: 1.42.0
     'GEOquery', version: 2.26.2
     'vsn', version: 3.32.0
  
  (*): Not on CRAN, has to be installed from source, corresponding windows binary 
       zip-folder "docval_1.1.6.zip" found in subfolder "AddOnFunctionsetc";
       Linux users may also download the respective tar.gz-folder from the following
       site: http://genomics.uni-regensburg.de/site/institute/software/docval-documentation-by-value
       (date: 2014/08/12);
       docval additionally requires the Bioconductor packages 'affyPLM' and 'vsn'.
       
  NOTE: Packages listed above depend on others, which might have to be installed
        manually in case this is not performed automatically.

- Platform: Linux (x86-64)





General instructions for reproducing the results:
#################################################

- The folder "CVIIMCodeData" this README is contained in has to be put into the R working directory.

- We use paths of the form "./CVIIMCodeData/...", wherefore all scripts have to be run
  in the R working directory.

- For an introductory demonstrative analysis using CVIIM consider the R script
  'RuntimeExample.R' which contains an artificial application example
  illustrating the general structure of the implementation of our
  real data analyses.

- IMPORTANT: All results obtained on real data presented in the main paper and
  Additional file 2 can be obtained using the R-scripts:

  "Combinations_EVALUATION.R" 
  "NormalizationStudy_EVALUATION.R" 
  "OtherSteps_EVALUATION.R" 
  "PrincipalComponentAnalysisStudy_EVALUATION.R"
  "VariableSelection_EVALUATION.R"

  Note that these scripts do not perform the repeated (incomplete) CVs, 
  but load Rda-files in which all calculated CV fold errors (including those for obtaining the 
  step-specific CV errors) are stored.
  By this the reader is allowed to work with the results without conducting the analyses, which are
  time- and computation power-intensive.
  All plots will be stored in the subfolder "Plots".

- Reproducing the PCA study and the simulation require an MPI-environment.
  Reproducing the normalization study requires an MPI-environment and a huge memory server.

- For reproducing the simulation results, the subfolder "SimulationCVIIM" has to be put 
  in the HOME DIRECTORY of a LINUX MACHINE.




Addon functions:
################

- The subfolder "AddOnFunctionsetc" mainly contains R-scripts containing functions 
  to perform the considered preparation steps, modified to return also all estimated 
  parameters necessary for addon preparation as well as the corresponding 
  addon functions. These scripts are sourced by the scripts performing the respective
  repeated (incomplete) CVs in the analyses.




Obtaining the data used in the analyses:
########################################

- Because of the large sizes of the datasets they could not be included
  in the electronic appendix.

- Exceptions are the datasets: "ProstatecTranscr.Rda" (subfolder: "DataRetrieval/DataSetsPCA"),
  "GenitInfCow.Rda" (subfolder: "DataSets_GenitInfCow_and_ProstatecMethyl") and
  "ProstatecMethyl.Rda" (subfolder: "DataSets_GenitInfCow_and_ProstatecMethyl")  

- The other datasets (all from ArrayExpress and NCBI GEO - see Table 1 in the main paper 
  for an overview) can be obtained automatically:

  - For the normalization study (takes long!):

    Run the R-Script "GetAllDataNormalization.R" (subfolder: "DataRetrieval/DataSetsNormalization").
    Alternatively, to spare computation time, run the 12 other R scripts found in the
    same folder in parallel. Each of the latter corresponds to a specific dataset.

    The CEL files of the individual datasets and the values of the target variable will be stored
    in the subfolders of the folder "GetAllDataNormalization.R" is contained in, with the same 
    names as the datasets. 

  - For the PCA study:

    Run the R-Script "GetAllDataPCA.R" (subfolder: "DataRetrieval/DataSetsPCA"). 
    Analoguous to the normalization study: alternatively, run the 19 R scripts corresponding to 
    the individual datasets in parallel.

    All datasets in the form of Rda-files of the same names as in the main paper will be stored
    in the same folder "GetAllDataPCA.R" is contained in.

    In the process of producing the *.Rda-datasets the processed data is downloaded into
    the subfolders of the same names as the datasets. The latter are only needed while
    producing the *.Rda-datasets and may be deleted afterwards.

  - NOTE: Only the scripts "E-GEOD-10320.R" (a dataset used in normalization)  
    and "E-GEOD-25609.R" (a dataset used in PCA) are commented. 
    However, because the individual scripts corresponding to the respective datasets
    are very similar, the commentation in "E-GEOD-10320.R" and "E-GEOD-25609.R" can be 
    largely conveyed to routines in the other scripts.

- The file "datainfoNormPCA.Rda" in the subfolder "DataRetrieval", generated by the script 
  "ExtractDataInfo.R" also found in this subfolder contains a meta-table providing informations 
  such as sample size and number of variables to all 32 data sets used in the normalization study and in
  the PCA study. Used for Table 1 in the main paper.




Instructions for reproducing the results for the individual analyses with
informations on the implementations:
##########################################################################


# Normalization study (subfolder: "NormalizationStudy"):

- The datasets differ in size. Six of the larger datasets were handled on
  a specific server with a huge shared memory. The package "parallel" is used
  in these analyses. The six remaining datasets could be handled in a more conventional
  MPI environment. In these analyses we use the R package "BatchExperiments".

- Because normalization on the whole data set has to be done only once, we perform
  this beforehand and save the corresponding results.
  For performing this the R scripts "NormalizationWholeDatasetBatchExperiments.R" 
  and "NormalizationWholeDatasetCluster.R" in the subfolder "NormalizationWholeDataset" have to be run.
  Here and in the following the substring "Cluster" corresponds to R scripts
  connected to analyses performed on the huge memory server and "BatchExperiments" to R scripts
  connected to analyses performed in the MPI-environment.

- The R scripts for reproducing the analyses with Nearest Shrunken Centroids
  as classification method can be found in the subfolder "NormalizationNSC".
  For performing the analyses in the MPI environment the script 
  "NormalizationBatchExperimentsNSC.R" has to be run.
  For performing the analyses on the huge memory server at first the
  R script "NormalizationScenarioScriptGenerationClusterNSC.R" has to
  be run. The latter script generates scripts
  "NormalizationScriptNSC1.R", ..., "NormalizationScriptNSC10.R" in the same folder. 
  Consecutively each of the latter has to be run.

- The R-scripts for reproducing the analyses with PLS followed by Linear
  Discriminant Analysis as classification method can be found in the subfolder "NormalizationPLSLDA".
  For performing the analyses in the MPI environment the script 
  "NormalizationBatchExperimentsPLSLDA.R" has to be run.
  For performing the RMA analyses on the huge memory server at first
  the script "NormalizationScenarioScriptGenerationClusterPLSLDARMA.R" has to be run,
  which generates scripts "NormalizationScriptPLSLDARMA1.R", ..., "NormalizationScriptPLSLDARMA10.R".
  Each of the latter has to be run.
  For performing the RMAglobalVSN analyses on the huge memory server at first
  the script "NormalizationScenarioScriptGenerationClusterPLSLDARMAglobalVSN.R" has to be run,
  which generates scripts "NormalizationScriptPLSLDARMAglobalVSN1.R", ..., "NormalizationScriptPLSLDARMAglobalVSN10.R".
  Each of the latter has to be run.

- The R-scripts "NormalizationfunctionsforBatchExperimentsNSC", "NormalizationFunctionsClusterNSC.R" 
  (subfolder: "NormalizationNSC"), "NormalizationfunctionsforBatchExperimentsPLSLDA.R" and
  "NormalizationFunctionsClusterPLSLDA.R" (subfolder: "NormalizationPLSLDA") contain
  functions for loading the datasets and performing the (incomplete) CVs.
  These are tailored to the respective R package used in the parallel executions
  ("BatchExperiments" and "parallel").

- All results are stored in the subfolder "ResultsNormalization".



# PCA study (subfolder: "PrincipalComponentAnalysisStudy"):

- The R-script "ScriptPCA.R" has to be run. In this analysis we use the R package "snowfall".

- For each CPU (1, ..., 145) informations on the settings are stored in the files "settingpca1.Rda", ...,
  "settingpca145.Rda" in the subfolder "Settings". The latter files have been generated
  by the R script "GenerateSettingsPCA.R" also found in the subfolder "Settings".

- All results are stored in the subfolder "ResultsPCA".



# Other steps and variable selection (subfolder: "OtherStepsAndVariableSelection"):

- Here we did not perform parallel computations, but (if necessary from a computational 
  point of view) used multiple R-scripts for the individuals scenarios, each performing 
  a certain number of repetitions in the repeated (incomplete) CVs.

- These scripts can be found in the subfolders: "Imputation", Tuning", "VariableFiltering" and "VariableSelection". 
  Each of these have to be run.

- All results are stored in the subfolder "ResultsOtherStepsAndVariableSelection".

- All R-script names contain the substring "K3", "K5" or "K10" standing for
  3-, 5- and 10-fold CV; moreover they contain two numbers separated by "_",
  here "i_j" indicates the j-th R-script to the i-th data set.

- NOTE: In each of the subfolders containing the R-scripts corresponding to a specific
  step, only specific scripts are commented: the first script to the first data set ("1_1")
  for 3-fold CV.



# Combination of several steps (subfolder: "Combinations"):

- The R scripts "CombineImpVarselTune1_1errorK3.R", "CombineImpVarselTune1_1errorK5.R",
  "CombineImpVarselTune1_1errorK10.R" perform the analyses described in Appendix F
  (Additional file 2). Only the script "CombineImpVarselTune1_1errorK3.R" is commented.

- All results are stored in the subfolder "ResultsCombinations".



# Simulation (subfolder: "SimulationCVIIM"):

- As noted further above: For reproducing the simulation results, the subfolder "SimulationCVIIM" has to be put 
  in the HOME DIRECTORY of a LINUX MACHINE.

- In the script "SimulationSetupScenarios.R" the two scenarios corresponding
  to the two different strengthes of the signal are set up. These are denoted as 
  "strong signal and "weak signal" in "MeanCov" in the Appendix A (Additional file 2).
  The script produces the files "simucorbigsignal.RData" and 
  "simucorweaksignal.RData" which contain the parameter values for the two 
  different strengthes of the signals.

- The simulation can be started using the file "run.sh". In this analysis we use the R package "pbdMPI" for
  parallel execution of the code. Note that before this "SimulationSetupScenarios.R"
  has to be executed to obtain the two Rda-files "simucorbigsignal.RData" and 
  "simucorweaksignal.RData" containing the parameter values for the two different 
  strenghts of the signals.

- Note that other than in the case of the real-data analyses, in the case of the
  simulation we do not include the results, since the corresponding RData-files
  would be too large. Therefore to reproduce the results presented in the
  Appendix A (Additional file 2), the simulation has to be performed.

- The script "SimulationScript.R" contains the R-Code used in the simulation.
  This script loads the RData-files "scenario1.RData", ..., "scenario24.RData",
  found in the subfolder "confs", which contain the setting parameters.
  Moreover, the Rda-files "simucorbigsignal.RData" and "simucorweaksignal.RData",
  in "confs", which have to be previously obtained by the script "SimulationSetupScenarios.R" 
  (see above), are loaded in "SimulationScript.R", which 
  contain the parameters specific to the scenarios with strong and weak effects. 
  The seeds used for the purpose of reproducibility for each of the 24 scenarios 
  are stored in "seeds.RData". "scdaCMA.R", also found in "confs" and sourced by 
  "SimulationScript.R"  contains a modified version of a CMA-function used for 
  Diagonal Linear Discriminant Analysis. The function was alternated to return 
  a function for classifying new observations.
  "SimulationScript.R" produces the RData-files "resultsscenario1.RData", ...,
  "resultsscenario24.RData" containing the results of the repeated (incomplete) CVs
   and the error estimates used for approximating the true average (incomplete)
   errors, see the Appendix A (Additional file 2) for a detailed description.

- The values for the 24 settings are tabularized in "ScenarioTable.txt" 
  found in "confs".

- The script "SimulationEvaluation.R" evaluates the simulation results, i.e.
  all plots presented in the Appendix A (Additional file 2) are produced. This code
  requires that "run.sh" has been executed beforehand.
  All plots will be stored in the subfolder "Plots".