###############################################################################################

Supplementary Material 2 - Electronic Appendix to the paper "Diversity Forests: Using Split Sampling
to Allow for Complex Split Procedures in Random Forest"

Roman Hornung 1,*

1 Institute for Medical Information Processing, Biometry and Epidemiology, 
  University of Munich, Marchioninistr. 15, 81377 Munich, Germany

* For questions, please contact: hornung@ibe.med.uni-muenchen.de

###############################################################################################



Program and Platform:
#####################

- Program: R, versions 3.5.0 and 3.6.0

- Used R packages available from CRAN:

     "asbio", version: 1.5-5
     "data.table", version: 1.11.8
     "ggplot2", version: 3.1.1
     "gridExtra", version: 2.3
     "nnet", version: 7.3-12
     "OpenML", version: 1.10
     "plyr", version: 1.8.4
     "ranger", version: 0.11.1
     "scales", version: 1.0.0
     "viridis", version: 0.5.1

- Further used R packages:

  "diversityForest":
  While the R package "diversityForest" is available 
  from CRAN and github, the specific version of the package that was 
  used for performing the analyses presented in the paper 
  and in the online appendices is not available from 
  CRAN or github.
  The version of the package used for the analyses was a 
  preliminary version in the sense that the documentation
  is incomplete and some function parameters have different
  names than in the versions published on CRAN and github.

  This preliminary package version is included in this
  Online Appendix (folder "suppmat2_hornung2020") as well:
  the corresponding file is called "diversityForest_0.11.0.tar.gz".

  It can be installed in R using the following R command:

  install.packages("PathToTarGzFile/diversityForest_0.11.0.tar.gz",
    repos=NULL, type="source")

  NOTE: This preliminary version of the package is not at 
  all recommended to be used in applications. Its sole
  purpose is to enable full reproduction of the analyses
  presented in the paper and in the online appendices by
  executing the corresponding R files (see Section
  "Full reproduction of the results" below for details).

- Platforms: Linux (x86-64)  (for the conduction of the analyses)
             Windows 7 64-bit (for the evaluation of the results)



General information and contents of this Electronic Appendix:
############################################################

- Paths of the form "./suppmat2_hornung2020/..." are used in all
  R scripts, where "./" is the R working directory. Therefore, the
  folder "suppmat2_hornung2020" this README file is contained in
  is to be put into the R working directory.
  Alternatively, the R working directory can be changed to an
  arbitrary directory that contains the folder "suppmat2_hornung2020".

- The following subfolders are found in "suppmat2_hornung2020":

  - "Data": This subfolder contains the subfolder "Datasets"
    and the files "DownloadData.R", "datainfo.Rda", "df_bmr.RData",
    and "MakeDataTable.R".

    The subfolder "Datasets" contains the processed data sets as
    used in the analyses in the form of Rda files. Note that
    this subfolder includes all 243 data sets used by Couronne et
    al. (2018), where for the analyses performed for the current
    paper the subset of 220 data sets were used that contains
    less than 10000 observations and a maximum of 500 covariates.

    The R script "DownloadData.R" was written to download and
    preprocess the data sets. This file also generated
    the Rda file "datainfo.Rda", also contained in "Data", which
    contains a data.frame with meta information on the data sets
    such as sample sizes and numbers of covariates.

    Tile RData file "df_bmr.RData" is used by "DownloadData.R"
    and contains a data.frame that provides the OpenML data set
    IDs necessary to download the data sets from OpenML.

    The R script "MakeDataTable.R" extracts the information
    presented in Tables S1 to S5 (Online Appendix 1) and
    also produces LaTeX code used to generate these tables.

  - "Evaluation": This subfolder contains two R scripts
    "Evaluation_DetermineTuningGrid.R" and "Evaluation_LargeStudy.R".

    "Evaluation_DetermineTuningGrid.R" was used to produce all results
    of the  Pre-study used to determine suitable grids for the
    tuning parameter values nsplits and proptry of diversity
    forests.

    "Evaluation_LargeStudy.R" was used to produce all results of the
    large scale comparison study of diversity forests against
    conventional random forests and random forests with extremely
    randomized trees.

    These R scripts also contain the R code used to produce the figures.

  - "Functions": This subfolder contains the R scripts 
    "Functions_DetermineTuningGrid.R" and "Functions_LargeStudy.R".

    These two R scripts contain the functions used in the pre-study
    ("Functions_DetermineTuningGrid.R") and in the large scale 
    comparison study ("Functions_LargeStudy.R").

    They are called by the corresponding R scripts found in the folder "JobScripts"
    that perform the pre-study and the large scale comparison study.

  - "JobScripts": This subfolder contains the R scripts "DetermineTuningGrid.R"
    and "LargeStudy.R", where the former performs the pre-study
    and the latter the large scale comparison study.

  - "Results": This folder contains the raw results of the pre-study
    and the large scale comparison study as well as the figures
    produces in the analyses.

    The subfolder "DetermineTuningGrid" contains the Rda file 
    "Results_DetermineTuningGrid.Rda" providing the raw results
    of the pre-study and the Rda file "scenariogrid.Rda", which 
    contains a data.frame providing information on the settings
    considered in the pre-study (used by "JobScripts/DetermineTuningGrid.R").

    The subfolder "LargeStudy" contains the Rda file 
    "Results_LargeStudy.Rda" providing the raw results
    of the large scale comparison study and the Rda file "scenariogrid.Rda", which 
    contains a data.frame providing information on the settings
    considered in the large scale comparison study (used by 
    "JobScripts/LargeStudy.R").

    The subfolder "Figures" contains all figures (in PDF format) produced by
    "Evaluation/Evaluation_DetermineTuningGrid.R" and 
    "Evaluation/Evaluation_LargeStudy.R".
    All Figures shown in the main paper and in Online Appendices 1
    and 3 are among the figures in this subfolder.



Evaluation of the results:
##########################

- For the evaluation of the results it is not necessary to re-perform the analyses:

  The R scripts "Evaluation_DetermineTuningGrid.R" and "Evaluation_LargeStudy.R"
  contained in the subfolder "Evaluation" produce all results shown in the main
  paper and in the online appendices without the need of re-performing 
  the analyses. These R scripts read in Rda files (stored in the subfolder "Results")
  that contain the raw results.



Full reproduction of the results:
#################################

- All R code needed to fully reproduce the analyses is available in 
  this electronic appendix.

- As a first step, the folder "suppmat2_hornung2020" this README is contained
  in has to be put into the home directory ("~/") of a Linux machine.

- An MPI environment is required.

- Note again that the preliminary version of the R package "diversityForest" used
  in the analyses is not available from CRAN or github, but contained
  in this Electronic Appendix (file "diversityForest_0.11.0.tar.gz" in the folder
  "suppmat2_hornung2020"). See again Section "Program and Platform" above for how
  to install this preliminary version of the package.

- The R scripts "JobScripts/DetermineTuningGrid.R" and "JobScripts/LargeStudy.R"
  perform the pre-study and the large scale comparisons study, respectively,
  in parallel.
  Before executing these scripts you need to install the RMPISNOW shell script 
  from the installed 'snow' R package or 'inst' directory of the package sources
  of the 'snow' R package in an appropriate location, preferably
  on your path. 
  See http://homepage.divms.uiowa.edu/~luke/R/cluster/cluster.html (last accessed: 
  17 December 2019) for more details.
  Subsequently, you need to create sh files, each for a different of the
  above R scripts. The following is the content of an example sh file "DetermineTuningGrid.sh":


  #!/bin/bash
  #SBATCH -o /myoutfiledirectory/myjob.%j.%N.out
  #SBATCH -D /myhomedirectory
  #SBATCH -J AnalysisCluster_1 
  #SBATCH --get-user-env 
  #SBATCH --clusters=myclustername
  #SBATCH --nodes=??
  #SBATCH --tasks-per-node=??
  #SBATCH --mail-type=end
  #SBATCH --mail-user=my@mail.de
  #SBATCH --time=??:??:??

  mpirun RMPISNOW < ./suppmat2_hornung2020/JobScripts/DetermineTuningGrid.R


  The above sh-file of course has to be adjusted to be useable (e.g., the "?"s have
  to replaced by actual numbers, the directories have to be adjusted and
  you need to specify your e-mail address; an e-mail will be sent to this address
  once the job is finished).
