hex.modelselection

Class ModelSelection

java.lang.Object

water.Iced

hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

hex.modelselection.ModelSelection

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, water.Freezable

public class ModelSelection
extends hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	ModelSelection.ModelSelectionDriver
static class	ModelSelection.SweepModel Contains information of a predictor subsets like predictor indices of the subset (with the newest predictor as the last element of the array), CPM associated with predictor subset minus the latest element and the error variance of the CPM.

Nested classes/interfaces inherited from class hex.ModelBuilder

hex.ModelBuilder.BuilderVisibility, hex.ModelBuilder.Driver, hex.ModelBuilder.FilterCols, hex.ModelBuilder.ModelTrainingCoordinator, hex.ModelBuilder.ValidationMessage

Field Summary

Fields

Modifier and Type	Field and Description
java.lang.String[][]	_bestModelPredictors
double[]	_bestR2Values
int	_glmNFolds
int	_numPredictors
java.lang.String[]	_predictorNames
java.lang.String[][]	_predictorsAdd
java.lang.String[][]	_predictorsRemoved

Fields inherited from class hex.ModelBuilder

_coordinator, _desc, _eventPublisher, _fold, _input_parms, _job, _messages, _nclass, _offset, _orig_projection_array, _origDomains, _origNames, _origTrain, _parms, _priorClassDist, _removedCols, _response, _result, _startUpOnceModelBuilder, _train, _treatment, _valid, _vresponse, _weights

Constructor Summary

Constructors

Constructor and Description
ModelSelection(boolean startup_once)
ModelSelection(ModelSelectionModel.ModelSelectionParameters parms)
ModelSelection(ModelSelectionModel.ModelSelectionParameters parms, water.Key<ModelSelectionModel> key)

Method Summary

Modifier and Type	Method and Description
static GLMModel	buildExtractBestR2Model(water.fvec.Frame[] trainingFrames, ModelSelectionModel.ModelSelectionParameters parms, int glmNFolds, java.lang.String foldColumn, hex.Model.Parameters.FoldAssignmentScheme foldAssignment) Given the training Frame array, build models for each training frame and return the GLMModel with the best R2 values.
hex.ModelCategory[]	can_build()
protected void	checkMemoryFootPrint(int p)
ModelSelection.SweepModel	forwardStep(java.util.List<java.lang.Integer> currSubsetIndices, java.util.List<java.lang.Integer> validSubsets, java.util.Set<java.util.BitSet> usedCombo, java.util.BitSet predIndices, int[][] predInd2CPMInd, ModelSelection.SweepModel bestModel, boolean hasIntercept) Given current predictor subset in currSubsetIndices, this method will add one more predictor to the subset and choose the one that will increase the R2 by the most.
static GLMModel	forwardStep(java.util.List<java.lang.Integer> currSubsetIndices, java.util.List<java.lang.String> coefNames, int predPos, java.util.List<java.lang.Integer> validSubsets, ModelSelectionModel.ModelSelectionParameters parms, java.lang.String foldColumn, int glmNFolds, hex.Model.Parameters.FoldAssignmentScheme foldAssignment)
static GLMModel	forwardStep(java.util.List<java.lang.Integer> currSubsetIndices, java.util.List<java.lang.String> coefNames, int predPos, java.util.List<java.lang.Integer> validSubsets, ModelSelectionModel.ModelSelectionParameters parms, java.lang.String foldColumn, int glmNFolds, hex.Model.Parameters.FoldAssignmentScheme foldAssignment, java.util.Set<java.util.BitSet> usedCombo) Given a predictor subset with indices stored in currSubsetIndices, one more predictor from the coefNames that was not found in currSubsetIndices was added to the subset to form a new Training frame.
ModelSelection.SweepModel	forwardStepR(java.util.List<java.lang.Integer> currSubsetIndices, java.util.List<java.lang.Integer> validSubsets, java.util.Set<java.util.BitSet> usedCombo, java.util.BitSet predIndices, int[][] predInd2CPMInd, ModelSelection.SweepModel bestModel, double bestErrVar, int predPos) Given a currSubsetIndices and a predPos, this function will try to look for new predictor that will decrease the error variance compared to bestErrVar.
boolean	haveMojo()
boolean	havePojo()
void	init(boolean expensive)
boolean	isSupervised()
protected int	nModelsInParallel(int folds)
ModelSelection.SweepModel	replacement(java.util.List<java.lang.Integer> currSubsetIndices, java.util.List<java.lang.Integer> validSubset, java.util.Set<java.util.BitSet> usedCombos, java.util.BitSet predIndices, ModelSelection.SweepModel bestModel, int[][] predictorIndex2CPMIndices) consider the predictors in subset as pred0, pred1, pred2 (using subset size 3 as example): a.
static GLMModel	replacement(java.util.List<java.lang.Integer> currSubsetIndices, java.util.List<java.lang.String> coefNames, double bestR2, ModelSelectionModel.ModelSelectionParameters parms, int glmNFolds, java.lang.String foldColumn, java.util.List<java.lang.Integer> validSubset, hex.Model.Parameters.FoldAssignmentScheme foldAssignment, java.util.Set<java.util.BitSet> usedCombos) consider the predictors in subset as pred0, pred1, pred2 (using subset size 3 as example): a.
protected ModelSelection.ModelSelectionDriver	trainModelImpl()

Methods inherited from class hex.ModelBuilder

algoName, algos, builderVisibility, canLearnFromNAs, checkCustomMetricForEarlyStopping, checkDistributions, checkEarlyStoppingReproducibility, checkMemoryFootPrint_impl, checkMemoryFootPrint, checkResponseVariable, clearInitState, clearValidationErrors, computeCrossValidation, computePriorClassDistribution, cv_buildModels, cv_canBuildMainModelInParallel, cv_computeAndSetOptimalParameters, cv_initStoppingParameters, cv_mainModelScores, cv_makeAggregateModelMetrics, cv_scoreCVModels, cv_updateOptimalParameters, defaultKey, desiredChunks, dest, error_count, error, get, getMessagesByFieldAndSeverity, getName, getSysProperty, getToEigenVec, hasFoldCol, hasOffsetCol, hasTreatmentCol, hasWeightCol, hide, ignoreBadColumns, ignoreConstColumns, ignoreInvalidColumns, ignoreStringColumns, ignoreUuidColumns, info, init_adaptFrameToTrain, init_getNClass, initWorkspace, isClassifier, isResponseOptional, isStopped, javaName, logMe, make, make, make, makeCVMetrics, makeCVModelBuilder, makeParameters, makePojoWriter, message, nclasses, nFoldCV, nFoldWork, nModelsInParallel, nModelsInParallel, numSpecialCols, paramName, raiseReproducibilityWarning, rebalance, remainingTimeSecs, response, schemaDirectory, separateFeatureVecs, setMaxRuntimeSecsForMainModel, setTrain, setValid, shouldReorder, smallDataSize, stop_requested, timeout, train, trainModel, trainModel, trainModelNested, trainModelNested, trainModelOnH2ONode, valid, validateBinaryResponse, validateStoppingMetric, validationErrors, validationWarnings, vresponse, warn

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_bestModelPredictors

public java.lang.String[][] _bestModelPredictors

_bestR2Values

public double[] _bestR2Values

_predictorsAdd

public java.lang.String[][] _predictorsAdd

_predictorsRemoved

public java.lang.String[][] _predictorsRemoved

_numPredictors

public int _numPredictors

_predictorNames

public java.lang.String[] _predictorNames

_glmNFolds

public int _glmNFolds

Constructor Detail

ModelSelection

public ModelSelection(boolean startup_once)

ModelSelection

public ModelSelection(ModelSelectionModel.ModelSelectionParameters parms)

ModelSelection

public ModelSelection(ModelSelectionModel.ModelSelectionParameters parms,
                      water.Key<ModelSelectionModel> key)

Method Detail

nModelsInParallel

protected int nModelsInParallel(int folds)

Overrides:

nModelsInParallel in class hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

trainModelImpl

protected ModelSelection.ModelSelectionDriver trainModelImpl()

Specified by:

trainModelImpl in class hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

can_build

public hex.ModelCategory[] can_build()

Specified by:

can_build in class hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

isSupervised

public boolean isSupervised()

Specified by:

isSupervised in class hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

haveMojo

public boolean haveMojo()

Overrides:

haveMojo in class hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

havePojo

public boolean havePojo()

Overrides:

havePojo in class hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

init

public void init(boolean expensive)

Overrides:

init in class hex.ModelBuilder<ModelSelectionModel,ModelSelectionModel.ModelSelectionParameters,ModelSelectionModel.ModelSelectionModelOutput>

checkMemoryFootPrint

protected void checkMemoryFootPrint(int p)

forwardStep

public ModelSelection.SweepModel forwardStep(java.util.List<java.lang.Integer> currSubsetIndices,
                                             java.util.List<java.lang.Integer> validSubsets,
                                             java.util.Set<java.util.BitSet> usedCombo,
                                             java.util.BitSet predIndices,
                                             int[][] predInd2CPMInd,
                                             ModelSelection.SweepModel bestModel,
                                             boolean hasIntercept)

Given current predictor subset in currSubsetIndices, this method will add one more predictor to the subset and choose the one that will increase the R2 by the most.

replacement

public ModelSelection.SweepModel replacement(java.util.List<java.lang.Integer> currSubsetIndices,
                                             java.util.List<java.lang.Integer> validSubset,
                                             java.util.Set<java.util.BitSet> usedCombos,
                                             java.util.BitSet predIndices,
                                             ModelSelection.SweepModel bestModel,
                                             int[][] predictorIndex2CPMIndices)

consider the predictors in subset as pred0, pred1, pred2 (using subset size 3 as example): a. Keep pred1, pred2 and replace pred0 with remaining predictors, find replacement with highest R2 b. keep pred0, pred2 and replace pred1 with remaining predictors, find replacement with highest R2 c. keep pred0, pred1 and replace pred2 with remaining predictors, find replacement with highest R2 d. from step 2a, 2b, 2c, choose the predictor subset with highest R2, say the subset is pred0, pred4, pred2 e. Take subset pred0, pred4, pred2 and go through steps a,b,c,d again and only stop when the best R2 does not improve anymore. see doc at https://github.com/h2oai/h2o-3/issues/7217 for details. The most important thing here is to make sure validSubset contains the true eligible predictors to choose from. Inside the for loop, I will remove and add predictors that have been chosen in oneLessSubset and add the removed predictor back to validSubset after it is no longer selected in the predictor subset. I also will reset the validSubset from time to time to just start to include all predictors as valid, then I will remove all predictors that have been chosen in currSubsetIndices.

forwardStepR

public ModelSelection.SweepModel forwardStepR(java.util.List<java.lang.Integer> currSubsetIndices,
                                              java.util.List<java.lang.Integer> validSubsets,
                                              java.util.Set<java.util.BitSet> usedCombo,
                                              java.util.BitSet predIndices,
                                              int[][] predInd2CPMInd,
                                              ModelSelection.SweepModel bestModel,
                                              double bestErrVar,
                                              int predPos)

Given a currSubsetIndices and a predPos, this function will try to look for new predictor that will decrease the error variance compared to bestErrVar. This is slightly different than the forwardStep. This is what happens, given the bestModel with the predictor subset stored in currSubsetIndices, it will do the following: 1. extract a subsetCPM which contains the cpm rows/columns corresponding to predictors in currSubsetIndices; 2. perform a sweeping for the removed predictors to undo the effect of the removed predictors. This subset is stored in subsetCPMO. 3. Next, in generateAllErrVarR, a new predictor is added to currentSubsetIndices to decide which one will provide the lowest error variance. 4. If the lowest error variance is lower than bestErrVar, the currentSubsetIndices will replace the predictor at predPos with the new predictor. 5. If the lowest error variance found from generateAllErrVarR is higher than bestErrVar, nothing is done.

buildExtractBestR2Model

public static GLMModel buildExtractBestR2Model(water.fvec.Frame[] trainingFrames,
                                               ModelSelectionModel.ModelSelectionParameters parms,
                                               int glmNFolds,
                                               java.lang.String foldColumn,
                                               hex.Model.Parameters.FoldAssignmentScheme foldAssignment)

Given the training Frame array, build models for each training frame and return the GLMModel with the best R2 values.

Parameters:

trainingFrames -

Returns:

forwardStep

public static GLMModel forwardStep(java.util.List<java.lang.Integer> currSubsetIndices,
                                   java.util.List<java.lang.String> coefNames,
                                   int predPos,
                                   java.util.List<java.lang.Integer> validSubsets,
                                   ModelSelectionModel.ModelSelectionParameters parms,
                                   java.lang.String foldColumn,
                                   int glmNFolds,
                                   hex.Model.Parameters.FoldAssignmentScheme foldAssignment,
                                   java.util.Set<java.util.BitSet> usedCombo)

Given a predictor subset with indices stored in currSubsetIndices, one more predictor from the coefNames that was not found in currSubsetIndices was added to the subset to form a new Training frame. An array of training frame are built training frames from all elligible predictors from coefNames. GLMParameters are built for all TrainingFrames, GLM models are built from those parameters. The GLM model with the highest R2 value is returned. The added predictor which resulted in the highest R2 value will be added to currSubsetIndices. see doc at https://github.com/h2oai/h2o-3/issues/7217 for details.

Parameters:

currSubsetIndices: - stored predictors that are chosen in the subset

coefNames: - predictor names of full training frame

predPos: - index/location of predictor to be added into currSubsetIndices

Returns:

GLMModel with highest R2 value

forwardStep

public static GLMModel forwardStep(java.util.List<java.lang.Integer> currSubsetIndices,
                                   java.util.List<java.lang.String> coefNames,
                                   int predPos,
                                   java.util.List<java.lang.Integer> validSubsets,
                                   ModelSelectionModel.ModelSelectionParameters parms,
                                   java.lang.String foldColumn,
                                   int glmNFolds,
                                   hex.Model.Parameters.FoldAssignmentScheme foldAssignment)

replacement

public static GLMModel replacement(java.util.List<java.lang.Integer> currSubsetIndices,
                                   java.util.List<java.lang.String> coefNames,
                                   double bestR2,
                                   ModelSelectionModel.ModelSelectionParameters parms,
                                   int glmNFolds,
                                   java.lang.String foldColumn,
                                   java.util.List<java.lang.Integer> validSubset,
                                   hex.Model.Parameters.FoldAssignmentScheme foldAssignment,
                                   java.util.Set<java.util.BitSet> usedCombos)

consider the predictors in subset as pred0, pred1, pred2 (using subset size 3 as example): a. Keep pred1, pred2 and replace pred0 with remaining predictors, find replacement with highest R2 b. keep pred0, pred2 and replace pred1 with remaining predictors, find replacement with highest R2 c. keeep pred0, pred1 and replace pred2 with remaining predictors, find replacement with highest R2 d. from step 2a, 2b, 2c, choose the predictor subset with highest R2, say the subset is pred0, pred4, pred2 e. Take subset pred0, pred4, pred2 and go through steps a,b,c,d again and only stop when the best R2 does not improve anymore. see doc at https://github.com/h2oai/h2o-3/issues/7217 for details.