import h2o
from h2o.automl import H2OAutoML

h2o.init()

Checking whether there is an H2O instance running at http://localhost:54321 . connected.
Warning: Version mismatch. H2O is version 3.37.0.99999, but the h2o-python package is version 0.0.local. This is a developer build, please contact your developer.

H2O_cluster_uptime:	21 mins 33 secs
H2O_cluster_timezone:	Europe/Prague
H2O_data_parsing_timezone:	UTC
H2O_cluster_version:	3.37.0.99999
H2O_cluster_version_age:	25 minutes
H2O_cluster_name:	tomasfryda
H2O_cluster_total_nodes:	1
H2O_cluster_free_memory:	1.309 Gb
H2O_cluster_total_cores:	16
H2O_cluster_allowed_cores:	8
H2O_cluster_status:	locked, healthy
H2O_connection_url:	http://localhost:54321
H2O_connection_proxy:	{"http": null, "https": null}
H2O_internal_security:	False
Python_version:	3.9.4 final

df = h2o.import_file("https://h2o-public-test-data.s3.amazonaws.com/smalldata/wine/winequality-redwhite-no-BOM.csv")

response = "quality"

predictors = [
  "fixed acidity", "volatile acidity", "citric acid", "residual sugar", "chlorides", "free sulfur dioxide",
  "total sulfur dioxide", "density", "pH", "sulphates", "alcohol",  "type"
]


train, test = df.split_frame(seed=1)

Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%

aml = H2OAutoML(max_runtime_secs=120, seed=1)
aml.train(x=predictors, y=response, training_frame=train)

AutoML progress: |███████████████████████████████████████████████████████████████| (done) 100%
Model Details
=============
H2OStackedEnsembleEstimator :  Stacked Ensemble
Model Key:  StackedEnsemble_AllModels_3_AutoML_3_20220408_140442

No model summary for this model

ModelMetricsRegressionGLM: stackedensemble
** Reported on train data. **

MSE: 0.023283411162563286
RMSE: 0.15258902700575586
MAE: 0.11065966448963627
RMSLE: 0.023320327709643703
R^2: 0.9696935442322193
Mean Residual Deviance: 0.023283411162563286
Null degrees of freedom: 4864
Residual degrees of freedom: 4856
Null deviance: 3737.612744090442
Residual deviance: 113.2737953058704
AIC: -4466.196859377104

ModelMetricsRegressionGLM: stackedensemble
** Reported on cross-validation data. **

MSE: 0.3759629223874193
RMSE: 0.6131581544654032
MAE: 0.42868816136684146
RMSLE: 0.0929837223844714
R^2: 0.5106342623893481
Mean Residual Deviance: 0.3759629223874193
Null degrees of freedom: 4864
Residual degrees of freedom: 4855
Null deviance: 3739.837050670749
Residual deviance: 1829.059617414795
AIC: 9069.013913851764

Cross-Validation Metrics Summary: 

		mean	sd	cv_1_valid	cv_2_valid	cv_3_valid	cv_4_valid	cv_5_valid
0	mae	0.429053	0.013194	0.418140	0.414085	0.432899	0.433157	0.446985
1	mean_residual_deviance	0.376736	0.026199	0.351823	0.353463	0.392439	0.372864	0.413090
2	mse	0.376736	0.026199	0.351823	0.353463	0.392439	0.372864	0.413090
3	null_deviance	747.967400	44.680550	758.488650	712.955260	794.441160	783.059700	690.892200
4	r2	0.508815	0.032326	0.530349	0.505313	0.523121	0.531222	0.454068
5	residual_deviance	365.811920	12.459985	356.044860	350.635000	378.703370	366.525540	377.150850
6	rmse	0.613494	0.021237	0.593147	0.594527	0.626449	0.610626	0.642721
7	rmsle	0.093029	0.004232	0.088082	0.090218	0.096522	0.092163	0.098159

aml.explain(test);

Leaderboard

Leaderboard shows models with their metrics. When provided with H2OAutoML object, the leaderboard shows 5-fold cross-validated metrics by default (depending on the H2OAutoML settings), otherwise it shows metrics computed on the frame. At most 20 models are shown by default.

model_id	rmse	mse	mae	rmsle	mean_residual_deviance	training_time_ms	predict_time_per_row_ms	algo
StackedEnsemble_AllModels_3_AutoML_3_20220408_140442	0.613158	0.375963	0.428688	0.0929837	0.375963	1334	0.072912	StackedEnsemble
StackedEnsemble_BestOfFamily_4_AutoML_3_20220408_140442	0.615253	0.378536	0.431953	0.093312	0.378536	423	0.027955	StackedEnsemble
StackedEnsemble_AllModels_2_AutoML_3_20220408_140442	0.61886	0.382988	0.439722	0.0937725	0.382988	146	0.033774	StackedEnsemble
StackedEnsemble_AllModels_1_AutoML_3_20220408_140442	0.619165	0.383365	0.440646	0.0937997	0.383365	146	0.02613	StackedEnsemble
StackedEnsemble_BestOfFamily_3_AutoML_3_20220408_140442	0.619706	0.384036	0.441241	0.0938909	0.384036	148	0.025569	StackedEnsemble
StackedEnsemble_BestOfFamily_2_AutoML_3_20220408_140442	0.620427	0.38493	0.442876	0.0939805	0.38493	138	0.023944	StackedEnsemble
DRF_1_AutoML_3_20220408_140442	0.622971	0.388092	0.449611	0.094527	0.388092	607	0.012088	DRF
GBM_grid_1_AutoML_3_20220408_140442_model_8	0.624575	0.390094	0.422475	0.0949575	0.390094	1812	0.013222	GBM
XRT_1_AutoML_3_20220408_140442	0.626012	0.391891	0.452358	0.0950673	0.391891	836	0.012583	DRF
GBM_grid_1_AutoML_3_20220408_140442_model_5	0.641773	0.411872	0.469973	0.0970923	0.411872	941	0.015306	GBM
GBM_4_AutoML_3_20220408_140442	0.64486	0.415845	0.484437	0.0975089	0.415845	329	0.009188	GBM
XGBoost_grid_1_AutoML_3_20220408_140442_model_19	0.646811	0.418365	0.479962	0.0976242	0.418365	785	0.005213	XGBoost
XGBoost_grid_1_AutoML_3_20220408_140442_model_2	0.648299	0.420292	0.44705	0.0980709	0.420292	580	0.005253	XGBoost
GBM_grid_1_AutoML_3_20220408_140442_model_1	0.648965	0.421155	0.478709	0.0982079	0.421155	553	0.014966	GBM
GBM_grid_1_AutoML_3_20220408_140442_model_10	0.65061	0.423293	0.478976	0.0985366	0.423293	1450	0.016868	GBM
StackedEnsemble_BestOfFamily_1_AutoML_3_20220408_140442	0.65196	0.425052	0.484097	0.0984392	0.425052	130	0.017711	StackedEnsemble
XGBoost_grid_1_AutoML_3_20220408_140442_model_28	0.652114	0.425252	0.48586	0.0983887	0.425252	1965	0.005989	XGBoost
GBM_3_AutoML_3_20220408_140442	0.654346	0.428168	0.497377	0.0989008	0.428168	267	0.010421	GBM
XGBoost_grid_1_AutoML_3_20220408_140442_model_24	0.655067	0.429113	0.455267	0.0990104	0.429113	1124	0.004858	XGBoost
XGBoost_grid_1_AutoML_3_20220408_140442_model_33	0.656874	0.431484	0.46009	0.099322	0.431484	560	0.002308	XGBoost

Residual Analysis

Residual Analysis plots the fitted values vs residuals on a test dataset. Ideally, residuals should be randomly distributed. Patterns in this plot can indicate potential problems with the model selection, e.g., using simpler model than necessary, not accounting for heteroscedasticity, autocorrelation, etc. Note that if you see "striped" lines of residuals, that is an artifact of having an integer valued (vs a real valued) response variable.

Variable Importance

The variable importance plot shows the relative importance of the most important variables in the model.

Variable Importance Heatmap

Variable importance heatmap shows variable importance across multiple models. Some models in H2O return variable importance for one-hot (binary indicator) encoded versions of categorical columns (e.g. Deep Learning, XGBoost). In order for the variable importance of categorical columns to be compared across all model types we compute a summarization of the the variable importance across all one-hot encoded features and return a single variable importance for the original categorical feature. By default, the models and variables are ordered by their similarity.

Model Correlation

This plot shows the correlation between the predictions of the models. For classification, frequency of identical predictions is used. By default, models are ordered by their similarity (as computed by hierarchical clustering). Interpretable models, such as GAM, GLM, and RuleFit are highlighted using red colored text.

SHAP Summary

SHAP summary plot shows the contribution of the features for each instance (row of data). The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function.

Partial Dependence Plots

Partial dependence plot (PDP) gives a graphical depiction of the marginal effect of a variable on the response. The effect of a variable is measured in change in the mean response. PDP assumes independence between the feature for which is the PDP computed and the rest.

Individual Conditional Expectation

An Individual Conditional Expectation (ICE) plot gives a graphical depiction of the marginal effect of a variable on the response. ICE plots are similar to partial dependence plots (PDP); PDP shows the average effect of a feature while ICE plot shows the effect for a single instance. This function will plot the effect for each decile. In contrast to the PDP, ICE plots can provide more insight, especially when there is stronger feature interaction.

aml.explain_row(test, row_index=42);

Leaderboard

Leaderboard shows models with their metrics and their predictions for a given row. When provided with H2OAutoML object, the leaderboard shows 5-fold cross-validated metrics by default (depending on the H2OAutoML settings), otherwise it shows metrics computed on the frame. At most 20 models are shown by default.

model_id	rmse	mse	mae	rmsle	mean_residual_deviance	training_time_ms	predict_time_per_row_ms	algo	predict
StackedEnsemble_AllModels_3_AutoML_3_20220408_140442	0.613158	0.375963	0.428688	0.0929837	0.375963	1334	0.072912	StackedEnsemble	5.27411
StackedEnsemble_BestOfFamily_4_AutoML_3_20220408_140442	0.615253	0.378536	0.431953	0.093312	0.378536	423	0.027955	StackedEnsemble	5.29165
StackedEnsemble_AllModels_2_AutoML_3_20220408_140442	0.61886	0.382988	0.439722	0.0937725	0.382988	146	0.033774	StackedEnsemble	5.36343
StackedEnsemble_AllModels_1_AutoML_3_20220408_140442	0.619165	0.383365	0.440646	0.0937997	0.383365	146	0.02613	StackedEnsemble	5.3616
StackedEnsemble_BestOfFamily_3_AutoML_3_20220408_140442	0.619706	0.384036	0.441241	0.0938909	0.384036	148	0.025569	StackedEnsemble	5.36032
StackedEnsemble_BestOfFamily_2_AutoML_3_20220408_140442	0.620427	0.38493	0.442876	0.0939805	0.38493	138	0.023944	StackedEnsemble	5.35617
DRF_1_AutoML_3_20220408_140442	0.622971	0.388092	0.449611	0.094527	0.388092	607	0.012088	DRF	5.37536
GBM_grid_1_AutoML_3_20220408_140442_model_8	0.624575	0.390094	0.422475	0.0949575	0.390094	1812	0.013222	GBM	5.26905
XRT_1_AutoML_3_20220408_140442	0.626012	0.391891	0.452358	0.0950673	0.391891	836	0.012583	DRF	5.50439
GBM_grid_1_AutoML_3_20220408_140442_model_5	0.641773	0.411872	0.469973	0.0970923	0.411872	941	0.015306	GBM	5.45243

SHAP Explanation

SHAP explanation shows contribution of features for a given instance. The sum of the feature contributions and the bias term is equal to the raw prediction of the model, i.e., prediction before applying inverse link function. H2O implements TreeSHAP which when the features are correlated, can increase contribution of a feature that had no influence on the prediction.