hex

Class ModelMetrics

java.lang.Object

water.Iced<T>

water.Keyed<ModelMetrics>

hex.ModelMetrics

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable<ModelMetrics>

Direct Known Subclasses:

ModelMetricsSupervised, ModelMetricsUnsupervised

public class ModelMetrics
extends Keyed<ModelMetrics>

Container to hold the metric for a model as scored on a specific frame. The MetricBuilder class is used in a hot inner-loop of a Big Data pass, and when given a class-distribution, can be used to compute CM's, and AUC's "on the fly" during ModelBuilding - or after-the-fact with a Model and a new Frame to be scored.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ModelMetrics.MetricBuilder<T extends ModelMetrics.MetricBuilder<T>> Class used to compute AUCs, CMs & HRs "on the fly" during other passes over Big Data.

Field Summary

Fields

Modifier and Type	Field and Description
CustomMetric	_custom_metric
java.lang.String	_description
double	_MSE
long	_nobs
long	_scoring_time

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
ModelMetrics(Model model, Frame frame, long nobs, double MSE, java.lang.String desc, CustomMetric customMetric)

Method Summary

Modifier and Type	Method and Description
protected java.lang.StringBuilder	appendToStringMetrics(java.lang.StringBuilder sb)
AUC2	auc_obj()
static Key<ModelMetrics>	buildKey(Key model_key, long model_checksum, Key frame_key, long frame_checksum)
static Key<ModelMetrics>	buildKey(Model model, Frame frame)
static TwoDimTable	calcVarImp(double[] rel_imp, java.lang.String[] coef_names)
static TwoDimTable	calcVarImp(double[] rel_imp, java.lang.String[] coef_names, java.lang.String table_header, java.lang.String[] col_headers)
static TwoDimTable	calcVarImp(float[] rel_imp, java.lang.String[] coef_names)
static TwoDimTable	calcVarImp(VarImp vi)
protected long	checksum_impl() High-quality 64-bit checksum of the content of the object.
ConfusionMatrix	cm()
double	custom_increasing()
double	custom()
ModelMetrics	deepCloneWithDifferentModelAndFrame(Model model, Frame frame) Utility used by code which creates metrics on a different frame and model than the ones that we want the metrics object to be accessible for.
static ModelMetrics	defaultModelMetrics(Model model)
Frame	frame()
static java.util.Set<java.lang.String>	getAllowedMetrics(Key<Model> key)
static ModelMetrics	getFromDKV(Model model, Frame frame)
static double	getMetricFromModel(Key<Model> key, java.lang.String criterion)
static double	getMetricFromModelMetric(ModelMetrics mm, java.lang.String criterion)
float[]	hr()
boolean	isForFrame(Frame f)
boolean	isForModel(Model m)
Model	model()
double	mse()
long	residual_degrees_of_freedom()
double	rmse()
void	setModelKey(Key modelKey)
static java.util.List<Key<Model>>	sortModelsByMetric(Frame frame, java.lang.String sort_by, boolean decreasing, java.util.List<Key<Model>> modelKeys) Return a new list of models sorted on metrics computed on the given frame, by the named criterion.
static java.util.List<Key<Model>>	sortModelsByMetric(java.lang.String sort_by, boolean decreasing, java.util.List<Key<Model>> modelKeys) Return a new list of models sorted on their xval, validation or training metrics, by the named criterion.
java.lang.String	toString()
ModelMetrics	withDescription(java.lang.String desc)
ModelMetrics	withModelAndFrame(Model model, Frame frame)

Methods inherited from class water.Keyed

checksum_impl, checksum, checksum, getKey, makeSchema, readAll_impl, readAll, remove_impl, remove_impl, remove_self_key_impl, remove, remove, remove, remove, remove, remove, removeQuietly, writeAll_impl, writeAll

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

_description

public java.lang.String _description

_scoring_time

public final long _scoring_time

_custom_metric

public final CustomMetric _custom_metric

_MSE

public final double _MSE

_nobs

public final long _nobs

Constructor Detail

ModelMetrics

public ModelMetrics(Model model,
                    Frame frame,
                    long nobs,
                    double MSE,
                    java.lang.String desc,
                    CustomMetric customMetric)

Method Detail

setModelKey

public final void setModelKey(Key modelKey)

withModelAndFrame

public final ModelMetrics withModelAndFrame(Model model,
                                            Frame frame)

withDescription

public ModelMetrics withDescription(java.lang.String desc)

deepCloneWithDifferentModelAndFrame

public ModelMetrics deepCloneWithDifferentModelAndFrame(Model model,
                                                        Frame frame)

Utility used by code which creates metrics on a different frame and model than the ones that we want the metrics object to be accessible for. An example is StackedEnsembleModel, which computes the metrics with a metalearner model.

Parameters:

model -

frame -

Returns:

residual_degrees_of_freedom

public long residual_degrees_of_freedom()

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

appendToStringMetrics

protected java.lang.StringBuilder appendToStringMetrics(java.lang.StringBuilder sb)

model

public final Model model()

frame

public final Frame frame()

custom

public double custom()

custom_increasing

public double custom_increasing()

mse

public double mse()

rmse

public double rmse()

cm

public ConfusionMatrix cm()

hr

public float[] hr()

auc_obj

public AUC2 auc_obj()

defaultModelMetrics

public static ModelMetrics defaultModelMetrics(Model model)

getMetricFromModel

public static double getMetricFromModel(Key<Model> key,
                                        java.lang.String criterion)

getMetricFromModelMetric

public static double getMetricFromModelMetric(ModelMetrics mm,
                                              java.lang.String criterion)

getAllowedMetrics

public static java.util.Set<java.lang.String> getAllowedMetrics(Key<Model> key)

sortModelsByMetric

public static java.util.List<Key<Model>> sortModelsByMetric(java.lang.String sort_by,
                                                            boolean decreasing,
                                                            java.util.List<Key<Model>> modelKeys)

Return a new list of models sorted on their xval, validation or training metrics, by the named criterion. The criterion (metric) can be such things as as "auc", mse", "hr", "err", "err_count", "accuracy", "specificity", "recall", "precision", "mcc", "max_per_class_error", "f1", "f2", "f0point5". . .

Parameters:

sort_by - criterion by which we should sort

decreasing - sort by decreasing metrics or not

modelKeys - keys of models to sortm

Returns:

keys of the models, sorted by the criterion

sortModelsByMetric

public static java.util.List<Key<Model>> sortModelsByMetric(Frame frame,
                                                            java.lang.String sort_by,
                                                            boolean decreasing,
                                                            java.util.List<Key<Model>> modelKeys)

Return a new list of models sorted on metrics computed on the given frame, by the named criterion. The criterion (metric) can be such things as as "auc", mse", "hr", "err", "err_count", "accuracy", "specificity", "recall", "precision", "mcc", "max_per_class_error", "f1", "f2", "f0point5". . .

Parameters:

frame - frame on which to compute the metrics; looked up in the DKV first to see if it was previously computed

sort_by - criterion by which we should sort

decreasing - sort by decreasing metrics or not

modelKeys - keys of models to sortm

Returns:

keys of the models, sorted by the criterion

calcVarImp

public static TwoDimTable calcVarImp(VarImp vi)

calcVarImp

public static TwoDimTable calcVarImp(float[] rel_imp,
                                     java.lang.String[] coef_names)

calcVarImp

public static TwoDimTable calcVarImp(double[] rel_imp,
                                     java.lang.String[] coef_names)

calcVarImp

public static TwoDimTable calcVarImp(double[] rel_imp,
                                     java.lang.String[] coef_names,
                                     java.lang.String table_header,
                                     java.lang.String[] col_headers)

buildKey

public static Key<ModelMetrics> buildKey(Key model_key,
                                         long model_checksum,
                                         Key frame_key,
                                         long frame_checksum)

buildKey

public static Key<ModelMetrics> buildKey(Model model,
                                         Frame frame)

isForModel

public boolean isForModel(Model m)

isForFrame

public boolean isForFrame(Frame f)

getFromDKV

public static ModelMetrics getFromDKV(Model model,
                                      Frame frame)

checksum_impl

protected long checksum_impl()

Description copied from class: Keyed

High-quality 64-bit checksum of the content of the object. Similar to hashcode(), but a long to reduce the chance of hash clashes. For composite objects this should be defined using the subcomponents' checksums (or hashcodes if not available). If two Keyed objects have the same checksum() there should be a 1 - 1/2^64 chance that they are the same object by value.

Overrides:

checksum_impl in class Keyed<ModelMetrics>