hex.leaderboard

Class Leaderboard

java.lang.Object

water.Iced<T>

water.Keyed<T>

water.Lockable<Leaderboard>

hex.leaderboard.Leaderboard

All Implemented Interfaces:

ModelContainer<Model>, java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable<Leaderboard>

public class Leaderboard
extends Lockable<Leaderboard>
implements ModelContainer<Model>

Utility to track all the models built for a given dataset type.

Note that if a new Leaderboard is made for the same project_name it'll keep using the old model list, which allows us to run AutoML multiple times and keep adding to the leaderboard.

The models are returned sorted by either an appropriate default metric for the model category (auc, mean per class error, or mean residual deviance), or by a metric that's set via #setMetricAndDirection.

TODO: make this robust against removal of models from the DKV.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Leaderboard.ScoreData What data should be used to generate leaderboard metrics.

Field Summary

Fields inherited from class water.Lockable

_lockers

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
Leaderboard(java.lang.String projectName, water.logging.Logger logger, Frame leaderboardFrame, java.lang.String sortMetric, Leaderboard.ScoreData scoreData) Constructs a new leaderboard (doesn't put it in DKV).

Method Summary

Modifier and Type	Method and Description
<M extends Model> void	addModel(Key<M> key)
void	addModels(Key<Model>[] modelKeys) Add the given models to the leaderboard.
void	ensureSorted()
java.util.Map<Key<Model>,LeaderboardCell[]>	getExtensionsAsMap()
static Leaderboard	getInstance(Key leaderboardKey, water.logging.Logger logger) Retrieves a leaderboard from DKV
static Leaderboard	getInstance(java.lang.String projectName, water.logging.Logger logger, Frame leaderboardFrame, java.lang.String sortMetric, Leaderboard.ScoreData scoreData) Retrieves a leaderboard from DKV
Model	getLeader()
java.lang.String[]	getMetrics() The sort metric is always the first element in the list of metrics.
double[]	getMetricValues(java.lang.String metricName)
int	getModelCount() Return the number of models in this Leaderboard.
Key<Model>[]	getModelKeys()
int	getModelRank(Key<Model> modelKey)
Model[]	getModels()
Model[]	getModelsSortedByMetric(java.lang.String metric)
ModelMetrics	getOrCreateModelMetrics(Key<Model> modelKey)
static Leaderboard	getOrMake(java.lang.String projectName, water.logging.Logger logger, Frame leaderboardFrame, java.lang.String sortMetric)
static Leaderboard	getOrMake(java.lang.String projectName, water.logging.Logger logger, Frame leaderboardFrame, java.lang.String sortMetric, Leaderboard.ScoreData scoreData) Retrieves a leaderboard from DKV or creates a fresh one and add it to DKV.
java.lang.String	getProject()
java.lang.String	getSortMetric() If no sort metric is provided when creating the leaderboard, then a default sort metric will be automatically chosen based on the problem type: binomial classification: auc multinomial classification: logloss regression: mean_residual_deviance
double[]	getSortMetricValues()
static java.lang.String	idForProject(java.lang.String project_name)
static java.lang.String	idForProject(java.lang.String project_name, Leaderboard.ScoreData score_data)
static boolean	isLossFunction(java.lang.String metric)
Frame	leaderboardFrame() Note: If no leaderboard was provided, then the models are sorted according to metrics obtained during training in the following priority order depending on availability: cross-validation metrics validation metrics training metrics
java.lang.String	rankTsv()
protected Futures	remove_impl(Futures fs, boolean cascade) Delete object and its dependencies from DKV, including models.
<M extends Model> void	removeModel(Key<M> key, boolean cascade)
void	removeModels(Key<Model>[] modelKeys, boolean cascade)
void	setExtensionsProvider(LeaderboardExtensionsProvider provider) Assign a LeaderboardExtensionsProvider to this leaderboard instance.
java.lang.String	toLogString()
java.lang.String	toString()
TwoDimTable	toTwoDimTable(java.lang.String... extensions) Creates a TwoDimTable representation of the leaderboard.

Methods inherited from class water.Lockable

delete_and_lock, delete_and_lock, delete_and_lock, delete_and_lock, delete, delete, delete, delete, read_lock, read_lock, read_lock, unlock_all, unlock, unlock, unlock, unlock, update, update, update, write_lock_to_read_lock, write_lock, write_lock, write_lock

Methods inherited from class water.Keyed

checksum_impl, checksum_impl, checksum, checksum, getKey, makeSchema, readAll_impl, readAll, remove_impl, remove_self_key_impl, remove, remove, remove, remove, remove, remove, removeQuietly, writeAll_impl, writeAll

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Leaderboard

public Leaderboard(java.lang.String projectName,
                   water.logging.Logger logger,
                   Frame leaderboardFrame,
                   java.lang.String sortMetric,
                   Leaderboard.ScoreData scoreData)

Constructs a new leaderboard (doesn't put it in DKV).

Parameters:

projectName -

logger -

leaderboardFrame -

sortMetric -

scoreData -

Method Detail

idForProject

public static java.lang.String idForProject(java.lang.String project_name)

Parameters:

project_name -

Returns:

a Leaderboard id for the project name

idForProject

public static java.lang.String idForProject(java.lang.String project_name,
                                            Leaderboard.ScoreData score_data)

Parameters:

project_name -

score_data - what metrics should be reported

Returns:

a Leaderboard id for the project name, when score_data == auto use idForProject(String) to generate id

isLossFunction

public static boolean isLossFunction(java.lang.String metric)

Parameters:

metric -

Returns:

true iff the metric is a loss function

getInstance

public static Leaderboard getInstance(Key leaderboardKey,
                                      water.logging.Logger logger)

Retrieves a leaderboard from DKV

Parameters:

leaderboardKey -

logger -

Returns:

an existing leaderboard if there's already one in DKV for this project, or null.

getInstance

public static Leaderboard getInstance(java.lang.String projectName,
                                      water.logging.Logger logger,
                                      Frame leaderboardFrame,
                                      java.lang.String sortMetric,
                                      Leaderboard.ScoreData scoreData)

Retrieves a leaderboard from DKV

Parameters:

projectName -

logger -

leaderboardFrame -

sortMetric -

scoreData -

Returns:

an existing leaderboard if there's already one in DKV for this project, or null.

getOrMake

public static Leaderboard getOrMake(java.lang.String projectName,
                                    water.logging.Logger logger,
                                    Frame leaderboardFrame,
                                    java.lang.String sortMetric,
                                    Leaderboard.ScoreData scoreData)

Retrieves a leaderboard from DKV or creates a fresh one and add it to DKV. Note that if the leaderboard is reused to add new models, we have to use the same leaderboard frame. IMPORTANT! if the leaderboard is created without leaderboardFrame, the models will be sorted according to their default metrics (in order of availability: cross-validation metrics, validation metrics, training metrics). Therefore, if some models were trained with/without cross-validation, or with different training or validation frames, then we can't guarantee the fairness of the leaderboard ranking.

Parameters:

projectName -

logger -

leaderboardFrame -

sortMetric -

scoreData -

Returns:

an existing leaderboard if there's already one in DKV for this project, or a new leaderboard added to DKV.

getOrMake

public static Leaderboard getOrMake(java.lang.String projectName,
                                    water.logging.Logger logger,
                                    Frame leaderboardFrame,
                                    java.lang.String sortMetric)

See Also:

getOrMake(String, Logger, Frame, String, ScoreData)

setExtensionsProvider

public void setExtensionsProvider(LeaderboardExtensionsProvider provider)

Assign a LeaderboardExtensionsProvider to this leaderboard instance.

Parameters:

provider - the provider used to generate the optional extension columns from the leaderboard.

See Also:

LeaderboardExtensionsProvider

getProject

public java.lang.String getProject()

getSortMetric

public java.lang.String getSortMetric()

If no sort metric is provided when creating the leaderboard, then a default sort metric will be automatically chosen based on the problem type:

binomial classification: auc

multinomial classification: logloss

regression: mean_residual_deviance

Returns:

the metric used to sort the models in the leaderboard.

getMetrics

public java.lang.String[] getMetrics()

The sort metric is always the first element in the list of metrics.

Returns:

the full list of metrics available in the leaderboard.

leaderboardFrame

public Frame leaderboardFrame()

Note: If no leaderboard was provided, then the models are sorted according to metrics obtained during training in the following priority order depending on availability:

cross-validation metrics

validation metrics

training metrics