water.util

Class MRUtils

java.lang.Object

water.util.MRUtils

public class MRUtils
extends java.lang.Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MRUtils.ClassDist Compute the class distribution from a class label vector (not counting missing values) Usage 1: Label vector is categorical ------------------------------------ Vec label = ...; assert(label.isCategorical()); double[] dist = new ClassDist(label).doAll(label).dist(); Usage 2: Label vector is numerical ---------------------------------- Vec label = ...; int num_classes = ...; assert(label.isInt()); double[] dist = new ClassDist(num_classes).doAll(label).dist();
static class	MRUtils.ClassDistQuasibinomial Compute the class distribution for qusibinomial distribution from a class label vector (not counting missing values)
static class	MRUtils.Dist

Constructor Summary

Constructors

Constructor and Description
MRUtils()

Method Summary

Modifier and Type	Method and Description
static Frame	sampleFrame(Frame fr, long rows, long seed) Sample rows from a frame.
static Frame	sampleFrame(Frame fr, long rows, java.lang.String weightColumn, long seed) Sample rows from a frame with weight column.
static Frame	sampleFrameSmall(Frame fr, int rows, long seed) Sample small number of rows from a frame.
static Frame	sampleFrameSmall(Frame fr, int rows, java.util.Random rand) Sample small number of rows from a frame.
static Frame	sampleFrameStratified(Frame fr, Vec label, Vec weights, float[] sampling_ratios, long seed, boolean debug, java.lang.String[] quasibinomialDomain) Stratified sampling
static Frame	sampleFrameStratified(Frame fr, Vec label, Vec weights, float[] sampling_ratios, long maxrows, long seed, boolean allowOversampling, boolean verbose) Stratified sampling for classifiers - FIXME: For weights, this is not accurate, as the sampling is done with uniform weights
static Frame	sampleFrameStratified(Frame fr, Vec label, Vec weights, float[] sampling_ratios, long maxrows, long seed, boolean allowOversampling, boolean verbose, java.lang.String[] quasibinomialDomain) Stratified sampling for classifiers - FIXME: For weights, this is not accurate, as the sampling is done with uniform weights
static Frame	shuffleFramePerChunk(Frame fr, long seed) Row-wise shuffle of a frame (only shuffles rows inside of each chunk)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MRUtils

public MRUtils()

Method Detail

sampleFrame

public static Frame sampleFrame(Frame fr,
                                long rows,
                                long seed)

Sample rows from a frame. Can be unlucky for small sampling fractions - will continue calling itself until at least 1 row is returned.

Parameters:

fr - Input frame

rows - Approximate number of rows to sample (across all chunks)

seed - Seed for RNG

Returns:

Sampled frame

shuffleFramePerChunk

public static Frame shuffleFramePerChunk(Frame fr,
                                         long seed)

Row-wise shuffle of a frame (only shuffles rows inside of each chunk)

Parameters:

fr - Input frame

Returns:

Shuffled frame

sampleFrameStratified

public static Frame sampleFrameStratified(Frame fr,
                                          Vec label,
                                          Vec weights,
                                          float[] sampling_ratios,
                                          long maxrows,
                                          long seed,
                                          boolean allowOversampling,
                                          boolean verbose)

Stratified sampling for classifiers - FIXME: For weights, this is not accurate, as the sampling is done with uniform weights

Parameters:

fr - Input frame

label - Label vector (must be categorical)

weights - Weights vector, can be null

sampling_ratios - Optional: array containing the requested sampling ratios per class (in order of domains), will be overwritten if it contains all 0s

maxrows - Maximum number of rows in the returned frame

seed - RNG seed for sampling

allowOversampling - Allow oversampling of minority classes

verbose - Whether to print verbose info

Returns:

Sampled frame, with approximately the same number of samples from each class (or given by the requested sampling ratios)

sampleFrameStratified

public static Frame sampleFrameStratified(Frame fr,
                                          Vec label,
                                          Vec weights,
                                          float[] sampling_ratios,
                                          long maxrows,
                                          long seed,
                                          boolean allowOversampling,
                                          boolean verbose,
                                          java.lang.String[] quasibinomialDomain)

Stratified sampling for classifiers - FIXME: For weights, this is not accurate, as the sampling is done with uniform weights

Parameters:

fr - Input frame

label - Label vector (must be categorical)

weights - Weights vector, can be null

sampling_ratios - Optional: array containing the requested sampling ratios per class (in order of domains), will be overwritten if it contains all 0s

maxrows - Maximum number of rows in the returned frame

seed - RNG seed for sampling

allowOversampling - Allow oversampling of minority classes

verbose - Whether to print verbose info

quasibinomialDomain - quasibinomial domain

Returns:

Sampled frame, with approximately the same number of samples from each class (or given by the requested sampling ratios)

sampleFrameStratified

public static Frame sampleFrameStratified(Frame fr,
                                          Vec label,
                                          Vec weights,
                                          float[] sampling_ratios,
                                          long seed,
                                          boolean debug,
                                          java.lang.String[] quasibinomialDomain)

Stratified sampling

Parameters:

fr - Input frame

label - Label vector (from the input frame)

weights - Weight vector (from the input frame), can be null

sampling_ratios - Given sampling ratios for each class, in order of domains

seed - RNG seed

debug - Whether to print debug info

quasibinomialDomain - quasibinomial domain

Returns:

Stratified frame

sampleFrame

public static Frame sampleFrame(Frame fr,
                                long rows,
                                java.lang.String weightColumn,
                                long seed)

Sample rows from a frame with weight column. Weights are used in the following manner: a row that has n-times higher weight should be n-times more likely to be picked. Can be unlucky for small sampling fractions - will continue calling itself until at least 1 row is returned.

Parameters:

fr - Input frame

rows - Approximate number of rows to sample (across all chunks)

weightColumn - Weight column name

seed - Seed for RNG

Returns:

Sampled frame

sampleFrameSmall

public static Frame sampleFrameSmall(Frame fr,
                                     int rows,
                                     long seed)

Sample small number of rows from a frame. Doesn't support weights. Meaning of "small" is relative, it shouldn't be more that 10k of rows.

Parameters:

fr - Input frame

rows - Exact number of rows to sample

seed - Seed for RNG

Returns:

Sampled frame, guaranteed to have exactly specified #rows (as long as the frame is large enough)

sampleFrameSmall

public static Frame sampleFrameSmall(Frame fr,
                                     int rows,
                                     java.util.Random rand)

Sample small number of rows from a frame. Doesn't support weights. Meaning of "small" is relative, it shouldn't be more that 10k of rows.

Parameters:

fr - Input frame

rows - Exact number of rows to sample

rand - Random Generator

Returns:

Sampled frame, guaranteed to have exactly specified #rows (as long as the frame is large enough)