DataInfo (h2o-algos 3.46.0 API)

java.lang.Object
- water.Iced<T>
- - water.Keyed<DataInfo>
  - - hex.DataInfo

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, water.Freezable<DataInfo>
```
public class DataInfo
extends water.Keyed<DataInfo>
```
Created by tomasnykodym on 1/29/15. Provides higher level interface for accessing data row-wise. Performs on the fly auto-expansion of categorical variables (to 1 hot encoding) and standardization ( or normalize/demean/descale/none) of predictors and response. Supports sparse data, sparse columns can be transformed to sparse rows on the fly with some (significant) memory overhead, as the data of the whole chunk(s) will be copied.

See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static interface`	`DataInfo.Imputer`
`static class`	`DataInfo.MeanImputer`
`class`	`DataInfo.Row`
`class`	`DataInfo.Rows`
`static class`	`DataInfo.TransformType`

Field Summary

Fields
Modifier and Type	Field and Description
`int[]`	`_activeCols`
`water.fvec.Frame`	`_adaptedFrame`
`int[][]`	`_catLvls`
`boolean[]`	`_catMissing`
`int[]`	`_catOffsets`
`int`	`_cats`
`java.lang.String[]`	`_coefNames`
`int[]`	`_coefOriginalIndices`
`boolean`	`_fold`
`boolean`	`_imputeMissing`
`hex.Model.InteractionPair[]`	`_interactions`
`hex.Model.InteractionSpec`	`_interactionSpec`
`int[]`	`_interactionVecs`
`boolean`	`_intercept`
`int[][]`	`_intLvls`
`double[]`	`_normMul`
`double[]`	`_normRespMul`
`double[]`	`_normRespSub`
`double[]`	`_normSigmaStandardizationOff`
`double[]`	`_normSub`
`double[]`	`_normSubStandardizationOff`
`double[]`	`_numMeans`
`double[]`	`_numNAFill`
`int[]`	`_numOffsets`
`int`	`_nums`
`boolean`	`_offset`
`int[]`	`_permutation`
`DataInfo.TransformType`	`_predictor_transform`
`DataInfo.TransformType`	`_response_transform`
`int`	`_responses`
`boolean`	`_skipMissing`
`boolean`	`_treatment`
`boolean`	`_useAllFactorLevels`
`boolean`	`_valid`
`boolean`	`_weights`

Fields inherited from class water.Keyed
_key

Constructor Summary

Constructors
Constructor and Description
`DataInfo(water.fvec.Frame train, water.fvec.Frame valid, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket)`
`DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold)`
`DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold, boolean intercept)`
`DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold, boolean treatment, hex.Model.InteractionSpec interactions)`
`DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, boolean missingBucket, boolean weight, boolean offset, boolean fold, hex.Model.InteractionSpec interactions)`
`DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, DataInfo.Imputer imputer, boolean missingBucket, boolean weight, boolean offset, boolean fold, boolean treatment, hex.Model.InteractionSpec interactions)` The train/valid Frame instances are sorted by categorical (themselves sorted by cardinality greatest to least) with all numerical columns following.
`DataInfo(water.fvec.Frame train, water.fvec.Frame valid, int nResponses, boolean useAllFactorLevels, DataInfo.TransformType predictor_transform, DataInfo.TransformType response_transform, boolean skipMissing, boolean imputeMissing, DataInfo.Imputer imputer, boolean missingBucket, boolean weight, boolean offset, boolean fold, hex.Model.InteractionSpec interactions)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`int[]`	`activeCols()`
`void`	`addOutput(java.lang.String name, water.fvec.Vec v)`
`void`	`addResponse(java.lang.String[] names, water.fvec.Vec[] vecs)`
`int[]`	`catNAFill()`
`int`	`catNAFill(int cid)`
`protected long`	`checksum_impl()`
`java.lang.String[]`	`coefNames()`
`int[]`	`coefOriginalColumnIndices()`
`int[]`	`coefOriginalColumnIndices(water.fvec.Frame adaptedFrame)`
`java.lang.String[]`	`coefOriginalNames()`
`java.lang.String[]`	`coefOriginalNames(water.fvec.Frame adaptedFrame)`
`double[]`	`denormalizeBeta(double[] beta)`
`DataInfo`	`disableIntercept()`
`void`	`dropInteractions()`
`void`	`dropWeights()`
`DataInfo.Row`	`extractDenseRow(water.fvec.Chunk[] chunks, int rid, DataInfo.Row row)`
`DataInfo.Row[]`	`extractSparseRows(water.fvec.Chunk[] chunks)` Extract (sparse) rows from given chunks.
`DataInfo`	`filterExpandedColumns(int[] cols)` Filter the _adaptedFrame so that it contains only the Vecs referenced by the cols parameter.
`int`	`foldChunkId()`
`protected int[]`	`fullCatOffsets()`
`int`	`fullN()` Get the fully expanded number of predictor columns.
`int`	`getCategoricalId(int cid, double val)`
`int`	`getCategoricalId(int cid, int val)` Get the offset into the expanded categorical
`int`	`getCategoricalIdFromInteraction(int cid, int val)`
`int`	`getInteractionOffset(water.fvec.Chunk[] chunks, int cid, int rid)`
`water.fvec.Vec`	`getOffsetVec()`
`water.fvec.Vec`	`getOutputVec(int i)`
`water.fvec.Vec`	`getWeightsVec()`
`static int`	`imputeCat(water.fvec.Vec v)`
`static int`	`imputeCat(water.fvec.Vec v, boolean useAllFactorLevels)`
`boolean`	`isInteractionVec(int colid)`
`int`	`largestCat()`
`int[]`	`mapNames(java.lang.String[] names)`
`DataInfo.Row`	`newDenseRow()`
`DataInfo.Row`	`newDenseRow(double[] numVals, long start)`
`int`	`nextNumericIdx(int currentColIdx)` Get the next expanded number-column index.
`double[]`	`normalizeBeta(double[] beta, boolean standardize)`
`double`	`normMul(int i)`
`double`	`normSub(int i)`
`int`	`numCats()`
`double[]`	`numNAFill()`
`double`	`numNAFill(int nid)`
`int`	`numNums()`
`int`	`numStart()`
`int`	`offsetChunkId()`
`int`	`outputChunkId()`
`int`	`outputChunkId(int n)`
`int`	`responseChunkId(int n)`
`DataInfo.Rows`	`rows(water.fvec.Chunk[] chks)`
`DataInfo.Rows`	`rows(water.fvec.Chunk[] chks, boolean sparse)`
`DataInfo`	`scoringInfo()` Creates a DataInfo for scoring on a test Frame from a DataInfo instance created during model training This is a lightweight version of the method only usable for models that don't use advanced features of DataInfo (eg.
`DataInfo`	`scoringInfo(java.lang.String[] names, water.fvec.Frame adaptFrame)`
`DataInfo`	`scoringInfo(java.lang.String[] names, water.fvec.Frame adaptFrame, int nResponses, boolean fixIVW)` Creates a scoringInfo from a DataInfo instance created during model training
`void`	`setCatNAFill(int[] catNAFill)`
`void`	`setPredictorTransform(DataInfo.TransformType t)`
`void`	`setResponse(java.lang.String name, water.fvec.Vec v)`
`void`	`setResponse(java.lang.String name, water.fvec.Vec v, int n)`
`void`	`setResponseTransform(DataInfo.TransformType t)`
`water.fvec.Vec`	`setWeights(java.lang.String name, water.fvec.Vec vec)`
`int`	`treatmentChunkId()`
`void`	`unScaleNumericals(double[] in, double[] out)` Undo the standardization/normalization of numerical columns
`void`	`updateWeightedSigmaAndMean(double[] sigmas, double[] mean)`
`void`	`updateWeightedSigmaAndMeanForResponse(double[] sigmas, double[] mean)`
`DataInfo`	`validDinfo(water.fvec.Frame valid)`
`int`	`weightChunkId()`

Methods inherited from class water.Keyed
checksum_impl, checksum, checksum, getKey, makeSchema, readAll_impl, readAll, remove_impl, remove_impl, remove_self_key_impl, remove, remove, remove, remove, remove, remove, removeQuietly, writeAll_impl, writeAll

Methods inherited from class water.Iced
asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

_activeCols
```
public int[] _activeCols
```

_adaptedFrame
```
public water.fvec.Frame _adaptedFrame
```

_responses
```
public int _responses
```

_predictor_transform

public DataInfo.TransformType _predictor_transform

_response_transform

public DataInfo.TransformType _response_transform

_useAllFactorLevels
```
public boolean _useAllFactorLevels
```

_nums
```
public int _nums
```

_cats
```
public int _cats
```

_catOffsets
```
public int[] _catOffsets
```

_catMissing
```
public boolean[] _catMissing
```

_numNAFill
```
public double[] _numNAFill
```

_permutation
```
public int[] _permutation
```

_normMul
```
public double[] _normMul
```

_normSub
```
public double[] _normSub
```

_normSigmaStandardizationOff

public double[] _normSigmaStandardizationOff

_normSubStandardizationOff

public double[] _normSubStandardizationOff

_normRespMul
```
public double[] _normRespMul
```

_normRespSub
```
public double[] _normRespSub
```

_numMeans
```
public double[] _numMeans
```

_intercept
```
public boolean _intercept
```

_offset
```
public boolean _offset
```

_weights
```
public boolean _weights
```

_fold
```
public boolean _fold
```

_treatment
```
public boolean _treatment
```

_interactions

public hex.Model.InteractionPair[] _interactions

_interactionSpec

public hex.Model.InteractionSpec _interactionSpec

_interactionVecs
```
public int[] _interactionVecs
```

_numOffsets
```
public int[] _numOffsets
```

_skipMissing
```
public final boolean _skipMissing
```

_imputeMissing
```
public final boolean _imputeMissing
```

_valid
```
public boolean _valid
```

_catLvls
```
public final int[][] _catLvls
```

_intLvls
```
public final int[][] _intLvls
```

_coefNames
```
public java.lang.String[] _coefNames
```

_coefOriginalIndices
```
public int[] _coefOriginalIndices
```

Constructor Detail

DataInfo

public DataInfo(water.fvec.Frame train,
                water.fvec.Frame valid,
                boolean useAllFactorLevels,
                DataInfo.TransformType predictor_transform,
                boolean skipMissing,
                boolean imputeMissing,
                boolean missingBucket)

DataInfo

public DataInfo(water.fvec.Frame train,
                water.fvec.Frame valid,
                int nResponses,
                boolean useAllFactorLevels,
                DataInfo.TransformType predictor_transform,
                DataInfo.TransformType response_transform,
                boolean skipMissing,
                boolean imputeMissing,
                boolean missingBucket,
                boolean weight,
                boolean offset,
                boolean fold)

DataInfo

public DataInfo(water.fvec.Frame train,
                water.fvec.Frame valid,
                int nResponses,
                boolean useAllFactorLevels,
                DataInfo.TransformType predictor_transform,
                DataInfo.TransformType response_transform,
                boolean skipMissing,
                boolean imputeMissing,
                boolean missingBucket,
                boolean weight,
                boolean offset,
                boolean fold,
                hex.Model.InteractionSpec interactions)

DataInfo

public DataInfo(water.fvec.Frame train,
                water.fvec.Frame valid,
                int nResponses,
                boolean useAllFactorLevels,
                DataInfo.TransformType predictor_transform,
                DataInfo.TransformType response_transform,
                boolean skipMissing,
                boolean imputeMissing,
                boolean missingBucket,
                boolean weight,
                boolean offset,
                boolean fold,
                boolean treatment,
                hex.Model.InteractionSpec interactions)

DataInfo

public DataInfo(water.fvec.Frame train,
                water.fvec.Frame valid,
                int nResponses,
                boolean useAllFactorLevels,
                DataInfo.TransformType predictor_transform,
                DataInfo.TransformType response_transform,
                boolean skipMissing,
                boolean imputeMissing,
                DataInfo.Imputer imputer,
                boolean missingBucket,
                boolean weight,
                boolean offset,
                boolean fold,
                hex.Model.InteractionSpec interactions)

DataInfo

public DataInfo(water.fvec.Frame train,
                water.fvec.Frame valid,
                int nResponses,
                boolean useAllFactorLevels,
                DataInfo.TransformType predictor_transform,
                DataInfo.TransformType response_transform,
                boolean skipMissing,
                boolean imputeMissing,
                DataInfo.Imputer imputer,
                boolean missingBucket,
                boolean weight,
                boolean offset,
                boolean fold,
                boolean treatment,
                hex.Model.InteractionSpec interactions)

The train/valid Frame instances are sorted by categorical (themselves sorted by cardinality greatest to least) with all numerical columns following. The response column(s) are placed at the end. Interactions: 1. Num-Num (Note: N(0,1) * N(0,1) ~ N(0,1) ) 2. Num-Enum 3. Enum-Enum Interactions are produced on the fly and are dense (in all 3 cases). Consumers of DataInfo should not have to care how these interactions are generated. Any heuristic using the fullN value should continue functioning the same. Interactions are specified in two ways: A. As a list of pairs of column indices. B. As a list of pairs of column indices with limited enums.

DataInfo

public DataInfo(water.fvec.Frame train,
                water.fvec.Frame valid,
                int nResponses,
                boolean useAllFactorLevels,
                DataInfo.TransformType predictor_transform,
                DataInfo.TransformType response_transform,
                boolean skipMissing,
                boolean imputeMissing,
                boolean missingBucket,
                boolean weight,
                boolean offset,
                boolean fold,
                boolean intercept)

Method Detail

setWeights

public water.fvec.Vec setWeights(java.lang.String name,
                                 water.fvec.Vec vec)

dropWeights
```
public void dropWeights()
```

dropInteractions
```
public void dropInteractions()
```

activeCols
```
public int[] activeCols()
```

addResponse

public void addResponse(java.lang.String[] names,
                        water.fvec.Vec[] vecs)

catNAFill
```
public int[] catNAFill()
```

catNAFill
```
public int catNAFill(int cid)
```

numNAFill
```
public double[] numNAFill()
```

numNAFill
```
public double numNAFill(int nid)
```

setCatNAFill

public void setCatNAFill(int[] catNAFill)

normSub
```
public double normSub(int i)
```

normMul
```
public double normMul(int i)
```

responseChunkId
```
public int responseChunkId(int n)
```

treatmentChunkId
```
public int treatmentChunkId()
```

foldChunkId
```
public int foldChunkId()
```

offsetChunkId
```
public int offsetChunkId()
```

weightChunkId
```
public int weightChunkId()
```

outputChunkId
```
public int outputChunkId()
```

outputChunkId
```
public int outputChunkId(int n)
```

addOutput

public void addOutput(java.lang.String name,
                      water.fvec.Vec v)

getOutputVec

public water.fvec.Vec getOutputVec(int i)

setResponse

public void setResponse(java.lang.String name,
                        water.fvec.Vec v)

setResponse

public void setResponse(java.lang.String name,
                        water.fvec.Vec v,
                        int n)

checksum_impl
```
protected long checksum_impl()
```
Overrides:

checksum_impl in class water.Keyed<DataInfo>

disableIntercept
```
public DataInfo disableIntercept()
```

validDinfo

public DataInfo validDinfo(water.fvec.Frame valid)

denormalizeBeta

public double[] denormalizeBeta(double[] beta)

normalizeBeta

public double[] normalizeBeta(double[] beta,
                              boolean standardize)

fullCatOffsets
```
protected int[] fullCatOffsets()
```

imputeCat

public static int imputeCat(water.fvec.Vec v)

imputeCat

public static int imputeCat(water.fvec.Vec v,
                            boolean useAllFactorLevels)

filterExpandedColumns
```
public DataInfo filterExpandedColumns(int[] cols)
```
Filter the _adaptedFrame so that it contains only the Vecs referenced by the cols parameter. This is done by recording the ignored columns in array ignoredCols. The enum columns are not expanded and considered as one column. However, it is possible that a level inside the enum column can be ignored. In this case, the enum levels are adjusted accordingly.

Parameters:

cols - Array of the expanded column indices to keep.

Returns:

A DataInfo with _activeCols specifying the active columns

updateWeightedSigmaAndMean

public void updateWeightedSigmaAndMean(double[] sigmas,
                                       double[] mean)

updateWeightedSigmaAndMeanForResponse

public void updateWeightedSigmaAndMeanForResponse(double[] sigmas,
                                                  double[] mean)

setPredictorTransform

public void setPredictorTransform(DataInfo.TransformType t)

setResponseTransform

public void setResponseTransform(DataInfo.TransformType t)

isInteractionVec

public boolean isInteractionVec(int colid)

fullN
```
public final int fullN()
```
Get the fully expanded number of predictor columns. Note that this value does not include: response column(s) weight column offset column fold column treatment column

Returns:

expanded number of columns in the underlying frame

largestCat
```
public final int largestCat()
```

numStart
```
public final int numStart()
```

numCats
```
public final int numCats()
```

numNums
```
public final int numNums()
```

nextNumericIdx
```
public final int nextNumericIdx(int currentColIdx)
```
Get the next expanded number-column index.

coefNames

public final java.lang.String[] coefNames()

coefOriginalColumnIndices

public final int[] coefOriginalColumnIndices(water.fvec.Frame adaptedFrame)

coefOriginalColumnIndices

public final int[] coefOriginalColumnIndices()

coefOriginalNames

public final java.lang.String[] coefOriginalNames(water.fvec.Frame adaptedFrame)

coefOriginalNames

public final java.lang.String[] coefOriginalNames()

mapNames

public int[] mapNames(java.lang.String[] names)

unScaleNumericals
```
public final void unScaleNumericals(double[] in,
                                    double[] out)
```
Undo the standardization/normalization of numerical columns

Parameters:

in - input values

out - output values (can be the same as input)

getCategoricalId

public final int getCategoricalId(int cid,
                                  double val)

getCategoricalId
```
public final int getCategoricalId(int cid,
                                  int val)
```
Get the offset into the expanded categorical

Parameters:

cid - the column id

val - the integer representation of the categorical level

Returns:

offset into the fullN set of columns

getCategoricalIdFromInteraction

public final int getCategoricalIdFromInteraction(int cid,
                                                 int val)

extractDenseRow

public final DataInfo.Row extractDenseRow(water.fvec.Chunk[] chunks,
                                          int rid,
                                          DataInfo.Row row)

getInteractionOffset

public int getInteractionOffset(water.fvec.Chunk[] chunks,
                                int cid,
                                int rid)

getWeightsVec
```
public water.fvec.Vec getWeightsVec()
```

getOffsetVec
```
public water.fvec.Vec getOffsetVec()
```

newDenseRow
```
public DataInfo.Row newDenseRow()
```

newDenseRow

public DataInfo.Row newDenseRow(double[] numVals,
                                long start)

rows

public DataInfo.Rows rows(water.fvec.Chunk[] chks)

rows

public DataInfo.Rows rows(water.fvec.Chunk[] chks,
                          boolean sparse)

extractSparseRows
```
public final DataInfo.Row[] extractSparseRows(water.fvec.Chunk[] chunks)
```
Extract (sparse) rows from given chunks. Note: 0 remains 0 - _normSub of DataInfo isn't used (mean shift during standarization is not reverted) - UNLESS offset is specified (for GLM only) Essentially turns the dataset 90 degrees.

Parameters:

chunks - - chunk of dataset

Returns:

array of sparse rows

scoringInfo

public DataInfo scoringInfo(java.lang.String[] names,
                            water.fvec.Frame adaptFrame)

scoringInfo
```
public DataInfo scoringInfo(java.lang.String[] names,
                            water.fvec.Frame adaptFrame,
                            int nResponses,
                            boolean fixIVW)
```
Creates a scoringInfo from a DataInfo instance created during model training

Parameters:

names - column names

adaptFrame - adapted frame

nResponses - number of responses (-1 indicates autodetect: 0/1 based on presence of a single response)

fixIVW - whether to force global useFactorLevels flag to InteractionWrappedVecs (GLM behavior)

Returns:

scoringInfo
```
public DataInfo scoringInfo()
```
Creates a DataInfo for scoring on a test Frame from a DataInfo instance created during model training This is a lightweight version of the method only usable for models that don't use advanced features of DataInfo (eg. interaction terms)

Returns:

DataInfo for scoring

Class DataInfo

Nested Class Summary

Field Summary

Fields inherited from class water.Keyed

Constructor Summary

Method Summary

Methods inherited from class water.Keyed

Methods inherited from class water.Iced

Methods inherited from class java.lang.Object

Field Detail

_activeCols

_adaptedFrame

_responses

_predictor_transform

_response_transform

_useAllFactorLevels

_nums

_cats

_catOffsets

_catMissing

_numNAFill

_permutation

_normMul

_normSub

_normSigmaStandardizationOff

_normSubStandardizationOff

_normRespMul

_normRespSub

_numMeans

_intercept

_offset

_weights

_fold

_treatment

_interactions

_interactionSpec

_interactionVecs

_numOffsets

_skipMissing

_imputeMissing

_valid

_catLvls

_intLvls

_coefNames

_coefOriginalIndices

Constructor Detail

DataInfo

DataInfo

DataInfo

DataInfo

DataInfo

DataInfo

DataInfo

Method Detail

setWeights

dropWeights

dropInteractions

activeCols

addResponse

catNAFill

catNAFill

numNAFill

numNAFill

setCatNAFill

normSub

normMul

responseChunkId

treatmentChunkId

foldChunkId

offsetChunkId

weightChunkId

outputChunkId

outputChunkId

addOutput

getOutputVec

setResponse

setResponse

checksum_impl

disableIntercept

validDinfo