Vec (h2o-core 3.46.0 API)

java.lang.Object
- water.Iced<T>
- - water.Keyed<Vec>
  - - water.fvec.Vec

All Implemented Interfaces:: java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable<Vec>

Direct Known Subclasses:: AppendableVec, ByteVec, CategoricalWrappedVec, InteractionWrappedVec, OneChunkVec, SubsetVec, TransformWrappedVec

public class Vec
extends Keyed<Vec>

A distributed vector/array/column of uniform data.

A distributed vector has a count of elements, an element-to-chunk mapping, a Java-like type (mostly determines rounding on store and display), and functions to directly load elements without further indirections. The data is compressed, or backed by disk or both.

A Vec is a collection of Chunks, each of which holds between 1,000 and 1,000,000 elements. Operations on a Chunk are intended to be single-threaded; operations on a Vec are intended to be parallel and distributed on Chunk granularities, with each Chunk being manipulated by a separate CPU. The standard Map/Reduce (MRTask) paradigm handles parallel and distributed Chunk access well.

Individual elements can be directly accessed like a (very large and distributed) array - however this is not the fastest way to access the data. Direct access from Chunks is faster, avoiding several layers of indirection. In particular accessing a random row from the Vec will force the containing Chunk data to be cached locally (and network traffic to bring it local); accessing all rows from a single machine will force all the Big Data to be pulled local typically resulting in swapping and very poor performance. The main API is provided for ease of small-data manipulations and is fairly slow for writing; when touching ALL the data you are much better off using e.g. MRTask.

The main API is at(long), set(long, long), and isNA(long):

Returns	Call	Missing?	Notes
`double`	`at(long)`	`NaN`
`long`	`at8(long)`	throws
`long`	`at16l(long)`	throws	Low half of 128-bit UUID
`long`	`at16h(long)`	throws	High half of 128-bit UUID
`BufferedString`	`atStr(water.parser.BufferedString, long)`	`null`	Updates BufferedString in-place and returns it for flow-coding
`boolean`	`isNA(long)`	`true`
	`set(long,double)`	`NaN`
	`set(long,float)`	`NaN`	Limited precision takes less memory
	`set(long,long)`	Cannot set
	`set(long,String)`	`null`	Convenience wrapper for String
`set(long,UUID)`	`null`
	`setNA(long)`

Example manipulating some individual elements:

    double r1 = vec.at(0x123456789L);  // Access element 0x1234567889 as a double
    double r2 = vec.at(-1);            // Throws AIOOBE
    long   r3 = vec.at8_abs(1);        // Element #1, as a long
    vec.set(2,r1+r3);                  // Set element #2, as a double

Vecs have a loosely enforced type: one of numeric, UUID or String. Numeric types are further broken down into integral (long) and real (double) types. The categorical type is an integral type, with a String mapping side-array. Most of the math algorithms will treat categoricals as small dense integers, and most categorical printouts will use the String mapping. Time is another special integral type: it is represented as milliseconds since the unix epoch, and is mostly treated as an integral type when doing math but it has special time-based printout formatting. All types support the notion of a missing element; for real types this is always NaN. It is an error to attempt to fetch a missing integral type, and isNA(long) must be called first. Integral types are losslessly compressed. Real types may lose 1 or 2 ULPS due to compression.

Reading elements as doubles, or checking for an element missing is always safe. Reading a missing integral type throws an exception, since there is no NaN equivalent in the integer domain. Writing to elements may throw if the backing data is read-only (file backed), and otherwise is fully supported.

Note this dangerous scenario: loading a missing value as a double, and setting it as a long:

   set(row,(long)at(row)); // Danger!

The cast from a Double.NaN to a long produces a zero! This code will silently replace a missing value with a zero.

Vecs have a lazily computed RollupStats object and Key. The RollupStats give fast access to the common metrics: min(), max(), mean(), sigma(), the count of missing elements (naCnt()) and non-zeros (nzCnt()), amongst other stats. They are cleared if the Vec is modified and lazily recomputed after the modified Vec is closed. Clearing the RollupStats cache is fairly expensive for individual set(long, long) calls but is easy to amortize over a large count of writes; i.e., batch writing is efficient. This is normally handled by the MRTask framework; the Vec.Writer framework allows single-threaded efficient batch writing for smaller Vecs.

Example usage of common stats:

    double mean = vec.mean();  // Vec's mean; first touch computes and caches rollups
    double min  = vec.min();   // Smallest element; already computed
    double max  = vec.max();   // Largest element; already computed
    double sigma= vec.sigma(); // Standard deviation; already computed

Example: Impute (replace) missing values with the mean. Note that the use of vec.mean() in the constructor uses (and computes) the general RollupStats before the MRTask starts. Setting a value in the Chunk clears the RollupStats (since setting any value but the mean will change the mean); they will be recomputed at the next use after the MRTask.

    new MRTask{} { final double _mean = vec.mean();
      public void map( Chunk chk ) {
        for( int row=0; row < chk._len; row++ )
          if( chk.isNA(row) ) chk.set(row,_mean);
      }
    }.doAll(vec);

Vecs have a Vec.VectorGroup. Vecs in the same VectorGroup have the same Chunk and row alignment - that is, Chunks with the same index are homed to the same Node and have the same count of rows-per-Chunk. Frames are only composed of Vecs of the same VectorGroup (or very small Vecs) guaranteeing that all elements of each row are homed to the same Node and set of Chunks - such that a simple for loop over a set of Chunks all operates locally. See the example in the Chunk class.

It is common and cheap to make new Vecs in the same VectorGroup as an existing Vec and initialized to e.g. zero. Such Vecs are often used as temps, and usually immediately set to interest values in a later MRTask pass.

Example creation of temp Vecs:

    Vec tmp0 = vec.makeZero();         // New Vec with same VectorGroup and layout as vec, filled with zero
    Vec tmp1 = vec.makeCon(mean);      // Filled with 'mean'
    assert tmp1.at(0x123456789)==mean; // All elements filled with 'mean'
    for( int i=0; i<100; i++ )         // A little math on the first 100 elements
      tmp0.set(i,tmp1.at(i)+17);       // ...set into the tmp0 vec

Vec Keys have a special layout (enforced by the various Vec constructors) so there is a direct Key mapping from a Vec to a numbered Chunk and vice-versa. This mapping is crucially used in all sorts of places, basically representing a global naming scheme across a Vec and the Chunks that make it up. The basic layout created by newKey():

              byte:    0      1   2 3 4 5  6 7 8 9  10+
  Vec   Key layout: Key.VEC  -1   vec#grp    -1     normal Key bytes; often e.g. a function of original file name
  Chunk Key layout: Key.CHK  -1   vec#grp  chunk#   normal Key bytes; often e.g. a function of original file name
  RollupStats Key : Key.CHK  -1   vec#grp    -2     normal Key bytes; often e.g. a function of original file name
  Group Key layout: Key.GRP  -1     -1       -1     normal Key bytes; often e.g. a function of original file name
  ESPC  Key layout: Key.GRP  -1     -1       -2     normal Key bytes; often e.g. a function of original file name

See Also:: Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`Vec.ESPC`
`static interface`	`Vec.Holder`
`class`	`Vec.Reader` A more efficient way to read randomly to a Vec - still single-threaded, but much faster than Vec.at(i).
`static class`	`Vec.VectorGroup` Class representing the group of vectors.
`class`	`Vec.Writer` A more efficient way to write randomly to a Vec - still single-threaded, still slow, but much faster than Vec.set().

Field Summary

Fields
Modifier and Type	Field and Description
`int[]`	`_cids`
`int`	`_rowLayout` Element-start per chunk, i.e.
`static boolean`	`DO_HISTOGRAMS`
`static int`	`KEY_PREFIX_LEN` Internally used to help build Vec and Chunk Keys; public to help PersistNFS build file mappings.
`static double[]`	`PERCENTILES` Default percentiles for approximate (single-pass) quantile computation (histogram-based).
`static byte`	`T_BAD`
`static byte`	`T_CAT`
`static byte`	`T_NUM`
`static byte`	`T_STR`
`static byte`	`T_TIME`
`static byte`	`T_UUID`
`static java.lang.String[]`	`TYPE_STR`

Fields inherited from class water.Keyed
_key

Constructor Summary

Constructors
Constructor and Description
`Vec(Key<Vec> key, int rowLayout)` Build a numeric-type Vec; the caller understands Chunk layout (via the `espc` array).
`Vec(Key<Vec> key, int rowLayout, java.lang.String[] domain, byte type)` Main default constructor; the caller understands Chunk layout (via the `espc` array), plus categorical/factor the `domain` (or null for non-categoricals), and the Vec type.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`Vec`	`adaptTo(java.lang.String[] domain)` Make a Vec adapting this cal vector to the 'to' categorical Vec.
`Vec`	`align(Vec vec)` Always makes a copy of the given vector which shares the same group as this Vec.
`void`	`asDouble()`
`double`	`at(long i)` Fetch element the slow way, as a double, or Double.NaN is missing.
`long`	`at16h(long i)` Fetch element the slow way, as the high half of a UUID.
`long`	`at16l(long i)` Fetch element the slow way, as the low half of a UUID.
`long`	`at8(long i)` Fetch element the slow way, as a long.
`BufferedString`	`atStr(BufferedString bStr, long i)` Fetch element the slow way, as a `BufferedString` or null if missing.
`double`	`base()` The `base` for a simple and cheap histogram of the Vec, useful for getting a broad overview of the data.
`long[]`	`bins()` A simple and cheap histogram of the Vec, useful for getting a broad overview of the data.
`static void`	`bulk_remove(Key<Vec>[] keys, int ncs)`
`long`	`byteSize()` Size of compressed vector data.
`int`	`cardinality()` Returns cardinality for categorical domain or -1 for other types.
`protected long`	`checksum_impl()` A high-quality 64-bit checksum of the Vec's content, useful for establishing dataset identity.
`Chunk`	`chunkForChunkIdx(int cidx)` The Chunk for a chunk#.
`Chunk`	`chunkForRow(long i)` The Chunk for a row#.
`Key`	`chunkKey(int cidx)` Get a Chunk Key from a chunk-index.
`static Key`	`chunkKey(Key veckey, int cidx)` Get a Chunk Key from a chunk-index and a Vec Key, without needing the actual Vec object.
`int`	`chunkLen(int cidx)` Number of rows in chunk.
`Vec`	`doCopy()`
`java.lang.String[]`	`domain()` Returns the categorical toString mapping array, or null if not an categorical column.
`int`	`elem2ChunkIdx(long i)` Convert a row# to a chunk#.
`boolean`	`equals(java.lang.Object o)` True if two Vecs are equal.
`long[]`	`espc()`
`java.lang.String`	`factor(long i)` Returns the `i`th factor for this categorical column.
`java.lang.String`	`get_type_str()`
`byte`	`get_type()` Get the column type.
`static Key`	`getVecKey(Key chk_key)` Get a Vec Key from Chunk Key, without loading the Chunk.
`Vec.VectorGroup`	`group()` Get the group this vector belongs to.
`int`	`hashCode()` Vec's hashcode, which is just the Vec Key hashcode.
`boolean`	`isBad()` True if the column contains only NAs
`boolean`	`isBinary()`
`boolean`	`isBinary(boolean strict)` Strict version of binary check
`boolean`	`isBinaryOnes()` Check binary vector case -1/1
`boolean`	`isCategorical()` True if this is an categorical column.
`boolean`	`isCompatibleWith(Vec v)` Check that row-layouts are compatible.
`boolean`	`isConst()` True if the column contains only a constant value and it is not full of NAs
`boolean`	`isConst(boolean includeNAs)` True if the column contains only a constant value and it is not full of NAs
`boolean`	`isDomainTruncated(int expectedCardinality)`
`boolean`	`isHomedLocally(int cidx)`
`boolean`	`isInt()` isInt is a property of numeric Vecs and not a type; this property can be changed by assigning non-integer values into the Vec (or restored by overwriting non-integer values with integers).
`boolean`	`isNA(long row)` Fetch the missing-status the slow way.
`boolean`	`isNumeric()` True if this is a numeric column, excluding categorical and time types.
`boolean`	`isString()` True if this is a String column.
`boolean`	`isTime()` True if this is a time column.
`boolean`	`isUUID()` True if this is a UUID column.
`boolean`	`isVolatile()`
`long[]`	`lazy_bins()` Optimistically return the histogram bins, or null if not computed
`long`	`length()` Number of elements in the vector; returned as a `long` instead of an `int` because Vecs support more than 2^32 elements.
`Vec`	`makeCon(byte type)`
`Vec`	`makeCon(double d)` Make a new vector with the same size and data layout as the current one, and initialized to the given constant value.
`Vec`	`makeCon(double d, byte type)`
`static Vec`	`makeCon(double x, long len)` Make a new constant vector with the given row count, and redistribute the data evenly around the cluster.
`static Vec`	`makeCon(double x, long len, boolean redistribute)` Make a new constant vector with the given row count.
`static Vec`	`makeCon(double x, long len, boolean redistribute, byte typeCode)`
`static Vec`	`makeCon(double x, long len, byte type)` Make a new constant vector of the specified type.
`static Vec`	`makeCon(double x, long len, int log_rows_per_chunk)` Make a new constant vector with the given row count, and redistribute the data evenly around the cluster.
`static Vec`	`makeCon(double x, long len, int log_rows_per_chunk, boolean redistribute)` Make a new constant vector with the given row count.
`static Vec`	`makeCon(double x, long len, int log_rows_per_chunk, boolean redistribute, byte type)`
`static Vec`	`makeCon(Key<Vec> k, double... rows)` A Vec from an array of doubles
`static Vec`	`makeCon(long totSize, long len)` Make a new constant vector with minimal number of chunks.
`static Vec`	`makeCon(long l, java.lang.String[] domain, Vec.VectorGroup group, int rowLayout)`
`Vec`	`makeCon(java.lang.String s)`
`static Vec`	`makeConN(long len, int nchunks)` Make a new constant vector with fixed number of chunks.
`static Vec[]`	`makeCons(double x, long len, int n)`
`Vec[]`	`makeCons(int n, long l, java.lang.String[][] domains, byte[] types)`
`Vec`	`makeCopy()` A new vector which is a copy of `this` one.
`Vec`	`makeCopy(java.lang.String[] domain)` A new vector which is a copy of `this` one.
`Vec`	`makeCopy(java.lang.String[] domain, byte type)`
`Vec[]`	`makeDoubles(int n, double[] values)`
`static Vec`	`makeOne(long len)`
`static Vec`	`makeOne(long len, byte typeCode)`
`Vec[]`	`makeOnes(int n)`
`Vec[]`	`makeOnes(int n, java.lang.String[][] domain, byte[] types)`
`Vec`	`makeRand(long seed)` Make a new vector initialized to random numbers with the given seed
`static Vec`	`makeRepSeq(long len, long repeat)` Make a new vector initialized to increasing integers mod `repeat`.
`static Vec`	`makeSeq(long len, boolean redistribute)` Make a new vector initialized to increasing integers, starting with 1.
`static Vec`	`makeSeq(long min, long len)` Make a new vector initialized to increasing integers, starting with `min`.
`static Vec`	`makeSeq(long min, long len, boolean redistribute)` Make a new vector initialized to increasing integers, starting with `min`.
`static Vec`	`makeTimeVec(double[] vals, Key<Vec> vecKey)`
`static Vec`	`makeVec(double[] vals, Key<Vec> vecKey)`
`static Vec`	`makeVec(double[] vals, java.lang.String[] domain, Key<Vec> vecKey)`
`static Vec`	`makeVec(float[] vals, Key<Vec> vecKey)`
`static Vec`	`makeVec(float[] vals, java.lang.String[] domain, Key<Vec> vecKey)`
`static Vec`	`makeVec(long[] vals, java.lang.String[] domain, Key<Vec> vecKey)`
`static Vec`	`makeVec(java.lang.String[] vals, Key<Vec> vecKey)`
`Vec[]`	`makeVolatileDoubles(int n)` Make a temporary work vec of double [] .
`Vec[]`	`makeVolatileInts(int[] cons)` Make a temporary work vec of int [] .
`Vec`	`makeZero()` Make a new vector with the same size and data layout as the current one, and initialized to zero.
`static Vec`	`makeZero(long len)` Make a new zero-filled vector with the given row count.
`static Vec`	`makeZero(long len, boolean redistribute)` Make a new zero-filled vec
`static Vec`	`makeZero(long len, byte typeCode)`
`Vec`	`makeZero(java.lang.String[] domain)` A new vector with the same size and data layout as the current one, and initialized to zero, with the given categorical domain.
`Vec[]`	`makeZeros(int n)`
`Vec[]`	`makeZeros(int n, java.lang.String[][] domain, byte[] types)`
`double`	`max()` Vec's maximum value
`double[]`	`maxs()` Vec's 5 largest values
`double`	`mean()` Vecs's mean
`double`	`min()` Vec's minimum value
`double[]`	`mins()` Vec's 5 smallest values
`int`	`mode()` Vecs's mode
`long`	`naCnt()` Count of missing elements
`int`	`nChunks()` Number of chunks, returned as an `int` - Chunk count is limited by the max size of a Java `long[]`.
`static int`	`nChunksFor(long len, int log_rows_per_chunk, boolean redistribute)`
`static Key<Vec>`	`newKey()` Make a new random Key that fits the requirements for a Vec key.
`long`	`ninfs()` Count of negative infinities
`int`	`nonEmptyChunks()` Number of non-empty chunks, also see `nChunks()`
`long`	`nzCnt()` Count of non-zero elements
`Vec.Writer`	`open()` Create a writer for bulk serial writes into this Vec.
`double[]`	`pctiles()` A simple and cheap percentiles of the Vec, useful for getting a broad overview of the data.
`long`	`pinfs()` Count of positive infinities
`Futures`	`postWrite(Futures fs)` Stop writing into this Vec.
`void`	`preWriting()` Begin writing into this Vec.
`protected Keyed`	`readAll_impl(AutoBuffer ab, Futures fs)`
`Futures`	`remove_impl(Futures fs, boolean cascade)` Remove associated Keys when this guy removes.
`protected Futures`	`remove_self_key_impl(Futures fs)` Remove my own key from DKV.
`Key`	`rollupStatsKey()`
`void`	`set(long i, double d)` Write element the slow way, as a double.
`void`	`set(long i, float f)` Write element the slow way, as a float.
`void`	`set(long i, long l)` Write element the slow way, as a long.
`void`	`set(long i, java.lang.String str)` Write element the slow way, as a String.
`void`	`set(long i, java.util.UUID uuid)`
`void`	`setBad()`
`static Key`	`setChunkIdx(Key k, int cidx)`
`void`	`setDomain(java.lang.String[] domain)` Set the categorical/factor names.
`void`	`setNA(long i)` Set the element as missing the slow way.
`double`	`sigma()` Vecs's standard deviation
`double`	`sparseRatio()`
`void`	`startRollupStats(Futures fs)`
`void`	`startRollupStats(Futures fs, boolean doHisto)` Check if we have local cached copy of basic Vec stats (including histogram if requested) and if not start task to compute and fetch them; useful when launching a bunch of them in parallel to avoid single threaded execution later (e.g.
`double`	`stride()` The `stride` for a a simple and cheap histogram of the Vec, useful for getting a broad overview of the data.
`java.lang.String`	`stringAt(long i)`
`Vec`	`toCategoricalVec()` Convenience method for converting to a categorical vector.
`Vec`	`toIntegerVec()`
`Vec`	`toNumericVec()` Convenience method for converting to a numeric vector.
`java.lang.String`	`toString()` Pretty print the Vec: `[#elems, min/mean/max]{chunks,...}`
`Vec`	`toStringVec()` Convenience method for converting to a string vector.
`TwoDimTable`	`toTwoDimTable()` Turn this Vec into a TwoDimTable of a 1-column Frame
`TwoDimTable`	`toTwoDimTable(int off, int len)` Turn this Vec into a TwoDimTable of a 1-column Frame
`protected AutoBuffer`	`writeAll_impl(AutoBuffer ab)` Write out K/V pairs

Methods inherited from class water.Keyed
checksum_impl, checksum, checksum, getKey, makeSchema, readAll, remove_impl, remove, remove, remove, remove, remove, remove, removeQuietly, writeAll

Methods inherited from class water.Iced
asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object
finalize, getClass, notify, notifyAll, wait, wait, wait

- Field Detail
  - _rowLayout
```
public int _rowLayout
```
    Element-start per chunk, i.e. the row layout. Defined in the VectorGroup. This field is dead/ignored in subclasses that are guaranteed to have fixed-sized chunks such as file-backed Vecs.
  - T_BAD
```
public static final byte T_BAD
```
    See Also:
    
    Constant Field Values
  - T_UUID
```
public static final byte T_UUID
```
    See Also:
    
    Constant Field Values
  - T_STR
```
public static final byte T_STR
```
    See Also:
    
    Constant Field Values
  - T_NUM
```
public static final byte T_NUM
```
    See Also:
    
    Constant Field Values
  - T_CAT
```
public static final byte T_CAT
```
    See Also:
    
    Constant Field Values
  - T_TIME
```
public static final byte T_TIME
```
    See Also:
    
    Constant Field Values
  - TYPE_STR
```
public static final java.lang.String[] TYPE_STR
```
  - DO_HISTOGRAMS
```
public static final boolean DO_HISTOGRAMS
```
    See Also:
    
    Constant Field Values
  - PERCENTILES
```
public static final double[] PERCENTILES
```
    Default percentiles for approximate (single-pass) quantile computation (histogram-based).
  - _cids
```
public transient int[] _cids
```
  - KEY_PREFIX_LEN
```
public static final int KEY_PREFIX_LEN
```
    Internally used to help build Vec and Chunk Keys; public to help PersistNFS build file mappings. Not intended as a public field.
    
    See Also:
    
    Constant Field Values
- Constructor Detail
  - Vec
```
public Vec(Key<Vec> key,
           int rowLayout)
```
    Build a numeric-type Vec; the caller understands Chunk layout (via the espc array).
  - Vec
```
public Vec(Key<Vec> key,
           int rowLayout,
           java.lang.String[] domain,
           byte type)
```
    Main default constructor; the caller understands Chunk layout (via the espc array), plus categorical/factor the domain (or null for non-categoricals), and the Vec type.
- Method Detail
  - domain
```
public java.lang.String[] domain()
```
    Returns the categorical toString mapping array, or null if not an categorical column. Not a defensive clone (to expensive to clone; coding error to change the contents).
    
    Returns:
    
    the categorical / factor / categorical mapping array, or null if not a categorical column
  - factor
```
public final java.lang.String factor(long i)
```
    Returns the ith factor for this categorical column.
    
    Returns:
    
    The ith factor
  - setDomain
```
public final void setDomain(java.lang.String[] domain)
```
    Set the categorical/factor names. No range-checking on the actual underlying numeric domain; user is responsible for maintaining a mapping which is coherent with the Vec contents.
  - cardinality
```
public final int cardinality()
```
    Returns cardinality for categorical domain or -1 for other types.
  - isDomainTruncated
```
public final boolean isDomainTruncated(int expectedCardinality)
```
    Returns:
    
    true iff the domain has been truncated (usually during encoding).
    
    See Also:
    
    CreateInteractions.makeDomain(java.util.Map, String[], String[])
  - isCategorical
```
public final boolean isCategorical()
```
    True if this is an categorical column. All categorical columns are also isInt(), but not vice-versa.
    
    Returns:
    
    true if this is an categorical column.
  - sparseRatio
```
public final double sparseRatio()
```
  - isUUID
```
public final boolean isUUID()
```
    True if this is a UUID column.
    
    Returns:
    
    true if this is a UUID column.
  - isString
```
public final boolean isString()
```
    True if this is a String column.
    
    Returns:
    
    true if this is a String column.
  - isNumeric
```
public final boolean isNumeric()
```
    True if this is a numeric column, excluding categorical and time types.
    
    Returns:
    
    true if this is a numeric column, excluding categorical and time types
  - isTime
```
public final boolean isTime()
```
    True if this is a time column. All time columns are also isInt(), but not vice-versa.
    
    Returns:
    
    true if this is a time column.
  - espc
```
public long[] espc()
```
  - length
```
public long length()
```
    Number of elements in the vector; returned as a long instead of an int because Vecs support more than 2^32 elements. Overridden by subclasses that compute length in an alternative way, such as file-backed Vecs.
    
    Returns:
    
    Number of elements in the vector
  - nChunks
```
public int nChunks()
```
    Number of chunks, returned as an int - Chunk count is limited by the max size of a Java long[]. Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.
    
    Returns:
    
    Number of chunks
  - nonEmptyChunks
```
public int nonEmptyChunks()
```
    Number of non-empty chunks, also see nChunks()
    
    Returns:
    
    Number of non-empty chunks
  - chunkLen
```
public int chunkLen(int cidx)
```
    Number of rows in chunk. Does not fetch chunk content.
  - isCompatibleWith
```
public boolean isCompatibleWith(Vec v)
```
    Check that row-layouts are compatible.
  - setBad
```
public void setBad()
```
  - get_type
```
public byte get_type()
```
    Get the column type.
  - get_type_str
```
public java.lang.String get_type_str()
```
  - isBinary
```
public boolean isBinary()
```
  - isBinary
```
public boolean isBinary(boolean strict)
```
    Strict version of binary check
    
    Parameters:
    
    strict - If true check also -1/1 case and if the vec is constant
    
    Returns:
    
    true if the vector is binary
  - isBinaryOnes
```
public boolean isBinaryOnes()
```
    Check binary vector case -1/1
    
    Returns:
    
    true if the vector is consist of -1/1 only
  - makeZero
```
public static Vec makeZero(long len,
                           boolean redistribute)
```
    Make a new zero-filled vec
  - makeZero
```
public static Vec makeZero(long len)
```
    Make a new zero-filled vector with the given row count.
    
    Returns:
    
    New zero-filled vector with the given row count.
  - makeOne
```
public static Vec makeOne(long len)
```
  - makeOne
```
public static Vec makeOne(long len,
                          byte typeCode)
```
  - makeZero
```
public static Vec makeZero(long len,
                           byte typeCode)
```
  - makeCon
```
public static Vec makeCon(double x,
                          long len)
```
    Make a new constant vector with the given row count, and redistribute the data evenly around the cluster.
    
    Parameters:
    
    x - The value with which to fill the Vec.
    
    len - Number of rows.
    
    Returns:
    
    New cosntant vector with the given len.
  - makeCon
```
public static Vec makeCon(double x,
                          long len,
                          byte type)
```
    Make a new constant vector of the specified type.
    
    Parameters:
    
    x - The value with which to fill the Vec.
    
    len - The number of rows in the produced Vec.
    
    type - Type of the Vec to construct.
  - makeCon
```
public static Vec makeCon(double x,
                          long len,
                          boolean redistribute)
```
    Make a new constant vector with the given row count.
    
    Returns:
    
    New constant vector with the given row count.
  - makeCon
```
public static Vec makeCon(double x,
                          long len,
                          boolean redistribute,
                          byte typeCode)
```
  - makeCon
```
public static Vec makeCon(double x,
                          long len,
                          int log_rows_per_chunk)
```
    Make a new constant vector with the given row count, and redistribute the data evenly around the cluster.
    
    Returns:
    
    New constant vector with the given row count.
  - makeCon
```
public static Vec makeCon(long totSize,
                          long len)
```
    Make a new constant vector with minimal number of chunks. Used for importing SQL tables.
    
    Returns:
    
    New constant vector with the given row count.
  - makeConN
```
public static Vec makeConN(long len,
                           int nchunks)
```
    Make a new constant vector with fixed number of chunks.
    
    Returns:
    
    New constant vector with the given chunks number.
  - nChunksFor
```
public static int nChunksFor(long len,
                             int log_rows_per_chunk,
                             boolean redistribute)
```
    Returns:
    
    the number of chunks that would be required when creating a Vec with given length and rows
  - makeCon
```
public static Vec makeCon(double x,
                          long len,
                          int log_rows_per_chunk,
                          boolean redistribute)
```
    Make a new constant vector with the given row count.
    
    Returns:
    
    New constant vector with the given row count.
  - makeCon
```
public static Vec makeCon(double x,
                          long len,
                          int log_rows_per_chunk,
                          boolean redistribute,
                          byte type)
```
  - makeDoubles
```
public Vec[] makeDoubles(int n,
                         double[] values)
```
  - makeZero
```
public Vec makeZero()
```
    Make a new vector with the same size and data layout as the current one, and initialized to zero.
    
    Returns:
    
    A new vector with the same size and data layout as the current one, and initialized to zero.
  - makeZero
```
public Vec makeZero(java.lang.String[] domain)
```
    A new vector with the same size and data layout as the current one, and initialized to zero, with the given categorical domain.
    
    Returns:
    
    A new vector with the same size and data layout as the current one, and initialized to zero, with the given categorical domain.
  - makeCopy
```
public Vec makeCopy()
```
    A new vector which is a copy of this one.
    
    Returns:
    
    a copy of the vector.
  - makeCopy
```
public Vec makeCopy(java.lang.String[] domain)
```
    A new vector which is a copy of this one.
    
    Returns:
    
    a copy of the vector.
  - makeCopy
```
public Vec makeCopy(java.lang.String[] domain,
                    byte type)
```
  - doCopy
```
public Vec doCopy()
```
  - makeCon
```
public static Vec makeCon(long l,
                          java.lang.String[] domain,
                          Vec.VectorGroup group,
                          int rowLayout)
```
  - makeTimeVec
```
public static Vec makeTimeVec(double[] vals,
                              Key<Vec> vecKey)
```
  - makeVec
```
public static Vec makeVec(double[] vals,
                          Key<Vec> vecKey)
```
  - makeVec
```
public static Vec makeVec(float[] vals,
                          Key<Vec> vecKey)
```
  - makeVec
```
public static Vec makeVec(java.lang.String[] vals,
                          Key<Vec> vecKey)
```
  - makeVec
```
public static Vec makeVec(float[] vals,
                          java.lang.String[] domain,
                          Key<Vec> vecKey)
```
  - makeVec
```
public static Vec makeVec(double[] vals,
                          java.lang.String[] domain,
                          Key<Vec> vecKey)
```
  - makeVec
```
public static Vec makeVec(long[] vals,
                          java.lang.String[] domain,
                          Key<Vec> vecKey)
```
  - makeCons
```
public static Vec[] makeCons(double x,
                             long len,
                             int n)
```
  - makeCon
```
public Vec makeCon(double d)
```
    Make a new vector with the same size and data layout as the current one, and initialized to the given constant value.
    
    Returns:
    
    A new vector with the same size and data layout as the current one, and initialized to the given constant value.
  - makeCon
```
public Vec makeCon(double d,
                   byte type)
```
  - makeCon
```
public Vec makeCon(byte type)
```
  - makeCon
```
public Vec makeCon(java.lang.String s)
```
  - makeZeros
```
public Vec[] makeZeros(int n)
```
  - makeOnes
```
public Vec[] makeOnes(int n)
```
  - makeVolatileDoubles
```
public Vec[] makeVolatileDoubles(int n)
```
    Make a temporary work vec of double [] . Volatile vecs can only be used locally (chunks do not serialize) and are assumed to change frequently(MRTask call preWiting() by default). Chunks stores as C8DVolatileChunk - expose data directly as double [].
    
    Parameters:
    
    n - number of columns
    
    Returns:
  - makeVolatileInts
```
public Vec[] makeVolatileInts(int[] cons)
```
    Make a temporary work vec of int [] . Volatile vecs can only be used locally (chunks do not serialize) and are assumed to change frequently(MRTask call preWiting() by default). Chunks stores as C4VolatileChunk - expose data directly as int [].
    
    Parameters:
    
    cons - integer array with constant used to fill each column.
    
    Returns:
  - makeZeros
```
public Vec[] makeZeros(int n,
                       java.lang.String[][] domain,
                       byte[] types)
```
  - makeOnes
```
public Vec[] makeOnes(int n,
                      java.lang.String[][] domain,
                      byte[] types)
```
  - makeCons
```
public Vec[] makeCons(int n,
                      long l,
                      java.lang.String[][] domains,
                      byte[] types)
```
  - makeCon
```
public static Vec makeCon(Key<Vec> k,
                          double... rows)
```
    A Vec from an array of doubles
    
    Parameters:
    
    rows - Data
    
    Returns:
    
    The Vec
  - makeSeq
```
public static Vec makeSeq(long len,
                          boolean redistribute)
```
    Make a new vector initialized to increasing integers, starting with 1.
    
    Returns:
    
    A new vector initialized to increasing integers, starting with 1.
  - makeSeq
```
public static Vec makeSeq(long min,
                          long len)
```
    Make a new vector initialized to increasing integers, starting with `min`.
    
    Returns:
    
    A new vector initialized to increasing integers, starting with `min`.
  - makeSeq
```
public static Vec makeSeq(long min,
                          long len,
                          boolean redistribute)
```
    Make a new vector initialized to increasing integers, starting with `min`.
    
    Returns:
    
    A new vector initialized to increasing integers, starting with `min`.
  - makeRepSeq
```
public static Vec makeRepSeq(long len,
                             long repeat)
```
    Make a new vector initialized to increasing integers mod repeat.
    
    Returns:
    
    A new vector initialized to increasing integers mod repeat.
  - makeRand
```
public Vec makeRand(long seed)
```
    Make a new vector initialized to random numbers with the given seed
  - min
```
public double min()
```
    Vec's minimum value
    
    Returns:
    
    Vec's minimum value
  - mins
```
public double[] mins()
```
    Vec's 5 smallest values
    
    Returns:
    
    Vec's 5 smallest values
  - max
```
public double max()
```
    Vec's maximum value
    
    Returns:
    
    Vec's maximum value
  - maxs
```
public double[] maxs()
```
    Vec's 5 largest values
    
    Returns:
    
    Vec's 5 largeest values
  - isConst
```
public final boolean isConst()
```
    True if the column contains only a constant value and it is not full of NAs
    
    Returns:
    
    True if the column is constant
  - isConst
```
public final boolean isConst(boolean includeNAs)
```
    True if the column contains only a constant value and it is not full of NAs
    
    Returns:
    
    True if the column is constant
  - isBad
```
public final boolean isBad()
```
    True if the column contains only NAs
    
    Returns:
    
    True if the column contains only NAs
  - mean
```
public double mean()
```
    Vecs's mean
    
    Returns:
    
    Vec's mean
  - sigma
```
public double sigma()
```
    Vecs's standard deviation
    
    Returns:
    
    Vec's standard deviation
  - mode
```
public int mode()
```
    Vecs's mode
    
    Returns:
    
    Vec's mode
  - naCnt
```
public long naCnt()
```
    Count of missing elements
    
    Returns:
    
    Count of missing elements
  - nzCnt
```
public long nzCnt()
```
    Count of non-zero elements
    
    Returns:
    
    Count of non-zero elements
  - pinfs
```
public long pinfs()
```
    Count of positive infinities
    
    Returns:
    
    Count of positive infinities
  - ninfs
```
public long ninfs()
```
    Count of negative infinities
    
    Returns:
    
    Count of negative infinities
  - isInt
```
public boolean isInt()
```
    isInt is a property of numeric Vecs and not a type; this property can be changed by assigning non-integer values into the Vec (or restored by overwriting non-integer values with integers). This is a strong type for isCategorical() and isTime() Vecs.
    
    Returns:
    
    true if the Vec is all integers
  - byteSize
```
public long byteSize()
```
    Size of compressed vector data.
  - bins
```
public long[] bins()
```
    A simple and cheap histogram of the Vec, useful for getting a broad overview of the data. Each bin is row-counts for the bin's range. The bin's range is computed from base() and stride(). The histogram is computed on first use and cached thereafter.
    
    Returns:
    
    A set of histogram bins, or null for String columns
  - lazy_bins
```
public long[] lazy_bins()
```
    Optimistically return the histogram bins, or null if not computed
    
    Returns:
    
    the histogram bins, or null if not computed
  - base
```
public double base()
```
    The base for a simple and cheap histogram of the Vec, useful for getting a broad overview of the data. This returns the base of bins()[0].
    
    Returns:
    
    the base of bins()[0]
  - stride
```
public double stride()
```
    The stride for a a simple and cheap histogram of the Vec, useful for getting a broad overview of the data. This returns the stride between any two bins.
    
    Returns:
    
    the stride between any two bins
  - pctiles
```
public double[] pctiles()
```
    A simple and cheap percentiles of the Vec, useful for getting a broad overview of the data. The specific percentiles are take from PERCENTILES.
    
    Returns:
    
    A set of percentiles
  - startRollupStats
```
public void startRollupStats(Futures fs)
```
  - startRollupStats
```
public void startRollupStats(Futures fs,
                             boolean doHisto)
```
    Check if we have local cached copy of basic Vec stats (including histogram if requested) and if not start task to compute and fetch them; useful when launching a bunch of them in parallel to avoid single threaded execution later (e.g. for-loop accessing min/max of every vec in a frame). Does *not* guarantee rollup stats will be cached locally when done since there may be racy changes to vec which will flush local caches.
    
    Parameters:
    
    fs - Futures allow to wait for this task to finish.
    
    doHisto - Also compute histogram, requires second pass over data amd is not computed by default.
  - checksum_impl
```
protected long checksum_impl()
```
    A high-quality 64-bit checksum of the Vec's content, useful for establishing dataset identity.
    
    Overrides:
    
    checksum_impl in class Keyed<Vec>
    
    Returns:
    
    Checksum of the Vec's content
  - isVolatile
```
public boolean isVolatile()
```
  - preWriting
```
public void preWriting()
```
    Begin writing into this Vec. Immediately clears all the rollup stats (min(), max(), mean(), etc) since such values are not meaningful while the Vec is being actively modified. Can be called repeatedly. Per-chunk row-counts will not be changing, just row contents.
  - postWrite
```
public Futures postWrite(Futures fs)
```
    Stop writing into this Vec. Rollup stats will again (lazily) be computed.
  - elem2ChunkIdx
```
public int elem2ChunkIdx(long i)
```
    Convert a row# to a chunk#. For constant-sized chunks this is a little shift-and-add math. For variable-sized chunks this is a binary search, with a sane API (JDK has an insane API). Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.
  - getVecKey
```
public static Key getVecKey(Key chk_key)
```
    Get a Vec Key from Chunk Key, without loading the Chunk.
    
    Returns:
    
    the Vec Key for the Chunk Key
  - chunkKey
```
public Key chunkKey(int cidx)
```
    Get a Chunk Key from a chunk-index. Basically the index-to-key map.
    
    Returns:
    
    Chunk Key from a chunk-index
  - chunkKey
```
public static Key chunkKey(Key veckey,
                           int cidx)
```
    Get a Chunk Key from a chunk-index and a Vec Key, without needing the actual Vec object. Basically the index-to-key map.
    
    Returns:
    
    Chunk Key from a chunk-index and Vec Key
  - rollupStatsKey
```
public Key rollupStatsKey()
```
  - newKey
```
public static Key<Vec> newKey()
```
    Make a new random Key that fits the requirements for a Vec key.
    
    Returns:
    
    A new random Vec Key
  - group
```
public final Vec.VectorGroup group()
```
    Get the group this vector belongs to. In case of a group with only one vector, the object actually does not exist in KV store. This is the ONLY place VectorGroups are fetched.
    
    Returns:
    
    VectorGroup this vector belongs to
  - chunkForChunkIdx
```
public Chunk chunkForChunkIdx(int cidx)
```
    The Chunk for a chunk#. Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!
    
    Returns:
    
    Chunk for a chunk#
  - chunkForRow
```
public final Chunk chunkForRow(long i)
```
    The Chunk for a row#. Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!
    
    Returns:
    
    Chunk for a row#
  - at8
```
public final long at8(long i)
```
    Fetch element the slow way, as a long. Floating point values are silently rounded to an integer. Throws if the value is missing.
    
    Returns:
    
    ith element as a long, or throw if missing
  - at
```
public final double at(long i)
```
    Fetch element the slow way, as a double, or Double.NaN is missing.
    
    Returns:
    
    ith element as a double, or Double.NaN if missing
  - isNA
```
public final boolean isNA(long row)
```
    Fetch the missing-status the slow way.
    
    Returns:
    
    the missing-status the slow way
  - at16l
```
public final long at16l(long i)
```
    Fetch element the slow way, as the low half of a UUID. Throws if the value is missing or not a UUID.
    
    Returns:
    
    ith element as a UUID low half, or throw if missing
  - at16h
```
public final long at16h(long i)
```
    Fetch element the slow way, as the high half of a UUID. Throws if the value is missing or not a UUID.
    
    Returns:
    
    ith element as a UUID high half, or throw if missing
  - atStr
```
public final BufferedString atStr(BufferedString bStr,
                                  long i)
```
    Fetch element the slow way, as a BufferedString or null if missing. Throws if the value is not a String. BufferedStrings are String-like objects than can be reused in-place, which is much more efficient than constructing Strings.
    
    Returns:
    
    ith element as BufferedString or null if missing, or throw if not a String
  - stringAt
```
public java.lang.String stringAt(long i)
```
  - set
```
public final void set(long i,
                      long l)
```
    Write element the slow way, as a long. There is no way to write a missing value with this call. Under rare circumstances this can throw: if the long does not fit in a double (value is larger magnitude than 2^52), AND float values are stored in Vec. In this case, there is no common compatible data representation.
  - set
```
public final void set(long i,
                      double d)
```
    Write element the slow way, as a double. Double.NaN will be treated as a set of a missing element.
  - set
```
public final void set(long i,
                      float f)
```
    Write element the slow way, as a float. Float.NaN will be treated as a set of a missing element.
  - setNA
```
public final void setNA(long i)
```
    Set the element as missing the slow way.
  - set
```
public final void set(long i,
                      java.lang.String str)
```
    Write element the slow way, as a String. null will be treated as a set of a missing element.
  - set
```
public final void set(long i,
                      java.util.UUID uuid)
```
  - open
```
public final Vec.Writer open()
```
    Create a writer for bulk serial writes into this Vec.
    
    Returns:
    
    A Writer for bulk serial writes
  - toString
```
public java.lang.String toString()
```
    Pretty print the Vec: [#elems, min/mean/max]{chunks,...}
    
    Overrides:
    
    toString in class java.lang.Object
    
    Returns:
    
    Brief string representation of a Vec
  - toTwoDimTable
```
public TwoDimTable toTwoDimTable(int off,
                                 int len)
```
    Turn this Vec into a TwoDimTable of a 1-column Frame
    
    Parameters:
    
    off - index of first element to include
    
    len - number of elements to include
    
    Returns:
    
    TwoDimTable that can be toString()'ed, etc.
  - toTwoDimTable
```
public TwoDimTable toTwoDimTable()
```
    Turn this Vec into a TwoDimTable of a 1-column Frame
    
    Returns:
    
    TwoDimTable that can be toString()'ed, etc.
  - toCategoricalVec
```
public Vec toCategoricalVec()
```
    Convenience method for converting to a categorical vector.
    
    Returns:
    
    A categorical vector based on the contents of the original vector.
  - toIntegerVec
```
public Vec toIntegerVec()
```
  - asDouble
```
public void asDouble()
```
  - toStringVec
```
public Vec toStringVec()
```
    Convenience method for converting to a string vector.
    
    Returns:
    
    A string vector based on the contents of the original vector.
  - toNumericVec
```
public Vec toNumericVec()
```
    Convenience method for converting to a numeric vector.
    
    Returns:
    
    A numeric vector based on the contents of the original vector.
  - equals
```
public boolean equals(java.lang.Object o)
```
    True if two Vecs are equal. Checks for equal-Keys only (so it is fast) and not equal-contents.
    
    Overrides:
    
    equals in class java.lang.Object
    
    Returns:
    
    True if two Vecs are equal
  - hashCode
```
public int hashCode()
```
    Vec's hashcode, which is just the Vec Key hashcode.
    
    Overrides:
    
    hashCode in class java.lang.Object
    
    Returns:
    
    Vec's hashcode
  - remove_impl
```
public Futures remove_impl(Futures fs,
                           boolean cascade)
```
    Remove associated Keys when this guy removes. For Vecs, remove all associated Chunks.
    
    Overrides:
    
    remove_impl in class Keyed<Vec>
    
    Returns:
    
    Passed in Futures for flow-coding
  - remove_self_key_impl
```
protected Futures remove_self_key_impl(Futures fs)
```
    Description copied from class: Keyed
    
    Remove my own key from DKV.
    
    Overrides:
    
    remove_self_key_impl in class Keyed<Vec>
  - bulk_remove
```
public static void bulk_remove(Key<Vec>[] keys,
                               int ncs)
```
  - writeAll_impl
```
protected AutoBuffer writeAll_impl(AutoBuffer ab)
```
    Write out K/V pairs
    
    Overrides:
    
    writeAll_impl in class Keyed<Vec>
  - readAll_impl
```
protected Keyed readAll_impl(AutoBuffer ab,
                             Futures fs)
```
    Overrides:
    
    readAll_impl in class Keyed<Vec>
  - align
```
public Vec align(Vec vec)
```
    Always makes a copy of the given vector which shares the same group as this Vec. This can be expensive operation since it can force copy of data among nodes.
    
    Parameters:
    
    vec - vector which is intended to be copied
    
    Returns:
    
    a copy of vec which shared the same Vec.VectorGroup with this vector
  - adaptTo
```
public Vec adaptTo(java.lang.String[] domain)
```
    Make a Vec adapting this cal vector to the 'to' categorical Vec. The adapted CategoricalWrappedVec has 'this' as it's masterVec, but returns results in the 'to' newDomain (or just past it, if 'this' has elements not appearing in the 'to' newDomain).
  - setChunkIdx
```
public static Key setChunkIdx(Key k,
                              int cidx)
```
  - isHomedLocally
```
public boolean isHomedLocally(int cidx)
```

Class Vec

Nested Class Summary

Field Summary

Fields inherited from class water.Keyed

Constructor Summary

Method Summary

Methods inherited from class water.Keyed

Methods inherited from class water.Iced

Methods inherited from class java.lang.Object

Field Detail

_rowLayout

T_BAD

T_UUID

T_STR

T_NUM

T_CAT

T_TIME

TYPE_STR

DO_HISTOGRAMS

PERCENTILES

_cids

KEY_PREFIX_LEN

Constructor Detail

Vec

Vec

Method Detail

domain

factor

setDomain

cardinality

isDomainTruncated

isCategorical

sparseRatio

isUUID

isString

isNumeric

isTime

espc

length

nChunks

nonEmptyChunks

chunkLen

isCompatibleWith

setBad

get_type

get_type_str

isBinary

isBinary

isBinaryOnes

makeZero

makeZero

makeOne

makeOne

makeZero

makeCon

makeCon

makeCon

makeCon

makeCon

makeCon

makeConN

nChunksFor

makeCon

makeCon

makeDoubles

makeZero

makeZero

makeCopy

makeCopy

makeCopy

doCopy

makeCon

makeTimeVec

makeVec

makeVec

makeVec

makeVec

makeVec

makeVec

makeCons