water.fvec

Class AppendableVec

java.lang.Object

water.Iced<T>

water.Keyed<Vec>

water.fvec.Vec

water.fvec.AppendableVec

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable<Vec>

public class AppendableVec
extends Vec

A NEW single distributed vector column. The NEW vector has no data, and takes no space. It supports distributed parallel writes to it, via calls to append2. Such writes happen in parallel and all writes are ordered. Writes *will* be local to the node doing them, specifically to allow control over locality. By default, writes will go local-homed chunks with no compression; there is a final 'close' to the NEW vector which may do compression; the final 'close' will return some other Vec type. NEW Vectors do NOT support reads!

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class water.fvec.Vec

Vec.ESPC, Vec.Holder, Vec.Reader, Vec.VectorGroup, Vec.Writer

Field Summary

Fields

Modifier and Type	Field and Description
int	_chunkOff
long[]	_tmp_espc

Fields inherited from class water.fvec.Vec

_cids, _rowLayout, DO_HISTOGRAMS, KEY_PREFIX_LEN, PERCENTILES, T_BAD, T_CAT, T_NUM, T_STR, T_TIME, T_UUID, TYPE_STR

Fields inherited from class water.Keyed

_key

Constructor Summary

Constructors

Constructor and Description
AppendableVec(Key<Vec> key, byte type)
AppendableVec(Key<Vec> key, long[] tmp_espc, byte type, int chunkOff)

Method Summary

Modifier and Type	Method and Description
long	byteSize() Size of compressed vector data.
protected long	chunk2StartElem(int cidx) Convert a chunk-index into a starting row #.
NewChunk	chunkForChunkIdx(int cidx) The Chunk for a chunk#.
Value	chunkIdx(int cidx) Get a Chunk's Value by index.
Vec	close(int rowLayout, Futures fs)
static Vec[]	closeAll(AppendableVec[] avs)
static Vec[]	closeAll(AppendableVec[] avs, Futures fs)
int	compute_rowLayout()
int	elem2ChunkIdx(long i) Convert a row# to a chunk#.
Vec	layout_and_close(Futures fs)
long	length() Number of elements in the vector; returned as a long instead of an int because Vecs support more than 2^32 elements.
int	nChunks() Number of chunks, returned as an int - Chunk count is limited by the max size of a Java long[].
int	nonEmptyChunks() Number of non-empty chunks, also see Vec.nChunks()
protected boolean	readable() Default read/write behavior for Vecs.
void	reduce(AppendableVec nv)
java.lang.String	toString() Pretty print the Vec: [#elems, min/mean/max]{chunks,...}
protected boolean	writable() Default read/write behavior for Vecs.

Methods inherited from class water.fvec.Vec

adaptTo, align, asDouble, at, at16h, at16l, at8, atStr, base, bins, bulk_remove, cardinality, checksum_impl, chunkForRow, chunkKey, chunkKey, chunkLen, doCopy, domain, equals, espc, factor, get_type_str, get_type, getVecKey, group, hashCode, isBad, isBinary, isBinary, isBinaryOnes, isCategorical, isCompatibleWith, isConst, isConst, isDomainTruncated, isHomedLocally, isInt, isNA, isNumeric, isString, isTime, isUUID, isVolatile, lazy_bins, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeCon, makeConN, makeCons, makeCons, makeCopy, makeCopy, makeCopy, makeDoubles, makeOne, makeOne, makeOnes, makeOnes, makeRand, makeRepSeq, makeSeq, makeSeq, makeSeq, makeTimeVec, makeVec, makeVec, makeVec, makeVec, makeVec, makeVec, makeVolatileDoubles, makeVolatileInts, makeZero, makeZero, makeZero, makeZero, makeZero, makeZeros, makeZeros, max, maxs, mean, min, mins, mode, naCnt, nChunksFor, newKey, ninfs, nzCnt, open, pctiles, pinfs, postWrite, preWriting, readAll_impl, remove_impl, remove_self_key_impl, rollupStatsKey, set, set, set, set, set, setBad, setChunkIdx, setDomain, setNA, sigma, sparseRatio, startRollupStats, startRollupStats, stride, stringAt, toCategoricalVec, toIntegerVec, toNumericVec, toStringVec, toTwoDimTable, toTwoDimTable, writeAll_impl

Methods inherited from class water.Keyed

checksum_impl, checksum, checksum, getKey, makeSchema, readAll, remove_impl, remove, remove, remove, remove, remove, remove, removeQuietly, writeAll

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

_tmp_espc

public long[] _tmp_espc

_chunkOff

public final int _chunkOff

Constructor Detail

AppendableVec

public AppendableVec(Key<Vec> key,
                     byte type)

AppendableVec

public AppendableVec(Key<Vec> key,
                     long[] tmp_espc,
                     byte type,
                     int chunkOff)

Method Detail

closeAll

public static Vec[] closeAll(AppendableVec[] avs)

closeAll

public static Vec[] closeAll(AppendableVec[] avs,
                             Futures fs)

reduce

public void reduce(AppendableVec nv)

layout_and_close

public Vec layout_and_close(Futures fs)

compute_rowLayout

public int compute_rowLayout()

close

public Vec close(int rowLayout,
                 Futures fs)

readable

protected boolean readable()

Description copied from class: Vec

Default read/write behavior for Vecs. File-backed Vecs are read-only.

writable

protected boolean writable()

Description copied from class: Vec

Default read/write behavior for Vecs. AppendableVecs are write-only.

chunkForChunkIdx

public NewChunk chunkForChunkIdx(int cidx)

Description copied from class: Vec

The Chunk for a chunk#. Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!

Overrides:

chunkForChunkIdx in class Vec

Returns:

Chunk for a chunk#

chunkIdx

public Value chunkIdx(int cidx)

Description copied from class: Vec

Get a Chunk's Value by index. Basically the index-to-key map, plus the DKV.get(). Warning: this pulls the data locally; using this call on every Chunk index on the same node will probably trigger an OOM!

length

public long length()

Description copied from class: Vec

Number of elements in the vector; returned as a long instead of an int because Vecs support more than 2^32 elements. Overridden by subclasses that compute length in an alternative way, such as file-backed Vecs.

Overrides:

length in class Vec

Returns:

Number of elements in the vector

nChunks

public int nChunks()

Description copied from class: Vec

Number of chunks, returned as an int - Chunk count is limited by the max size of a Java long[]. Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.

Overrides:

nChunks in class Vec

Returns:

Number of chunks

nonEmptyChunks

public int nonEmptyChunks()

Description copied from class: Vec

Number of non-empty chunks, also see Vec.nChunks()

Overrides:

nonEmptyChunks in class Vec

Returns:

Number of non-empty chunks

elem2ChunkIdx

public int elem2ChunkIdx(long i)

Description copied from class: Vec

Convert a row# to a chunk#. For constant-sized chunks this is a little shift-and-add math. For variable-sized chunks this is a binary search, with a sane API (JDK has an insane API). Overridden by subclasses that compute chunks in an alternative way, such as file-backed Vecs.

Overrides:

elem2ChunkIdx in class Vec

chunk2StartElem

protected long chunk2StartElem(int cidx)

Description copied from class: Vec

Convert a chunk-index into a starting row #. For constant-sized chunks this is a little shift-and-add math. For variable-sized chunks this is a table lookup.

byteSize

public long byteSize()

Description copied from class: Vec

Size of compressed vector data.

Overrides:

byteSize in class Vec

toString

public java.lang.String toString()

Description copied from class: Vec

Pretty print the Vec: [#elems, min/mean/max]{chunks,...}

Overrides:

toString in class Vec

Returns:

Brief string representation of a Vec