hex

Class RegexTokenizer

java.lang.Object

jsr166y.ForkJoinTask<java.lang.Void>

jsr166y.CountedCompleter

water.H2O.H2OCountedCompleter<T>

water.DTask<T>

water.MRTask<RegexTokenizer>

hex.RegexTokenizer

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, java.util.concurrent.Future<java.lang.Void>, ForkJoinPool.ManagedBlocker, Freezable<RegexTokenizer>

public class RegexTokenizer
extends MRTask<RegexTokenizer>

RegexTokenizer splits rows of a given Frame into delimited sequences of tokens using a regular expression. The output structure is suitable for use in the word2vec algorithm.

Example usage:

 final RegexTokenizer tokenizer = new RegexTokenizer.Builder()
   .setRegex("[,;]")
   .setMinLength(2)
   .setToLowercase(true)
   .create();
 final Frame tokens = tokenizer.transform(inputFrame);
 
 

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RegexTokenizer.Builder

Nested classes/interfaces inherited from class water.MRTask

MRTask.PostMapAction<T extends MRTask.PostMapAction<T>>

Nested classes/interfaces inherited from class water.DTask

DTask.DKeyTask<T extends DTask.DKeyTask,V extends Keyed>, DTask.RemoveCall

Field Summary

Fields inherited from class water.MRTask

_appendables, _fr, _fs, _hi, _keys, _left, _lo, _nhi, _nleft, _nlo, _nrite, _rite, _run_local, _topLocal

Fields inherited from class water.DTask

_ex

Constructor Summary

Constructors

Constructor and Description
RegexTokenizer(java.lang.String regex)

Method Summary

Modifier and Type	Method and Description
void	map(Chunk[] cs, NewChunk nc) The handy method to generate a new vector based on existing vectors.
Frame	transform(Frame input) Tokenizes a given Frame

Methods inherited from class water.MRTask

appendables, asyncExecOnAllNodes, block, closeLocal, compute2, dfork, dfork, dfork, dfork, dfork, dinvoke, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAll, doAllNodes, getResult, getResult, isReleasable, map, map, map, map, map, map, map, map, map, map, map, modifiesVolatileVecs, onCompletion, onExceptionalCompletion, outputFrame, outputFrame, outputFrame, postGlobal, profile, profString, reduce, self, setupLocal, withPostMapAction

Methods inherited from class water.DTask

copyOver, getDException, hasException, logVerbose, onAck, onAckAck, setException

Methods inherited from class water.H2O.H2OCountedCompleter

asBytes, clone, compute, compute1, currThrPriority, frozenType, icer, priority, read, readJSON, reloadFromBytes, write, writeJSON

Methods inherited from class jsr166y.CountedCompleter

__tryComplete, addToPendingCount, compareAndSetPendingCount, complete, exec, getCompleter, getPendingCount, getRawResult, setCompleter, setPendingCount, setRawResult, tryComplete

Methods inherited from class jsr166y.ForkJoinTask

adapt, adapt, adapt, cancel, compareAndSetForkJoinTaskTag, completeExceptionally, fork, get, get, get, getException, getForkJoinTaskTag, getPool, getQueuedTaskCount, getSurplusQueuedTaskCount, helpQuiesce, inForkJoinPool, invoke, invokeAll, invokeAll, invokeAll, isCancelled, isCompletedAbnormally, isCompletedNormally, isDone, join, peekNextLocalTask, pollNextLocalTask, pollTask, quietlyComplete, quietlyInvoke, quietlyJoin, reinitialize, setForkJoinTaskTag, tryUnfork

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RegexTokenizer

public RegexTokenizer(java.lang.String regex)

Method Detail

map

public void map(Chunk[] cs,
                NewChunk nc)

Description copied from class: MRTask

The handy method to generate a new vector based on existing vectors. Note: This method is used by Sparkling Water examples.

Overrides:

map in class MRTask<RegexTokenizer>

Parameters:

cs - input vectors

nc - output vector

transform

public Frame transform(Frame input)

Tokenizes a given Frame

Parameters:

input - Input Frame is expected to only contain String columns. Each row of the Frame represents a logical sentence. The sentence can span one or more cells of the row.

Returns:

Frame made of a single String column where original sentences are split into tokens and delimited by NAs.