water.parser

Class ParseSetup

java.lang.Object

water.Iced

water.parser.ParseSetup

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable

public class ParseSetup
extends Iced

A generic configuration and base guesser for a parser.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ParseSetup.GuessSetupTsk Try to determine the ParseSetup on a file by file basis and merge results.

Field Summary

Fields

Modifier and Type	Field and Description
int	_chunk_size
static byte	DEFAULT_ESCAPE_CHAR
boolean	disableParallelParse
static int	GUESS_COL_CNT
static int	GUESS_HEADER
static byte	GUESS_SEP
static int	HAS_HEADER
static int	NO_HEADER
java.lang.String[]	parquetColumnTypes

Constructor Summary

Constructors

Constructor and Description
ParseSetup() Create a default ParseSetup Used by Ray's schema magic
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[][] data, ParseWriter.ParseErr[] errs) Create a ParseSetup without any column information
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data)
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, byte escapeChar) Create a ParseSetup with all parameters except chunk size.
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, byte[] nonDataLineMarkers, byte escapeChar) Create a ParseSetup with all parameters except chunk size.
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, ParseWriter.ParseErr[] errs)
ParseSetup(ParserInfo parseType, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, ParseWriter.ParseErr[] errs, byte[] nonDataLineMarkers)
ParseSetup(ParserInfo parse_type, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, ParseWriter.ParseErr[] errs, int chunkSize, byte[] nonDataLineMarkers, byte escapeChar)
ParseSetup(ParserInfo parse_type, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, ParseWriter.ParseErr[] errs, int chunkSize, Key<DecryptionTool> decrypt_tool, int[] skipped_columns, byte[] nonDataLineMarkers, byte escapeChar)
ParseSetup(ParserInfo parse_type, byte sep, boolean singleQuotes, int checkHeader, int ncols, java.lang.String[] columnNames, byte[] ctypes, java.lang.String[][] domains, java.lang.String[][] naStrings, java.lang.String[][] data, ParseWriter.ParseErr[] errs, int chunkSize, Key<DecryptionTool> decrypt_tool, int[] skipped_columns, byte[] nonDataLineMarkers, byte escapeChar, boolean force_col_types)
ParseSetup(ParseSetup ps)
ParseSetup(ParseSetupV3 ps) Create a ParseSetup with parameters from the client.

Method Summary

Modifier and Type	Method and Description
void	addErrs(ParseWriter.ParseErr... errs)
<T extends ParseSetup> T	copyTo(T setup) Copies the common setup to another object (that is possibly and extension of the base setup).
static java.lang.String	createHexName(java.lang.String n) Cleans up the file name to make .hex name to be used as a destination key.
ParseWriter.ParseErr[]	errs()
int[]	get_parse_columns_indices()
java.lang.String[]	getColumnNames()
byte[]	getColumnTypes()
java.lang.String[]	getColumnTypeStrings()
java.lang.String[][]	getData()
DecryptionTool	getDecryptionTool()
ParseSetup	getFinalSetup(Key[] inputKeys, ParseSetup demandedSetup) Return create a final parser-specific setup for this configuration.
boolean	getForceColTypes()
int	getNumberColumns()
java.lang.String[]	getOrigColumnTypes()
java.lang.String[]	getParquetColumnTypes()
ParserInfo	getParseType()
int[]	getSkippedColumns()
static ParseSetup	guessSetup(ByteVec bv, byte[] bits, ParseSetup userSetup) Guess everything from a single pile-o-bits.
static ParseSetup	guessSetup(Key[] fkeys, boolean singleQuote, int checkHeader) Used by test harnesses for simple parsing of test data.
static ParseSetup	guessSetup(Key[] fkeys, ParseSetup userSetup) Discover the parse setup needed to correctly parse all files.
protected boolean	isCompatible(ParseSetup setupB)
boolean	isNA(int colIdx, BufferedString str) Tests whether a given string represents a NA in a given column.
static ParseSetup	makeSVMLightSetup()
ParserInfo.ParseMethod	parseMethod(int nfiles, Vec v)
protected Parser	parser(Key jobKey) This is a single entry-point to create a parser.
ParseSetup	setCheckHeader(int check_header)
ParseSetup	setChunkSize(int chunk_size)
ParseSetup	setColumnNames(java.lang.String[] column_names)
ParseSetup	setColumnTypes(byte[] column_types)
ParseSetup	setDecryptTool(Key<DecryptionTool> decrypt_tool)
ParseSetup	setDomains(java.lang.String[][] domains)
void	setFileName(java.lang.String name)
ParseSetup	setForceColTypes(boolean force_col_types)
ParseSetup	setNAStrings(java.lang.String[][] na_strings)
ParseSetup	setNumberColumns(int number_columns)
ParseSetup	setOrigColumnTypes(java.lang.String[] col_types)
void	setParquetColumnTypes(java.lang.String[] columnTypes)
void	setParseColumnIndices(int ncols, int[] skipped_columns)
ParseSetup	setParseType(ParserInfo parse_type)
ParseSetup	setSeparator(byte separator)
ParseSetup	setSingleQuotes(boolean single_quotes)
ParseSetup	setSkippedColumns(int[] skipped_columns)
void	setSyntheticColumns(java.lang.String[] names, java.lang.String[][] valueMapping, byte synthetic_column_type)
static byte[]	strToColumnTypes(java.lang.String[] strs)
java.lang.String	toString()

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

GUESS_SEP

public static final byte GUESS_SEP

See Also:

Constant Field Values

NO_HEADER

public static final int NO_HEADER

See Also:

Constant Field Values

GUESS_HEADER

public static final int GUESS_HEADER

See Also:

Constant Field Values

HAS_HEADER

public static final int HAS_HEADER

See Also:

Constant Field Values

GUESS_COL_CNT

public static final int GUESS_COL_CNT

See Also:

Constant Field Values

DEFAULT_ESCAPE_CHAR

public static final byte DEFAULT_ESCAPE_CHAR

See Also:

Constant Field Values

disableParallelParse

public boolean disableParallelParse

_chunk_size

public int _chunk_size

parquetColumnTypes

public java.lang.String[] parquetColumnTypes

Constructor Detail

ParseSetup

public ParseSetup(ParseSetup ps)

ParseSetup

public ParseSetup(ParserInfo parse_type,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[] columnNames,
                  byte[] ctypes,
                  java.lang.String[][] domains,
                  java.lang.String[][] naStrings,
                  java.lang.String[][] data,
                  ParseWriter.ParseErr[] errs,
                  int chunkSize,
                  byte[] nonDataLineMarkers,
                  byte escapeChar)

ParseSetup

public ParseSetup(ParserInfo parse_type,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[] columnNames,
                  byte[] ctypes,
                  java.lang.String[][] domains,
                  java.lang.String[][] naStrings,
                  java.lang.String[][] data,
                  ParseWriter.ParseErr[] errs,
                  int chunkSize,
                  Key<DecryptionTool> decrypt_tool,
                  int[] skipped_columns,
                  byte[] nonDataLineMarkers,
                  byte escapeChar)

ParseSetup

public ParseSetup(ParserInfo parse_type,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[] columnNames,
                  byte[] ctypes,
                  java.lang.String[][] domains,
                  java.lang.String[][] naStrings,
                  java.lang.String[][] data,
                  ParseWriter.ParseErr[] errs,
                  int chunkSize,
                  Key<DecryptionTool> decrypt_tool,
                  int[] skipped_columns,
                  byte[] nonDataLineMarkers,
                  byte escapeChar,
                  boolean force_col_types)

ParseSetup

public ParseSetup(ParseSetupV3 ps)

Create a ParseSetup with parameters from the client. Typically used to guide sampling in the data to verify chosen settings, and fill in missing settings.

Parameters:

ps - Parse setup settings from client

ParseSetup

public ParseSetup(ParserInfo parseType,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[] columnNames,
                  byte[] ctypes,
                  java.lang.String[][] domains,
                  java.lang.String[][] naStrings,
                  java.lang.String[][] data,
                  byte[] nonDataLineMarkers,
                  byte escapeChar)

Create a ParseSetup with all parameters except chunk size.

Typically used by file type parsers for returning final valid results _chunk_size will be set later using results from all files.

ParseSetup

public ParseSetup(ParserInfo parseType,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[] columnNames,
                  byte[] ctypes,
                  java.lang.String[][] domains,
                  java.lang.String[][] naStrings,
                  java.lang.String[][] data,
                  byte escapeChar)

Create a ParseSetup with all parameters except chunk size.

Typically used by file type parsers for returning final valid results _chunk_size will be set later using results from all files.

ParseSetup

public ParseSetup(ParserInfo parseType,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[] columnNames,
                  byte[] ctypes,
                  java.lang.String[][] domains,
                  java.lang.String[][] naStrings,
                  java.lang.String[][] data)

ParseSetup

public ParseSetup(ParserInfo parseType,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[] columnNames,
                  byte[] ctypes,
                  java.lang.String[][] domains,
                  java.lang.String[][] naStrings,
                  java.lang.String[][] data,
                  ParseWriter.ParseErr[] errs,
                  byte[] nonDataLineMarkers)

ParseSetup

public ParseSetup(ParserInfo parseType,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[] columnNames,
                  byte[] ctypes,
                  java.lang.String[][] domains,
                  java.lang.String[][] naStrings,
                  java.lang.String[][] data,
                  ParseWriter.ParseErr[] errs)

ParseSetup

public ParseSetup(ParserInfo parseType,
                  byte sep,
                  boolean singleQuotes,
                  int checkHeader,
                  int ncols,
                  java.lang.String[][] data,
                  ParseWriter.ParseErr[] errs)

Create a ParseSetup without any column information

Typically used by file type parsers for returning final invalid results

ParseSetup

public ParseSetup()

Create a default ParseSetup Used by Ray's schema magic

Method Detail

setFileName

public void setFileName(java.lang.String name)

errs

public final ParseWriter.ParseErr[] errs()

addErrs

public void addErrs(ParseWriter.ParseErr... errs)

makeSVMLightSetup

public static ParseSetup makeSVMLightSetup()

setParseColumnIndices

public void setParseColumnIndices(int ncols,
                                  int[] skipped_columns)

setSyntheticColumns

public void setSyntheticColumns(java.lang.String[] names,
                                java.lang.String[][] valueMapping,
                                byte synthetic_column_type)

setParquetColumnTypes

public void setParquetColumnTypes(java.lang.String[] columnTypes)

getColumnNames

public java.lang.String[] getColumnNames()

getSkippedColumns

public int[] getSkippedColumns()

get_parse_columns_indices

public int[] get_parse_columns_indices()

getData

public java.lang.String[][] getData()

getColumnTypeStrings

public java.lang.String[] getColumnTypeStrings()

getOrigColumnTypes

public java.lang.String[] getOrigColumnTypes()

getForceColTypes

public boolean getForceColTypes()

getColumnTypes

public byte[] getColumnTypes()

strToColumnTypes

public static byte[] strToColumnTypes(java.lang.String[] strs)

parser

protected Parser parser(Key jobKey)

This is a single entry-point to create a parser. Should be override in subclasses.

getFinalSetup

public final ParseSetup getFinalSetup(Key[] inputKeys,
                                      ParseSetup demandedSetup)

Return create a final parser-specific setup for this configuration.

Parameters:

inputKeys - inputs

demandedSetup - setup demanded by a user

Returns:

a parser specific setup based on demanded setup

getNumberColumns

public int getNumberColumns()

getDecryptionTool

public final DecryptionTool getDecryptionTool()

getParquetColumnTypes

public final java.lang.String[] getParquetColumnTypes()

parseMethod

public final ParserInfo.ParseMethod parseMethod(int nfiles,
                                                Vec v)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

guessSetup

public static ParseSetup guessSetup(Key[] fkeys,
                                    boolean singleQuote,
                                    int checkHeader)

Used by test harnesses for simple parsing of test data. Presumes auto-detection for file and separator types.

Parameters:

fkeys - Keys to input vectors to be parsed

singleQuote - single quotes quote fields

checkHeader - check for a header

Returns:

ParseSetup settings from looking at all files

guessSetup

public static ParseSetup guessSetup(Key[] fkeys,
                                    ParseSetup userSetup)

Discover the parse setup needed to correctly parse all files. This takes a ParseSetup as guidance. Each file is examined individually and then results merged. If a conflict exists between any results all files are re-examined using the best guess from the first examination.

Parameters:

fkeys - Keys to input vectors to be parsed

userSetup - Setup guidance from user

Returns:

ParseSetup settings from looking at all files

isCompatible

protected boolean isCompatible(ParseSetup setupB)

guessSetup

public static ParseSetup guessSetup(ByteVec bv,
                                    byte[] bits,
                                    ParseSetup userSetup)

Guess everything from a single pile-o-bits. Used in tests, or in initial parser inspections when the user has not told us anything about separators or headers.

Parameters:

bits - Initial bytes from a parse source

Returns:

ParseSetup settings from looking at all files

createHexName

public static java.lang.String createHexName(java.lang.String n)

Cleans up the file name to make .hex name to be used as a destination key. Eliminates common file extensions, and replaces odd characters.

Parameters:

n - filename to be cleaned

Returns:

cleaned name

copyTo

public <T extends ParseSetup> T copyTo(T setup)

Copies the common setup to another object (that is possibly and extension of the base setup). Note: this method only copies fields directly declared in ParseSetup class, it doesn't handle fields that are declared in classes derived from ParseSetup.

Type Parameters:

T - class derived from ParseSetup

Parameters:

setup - target setup object

Returns:

the target setup object (for convenience)

isNA

public boolean isNA(int colIdx,
                    BufferedString str)

Tests whether a given string represents a NA in a given column. Note: NAs are expected to be made ONLY of ASCII (7-bit) characters, NA constants in unicode won't be recognized.

Parameters:

colIdx - index of the column

str - string to be tested for NA

Returns:

true - string is one of the column's NAs, false otherwise

getParseType

public ParserInfo getParseType()

setParseType

public ParseSetup setParseType(ParserInfo parse_type)

setSeparator

public ParseSetup setSeparator(byte separator)

setSingleQuotes

public ParseSetup setSingleQuotes(boolean single_quotes)

setCheckHeader

public ParseSetup setCheckHeader(int check_header)

setNumberColumns

public ParseSetup setNumberColumns(int number_columns)

setColumnNames

public ParseSetup setColumnNames(java.lang.String[] column_names)

setSkippedColumns

public ParseSetup setSkippedColumns(int[] skipped_columns)

setColumnTypes

public ParseSetup setColumnTypes(byte[] column_types)

setOrigColumnTypes

public ParseSetup setOrigColumnTypes(java.lang.String[] col_types)

setForceColTypes

public ParseSetup setForceColTypes(boolean force_col_types)

setDomains

public ParseSetup setDomains(java.lang.String[][] domains)

setNAStrings

public ParseSetup setNAStrings(java.lang.String[][] na_strings)

setChunkSize

public ParseSetup setChunkSize(int chunk_size)

setDecryptTool

public ParseSetup setDecryptTool(Key<DecryptionTool> decrypt_tool)