water.parser

Class CsvParser

java.lang.Object

water.Iced

water.parser.Parser

water.parser.CsvParser

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, Freezable

public class CsvParser
extends Parser

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static byte	HIVE_SEP Separators recognized by the CSV parser.

Fields inherited from class water.parser.Parser

_jobKey, _keepColumns, _setup, CHAR_DECIMAL_SEP, CHAR_ESCAPE, CHAR_SEPARATOR, COND_QUOTE, COND_QUOTED_NUMBER_END, COND_QUOTED_TOKEN, EOL, EXPECT_COND_LF, HASHTAG, LARGEST_DIGIT_NUMBER, NUMBER, NUMBER_END, NUMBER_EXP, NUMBER_EXP_START, NUMBER_FRACTION, NUMBER_SKIP, NUMBER_SKIP_NO_DOT, POSSIBLE_CURRENCY, POSSIBLE_EMPTY_LINE, POSSIBLE_ESCAPED_QUOTE, SEPARATOR_OR_EOL, SKIP_LINE, STRING, STRING_END, TOKEN, WHITESPACE_BEFORE_TOKEN

Method Summary

Modifier and Type	Method and Description
static java.lang.String[]	determineTokens(java.lang.String from, byte separator, boolean singleQuotes, byte escapechar) Determines the tokens that are inside a line and returns them as strings in an array.
static java.lang.String[]	determineTokens(java.lang.String from, byte separator, byte quoteChar, byte escapechar)
protected int	fileHasHeader(byte[] bits, ParseSetup ps)
static int	guessNcols(java.lang.String[] columnNames, java.lang.String[][] data)
static byte	guessSeparator(java.lang.String l1, java.lang.String l2, boolean singleQuotes, byte escapechar)
ParseWriter	parseChunk(int cidx, ParseReader din, ParseWriter dout)

Methods inherited from class water.parser.Parser

isEOL, sequentialParse, streamParse, streamParseZip

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HIVE_SEP

public static final byte HIVE_SEP

Separators recognized by the CSV parser. You can add new separators to this list and the parser will automatically attempt to recognize them. In case of doubt the separators are listed in descending order of probability, with space being the last one - space must always be the last one as it is used if all other fails because multiple spaces can be used as a single separator.

See Also:

Constant Field Values

Method Detail

parseChunk

public ParseWriter parseChunk(int cidx,
                              ParseReader din,
                              ParseWriter dout)

Specified by:

parseChunk in class Parser

fileHasHeader

protected int fileHasHeader(byte[] bits,
                            ParseSetup ps)

Overrides:

fileHasHeader in class Parser

determineTokens

public static java.lang.String[] determineTokens(java.lang.String from,
                                                 byte separator,
                                                 boolean singleQuotes,
                                                 byte escapechar)

Determines the tokens that are inside a line and returns them as strings in an array. Assumes the given separator.

determineTokens

public static java.lang.String[] determineTokens(java.lang.String from,
                                                 byte separator,
                                                 byte quoteChar,
                                                 byte escapechar)

guessSeparator

public static byte guessSeparator(java.lang.String l1,
                                  java.lang.String l2,
                                  boolean singleQuotes,
                                  byte escapechar)

guessNcols

public static int guessNcols(java.lang.String[] columnNames,
                             java.lang.String[][] data)