water.parser

Class ParserProvider

java.lang.Object

water.parser.ParserProvider

Direct Known Subclasses:

BinaryParserProvider, DefaultParserProviders.ArffParserProvider, DefaultParserProviders.CsvParserProvider, DefaultParserProviders.GuessParserProvider, DefaultParserProviders.SVMLightParserProvider, DefaultParserProviders.XlsParserProvider

public abstract class ParserProvider
extends java.lang.Object

Generic Parser provider.

Constructor Summary

Constructors

Constructor and Description
ParserProvider()

Method Summary

Modifier and Type	Method and Description
abstract Parser	createParser(ParseSetup setup, Key<Job> jobKey) Create a new parser
abstract ParseSetup	createParserSetup(Key[] inputs, ParseSetup requiredSetup) Create a parser specific setup.
ParseSetup	guessFinalSetup(ByteVec v, byte[] bits, ParseSetup ps) Finalizes ParseSetup created by guessInitSetup using data read from a given ByteVec/bits.
ParseSetup	guessInitSetup(ByteVec v, byte[] bits, ParseSetup userSetup) Constructs initial ParseSetup from a given user setup Any exception thrown by this method will signal that this ParserProvider doesn't support the input data.
protected ParseSetup	guessSetup_impl(ByteVec v, byte[] bits, ParseSetup userSetup) Actual implementation of the guessSetup method.
ParseSetup	guessSetup(ByteVec v, byte[] bits, byte sep, int ncols, boolean singleQuotes, int checkHeader, java.lang.String[] columnNames, byte[] columnTypes, java.lang.String[][] domains, java.lang.String[][] naStrings) Returns parser setup of throws exception if input is not recognized
ParseSetup	guessSetup(ByteVec v, byte[] bits, ParseSetup userSetup)
abstract ParserInfo	info() Technical information for this parser
ParseSetup	setupLocal(Vec v, ParseSetup setup) Executed exactly once per-file-per-node during parse.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParserProvider

public ParserProvider()

Method Detail

info

public abstract ParserInfo info()

Technical information for this parser

createParser

public abstract Parser createParser(ParseSetup setup,
                                    Key<Job> jobKey)

Create a new parser

guessSetup

public final ParseSetup guessSetup(ByteVec v,
                                   byte[] bits,
                                   ParseSetup userSetup)

Parameters:

v - optional ByteVec (can be null if extracting eg. from a compressed file)

bits - first bytes of the file

userSetup - user specified setup

Returns:

derived ParseSetup

guessSetup_impl

protected ParseSetup guessSetup_impl(ByteVec v,
                                     byte[] bits,
                                     ParseSetup userSetup)

Actual implementation of the guessSetup method. Should almost never be overridden (the only exception is the GuessParserProvider).

Parameters:

v -

bits -

userSetup -

Returns:

guessInitSetup

public ParseSetup guessInitSetup(ByteVec v,
                                 byte[] bits,
                                 ParseSetup userSetup)

Constructs initial ParseSetup from a given user setup Any exception thrown by this method will signal that this ParserProvider doesn't support the input data. Parsers of data formats that provide metadata (eg. a binary file formats like Parquet) should use the file metadata to identify the parse type and possibly other properties of the ParseSetup that can be determined just from the metadata itself. The goal should be perform the least amount of operations to correctly determine the ParseType (any exception means - format is not supported!). Note: Some file formats like CSV don't provide any metadata. In that case this method can return the final ParseSetup.

Parameters:

v - optional ByteVec

bits - first bytes of the file

userSetup - user specified setup

Returns:

null if this Provider cannot provide a parser for this file, otherwise an instance of ParseSetup with correct setting of ParseSetup._parse_type

guessFinalSetup

public ParseSetup guessFinalSetup(ByteVec v,
                                  byte[] bits,
                                  ParseSetup ps)

Finalizes ParseSetup created by guessInitSetup using data read from a given ByteVec/bits.

Parameters:

v - optional ByteVec

bits - first bytes of the file

ps - parse setup as created by guessInitSetup

Returns:

fully initialized ParseSetup

guessSetup

public ParseSetup guessSetup(ByteVec v,
                             byte[] bits,
                             byte sep,
                             int ncols,
                             boolean singleQuotes,
                             int checkHeader,
                             java.lang.String[] columnNames,
                             byte[] columnTypes,
                             java.lang.String[][] domains,
                             java.lang.String[][] naStrings)

Returns parser setup of throws exception if input is not recognized

createParserSetup

public abstract ParseSetup createParserSetup(Key[] inputs,
                                             ParseSetup requiredSetup)

Create a parser specific setup. Useful if parser need a single

Parameters:

inputs - input keys

requiredSetup - user given parser setup

Returns:

parser specific setup

setupLocal

public ParseSetup setupLocal(Vec v,
                             ParseSetup setup)

Executed exactly once per-file-per-node during parse. Do any file-related non-distributed setup here. E.g. ORC reader creates node-shared instance of a (non-serializable) Reader.

Parameters:

v -

setup -