Skip to main content
Version: 1.2.0

Logging

By default, all Feature Store services produce logs to stdout in JSON format.

Log structure​

We use the following JSON template:

{
  "time": {
    "$resolver": "timestamp",
    "pattern": {
      "format": "yyyy-MM-dd'T'HH:mm:ss.SS'Z'",
      "timeZone": "UTC"
    }
  },
  "level": {
    "$resolver": "level",
    "field": "name"
  },
  "message": {
    "$resolver": "message",
    "stringified": true
  },
  "error": {
    "$resolver": "exception",
    "field": "message",
    "stringified": true
  },
  "user": {
    "$resolver": "mdc",
    "key": "userId"
  },
  "grpc_server_method": {
    "$resolver": "mdc",
    "key": "grpcServerMethod"
  },
  "thread_id": {
    "$resolver": "thread",
    "field": "name"
  },
  "logger": {
    "$resolver": "logger",
    "field": "name"
  },
  "stacktrace": {
    "$resolver": "exception",
    "field": "stackTrace",
    "stackTrace": {
      "stringified": true
    }
  }
}

Customize log format​

It is s possible to provide a custom log4j2 configuration. To overwrite the log4j2.properties file (after installation), edit default log4j config map - {{ .Release.Name }}-log4j-config

Also, it is possible to provide a custom config map with log4j2 configuration. Prepare the new log4j2.properties file you want to use. For example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: log4j-config
data:
  log4j2.properties: |
    monitorInterval=5

    # Root logger option
    rootLogger.level=INFO
    rootLogger.appenderRefs=stdout
    rootLogger.appenderRef.stdout.ref=STDOUT
    appenders=console

    # Direct log messages to stdout
    appender.console.type=Console
    appender.console.name=STDOUT
    appender.console.layout.type = PatternLayout
    appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

The next step is to configure Helm chart to use that config map. To override log4j configuration for all components, put name of custom config map to global.log4jConfigMapOverride value. To provide different config maps for different components put name of custom config maps to following values:

  • for core pods - core.log4jConfigMapOverride
  • for online store pods - onlinestore.log4jConfigMapOverride
  • for spark operator pod - sparkoperator.log4jConfigMapOverride
  • for spark driver/executor pods - sparkoperator.config.spark.log4jConfigMapOverride
  • for telemetry pod - telemetry.log4jConfigMapOverride

Use different file for log4j configuration​

By defaults feature store are looking for log4j2.properties file in /opt/h2oai/log4j directory. If it is required to use different file instead (e.g. log4j2.xml), custom config map should contain that file and JAVA_LOG4J_PROPERTIES environment variable has to be updated:

  • fore core pods:
core:
  env:
    JAVA_LOG4J_PROPERTIES: /opt/h2oai/log4j/log4j2.xml
  • for online store pods:
onlinestore:
  env:
    JAVA_LOG4J_PROPERTIES: /opt/h2oai/log4j/log4j2.xml
  • for sparkoperator store pods:
sparkoperator:
  env:
    JAVA_LOG4J_PROPERTIES: /opt/h2oai/log4j/log4j2.xml
  • for spark driver/executor pods
sparkoperator:
  config:
    spark:
      extraOptions:
        - spark.driver.extraJavaOptions="-Dlog4j2.configurationFile=file:///opt/h2oai/log4j/log4j2.xml"
        - spark.executor.extraJavaOptions="-Dlog4j2.configurationFile=file:///opt/h2oai/log4j/log4j2.xml"
  • for telemetry store pods:
telemetry:
  env: /opt/h2oai/log4j/log4j2.xml
Feedback