Driverless AI Logs

Driverless AI provides several logs that can be viewed and/or retrieved when performing different tasks. All content in the logs are labeled with INFO, DATA, WARNING and ERROR tags. Driverless AI Modeling and MLI experiments also provide access to anonymized logs that do not contain contents from the DATA tag.

Available Log Files

The following is a list of available Driverless AI log files.

dai.log

Experiment Log

MLI Logs

Auto Visualization Logs

h2oai_server Log

Audit Logs

Sending Logs to support@H2O.ai

This section describes what logs to send in the event of failures when running Driverless AI. All content in the logs are labeled with INFO, DATA, WARNING and ERROR tags. Driverless AI Modeling and MLI experiments also provides access to anonymized logs that do not contain contents from the DATA tag.

  • Driverless AI starting Failures: This requires inspection of System Logs like dai.log file.

  • Dataset Failures: A simple error stack trace is displayed on the GUI in case of datasets failures like Adding Datasets or viewing Dataset Details and detailed logs are registered as h2oai_server logs that requires admin access.

  • Auto Viz Failures: Auto Visualization Logs on datasets can be accessed from the GUI. A full detailed stacktrace is also available in the h2oai_server.log file in ./tmp folder of DAI that requires admin access.

  • Experiment Failures: User needs to send the experiment logs. In some cases, for in depth analysis, support@h2o.ai may request dai.logs that requires admin access to retrieve.

  • MLI Failures: See MLI Logs for details.

  • Custom Recipes Failures: If a Custom Recipe is producing errors, the entire zip file obtained by clicking on the Download Summary & Logs button on the experiment page, can be sent for troubleshooting. Note that these files may contain information that is not anonymized.

System Logs

System logs include useful information about Driverless AI. Driverless AI solution needs following set of services to work-

  • Driverless AI server: This is a python code, that internally starts a local worker to start a web server for UI pages (DAI GUI) and runs the actual experiment work.

  • h2o3 server: This is needed for MLI and recipes that use h2o3.

  • procsy: This handles the communication between the DAI server (python code) and other binaries or java jar files, like data connectors or the vis-sever.

  • vis-server: This is needed for Auto visualization of Datasets, DAI sends a request to procsy, which in turn will query the vis-server to make the computations necessary for autoviz.

  • redis-server: It is used as a communication bus between the backend (DAI) server and the local worker or remote workers (in case of DAI multinode set up).

  • minio: This is needed in multinode setup, and is used for data storage, for example, when running an experiment on a remote node, the remote worker gets the experiment configuration details via redis, and the actual dataset, is pushed to minio and the remote worker is instructed to fetch it. When experiment finishes, the model is sent back to the main server from the remote node via minio (upload and download).

Each of these services creates a log file. Collecting these system logs varies depending on the way Driverless AI is installed and requires admin access to DAI install location.

  • For Docker installs, the system logs reside in the ./log/{date_time}/ folder of Driverless AI. For example-

x@a_machine:~/dai_1.10/log/20210720-xyz$ ls
dai.log  h2o.log  jupyter.log  procsy.log  redis-server.log  vis-server.log  minio.log
  • For RPM and Deb installs that do NOT use systemD, the log file will be in opt/h2oai/dai/log. For example:

sudo less /opt/h2oai/dai/log/dai.log
sudo less /opt/h2oai/dai/log/h2o.log
sudo less /opt/h2oai/dai/log/procsy.log
sudo less /opt/h2oai/dai/log/vis-server.log
  • For RPM and Deb installs that use systemD, you can use journalctl to view logs collected by systemd. For example:

sudo systemctl status dai-dai
sudo systemctl status dai-h2o
sudo systemctl status dai-procsy
sudo systemctl status dai-vis-server
sudo journalctl -u dai-dai
sudo journalctl -u dai-h2o
sudo journalctl -u dai-procsy
sudo journalctl -u dai-vis-server
  • For Tar installs, use the following commands to view system logs:

less log/dai.log
less log/h2o.log
less log/procsy.log
less log/vis-server.log