Driverless AI Logs¶
Driverless AI provides several logs that can be viewed and/or retrieved when performing different tasks. All content in the logs are labeled with INFO, DATA, WARNING and ERROR tags. Driverless AI Modeling and MLI experiments also provide access to anonymized logs that do not contain contents from the DATA tag.
Available Log Files¶
The following is a list of available Driverless AI log files.
dai.log¶
dai.log are part of Driverless AI System Logs. They are generated as part of stderr/stdout and are useful for debugging or detailed support in case of issues. If needed, the verbosity or logging level of this log file can be toggled using config.toml settings.
Admin access to Driverless AI installation location is required to obtain these logs. See System Logs section on steps to obtain them.
Experiment Log¶
This log store information about an experiment’s parameters, model runs, performance and more. It helps with understanding the run details and debugging experiment related issues. The log file naming convention is h2oai_experiment_{experiment_ID}.log and the content is labeled with INFO, DATA, WARNING and ERROR tags.
Users can download these log directly from the experiment page of the Driverless AI GUI. For an experiment in progress, logs can be accessed from under the Log tab to the right. For completed experiments, the logs reside with the summary zip file.
The zip also contains an anonymized version of experiment logs that does not report any information relating to the data used in the experiment (i.e no DATA label), such as column names and individual data points. And a details folder that comprises of error stack traces that may help with debugging.
MLI Logs¶
These logs cover the model interpretation process runs for surrogate models and explainer/recipe runs for Driverless AI Machine Learning Interpretability jobs.
MLI surrogate model run logs can be downloaded from the Action button on the MLI GUI page. This folder naming convention is mli_experiment_{experiment_ID}.log. It contains three files, the stdout/stderr log for full MLI process run, an anonymized copy (i.e no DATA label) of the same log file and surrogate model run logs.
The explainer or recipe logs are accessible from the task run button.
MLI uses H2O_3 (Java backend) to build surrogate models. Admins can access the h2o_3 server logs using System Logs commands in case of issues with starting the MLI server. The /tmp folder of DAI contains h2o_mli.log, that keeps track of rolling mli logs and are also admin accessible.
Auto Visualization Logs¶
This log store run information for automatic data visualization in Driverless AI. Users can obtain them from the Autoviz page of DAI GUI.
Admins can access the viz-server logs using System Logs commands in case of issues with starting of Viz server. The failure logs relating to data visualization are also available from the /tmp folder as h2oai_server.log and requires admin access.
h2oai_server Log¶
These logs register all issues relating to datasets like Adding Datasets or viewing Dataset Details or Auto Visualization of datasets. They are stored under the Driverless AI ./tmp folder. An anonymized copy (i.e no DATA label) of this log file is also available in the same folder.
Accessing h2oai_server log requires admin access to Driverless AI.
Audit Logs¶
Audit logs register all user interactions with the Driverless AI system like login/logout, downloads/uploads, experiment creation/deletion etc.
Admins can access them from /tmp folder of Driverless AI.
Sending Logs to support@H2O.ai¶
This section describes what logs to send in the event of failures when running Driverless AI. All content in the logs are labeled with INFO, DATA, WARNING and ERROR tags. Driverless AI Modeling and MLI experiments also provides access to anonymized logs that do not contain contents from the DATA tag.
Driverless AI starting Failures: This requires inspection of System Logs like dai.log file.
Dataset Failures: A simple error stack trace is displayed on the GUI in case of datasets failures like Adding Datasets or viewing Dataset Details and detailed logs are registered as h2oai_server logs that requires admin access.
Auto Viz Failures: Auto Visualization Logs on datasets can be accessed from the GUI. A full detailed stacktrace is also available in the h2oai_server.log file in ./tmp folder of DAI that requires admin access.
Experiment Failures: User needs to send the experiment logs. In some cases, for in depth analysis, support@h2o.ai may request dai.logs that requires admin access to retrieve.
MLI Failures: See MLI Logs for details.
Custom Recipes Failures: If a Custom Recipe is producing errors, the entire zip file obtained by clicking on the Download Summary & Logs button on the experiment page, can be sent for troubleshooting. Note that these files may contain information that is not anonymized.
System Logs¶
System logs include useful information about Driverless AI. Driverless AI solution needs following set of services to work-
Driverless AI server: This is a python code, that internally starts a local worker to start a web server for UI pages (DAI GUI) and runs the actual experiment work.
h2o3 server: This is needed for MLI and recipes that use h2o3.
procsy: This handles the communication between the DAI server (python code) and other binaries or java jar files, like data connectors or the vis-sever.
vis-server: This is needed for Auto visualization of Datasets, DAI sends a request to procsy, which in turn will query the vis-server to make the computations necessary for autoviz.
redis-server: It is used as a communication bus between the backend (DAI) server and the local worker or remote workers (in case of DAI multinode set up).
minio: This is needed in multinode setup, and is used for data storage, for example, when running an experiment on a remote node, the remote worker gets the experiment configuration details via redis, and the actual dataset, is pushed to minio and the remote worker is instructed to fetch it. When experiment finishes, the model is sent back to the main server from the remote node via minio (upload and download).
Each of these services creates a log file. Collecting these system logs varies depending on the way Driverless AI is installed and requires admin access to DAI install location.
For Docker installs, the system logs reside in the ./log/{date_time}/ folder of Driverless AI. For example-
x@a_machine:~/dai_1.10/log/20210720-xyz$ ls dai.log h2o.log jupyter.log procsy.log redis-server.log vis-server.log minio.log
For RPM and Deb installs that do NOT use systemD, the log file will be in opt/h2oai/dai/log. For example:
sudo less /opt/h2oai/dai/log/dai.log sudo less /opt/h2oai/dai/log/h2o.log sudo less /opt/h2oai/dai/log/procsy.log sudo less /opt/h2oai/dai/log/vis-server.log
For RPM and Deb installs that use systemD, you can use journalctl to view logs collected by systemd. For example:
sudo systemctl status dai-dai sudo systemctl status dai-h2o sudo systemctl status dai-procsy sudo systemctl status dai-vis-server sudo journalctl -u dai-dai sudo journalctl -u dai-h2o sudo journalctl -u dai-procsy sudo journalctl -u dai-vis-server
For Tar installs, use the following commands to view system logs:
less log/dai.log less log/h2o.log less log/procsy.log less log/vis-server.log