Driverless AI logs
Driverless AI provides several logs that can be viewed and/or retrieved when performing different tasks. All content in the logs are labeled with INFO, DATA, WARNING and ERROR tags. Driverless AI Modeling and MLI experiments also provide access to anonymized logs that do not contain contents from the DATA tag.
Note: For RPM or DEB package installations, log files are located in the following directories:
Startup and Daemon logs are accessible through
journalctl.Experiment logs are located in
/opt/h2oai/dai/tmp/<user_logged_in_DAI>/.Redis/server-related standard output and standard error logs are located in
/opt/h2oai/dai/tmp/server_log/.
The process IDs (PIDs) of the services are stored in /opt/h2oai/dai/tmp/pids/.
Available log files
The following is a list of available Driverless AI log files.
dai.log
dai.log are part of Driverless AI System Logs. They are generated as part of stderr/stdout and are useful for debugging or detailed support in case of issues. If needed, the verbosity or logging level of this log file can be toggled using config.toml settings.
Admin access to Driverless AI installation location is required to obtain these logs. See System Logs section on steps to obtain them.
Experiment log
This log store information about an experiment’s parameters, model runs, performance and more. It helps with understanding the run details and debugging experiment related issues. The log file naming convention is h2oai_experiment_{experiment_ID}.log and the content is labeled with INFO, DATA, WARNING and ERROR tags.
Users can download these log directly from the experiment page of the Driverless AI GUI. For an experiment in progress, logs can be accessed from under the Log tab to the right. For completed experiments, the logs reside with the summary zip file.
The zip also contains an anonymized version of experiment logs that does not report any information relating to the data used in the experiment (i.e no DATA label), such as column names and individual data points. And a details folder that comprises of error stack traces that may help with debugging.
MLI logs
These logs cover the model interpretation process runs for surrogate models and explainer/recipe runs for Driverless AI Machine Learning Interpretability jobs.
MLI surrogate model run logs can be downloaded from the Action button on the MLI GUI page. This folder naming convention is mli_experiment_{experiment_ID}.log. It contains three files, the stdout/stderr log for full MLI process run, an anonymized copy (i.e no DATA label) of the same log file and surrogate model run logs.
The explainer or recipe logs are accessible from the task run button.
MLI uses H2O_3 (Java backend) to build surrogate models. Admins can access the h2o_3 server logs using System Logs commands in case of issues with starting the MLI server. The /tmp folder of DAI contains h2o_mli.log, that keeps track of rolling mli logs and are also admin accessible.
Auto Visualization logs
This log stores run information for automatic data visualization in Driverless AI. Users can obtain them from the Autoviz page of DAI GUI.
Admin users can access failure logs relating to data visualization from the /tmp folder as h2oai_server.log.
h2oai_server log
These logs register all issues relating to datasets like Adding Datasets or viewing Dataset Details or Auto Visualization of datasets. They are stored under the Driverless AI ./tmp folder. An anonymized copy (i.e no DATA label) of this log file is also available in the same folder.
Accessing h2oai_server log requires admin access to Driverless AI.
Audit logs
Audit logs register all user interactions with the Driverless AI system like login/logout, downloads/uploads, experiment creation/deletion etc.
Admins can access them from /tmp folder of Driverless AI.
Send logs to support@h2o.ai
This section describes what logs to send in the event of failures when running Driverless AI. All content in the logs are labeled with INFO, DATA, WARNING and ERROR tags. Driverless AI Modeling and MLI experiments also provide access to anonymized logs that do not contain contents from the DATA tag.
Driverless AI starting Failures: This requires inspection of System Logs like dai.log file.
Dataset Failures: A simple error stack trace is displayed on the GUI in case of datasets failures like Adding Datasets or viewing Dataset Details and detailed logs are registered as h2oai_server logs that requires admin access.
Auto Viz Failures: Auto Visualization logs on datasets can be accessed from the GUI. A full detailed stacktrace is also available in the h2oai_server.log file in ./tmp folder of DAI that requires admin access.
Experiment Failures: User needs to send the experiment logs. In some cases, for in depth analysis, support@h2o.ai may request dai.logs that requires admin access to retrieve.
MLI Failures: See MLI Logs for details.
Custom Recipes Failures: If a Custom Recipe is producing errors, the entire zip file obtained by clicking on the Download Summary & Logs button on the experiment page, can be sent for troubleshooting. Note that these files may contain information that is not anonymized.
System logs
This section describes how to collect system logs, which include useful information about Driverless AI. Note that the method for collecting logs varies depending on the way Driverless AI is installed and that admin access to the DAI install location is required in order to collect logs.
For Docker installs, the system logs reside in the ./log/{date_time}/ folder of Driverless AI. For example-
x@a_machine:~/dai_1.10/log/20210720-xyz$ ls dai.log
For RPM and DEB installs that use systemd, you can use journalctl to view logs collected by systemd. For example:
sudo systemctl status dai-dai sudo journalctl -u dai-dai
For Tar installs, use the following commands to view system logs:
less log/dai.log
Download server logs
Note: This feature is not available for standalone installations.
To download server logs from the UI, click User > Logs in the main navigation. This opens the Logs page, which contains a list of downloadable server logs. Note that you must be an admin user to view the list of server logs.
