Skip to main content

Create your own evaluation datasets

The Custom Eval feature in H2O LLM DataStudio enables you to create your own evaluation datasets from various document formats (such as PDFs and DOC files), from audio and video files and existing datasets. These evaluation datasets can be downloaded in JSON formats for each evaluation type, allowing for seamless integration with H2O Eval Studio.

note

Custom Eval only supports English language.

Instructions​

To create your own evaluation dataset, consider the following instructions:

  1. On the H2O LLM DataStudio left navigation menu, click Custom Eval.
  2. On the Create Your Own Eval Datasets page, click New.
  3. On the Project name text box, enter a name for the project.
  4. On the Description text box, enter a description for the project.
  5. On the Dataset type drop-down menu, select the evaluation dataset type. The available dataset types are,
    • Question type: In this dataset, each entry includes a specific question, the correct answer to that question, and a label indicating the type of question (e.g., simple, conditional).
    • Multi-Choice: In this dataset, each entry consists of a question followed by multiple answer choices, and the correct, or most appropriate answer.
    • Token presence: In this dataset, each entry includes a question, the correct answer, and a list of key tokens that are relevant to the answer.
  6. On the Do you already have a QA Dataset? drop-down menu, select the following options accordingly:
Select Yes already possess a question-answer dataset.

7. Click Next.

8. Click Browse to upload the Q&A dataset in CSV file format.

9. Click Upload.
  1. Under Configure columns section, select the columns which contain the context, question, and answer from the given options.
  2. Click Run pipeline.
Feedback