Skip to main content

Tests

Overview​

In H2O Eval Studio, a test is a collection of documents (that is, a corpus) along with prompts that are relevant to the corpus, ground truth, constraints, and other parameters that are used to evaluate a RAG or LLM model. This page describes how to import a Test Suite (a collection of Tests) or add a test and new test cases manually.

Import Test Suite​

The following steps describe how to import a Test Suite in H2O EvalStudio.

  1. In the main navigation, click Tests.

  2. Click the Import Test Suite button. Import Test Suite

  3. Enter a name prefix for the tests in the Test Suite.

  4. Enter a description of the tests in the Test Suite.

  5. Enter the Test Suite JSON or URL. The following is an example of how the Test Suite JSON should be formatted:

    {
       "name": "Sample Test Suite",
       "description": "This is a sample test suite.",
       "tests": [
           {
               "documents": [
               ],
               "test_cases": [
                   {
                       "prompt": "Sample prompt 1.",
                       "categories": [
                           "privacy", "security"
                       ],
                       "constraints": [
                       ],
                       "expected_output": "Expected output for sample prompt 1."
                   },
                   {
                       "prompt": "Sample prompt 2.",
                       "categories": [
                           "privacy", "security"
                       ],
                       "constraints": [
                       ],
                       "expected_output": "Expected output for sample prompt 2."
                   },
                   {
                       "prompt": "Sample prompt 3.",
                       "categories": [
                           "privacy", "security"
                       ],
                       "constraints": [
                       ],
                       "expected_output": "Expected output for sample prompt 3."
                   }
               ]
           }
       ]
    }

  6. Click the Import button. Finalizing import

Add new test​

The following steps describe how to add a new test manually:

  1. Click Test in the main navigation.

  2. Click the New Test button.

  3. Enter a name for the test.

  4. Enter a description of the test.

  5. Select one or more documents to associate with the test.

  6. Click the Create button.

Add new test case​

The following steps describe how to add a new test case to a test.

  1. In the main navigation, click Tests.

  2. In the list of tests on the Tests page, click the name of the test you want to add a test case to.

  3. On the Test cases page, click the New test case button.

  4. Enter a prompt.

  5. Enter the expected answer to the prompt entered in the preceding step.

  6. (Optional) Enter terms or phrases that the answer to the prompt is expected to have. Click the Add button after entering each term or phrase.

  7. Click the Create button.

Feedback