Skip to main content

Tests

Overview​

In H2O Eval Studio, a test is a collection of documents (that is, a corpus) along with prompts that are relevant to the corpus, ground truth, constraints, and other parameters that are used to evaluate a RAG or LLM model. There are two ways how to create a test:

Add a new test​

The following steps describe how to add a new test manually:

  1. Click Test in the main navigation.

  2. Click the New Test button. Add a new test

  3. Enter a name for the test.

  4. Enter a description of the test.

  5. Select one or more documents to associate with the test.

  6. Click the Create button.

Add new test case​

The following steps describe how to add a new test case to a test:

  1. In the main navigation, click Tests.

  2. In the list of tests on the Tests page, click the name of the test you want to add a test case to.

  3. On the Test cases page, click the New test case button. Add new test case

  4. Enter a prompt.

  5. Enter the expected answer to the prompt entered in the preceding step.

  6. (Optional) Enter terms or phrases that the answer to the prompt is expected to have - see conditions reference in the Text Matching evaluator documentation for more details. Click the Add button after entering each term or phrase.

  7. Click the Create button.

New test case is created in the Test.

Import new test case from library​

H2O Eval Studio provides a library with 1.000.000+ test cases which can be (re)used when creating new tests. The following steps describe how to import a test case from the library:

  1. In the main navigation, click Tests.

  2. In the list of tests on the Tests page, click the name of the test you want to add a test case to.

  3. On the Test cases page, click Actions button and choose Add from library drop down.

  4. Right side bar opens - choose the number of test cases to add (Count) and Test Suite from which to add the test cases. Add test cases

  5. Click Add button.

Test cases are added to the Test.

Import an existing test suite​

To import a Test Suite in H2O Eval Studio, follow these steps:

  1. In the main navigation, click Tests.

  2. Click the Import Test Suite button. Import Test Suite

  3. Enter a name prefix for the tests in the Test Suite.

  4. Enter a description of the tests in the Test Suite.

  5. Enter the Test Suite JSON or URL. The following is an example of how the Test Suite JSON should be formatted:

    {
       "name": "Sample Test Suite",
       "description": "This is a sample test suite.",
       "tests": [
           {
               "documents": [
               ],
               "test_cases": [
                   {
                       "prompt": "Sample prompt 1.",
                       "categories": [
                           "privacy", "security"
                       ],
                       "constraints": [
                       ],
                       "expected_output": "Expected output for sample prompt 1."
                   },
                   {
                       "prompt": "Sample prompt 2.",
                       "categories": [
                           "privacy", "security"
                       ],
                       "constraints": [
                       ],
                       "expected_output": "Expected output for sample prompt 2."
                   },
                   {
                       "prompt": "Sample prompt 3.",
                       "categories": [
                           "privacy", "security"
                       ],
                       "constraints": [
                       ],
                       "expected_output": "Expected output for sample prompt 3."
                   }
               ]
           }
       ]
    }

  6. Click the Import button. Finalizing import

Feedback