Add a model host

1. Click Model hosts in the main navigation.

2. Click the + New Model Host button.

3. Enter a name for the model host.

4. Enter a description of the model host.

5. Select one of the following model host types. For more information, see Supported LLM hosts.

   • h2oGPTe RAG
   • h2oGPTe LLM
   • h2oGPT LLM
   • h2o LLMOps
   • Ollama
   • OpenAI chat
   • OpenAI chat (Azure)
   • OpenAI API chat
   • OpenAI Assistant
   • Amazon Bedrock

6. Enter the URL address and/or access keys of the model host.

7. Click the Create button.

Supported LLM hosts

The following section describes the LLM hosts that are currently supported by H2O Eval Studio.

h2oGPTe RAG: Enterprise h2oGPTe LLM Host

Enterprise h2oGPTe is RAG product that uses LLMs to generate responses. You can use H2O Eval Studio to evaluate the performance of LLMs hosted by Enterprise h2oGPTe.

Parameters:

• Host URL: The URL address of the Enterprise h2oGPTe.
• API Key: The API key for the Enterprise h2oGPTe.
• Advanced Settings: Additional advanced settings for the Enterprise h2oGPTe. The setting described below are optional. They can be used to customize the h2oGPTe collections and chats creation. Use the h2oGPTe client documentation as the reference. Use only the settings that are relevant to your use case. The settings are in JSON format in order to be forward compatible with h2oGPTe API. The parameters are passed to the h2oGPTe client API as is:

{
    "embedding_model": null,
    "prompt_template_id": null,
    "system_prompt": null,
    "pre_prompt_query": null,
    "prompt_query": null,
    "pre_prompt_summary": null,
    "prompt_summary": null,
    "llm": null,
    "llm_args": {
        "temperature": 0.0,
        "seed": 0,
        "top_k": 1,
        "top_p": 1.0,
        "repetition_penalty": 1.07,
        "max_new_tokens": 1024,
        "min_max_new_tokens": 512
    },
    "self_reflection_config": null,
    "rag_config": null,
    "timeout": null
}

Advanced settings example - change RAG type to hyde2:

{
    "rag_config": {
        "rag_type":"hyde2"
    }
}

h2oGPTe LLM: Enterprise h2oGPTe LLM Host

Enterprise h2oGPTe is RAG product that uses LLMs to generate responses. You can use H2O Eval Studio to evaluate the performance of LLMs hosted by Enterprise h2oGPTe without the RAG functionality. This is useful when you want to evaluate the performance of LLMs without the RAG.

Parameters:

• Host URL: The URL address of the Enterprise h2oGPTe.
• API Key: The API key for the Enterprise h2oGPTe.
• Advanced Settings: Additional advanced settings for the Enterprise h2oGPTe. The setting described below are optional. They can be used to customize the h2oGPTe chats creation. Use h2oGPTe client documentation as the reference. Use only the settings that are relevant to your use case. The settings are in JSON format in order to be forward compatible with h2oGPTe API. The parameters are passed to the h2oGPTe client API as is:

{
    "temperature": 0.0,
    "seed": 0,
    "top_k": 1,
    "top_p": 1.0,
    "repetition_penalty": 1.07,
    "max_new_tokens": 1024,
    "min_max_new_tokens": 512
    "timeout": null
}

Advanced settings example - change RAG type to hyde2:

{
    "temperature": 9.0
}

h2oGPT LLM Host

h2oGPT is a product that is used to host LLMs. You can use H2O Eval Studio to evaluate the performance of LLMs hosted by the H2O GPT.

Parameters:

• Host URL: The URL address of the h2oGPT.
• API Key: The API key for the h2oGPT.
• Advanced Settings: Additional advanced settings for the h2oGPT. The setting described below are optional. They can be used to customize the h2oGPT chats. Use the h2oGPT client documentation as the reference. Use only the settings that are relevant to your use case. The settings are in JSON format in order to be forward compatible with h2oGPTe API. The parameters are passed to the h2oGPT client API as is:

{
    "messages": null,
    "frequency_penalty": null,
    "function_call": null,
    "functions": null,
    "logit_bias": null,
    "logprobs": null,
    "max_tokens": null,
    "n": null,
    "presence_penalty": null,
    "response_format": null,
    "seed": null,
    "stop": null,
    "stream": null,
    "temperature": null,
    "tool_choice": null,
    "tools": null,
    "top_logprobs": null,
    "top_p": null,
    "user": null,
    "extra_headers": null,
    "extra_query": null,
    "extra_body": null,
    "timeout": null
}

H2O LLMOps Host

H2O LLMOps is a product that is used to host LLMs. H2O Eval Studio can be used to evaluate the performance of LLMs hosted by the H2O LLMOps. LLMs can be deployed using the H2O LLMOps deployer.

Parameters:

• Host URL: The URL address of the H2O LLMOps.
• API Key: The API key for the H2O LLMOps.
• Advanced Settings: Additional advanced settings for the H2O LLMOps. The setting described below are optional. They can be used to customize the H2O LLMOps chats. The settings are in JSON format in order to be forward compatible with H2O LLMOps. The parameters are passed to the H2O LLMOps API as is:

{
    "messages": null,
    "frequency_penalty": null,
    "function_call": null,
    "functions": null,
    "logit_bias": null,
    "logprobs": null,
    "max_tokens": null,
    "n": null,
    "presence_penalty": null,
    "response_format": null,
    "seed": null,
    "stop": null,
    "stream": null,
    "temperature": null,
    "tool_choice": null,
    "tools": null,
    "top_logprobs": null,
    "top_p": null,
    "user": null,
    "extra_headers": null,
    "extra_query": null,
    "extra_body": null,
    "timeout": null
}

OpenAI Assistant: Open AI Assistants with File Search (formerly Retrieval) Tool LLM Host

OpenAI Assistants with the File Search tool (or deprecated Retrieval tool) is a RAG system from OpenAI which hosts LLMs to generate the answers. H2O Eval Studio can be used to evaluate the performance of LLMs hosted by Open AI Assistants with the tools.

OpenAI Assistants with Retrieval tool is available when openai client library version 1.20 and below is installed. Open AI Assistants with the File Search tool is available when newer openai client library is installed. H2O Eval Studio automatically detects the tool availability and uses the appropriate LLM/RAG client and tool when connecting to the OpenAI Assistants.

However, there are important limitations when using the OpenAI Assistants:

• The OpenAI Assistants version 2 with the File Search tool do not provide retrieved contexts when H2O Eval Studio builds the test lab. Therefore the retrieved_contexts field in the test lab will be empty and evaluators which require the retrieved contexts should not be used as they will not work as expected - their results will be based on the generated responses only and might be incorrect. H2O Eval Studio will generate problems for the test lab with the empty retrieved contexts.

• The OpenAI Assistants version 1 with the Retrieval tool is deprecated and will be removed in the future. OpenaAI’s endpoint provided the the retrieved context in the past. However, as part of the deprecation process the retrieved context is no longer provided by the OpenAI’s endpoint as well which brings evaluators accuracy issues described above.

Parameters:

• API Key: The API key for the Open AI Assistants.
• Advanced Settings: Additional advanced settings for the Open AI Assistants. The setting described below are optional. They can be used to customize the Open AI assistants, threads and chats creation. The settings are in JSON format in order to be forward compatible with Open AI Assistants API. The parameters are passed to the Open AI Assistants client API as is:

{
    "assistant_kwargs": {
        "name": null,
        "description": null,
        "instructions": null,
        "tools": null,
        "metadata": null,
        "extra_headers": null,
        "extra_query": null,
        "extra_body": null,
        "timeout": null
    },
    "thread_kwargs": {
        "messages": null,
        "metadata": null,
        "extra_headers": null,
        "extra_query": null,
        "extra_body": null,
        "timeout": null
    },
    "run_kwargs": {
        "additional_instructions": null,
        "additional_messages": null,
        "instructions": null,
        "max_completion_tokens": null,
        "max_prompt_tokens": null,
        "metadata": null,
        "response_format": null,
        "stream": null,
        "temperature": null,
        "tool_choice": null,
        "tools": null,
        "truncation_strategy": null,
        "extra_headers": null,
        "extra_query": null,
        "extra_body": null,
        "timeout": null
    }
}

OpenAI chat: Open AI Chat LLM Host

Open AI Chat is a product that hosts LLMs. You can use H2O Eval Studio to evaluate the performance of LLMs hosted by the Open AI Chat.

Parameters:

• API Key: The API key for the Open AI Chat.
• Advanced Settings: Additional advanced settings for the Open AI Chat. The setting described below are optional. They can be used to customize the Open AI Chat chats creation. The settings are in JSON format in order to be forward compatible with Open AI Chat API. The parameters are passed to the Open AI Chat API as is:

{
    "messages": null,
    "frequency_penalty": null,
    "function_call": null,
    "functions": null,
    "logit_bias": null,
    "logprobs": null,
    "max_tokens": null,
    "n": null,
    "presence_penalty": null,
    "response_format": null,
    "seed": null,
    "stop": null,
    "stream": null,
    "temperature": null,
    "tool_choice": null,
    "tools": null,
    "top_logprobs": null,
    "top_p": null,
    "user": null,
    "extra_headers": null,
    "extra_query": null,
    "extra_body": null,
    "timeout": null
}

OpenAI chat (Azure): Microsoft Azure Open AI Chat LLM Host

Microsoft Azure hosted Open AI Chat is a product that hosts LLMs. You can use H2O Eval Studio to evaluate the performance of LLMs hosted by the Open AI Chat.

Parameters:

• Host URL: The URL address of the Microsoft Azure Open AI Chat.
• API Key: The API key for the Microsoft Azure Open AI Chat.
• Environment ID:: ID of the Open AI Chat environment hosted by Microsoft Azure used as the LLM model name.
• Advanced Settings: Additional advanced settings for the Microsoft Azure Open AI Chat. The setting described below are optional. They can be used to customize the Microsoft Azure Open AI Chat chats creation. The settings are in JSON format in order to be forward compatible with Microsoft Azure Open AI Chat API. The parameters are passed to the Microsoft Azure Open AI Chat API as is:

{
    "messages": null,
    "frequency_penalty": null,
    "function_call": null,
    "functions": null,
    "logit_bias": null,
    "logprobs": null,
    "max_tokens": null,
    "n": null,
    "presence_penalty": null,
    "response_format": null,
    "seed": null,
    "stop": null,
    "stream": null,
    "temperature": null,
    "tool_choice": null,
    "tools": null,
    "top_logprobs": null,
    "top_p": null,
    "user": null,
    "extra_headers": null,
    "extra_query": null,
    "extra_body": null,
    "timeout": null
}

OpenAI API chat: Open AI Chat API Compatible LLM Host

H2O Eval Studio can be used to evaluate the performance of LLMs hosted by any OpenAI API compatible LLM host.

Parameters:

• Host URL: The URL address of the Open AI Chat API.
• API Key: The API key for the Open AI Chat API.
• Advanced Settings: Additional advanced settings for the Open AI Chat API. The setting described below are optional. They can be used to customize the Open AI Chat chats creation. The settings are in JSON format in order to be forward compatible with Open AI Chat API. The parameters are passed to the Open AI Chat API as is:

{
    "messages": null,
    "frequency_penalty": null,
    "function_call": null,
    "functions": null,
    "logit_bias": null,
    "logprobs": null,
    "max_tokens": null,
    "n": null,
    "presence_penalty": null,
    "response_format": null,
    "seed": null,
    "stop": null,
    "stream": null,
    "temperature": null,
    "tool_choice": null,
    "tools": null,
    "top_logprobs": null,
    "top_p": null,
    "user": null,
    "extra_headers": null,
    "extra_query": null,
    "extra_body": null,
    "timeout": null
}

ollama LLM Host

H2O Eval Studio can be used to evaluate the performance of LLMs hosted by ollama LLM hosts.

Before you can create a leaderboard, you need to add a model. This page describes how to add a new model in H2O Eval Studio.

Parameters:

• Host URL: The URL address of the ollama.
• Advanced Settings: Additional advanced settings for the ollama. The setting described below are optional. They can be used to customize the ollama chats creation. The settings are in JSON format in order to be forward compatible with ollama API. The parameters are passed to the ollama API as is:

{
    "messages": null,
    "frequency_penalty": null,
    "function_call": null,
    "functions": null,
    "logit_bias": null,
    "logprobs": null,
    "max_tokens": null,
    "n": null,
    "presence_penalty": null,
    "response_format": null,
    "seed": null,
    "stop": null,
    "stream": null,
    "temperature": null,
    "tool_choice": null,
    "tools": null,
    "top_logprobs": null,
    "top_p": null,
    "user": null,
    "extra_headers": null,
    "extra_query": null,
    "extra_body": null,
    "timeout": null
}

Amazon Bedrock LLM Host

The current implementation of the Amazon Bedrock client supports RAG with a predefined collection ID that corresponds to knowledgeBaseId. Only Anthropic Claude models are supported for usage in the RAG.

Parameters:

• AWS Access Key ID: The AWS Access Key ID for the Amazon Bedrock.
• AWS Secret Access Key: The AWS Secret Access Key for the Amazon Bedrock.
• AWS Session Token: The AWS Session Token for the Amazon Bedrock.
• AWS Region: The AWS Region for the Amazon Bedrock.
• Advanced Settings: Additional advanced settings for the Amazon Bedrock. The setting described below are optional. They can be used to customize the Amazon Bedrock chats creation. The settings are in JSON format in order to be forward compatible with Amazon Bedrock API. The parameters are passed to the Amazon Bedrock API as is:

{
     "guardrailConfiguration": {
        "guardrailId": "string",
        "guardrailVersion": "string"
     },
     "inferenceConfig": {
        "textInferenceConfig": {
           "maxTokens": number,
           "stopSequences": [ "string" ],
           "temperature": number,
           "topP": number
        }
     },
     "promptTemplate": {
        "textPromptTemplate": "string"
     },
     "orchestrationConfiguration": {
            "queryTransformationConfiguration": {
                "type": "QUERY_DECOMPOSITION"
            }
     },
     "retrievalConfiguration": {
        "vectorSearchConfiguration": {
           "filter": { ... },
           "numberOfResults": number,
           "overrideSearchType": "string"
        }
     }
}

---

Feedback

• Submit and view feedback for this page
• Send feedback about H2O Eval Studio to cloud-feedback@h2o.ai