Write test cases
Writing test scenarios to assess your LLM product behavior
A test case
is a specific set of conditions that outline the actions needed to be carried out successfully by your LLM product.
Test cases can be grouped into test suites
, which are groups of related test cases.
The test case repository
is where you create and manage all test cases and test suites, enabling you to better organize your test strategy as you scale.
Ottic currently supports 2 types of tests:
- LLM Test: tests the output generated directly by the LLM.
- Agent Test: tests the output produced by an agent interacting with the LLM.
Creating a test case
To create a test case, in the Test Cases menu:
- Click “Create new test case”.
- Name the test case.
- Write a short description for the test case
- Choose a type of test (LLM Test or Agent Test).
Creating a LLM test case
Choose LLM test, when creating a test case to test the LLM output.
You need to add more details to build you test case. While creating a LLM test case, you need to specify:
-
The prompt and version you’ll be evaluating
-
Input or conversations you´ll be testing
-
The pass criteria containing the expected outcomes
IMAGES
Creating an Agent test
Choose Agent test, when creating a test case to test an agent workflow.
While creating an Agent test case, you need to specify:
-
Agent URL endpoint
-
Input or conversations you´ll be testing
-
The pass criteria containing the expected results
Editing, cloning and deleting test cases
The test case repository
contains all your test cases and its details.
Clicking a test case opens a details drawer on the left, where you can:
-
Edit the test case
-
Clone the test case
-
Delete the test case
TC -XX
.This number cannot be changed.Viewing run history of a test case
With a test case open, click the Run history tab to view execution details:
-
When it was executed (Date and time)
-
Test runs it is associated with (ID and title of the Run)
-
Results history (passed of failed)
Organize test cases in test suites
As you test strategy scales, your number of test cases will grow.
You can use test suites
to organize test cases.
You can drag and drop test cases to different suites and positions.
Next steps
When you are ready to build your test cases, you can start testing them in the Test Run menu.