This quickstart guide walks you through each essential step in Ottic:

1

Insert your model provider API key

Go to LLM Models and create model with your model provider API key.

This is essential enabling you to run LLM-as-a-judge evaluations.

2

Create your first prompt

In the Prompts menu:

  • Click “Create new prompt”;
  • Add the prompt you will test;
  • Save the prompt you created.
See more on prompts in the section create prompts
3

Write your first test case

In the Test Cases menu:

  • (1) Click “Create new test case”;
  • (2) Define of details of your test case (title and description);
    • Choose the prompt you just created;
    • Add the input (the message you want to test);
    • Add the expected output (the response expected);
  • (3) Save your test case.

See more on test cases in the section write test cases
We provide an API to migrate your test cases to Ottic. See more on API reference
We provide white-glove support for test pipeline migration. Contact us.
4

Run your first test

In the Test Runs menu:

  • Click “Start new test run”;
    • Select the test case you just created;
    • Click, “Run” to run your test;
  • See the result (passed, failed);
  • Open the executed test case to explore details and reasoning.
See more on test runs in the section execute test runs
5

Publish your prompt to production

You can publish your prompt directly to production from Ottic.

In the Prompts menu:

  • Open the prompt you just tested;
  • Select the version you want to publish from the dropdown;
  • Click “Publish”.
To publish prompts to production from Ottic, install Ottic SDK. See more on Integrations

To unlock the full potential of Ottic, you’ll find detailed guides on the next sections.

IntroductionStart a project