Warmly’s AI Test & Train feature lets you safely test and refine your AI agent’s responses before it ever talks to a real visitor. Think of it like coaching: ask the agent a question, review its answer, give feedback, and watch it improve — all without affecting live conversations.

You’ll find Test & Train inside AI Studio, the central place where you configure your agent’s persona, knowledge base, and behavior.

An AI agent is only as good as the answers it gives. Launching one without training is like putting a new sales rep on the phones without onboarding — they’ll try, but they’ll miss important details.

With Test & Train, you can:

• Catch bad answers before prospects see them. If the agent mishandles a pricing question or goes off-brand, you’ll catch it before a lead does.

• Improve tone and accuracy quickly. No prompt engineering required — just tell the agent what to change in plain English.

• Create a library of “golden” responses. Approved answers become examples the agent learns from over time.

• Understand where answers come from. Source citations show which knowledge base documents informed each response, making it easy to fix incorrect or outdated content at the source.

Before you start

Make sure you've completed the following in AI Studio:

1. Set up your agent persona — name, persona, tone

2. Added your knowledge base — uploaded documents, website pages, or FAQs the agent can reference

3. Defined your agent rules — any instructions about what the agent should or shouldn't say

If you haven't done these yet, start with Setting Up Your AI Agent.

Step-by-Step Instructions

Step 1: Open AI Studio and navigate to Test & Train

  1. Go to Settings in your Warmly dashboard

  2. Click Inbound Chat Flow

  3. Select your chat flow and click into AI Studio

  4. Click the Training tab

  You'll see a list of your training sessions (or an empty state if this is your first time).

Step 2: Create a new training session

1. Click New Training Session

2. Enter a trigger question — this should be a question your prospects actually ask. Think: "What does your product do?", "How much does it cost?", "Do you integrate with Salesforce?"

3. Hit Submit

  The agent will immediately generate its first response using your knowledge base.

Step 3: Review the response

Read the agent’s response carefully and check a few things:

• Accuracy: Does the answer reflect what you actually offer or do?

• Tone: Does it sound natural, or is it too robotic or too casual?

• Relevance: Did it stay on topic, or drift into unrelated details?

• Sources: What documents did it cite? You can click any source to preview the exact section the agent used.

From here, you have two options:

If the response looks good → Approve it
Click the ✅ thumbs up to approve the response. This turns the session into a “golden example” the agent can reference in future training.

If the response needs improvement → Refine it
Move on to Step 4 to guide the agent toward a better answer.

Step 4: Give feedback and iterate

1. Click Refine

2. In the feedback box, explain what the agent should change — just write it in plain English. For example:

   • “Be more concise — keep it under 3 sentences.”

   • “Lead with the ROI instead of the features.”

   • “Don’t mention pricing — tell them to contact our team for pricing details.”

3. Click Submit.

The agent will generate a new response that incorporates your feedback. Each round builds on the previous one, so you’re improving the answer step by step rather than starting from scratch.

Repeat this process until the response looks the way you want.

Step 5: Approve the final response

Once the response looks right, click ✅ Approve.

The session will be saved, and the agent will use it as a reference example when answering similar questions in the future.

Step 6: Repeat for your top questions

Build a library of approved responses across the questions your prospects ask most often, such as:

• Product or feature questions

• Pricing objections

• Competitive comparisons

• Integration questions

• Meeting booking prompts

The more approved examples you create, the more consistent, accurate, and on-brand your agent will become.

Use Cases

Here’s how sales and marketing teams typically use Test & Train day to day:

• Pre-launch QA: Before launching a new agent or rolling out a major knowledge base update, run through your 10 most common prospect questions to make sure everything works as expected.

• After adding new content: Just added a pricing page or product doc to the knowledge base? Test the agent on pricing-related questions right away to confirm it’s pulling the correct information.

• Brand voice calibration: If the agent sounds too formal or too vague, run a few training sessions and guide it with tone feedback until it sounds like your best sales rep.

• Competitive positioning: Test how the agent responds to questions like “How do you compare to [Competitor]?” and refine the answer until it’s clear, confident, and on-message.

• Onboarding new agents: When creating a new AI agent persona for a specific segment or use case, run training sessions to make sure it handles that segment’s typical questions correctly before going live.