This document describes how to debug and test Agents in DataBuddy, covering ChatUI preview, Trace tracking, evaluators, evaluation runs, and intelligent tuning (ALHF) to help you quickly identify issues and continuously improve Agent quality.
Overview
Agent debugging and testing are accomplished through four levels:
Layer | Problem addressed | Entries |
ChatUI preview experience | Verify whether the Agent works properly through direct conversation. | Right side of the Agent overview page |
Trace | Troubleshoot why a specific call is slow / fails / returns incorrect answers. | MLflow experiment > Tracking |
Evaluator | Automated scoring (relevance, safety, correctness) and large-scale evaluation | MLflow experiment > Evaluator |
Intelligent Tuning (ALHF) | Convert human feedback into Agent behavior improvements. | Agent details page > Intelligent Tuning |
Prerequisites
An Agent has been created and started. For details, see Creating and Managing Agents.
The current account has CAN_EDIT permission on the MLflow experiment associated with the Agent.
Use Limits
Limit | Description |
Number of traces per experiment | Supports up to 100,000 traces. |
LLM evaluator type | Supports LLM evaluators (relevance / safety / correctness). Code evaluator support is coming soon. |
Evaluation run comparison | Supports single Run view. Cross-Run comparison will be supported in a later version. |
Evaluation feedback / expectation cards | For the upper limit on the number of feedback and expectation entries per Trace, see the platform documentation. |
ChatUI Preview Experience
Entries
When you go to the Agent Overview page, a ChatUI template preview window is embedded on the right by default. You can also open it independently in a new tab.
Capability List
Capability | Description |
Sending Messages | Enter text to input, press Enter to send, and press Shift + Enter to start a new line. You cannot send a new message while the model is responding. You can click Stop. |
Thinking process | Before the model responds, "Thinking" is displayed. After thinking is complete, intent understanding, execution steps, and model output are displayed. |
Tool Invocation Display | Distinguish between Tools / Agent calls. During the call, Loading is displayed. After completion, a check mark and the duration are shown. Click to expand and view input / output / duration. |
Streaming Output | Model output is rendered in streaming mode and supports Markdown rich text (code blocks / tables / lists). |
Response Actions | Like / Dislike (Results Are Sent to ML Trace) / Copy Response |
Session Context | The model automatically associates context within a single session. |
Create Session | Close the current session and start a new session. |
Session History | A database that can store session history must be specified. For detailed configuration, see the platform documentation. |
Recommended Questions | In the initial state, fixed recommended questions are displayed. Click a question to start a conversation directly. |
Tip
ChatUI is the conversation interface that users actually see after an Agent goes live. We recommend that you configure the Agent's opening message and recommended questions during the debugging phase to improve the first-use experience after launch.
Trace Tracking (Observability)
Each Agent Q&A session generates a Trace, which is automatically stored in the associated MLflow experiment. A Trace contains the complete call chain: model calls, tool calls, sub-Agent calls, Token consumption, duration, and more.
Accessing the Entry
1. On the Agent overview page, click the Tracking and Evaluation module shortcut.
2. Alternatively, open the experiment from the MLflow experiment list, and then go to Observability > Tracking on the left.
Trace List Fields
Field | Description |
Trace ID | Globally unique ID. Click to open Trace details. |
Request | Request content |
Response | Response result |
Session | Session ID |
User | Execution username |
Trace name | Trace name |
Version | Agent version. Click to go to the corresponding version. |
Tokens | Number of Tokens Consumed |
Execution time | Execution Duration |
Request Time | Request Time |
Run name | Associated run name (for example, from an evaluation run). If none, leave it empty. |
Source | Entry point or script name that generates the Trace |
State | Status: Normal / Error |
You can filter by custom fields and perform fuzzy searches by Request.
Trace Details: Overview
Click a Trace ID to open the details dialog. The Overview tab displays the following information:
Area | Description |
Basic Information | Status (OK / ERROR / IN_PROGRESS) / Trace ID / Token consumption / Trace duration |
Inputs | User input content and context |
Call chain | Actual call chain of the large model / sub-Agent / tools; collapsed by default, click to expand and view the input / output / duration of each step. |
Output | Final output from the Agent to the user |
Evaluation Item | Evaluation results such as relevance, safety, and user feedback (display only, not editable) |
Trace Details: Details & Timeline
Switch to the Details & Timeline Tab to view all Spans displayed in a tree structure. Each Span is an independent operation unit, and its types include model calls, tool calls, and Agent calls. Each Span provides:
Chat: A conversation view is automatically rendered when the Span input and output conform to the OpenAI Chat Completions message format.
Input and Output: Displays the Span input information and output content as needed, with JSON automatically formatted.
Attributes: Displays structured attributes of models, tools, and more in key-value pairs.
Events: Discrete events that occur during Span execution, such as exception stack traces and error messages.
You can search for keywords to filter the left-side directory tree and locate debugging information across nodes.
Trace Details: Right-Side Evaluation Area
The right side of the details dialog for each Trace is the evaluation area, where you can add the following to the Trace:
Evaluation Type | Purpose |
Feedback | Quality scoring of AI-generated answers (for example, "Is this answer good or not") |
Expectation | Constraints that the ideal answer to the question should satisfy (for example, "Must cite the knowledge base") |
Each feedback and expectation supports the following fields:
Field | Value Range |
Evaluation Type | Feedback / expectation |
Evaluation Name | No more than 100 characters; names containing . characters are not supported. |
Data types | Boolean / string / number |
Evaluation Value | String, no more than 500 characters |
Evaluation Basis | Provide the reason for this evaluation for subsequent tracing/training, no more than 500 characters. |
Feedback and expectation cards can be expanded, collapsed, edited, and deleted.
Evaluator
An evaluator is the logic for automated scoring of Agent output. Currently, LLM evaluators are supported, and code evaluators will be supported in a later version.
Accessing the Entry
1. Go to the MLflow experiment associated with the Agent.
2. In the left sidebar, go to Evaluation > Evaluators.
Preset LLM Evaluators
The platform provides three types of LLM evaluators:
Type | Evaluation Target | Output |
Correctness | Whether the answer is factually correct (no custom Prompt required; select from the dropdown). | yes / no |
Relevance | Whether the answer addresses the question (without considering correctness / completeness). | yes / no |
Security | Whether the content violates security policies (hate speech, harassment, incitement to violence, and so on). | yes / no |
Creating an LLM Evaluator
Step 1: Opening the Create Dialog
In the evaluator list, click New LLM Evaluator.
Step 2: Configuring General Settings, Evaluation Criteria, and Automatic Evaluation
Tab | Configuration Item |
General | Evaluator Name, Evaluation Description |
Evaluation Criteria | Select Evaluation Model (select an existing large model from the dropdown list), Select Built-in LLM Evaluator (correctness / relevance / safety). |
Automatic Evaluation | Whether to Enable Automatic Evaluation, Sampling Rate, Filter String (automatically run evaluation only for traces that meet the filter criteria) |
Step 3: Saving and Viewing the Evaluator Card
After saving, return to the evaluator list, where each card displays:
Evaluator name
Evaluator type (LLM evaluator)
Whether to run on all future traces
Sampling rate (calculated based on configured Trace results)
Evaluate the enabled or disabled status of tracking.
Edit button, More (Delete)
Cards can be expanded and collapsed, and edits take effect immediately.
Evaluation Runs
Evaluation runs are used to display and manage the results of multiple evaluations performed on an Agent, making it easy to compare the performance of different versions and prompt configurations.
Accessing the Entry
Go to MLflow Experiments → Evaluation > Evaluation Runs.
Evaluation Run List
Field | Description |
Run name | Custom name (for example, "Manual Review v3") |
Created at | Date and time when the evaluation was created |
Dataset | Name of the associated evaluation dataset, clickable to navigate |
Version | Associated Agent version. If none, - is displayed. |
Status color indicator | Colored dots in red / green / yellow / brown and other colors, used as custom markers |
Built-in LLM evaluation dimension | When the experiment is configured with three built-in evaluators, the list displays the evaluation results for the corresponding dimensions. |
Operations:
Delete: Select one or more runs to delete (confirmation required).
Compare: Select 2 runs to enter the comparison view (supported in a later version).
Refresh: Fetch the latest data.
Single Run Details: Sessions and Traces
After you click a run, all sessions and traces under that run are loaded on the right.
Field | Description |
Session | Displayed only when Group by session is enabled. |
Trace ID | Uniquely identifies a single model call. Click to expand Trace details. |
Request | User input |
Response time | Model response |
Execution Time | Accurate to milliseconds |
Status | Success / Error |
You can search by keywords such as Trace ID / request and response.
You can select the attributes, evaluation, and expectation fields to display through column configuration. By default, 3 evaluation and expectation items are displayed, and you can customize the number in the column configuration.
Intelligent Tuning (ALHF)
Intelligent tuning is a core differentiator of DataBuddy for business personnel, enabling Agents to optimize automatically through a closed loop of human feedback + AI recommendations.
Four Stages of Tuning
1. Configuration phase: Provide reference Q&A pairs (data sources: Trace history / file upload / manual input).
│
▼
2. Feedback phase: Evaluate recommended Q&A as "Good / Bad / Skip".
│
▼
3. Optimization phase: The system automatically extracts semantic memory and generates recommended Guidelines.
│
▼
4. Save and update: Automatically optimize the Agent's Prompt, MCP description, and other elements to improve Agent performance.
Entries
1. On the Agent overview page, go to the Intelligent Tuning entry card and click Start Tuning.
2. Alternatively, in the top navigation of the Agent details page, click the Intelligent Tuning Tab.
Step 1: Configuring Reference Q&A Pairs
Go to the Q&A Configuration panel and select the source of the Q&A pairs:
Source | Scenario | Key operation points |
Import from Trace history | Agent has existing online request records. | Select in the transfer dialog box, and choose at least 3 traces. |
Upload from local file. | Question-answer pairs are stored in local files. | Download the template, upload xlsx/csv files, with a single file no larger than 200 MB. |
Manual entry | Cold start with no historical data | Two input boxes per row (question + expected answer), with at least 3 complete pairs. |
On the preview page, you can confirm the quantity again and delete entries while keeping at least 3 entries. You can also manually add questions and expected answers.
Tip
When a save-and-overwrite operation is performed, the IDs of entries whose questions are not modified remain unchanged, and the original feedback records and feedback history are not lost.
Step 2: Recommending Feedback
The system recommends conversations for user feedback based on the following logic:
Typical scenarios not yet covered by the current Guidelines.
Outputs with low model confidence.
Cold start phase: Directly display the first 50 configured Q&A pairs.
Each card supports three operations:
Operation | Description |
Good | Positive sample, stored in episodic memory to reinforce Agent behavior |
Not Good | Negative sample. Expand the text box and press Ctrl + Enter to submit the reason for dissatisfaction and the expected answer. |
Skip | Recorded but not used as a basis for optimization; if the system determines that recommendation is still needed, it can continue to push recommendations later. |
After a card is evaluated, a fly-out animation is displayed, and the title bar shows "X Remaining" in real time. You can operate the next card only after the animation is completed.
Step 3: Q&A Inspection
View all configured Q&A pairs in the Q&A Check module and compare the Agent responses with the expected answers one by one:
Fuzzy search by keyword is supported, covering both questions and expected answers.
You can filter Q&A pairs that have already received feedback (with existing feedback records).
Single item details: Displays the question, expected answer, and actual Agent response. You can like or dislike the response, with the same interaction as in Recommended Feedback.
History Feedback: Displays the last 3 feedback entries, excluding "Skip".
Step 4: Evaluation Criteria and Prompts
Evaluation Guidelines
Define what constitutes a good answer. Each Guideline is also used to:
LLM Judge evaluates and scores.
Constraints for prompt optimization.
The system generates recommended Guidelines based on the following information:
Agent name and description.
Q&A pairs configured by users.
Feedback provided by users in Recommended Feedback and Q&A Inspection.
Operations:
Review each recommended Guideline one by one.
Click Accept to accept, or Reject to reject.
You can also click + Add to add manually.
After acceptance, the button changes to "Accepted" (Accept can no longer be clicked), but accepted Guidelines can be deleted in the editing area.
Instructions
Define the high-level goals and core tasks of the Agent (for example, "You are a professional customer service data analysis assistant..."), and control the response style and content boundaries.
Instructions are directional guidance.
Guidelines are specific evaluation and constraint criteria.
They complement each other, so it is recommended to maintain them together.
Step 5: Saving and Updating
Click the Save and update button in the lower-right corner, and the platform engine automatically breaks down the feedback and rule changes and applies them to:
Prompt
Vector Index
LLM Judge (evaluator)
Evaluation Dataset
MCP Tools (tool configuration)
Agent Config
Tip
There is a delay of about 30 seconds before the current update takes effect. After the update is saved successfully, a message will appear saying "Update saved. It is expected to take effect within 30 seconds. Please go to the preview later." A Go to preview shortcut link will also be provided.
After a successful save, the recommended feedback content is also refreshed asynchronously.
FAQs
Q: What are the differences between the Good, Bad, and Skip actions in intelligent tuning?
A: Good reinforces the current answer pattern (positive sample). Bad requires you to provide a reason and expected answer, which serve as a negative sample and improvement direction. Skip does not participate in optimization, but the system will recommend it again if it is still considered valuable.
Q: Why are the three evaluators for relevance, safety, and correctness preset?
A: These three cover the most common quality dimensions of LLM applications: relevance (whether the question is answered), safety (whether security policies are violated), and correctness (whether facts are accurate). They can meet most initial evaluation needs, and more granular evaluation dimensions can be implemented through custom LLM evaluators.
Q: How do I compare two versions of an Agent during an evaluation run?
A: Cross-Run comparison is not currently supported. In a later version, you will be able to select 2 runs to enter the comparison view. For now, you can assign different version identifiers to different versions and then view the Run details separately for manual comparison.
References
Creating and Managing Agents