User Chat Quality

The User Chat Quality metric evaluates the quality and clarity of user inputs in a chat use case. This metric assesses whether user messages are concise, relevant, and clearly convey intent to facilitate effective responses from the agent. An LLM is used as an evaluator to score user inputs on factors like clarity, completeness, and adherence to the conversational context.

Required Column in Dataset:

• Chat: A single column containing the full chat conversation, including both user messages and agent responses, to provide context for evaluating user input quality within the flow of interaction.

Interpretation:

A higher User Chat Quality score indicates that user inputs are clear, well-structured, and contribute positively to the conversation flow, aiding the agent in generating accurate responses. This metric helps identify potential user-side issues that may affect chat performance, such as ambiguous or incomplete messages.

Metric Execution via UI:

To execute this metric, select User Chat Quality from the list of available metrics in the UI, and configure settings to evaluate user inputs within the full conversation context.

Example:

• Chat:

  • User: “I need to know more about the Italian restaurants here.”

  • Agent Response: “Of course! Are you looking for something casual or fine dining?”

• Metric Score: 0.85 Reasoning: The user’s input is clear and relevant, specifying the interest in Italian restaurants, which provides a clear direction for the agent. However, it could be improved by specifying further details, such as preferences for dining type, which would yield an even higher score for optimal clarity.