# RagaAI ## RagaAI - [Welcome](https://docs.raga.ai/readme.md): Introduction of the RagaAI - An end to end testing platform for GenerativeAI and DiscriminativeAI models - LLM, Computer Vision, NLP and Tabular Data. - [RagaAI Catalyst](https://docs.raga.ai/ragaai-catalyst.md): The automated AI evaluation platform to build safe, reliable and cost efficient GenAI applications. - [User Quickstart](https://docs.raga.ai/ragaai-catalyst/user-quickstart.md): This guide will help you get started with RagaAI Catalyst using any sample dataset, straight from the RagaAI GUI or the Python environment. - [Concepts](https://docs.raga.ai/ragaai-catalyst/concepts.md): RagaAI Catalyst Concepts – GenAI Evaluation & Observability Overview - [Configure Your API Keys](https://docs.raga.ai/ragaai-catalyst/concepts/configure-your-api-keys.md): Set API keys for your LLM models to enable metric evaluations on the Catalyst platform. If you're trying to set up a custom gateway, refer the "Enable Custom Gateway" page. - [Supported LLMs](https://docs.raga.ai/ragaai-catalyst/concepts/supported-llms.md): Supported LLMs – Compatible Models in RagaAI Catalyst - [OpenAI](https://docs.raga.ai/ragaai-catalyst/concepts/supported-llms/openai.md): OpenAI Integration – RagaAI Catalyst Supported LLM - [Gemini](https://docs.raga.ai/ragaai-catalyst/concepts/supported-llms/gemini.md): Google Gemini – RagaAI Catalyst Supported LLM - [Azure](https://docs.raga.ai/ragaai-catalyst/concepts/supported-llms/azure.md): Azure OpenAI – RagaAI Catalyst Supported LLM - [AWS Bedrock](https://docs.raga.ai/ragaai-catalyst/concepts/supported-llms/aws-bedrock.md): AWS Bedrock – RagaAI Catalyst Supported LLM - [ANTHROPIC](https://docs.raga.ai/ragaai-catalyst/concepts/supported-llms/anthropic.md): Anthropic Claude – RagaAI Catalyst Supported LLM - [Catalyst Access/Secret Keys](https://docs.raga.ai/ragaai-catalyst/concepts/catalyst-access-secret-keys.md): Generate access and secret keys and quickly integrate Catalyst within your application's SDK flow. - [Enable Custom Gateway](https://docs.raga.ai/ragaai-catalyst/concepts/enable-custom-gateway.md): Run metric evaluations on your own LLM models, even if you use a custom gateway. - [Uploading Data](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data.md): Understand the process of creating a new project and importing your dataset, including file format requirements and tips to prepare your data for effective LLM testing in the platform. - [Create new project](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/create-new-project.md): Get started on evaluations by creating use-cases specific projects - [RAG Dataset](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/rag-datset.md): Once your project is created, you can upload datasets to it for evaluation. - [Chat Dataset](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/chat-dataset.md): Guide for uploading chat datasets to RagaAI Catalyst - [Prompt Format](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/chat-dataset/prompt-format.md): How to represent conversation prompts and responses when preparing your dataset, ensuring Catalyst correctly interprets each user query and assistant reply during evaluation. - [Logging traces (LlamaIndex, Langchain, etc.)](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/logging-traces.md): Learn how to log execution traces from LlamaIndex or LangChain into RagaAI Catalyst. Capture prompts, responses, and steps for easier debugging and evaluation. - [Trace Masking Functions](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/trace-masking-functions.md): Users can mask keywords and regex patterns using custom Python functions - [Trace Level Metadata](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/trace-level-metadata.md): Add metadata as key-value pairs for your real-time trace logs - [Correlating Traces with External IDs](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/correlating-traces-with-external-ids.md): Users can connect the traces logged on RagaAI Catalyst with their existing logs elsewhere - such as GCP, AWS, or other internal services - [Add Dataset](https://docs.raga.ai/ragaai-catalyst/concepts/uploading-data/add-dataset.md): How to upload or create datasets for RAG, chat testing, or custom evaluations. Follow steps for import and format verification. - [Running RagaAI Evals](https://docs.raga.ai/ragaai-catalyst/concepts/running-ragaai-evals.md): This section contains all information required by users to run RagaAI's automated evaluation metrics as well as guardrails. - [Executing Evaluations](https://docs.raga.ai/ragaai-catalyst/concepts/running-ragaai-evals/executing-evaluations.md): Run various cutting edge evaluation metrics out-of-the-box with a few simple steps - [Compare Datasets](https://docs.raga.ai/ragaai-catalyst/concepts/running-ragaai-evals/compare-datasets.md): Compare results from multiple datasets or runs side by side. Use diff view to spot performance differences across models and dataset versions easily. - [Analysis](https://docs.raga.ai/ragaai-catalyst/concepts/analysis.md): RagaAI Catalyst’s analysis dashboard helps interpret evaluation results with metrics, charts, and visualizations. Drill into test cases, spot failure patterns and gain insights to improve your LLM. - [Embeddings](https://docs.raga.ai/ragaai-catalyst/concepts/embeddings.md): Explore how RagaAI Catalyst uses vector embeddings for semantic analysis. Visualize spaces, compare query–document similarity, and apply metrics to evaluate LLM retrieval quality. - [RagaAI Metric Library](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library.md): This section highlights all the different kinds of evaluation metrics and guardrails available on the RagaAI platform. - [RAG Metrics](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics.md): Overview of RAG metrics in the RagaAI Metric Library. Measure hallucination, faithfulness, and context use to assess LLM accuracy and reliability with external knowledge. - [Hallucination RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/hallucination.md): Check if an LLM adds unsupported or fabricated details. A high hallucination score flags ungrounded content and helps developers spot false outputs. - [Faithfulness RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/faithfulness.md): Measure how well an LLM’s answer stays true to the source material. Ensure factual accuracy in RAG by avoiding distortions or errors. - [Response Correctness RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/response-correctness.md): Assess if the LLM’s response directly answers the user’s question using context. Evaluate accuracy to check if the model found the right information. - [Response Completeness RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/response-completeness.md): Check if the LLM’s answer fully addresses the question. Ensures responses cover all key points without missing details when context allows. - [False Refusal RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/false-refusal.md): Identify the instances where LLM improperly refuses to answer a question despite having sufficient information to do so. Understand how to fine-tune overly cautious or misaligned refusal behaviors. - [Context Relevancy RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/context-relevancy.md): Evaluate how relevant the retrieved context is to the user’s query and the LLM’s answer, ensuring the model uses appropriate and useful information. - [Context Precision RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/context-precision.md): Measure how much of the provided context was relevant to the answer. High context precision means the model used only the most relevant parts with minimal extra info. - [Context Recall RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/context-recall.md): Assess how well the LLM used relevant context. High recall means it captured key details without missing important information. - [PII Detection RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/pii-detection.md): Check LLM outputs for Personally Identifiable Information (names, emails, phone numbers) to prevent sensitive data leaks and ensure privacy compliance. - [Toxicity RAG Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/toxicity.md): Evaluate LLM responses for toxic or offensive language. Flag unsafe outputs and apply filters to ensure safe, respectful interactions. - [Chat Metrics](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/chat-metrics.md): Chat Metrics in the RagaAI Metric Library evaluate conversational AI quality. Measuring instruction adherence, user clarity, agent helpfulness, coherence, and overall dialogue performance. - [Agent Quality Chat Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/chat-metrics/agent-quality.md): Evaluate the AI assistant’s performance based on helpfulness, accuracy, and appropriateness across the full conversation, giving a holistic engagement score. - [Instruction Adherence Chat Metric](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/chat-metrics/instruction-adherence.md): Evaluate how well the AI follows instructions and system prompts. Ensure responses align with user requests, formats, and style guidelines. - [User Chat Quality](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/chat-metrics/user-chat-quality.md): Evaluate if user messages are clear, well-formed, and relevant. High-quality inputs enable better AI answers, while unclear queries may need refinement. - [Text-to-SQL](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-to-sql.md): Overview of Text-to-SQL metrics in RagaAI Metric Library, covering SQL correctness, query ambiguity, and alignment with user intent. - [SQL Response Correctness](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-to-sql/sql-response-correctness.md): Check if the LLM-generated SQL query retrieves the intended data. Evaluate accuracy of the model’s NL-to-SQL conversion against expected results. - [SQL Prompt Ambiguity](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-to-sql/sql-prompt-ambiguity.md): Assess ambiguity in Text-to-SQL queries. High scores show questions that are vague or multi-interpretable, helping flag those needing rephrasing or more context. - [SQL Context Ambiguity](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-to-sql/sql-context-ambiguity.md): Evaluate ambiguity in Text-to-SQL tasks. Check if unclear schema elements (tables, columns, etc.) cause multiple interpretations, requiring clearer context. - [SQL Context Sufficiency](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-to-sql/sql-context-sufficiency.md): Check whether the SQL context provided is enough for accurate answers. Learn to optimize context and schema inputs. - [SQL Prompt Injection](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-to-sql/sql-prompt-injection.md): Protect against prompt injection in SQL tasks. Detect malicious instructions and enforce guardrails for safer execution. - [Text Summarization](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization.md): Evaluate LLM-generated summaries for accuracy and readability. Learn to refine summarization for clarity and completeness. - [Summary Consistency](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/summary-consistency.md): Measure whether summaries stay faithful to the original text. Spot inconsistencies and improve alignment with source content. - [Summary Relevance](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/summary-relevance.md): Assess if generated summaries include key points. Identify missing elements and ensure better focus on essentials. - [Summary Fluency](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/summary-fluency.md): Analyze the readability and flow of summaries. Improve grammar and style for more natural outputs. - [Summary Coherence](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/summary-coherence.md): Check if summaries are logically connected and structured. Spot gaps in flow and enhance overall quality. - [SummaC](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/summac.md): Use SummaC to validate summary alignment with source. Ensure factual and semantic consistency in AI-generated outputs. - [QAG Score](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/qag-score.md): Apply QAG Score to measure summary reliability. Detect factual gaps and improve question-answer consistency. - [ROUGE](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/rouge.md): Evaluate summaries with ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metrics. Measure overlap with references to track relevance and coverage. - [BLEU](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/bleu.md): Assess LLM text outputs with BLEU (Bilingual Evaluation Understudy). Compare n-gram overlap for translation and summarization accuracy. - [METEOR](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/meteor.md): Improve evaluation with METEOR (Metric for Evaluation of Translation with Explicit Ordering) scores. Capture synonyms, stemming, and precision/recall beyond BLEU or ROUGE. - [BERTScore](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/text-summarization/bertscore.md): Use BERTScore to check semantic similarity in outputs. Ensure summaries align with original meaning beyond surface words. - [Information Extraction](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/information-extraction.md): Evaluate how well LLMs extract information from text. Detect missed entities or relations and enhance precision. - [MINEA](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/information-extraction/minea.md): Apply MINEA (Multiple Infused Needle Extraction Accuracy) to test subjective question correction. Identify biases and improve fairness in LLM responses. - [Subjective Question Correction](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/information-extraction/subjective-question-correction.md): Detect the errors in subjective question handling. Improve model performance on open-ended or opinion-based tasks. - [Precision@K](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/information-extraction/precision-k.md): Measure how often correct answers appear in top-K results. Track ranking performance for retrieval-augmented systems. - [Chunk Relevance](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/information-extraction/chunk-relevance.md): Evaluate whether retrieved text chunks are relevant. Identify irrelevant retrievals and refine data pipelines. - [Entity Co-occurrence](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/information-extraction/entity-co-occurrence.md): Test how accurately LLMs connect related entities. Spot gaps in recognition and strengthen relationship extraction. - [Fact Entropy](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/information-extraction/fact-entropy.md): Measure uncertainty in LLM facts. Use entropy scoring to identify unreliable answers and boost accuracy. - [Code Generation](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/code-generation.md): Evaluate LLMs on generating executable code. Spot syntax or logic errors and improve development workflows. - [Functional Correctness](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/code-generation/functional-correctness.md): Test whether generated code runs correctly. Identify functional errors and refine LLM outputs for accuracy. - [ChrF](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/code-generation/chrf.md): Apply ChrF metrics to measure text similarity. Use it for machine translation and summarization evaluation - [Ruby](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/code-generation/ruby.md): Test LLM outputs with Ruby metrics. Assess token overlap for translation accuracy and language precision. - [CodeBLEU](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/code-generation/codebleu.md): Evaluate LLM-generated code with CodeBLEU. Capture logic, syntax, and structure beyond surface similarity. - [Robust Pass@k](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/code-generation/robust-pass-k.md): Measure code generation reliability with Pass@k. Assess how often working solutions appear in multiple attempts. - [Robust Drop@k](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/code-generation/robust-drop-k.md): Track failure rates in code attempts with Drop@k. Identify instability in LLM coding performance. - [Pass-Ratio@n](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/code-generation/pass-ratio-n.md): Measure the ratio of successful generations at N tries. Track consistency in LLM code generation outputs. - [Marketing Content Evaluation](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/marketing-content-evaluation.md): Analyze AI-generated marketing content for engagement, clarity, and accuracy. Optimize content impact with metrics. - [Engagement Score](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/marketing-content-evaluation/engagement-score.md): Measure how engaging LLM-generated marketing content is. Track reader interest and optimize copy impact. - [Misattribution](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/marketing-content-evaluation/misattribution.md): Detect when LLMs misattribute facts in marketing text. Improve reliability and avoid misleading outputs. - [Readability](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/marketing-content-evaluation/readability.md): Assess readability of AI-generated marketing copy. Optimize clarity and user understanding across campaigns. - [Topic Coverage](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/marketing-content-evaluation/topic-coverage.md): Check whether AI-generated marketing text covers required topics. Identify missing content and improve coverage. - [Fabrication](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/marketing-content-evaluation/fabrication.md): Detect fabricated facts in marketing outputs. Ensure content accuracy and safeguard brand credibility. - [Learning Management System](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/learning-management-system.md): Evaluate AI in LMS tasks. Measure performance on content creation, quiz generation, and student interaction. - [Topic Coverage](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/learning-management-system/topic-coverage.md): Analyze how well AI-generated LMS content covers intended topics. Ensure completeness and reliability in educational material. - [Topic Redundancy](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/learning-management-system/topic-redundancy.md): Detect redundancy in AI-generated LMS content. Streamline outputs for efficiency and learner engagement. - [Question Redundancy](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/learning-management-system/question-redundancy.md): Identify repeated questions in LMS assessments. Refine datasets to reduce duplication and improve learning quality. - [Answer Correctness](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/learning-management-system/answer-correctness.md): Evaluate accuracy of AI-generated answers in LMS. Improve reliability of assessments and learning interactions. - [Source Citability](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/learning-management-system/source-citability.md): Test whether LMS content provides proper sources. Improve citation quality and academic reliability. - [Difficulty Level](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/learning-management-system/difficulty-level.md): Measure difficulty levels in LMS tasks. Balance AI-generated questions for fairness and learner growth. - [Additional Metrics](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics.md): Explore extra evaluation metrics for LLMs. Learn how advanced checks reveal hidden weaknesses in AI systems. - [Evaluation](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation.md) - [Chunk Impact](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/chunk-impact.md) - [Faithfulness](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/faithfulness.md) - [Contextual Relevancy](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/contextual-relevancy.md) - [Hallucination](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/hallucination.md) - [Consistency](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/consistency.md) - [Summarization](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/summarization.md) - [Coherence](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/coherence.md) - [Conciseness](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/conciseness.md) - [Grade Score](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/grade-score.md) - [Toxicity](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/toxicity.md) - [Cover](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/cover.md) - [Cosine Similarity](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/cosine-similarity.md) - [Sentiment Analysis](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/sentiment-analysis.md) - [Overall](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/overall.md) - [Winner](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/winner.md) - [POS](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/pos.md) - [Harmless](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/harmless.md) - [Maliciousness](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/maliciousness.md) - [Prompt Injection](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/prompt-injection.md) - [Bias](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/bias.md) - [Response Toxicity](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/response-toxicity.md) - [Readability](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/readability.md) - [Length](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/length.md) - [Contextual Recall](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/contextual-recall.md) - [Contextual Precision](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/contextual-precision.md) - [Correctness](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/evaluation/correctness.md) - [Guardrails](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails.md): Assess AI guardrails for enforcing safety. Explore methods to keep outputs aligned with rules and policies. - [Anonymize](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/anonymize.md): Use anonymization guardrails to hide sensitive data. Protect privacy while maintaining analysis accuracy. - [Deanonymize](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/deanonymize.md): Detect deanonymization risks in AI outputs. Safeguard data security by controlling re-identification attempts. - [Ban Competitors](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/ban-competitors.md): Apply guardrails to block competitor mentions. Control outputs to align with business or compliance goals. - [Ban Substrings](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/ban-substrings.md): Restrict specific substrings in outputs. Use this guardrail to prevent unsafe or undesired content. - [Ban Topics](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/ban-topics.md): Block entire topics in LLM outputs. Control model focus and enforce safety policies. - [Code](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/code.md): Apply code-specific guardrails to monitor outputs. Prevent unsafe, malicious, or irrelevant code generations. - [Invisible Text](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/invisible-text.md): Detect and prevent invisible or hidden text in outputs. Guard against manipulation or misuse. - [Language](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/language.md): Enforce language-specific guardrails. Restrict outputs to desired languages and filter unsafe terms. - [Secret](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/secret.md): Safeguard against LLMs leaking secrets. Use this guardrail to prevent exposure of sensitive information. - [Sentiment](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/sentiment.md): Track sentiment in LLM outputs. Use guardrails to manage tone and align with audience expectations. - [Factual Consistency](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/factual-consistency.md): Ensure AI outputs stay factually consistent. Detect contradictions and improve trustworthiness. - [Language Same](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/language-same.md): Guardrail ensuring language consistency across responses. Prevent mixed-language outputs and enforce clarity. - [No Refusal](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/no-refusal.md): Stop models from refusing legitimate tasks. Enforce responsiveness while balancing safety. - [Reading Time](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/reading-time.md): Estimate reading time for LLM outputs. Use this guardrail to optimize user experience and content design. - [Sensitive](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/sensitive.md): Flag sensitive content in LLM outputs. Strengthen moderation and ensure compliance with safety standards. - [URL Reachability](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/url-reachability.md): Check if generated URLs are reachable. Ensure AI outputs contain working, valid links. - [JSON Verify](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/guardrails/json-verify.md): Validate AI-generated JSON outputs. Use guardrails to prevent schema errors and broken integrations. - [Vulnerability Scanner](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner.md): Detect vulnerabilities in AI outputs. Protect against malicious injections and security risks. - [Bullying](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/bullying.md): Identify bullying or harassment language in outputs. Guard against harmful and unsafe responses. - [Deadnaming](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/deadnaming.md): Detect instances of deadnaming in AI outputs. Protect inclusivity and ensure respectful language use. - [SexualContent](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/sexualcontent.md): Block explicit or inappropriate sexual content in outputs. Enforce strict safety standards. - [Sexualisation](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/sexualisation.md): Detect harmful sexualisation in AI responses. Safeguard users and enforce responsible AI usage. - [SlurUsage](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/slurusage.md): Identify and block slurs in LLM outputs. Ensure respectful, non-discriminatory content generation. - [Profanity](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/profanity.md): Detect and restrict profanity in AI outputs. Enforce brand tone and create safer interactions. - [QuackMedicine](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/quackmedicine.md): Spot unverified or dangerous medical claims. Prevent misinformation and enforce healthcare content standards. - [DAN 11](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dan-11.md): Detect jailbreak prompts like DAN 11. Use guardrails to block unsafe or manipulative model instructions. - [DAN 10](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dan-10.md): Block prompt jailbreaks such as DAN 10. Strengthen safeguards and prevent misuse of AI models. - [DAN 9](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dan-9.md): Identify DAN 9 jailbreak attempts and reinforce safety by blocking malicious instructions. - [DAN 8](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dan-8.md): Detect older jailbreak prompts like DAN 8. Apply guardrails to stop manipulation of model behavior. - [DAN 7](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dan-7.md): Block prompt jailbreak strategies from DAN 7. Ensure AI outputs remain aligned with safety rules. - [DAN 6\_2](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dan-6_2.md): Guard against DAN 6\_2 jailbreak attempts. Enforce strict protections to avoid policy bypasses. - [DAN 6\_0](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dan-6_0.md): Identify and restrict DAN 6\_0 jailbreak prompts. Prevent unsafe AI responses and misuse. - [DUDE](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dude.md): Block DUDE jailbreak exploits. Apply safeguards to maintain AI integrity and safe outputs. - [STAN](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/stan.md): Detect STAN jailbreak attempts. Strengthen AI guardrails against manipulation. - [DAN\_JailBreak](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/dan_jailbreak.md): Identify and block DAN\_JailBreak exploits. Safeguard models from unsafe prompt manipulation. - [AntiDAN](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/antidan.md): Use AntiDAN protections to block jailbreaks. Ensure AI safety against adversarial prompt attacks. - [ChatGPT\_Developer\_Mode\_v2](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/chatgpt_developer_mode_v2.md): Detect developer Mode jailbreaks. Guard against unsafe manipulations of ChatGPT-like systems. - [ChatGPT\_Developer\_Mode\_RANTI](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/chatgpt_developer_mode_ranti.md): Spot RANTI jailbreak prompts. Apply strict protections to avoid safety bypasses. - [ChatGPT\_Image\_Markdown](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/chatgpt_image_markdown.md): Block unsafe image markdown jailbreak exploits. Enforce content safety in AI image generations. - [Ablation\_Dan\_11\_0](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/ablation_dan_11_0.md): Identify Ablation\_Dan\_11\_0 jailbreak attempts. Guard against malicious prompt bypasses. - [Anthropomorphisation](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/additional-metrics/vulnerability-scanner/anthropomorphisation.md): Detect anthropomorphisation in outputs. Keep AI responses objective and avoid misleading human-like traits. - [Guardrails](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails.md): Explore guardrails for AI outputs. Ensure safe, policy-aligned responses across applications. - [Competitor Check](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/competitor-check.md): Block competitor mentions in outputs. Apply guardrails to align with business strategy and compliance. - [Gibberish Check](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/gibberish-check.md): Detect nonsensical outputs from LLMs. Apply guardrails to ensure clarity and coherence. - [PII](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/pii.md): Flag and block personal data exposure. Protect privacy in AI-generated outputs. - [Regex Check](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/regex-check.md): Validate outputs using regex guardrails. Catch unsafe, malformed, or non-compliant responses automatically. - [Response Evaluator](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/response-evaluator.md): Use automated guardrails to evaluate responses. Ensure outputs align with expected standards. - [Toxicity](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/toxicity.md): Detect toxic language in LLM responses. Apply strict guardrails to keep interactions safe. - [Unusual Prompt](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/unusual-prompt.md): Flag unusual or suspicious prompts. Guard against adversarial inputs that could compromise AI behavior. - [Ban List](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/ban-list.md): Block specific banned terms or patterns. Use guardrails to control AI output safety and compliance. - [Detect Drug](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/detect-drug.md): Identify and block drug-related outputs. Enforce safe and compliant AI usage. - [Detect Redundancy](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/detect-redundancy.md): Spot redundant outputs in AI responses. Optimize for efficiency and clarity in generated content. - [Detect Secrets](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/detect-secrets.md): Detect secrets like API keys or passwords in model responses. Prevent accidental exposure with automated guardrails. - [Financial Tone Check](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/financial-tone-check.md): Keep AI-generated financial content professional and compliant. Detect tone mismatches and maintain credibility. - [Has Url](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/has-url.md): Automatically detect URLs in AI outputs. Ensure link presence is flagged and handled correctly. - [HTML Sanitisation](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/html-sanitisation.md): Strip unsafe HTML from model responses. Prevent injection risks and ensure content safety. - [Live URL](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/live-url.md): Check if AI-generated URLs are live and valid. Ensure outputs contain only working, trustworthy links. - [Logic Check](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/logic-check.md): Identify logical flaws or contradictions in model responses. Improve reliability with automated checks. - [Politeness Check](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/politeness-check.md): Detect impolite or harsh tones in AI responses. Guarantee positive, user-friendly interactions. - [Profanity Check](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/profanity-check.md): Automatically detect and block profanity in LLM responses. Maintain safe and professional content. - [Quote Price](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/quote-price.md): Ensure AI-generated content includes correct, valid price quotes. Improve trust in financial AI outputs. - [Restrict Topics](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/restrict-topics.md): Enforce topic restrictions in LLM outputs. Keep AI conversations safe, relevant, and policy-aligned. - [SQL Predicates Guard](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/sql-predicates-guard.md): Prevent dangerous SQL predicates in AI outputs. Protect databases from risky or malicious statements. - [Valid CSV](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/valid-csv.md): Ensure AI outputs produce correct CSV structure. Detect errors before data is processed or used - [Valid JSON](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/valid-json.md): Ensure AI outputs produce correct JSON structure. Detect errors before data is processed or used. - [Valid Python](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/valid-python.md): Detect syntax errors in AI-generated Python code. Guarantee safe, executable programming outputs. - [Valid Range](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/valid-range.md): Ensure numeric values generated by AI fall within allowed ranges. Keep calculations safe and accurate. - [Valid SQL](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/valid-sql.md): Check correctness of SQL outputs from AI models. Prevent invalid or broken database statements. - [Valid URL](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/valid-url.md): Automatically check if URLs generated by AI are valid. Reduce errors with automated guardrails. - [Cosine Similarity](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/cosine-similarity.md): Compare AI outputs with reference text using cosine similarity. Improve semantic consistency and accuracy. - [Honesty Detection](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/honesty-detection.md): Detect dishonest or misleading AI outputs. Improve transparency and build trust in model responses. - [Toxicity Hate Speech](https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/guardrails/toxicity-hate-speech.md): Identify toxic or hateful language in AI outputs. Enforce safe, respectful, and policy-compliant content. - [Prompt Playground](https://docs.raga.ai/ragaai-catalyst/prompt-playground.md): Test and refine prompts in a sandbox. Compare outputs, optimize instructions, and improve AI performance. - [Concepts](https://docs.raga.ai/ragaai-catalyst/prompt-playground/concepts.md): Explore the key concepts behind prompt experimentation. Understand structure, evaluation, and optimization. - [Single-Prompt Playground](https://docs.raga.ai/ragaai-catalyst/prompt-playground/single-prompt-playground.md): Run focused experiments with a single prompt. Evaluate AI outputs and fine-tune responses quickly. - [Multiple Prompt Playground](https://docs.raga.ai/ragaai-catalyst/prompt-playground/multiple-prompt-playground.md): Test and compare multiple prompts simultaneously. Identify which instructions deliver the best AI results. - [Run Evaluations](https://docs.raga.ai/ragaai-catalyst/prompt-playground/run-evaluations.md): Execute structured evaluations on prompts. Analyze model behavior and ensure reliable, consistent outputs. - [Using Prompt Slugs with Python SDK](https://docs.raga.ai/ragaai-catalyst/prompt-playground/using-prompt-slugs-with-python-sdk.md): Use prompt slugs in the Python SDK to run tests programmatically. Streamline AI evaluations at scale. - [Create with AI using Prompt Wizard](https://docs.raga.ai/ragaai-catalyst/prompt-playground/create-with-ai-using-prompt-wizard.md): Generate optimized prompts using the AI-powered wizard. Simplify experimentation and boost AI reliability. - [Prompt Diff View](https://docs.raga.ai/ragaai-catalyst/prompt-playground/prompt-diff-view.md): Visually compare AI responses to different prompts. Spot differences and refine your best-performing instructions. - [Synthetic Data Generation](https://docs.raga.ai/ragaai-catalyst/synthetic-data-generation.md): Generate synthetic data for training and evaluation. Strengthen LLM testing with diverse, controlled datasets. - [Gateway](https://docs.raga.ai/ragaai-catalyst/gateway.md): Use Gateway to integrate your LLMs with Catalyst. Streamline testing, monitoring, and evaluation pipelines. - [Quickstart](https://docs.raga.ai/ragaai-catalyst/gateway/quickstart.md): Follow the quickstart guide to set up Gateway. Link LLMs seamlessly and begin evaluating in minutes. - [Guardrails](https://docs.raga.ai/ragaai-catalyst/guardrails.md): Add safety guardrails to AI outputs. Prevent unsafe, biased, or invalid responses with automated checks. - [Quickstart](https://docs.raga.ai/ragaai-catalyst/guardrails/quickstart.md): Learn how to set up guardrails quickly. Protect your AI outputs with instant, reliable safety measures. - [Python SDK](https://docs.raga.ai/ragaai-catalyst/guardrails/python-sdk.md): Use the Python SDK to enforce guardrails in your applications. Automate safety checks at runtime. - [RagaAI Whitepapers](https://docs.raga.ai/ragaai-catalyst/ragaai-whitepapers.md): Access in-depth research and frameworks from RagaAI. Explore evaluation methods and insights for LLM testing. - [RagaAI RLEF (RAG LLM Evaluation Framework)](https://docs.raga.ai/ragaai-catalyst/ragaai-whitepapers/ragaai-rlef-rag-llm-evaluation-framework.md): Discover RagaAI’s RLEF whitepaper. Learn a structured framework for evaluating retrieval-augmented generation systems. - [Agentic Testing](https://docs.raga.ai/ragaai-catalyst/agentic-testing.md): Test agentic AI workflows for reliability and safety. Explore methods to validate multi-step reasoning and decisions. - [Quickstart](https://docs.raga.ai/ragaai-catalyst/agentic-testing/quickstart.md): Get started with agentic testing in minutes. Learn setup steps and quickly evaluate AI agent workflows. - [Concepts](https://docs.raga.ai/ragaai-catalyst/agentic-testing/concepts.md): Understand the key concepts of agentic testing. Explore workflows, trace analysis, and evaluation methods. - [Tracing](https://docs.raga.ai/ragaai-catalyst/agentic-testing/concepts/tracing.md): Track each step in AI agent workflows. Use tracing to debug errors, evaluate reasoning, and improve reliability. - [Langgraph (Agentic Tracing)](https://docs.raga.ai/ragaai-catalyst/agentic-testing/concepts/tracing/langgraph-agentic-tracing.md): Use Langgraph to visualize agent reasoning. Debug, monitor, and optimize multi-step agentic workflows. - [RagaAI Catalyst Tracing Guide for Azure OpenAI Users](https://docs.raga.ai/ragaai-catalyst/agentic-testing/concepts/tracing/ragaai-catalyst-tracing-guide-for-azure-openai-users.md): Learn how to set up tracing for Azure OpenAI models in RagaAI. Ensure detailed monitoring and evaluation. - [Dynamic Tracing](https://docs.raga.ai/ragaai-catalyst/agentic-testing/concepts/dynamic-tracing.md): Monitor AI agent behavior dynamically. Detect errors in real time and optimize workflow execution. - [Application Workflow](https://docs.raga.ai/ragaai-catalyst/agentic-testing/concepts/application-workflow.md): Learn how to design and manage agentic application workflows. Improve traceability and reliability in testing. - [Create New Dataset](https://docs.raga.ai/ragaai-catalyst/agentic-testing/create-new-dataset.md): Create datasets for testing AI models. Add structured data to evaluate performance across multiple scenarios. - [Metrics](https://docs.raga.ai/ragaai-catalyst/agentic-testing/metrics.md): Explore metrics that evaluate AI agents. Track hallucinations, honesty, similarity, and more in workflows. - [Hallucination](https://docs.raga.ai/ragaai-catalyst/agentic-testing/metrics/hallucination.md): Spot hallucinations in LLM outputs. Identify when models generate made-up content and apply corrections. - [Toxicity](https://docs.raga.ai/ragaai-catalyst/agentic-testing/metrics/toxicity.md): Identify toxic language in AI outputs. Use toxicity detection to enforce safe and respectful responses. - [Honesty](https://docs.raga.ai/ragaai-catalyst/agentic-testing/metrics/honesty.md): Detect dishonest or misleading responses. Strengthen AI reliability with honesty evaluation metrics. - [Cosine Similarity](https://docs.raga.ai/ragaai-catalyst/agentic-testing/metrics/cosine-similarity.md): Measure similarity between generated and reference texts. Improve alignment with semantic scoring. - [Compare Traces](https://docs.raga.ai/ragaai-catalyst/agentic-testing/compare-traces.md): Compare execution traces side by side. Identify issues and refine agent reasoning across tasks. - [Compare Experiments](https://docs.raga.ai/ragaai-catalyst/agentic-testing/compare-experiments.md): Analyze experiment results across multiple models. Compare workflows to choose the best-performing system. - [Add metrics locally](https://docs.raga.ai/ragaai-catalyst/agentic-testing/add-metrics-locally.md): Extend Catalyst by adding local metrics. Tailor AI evaluation to your organization’s specific needs. - [Custom Metric](https://docs.raga.ai/ragaai-catalyst/custom-metric.md): Create custom metrics for specialized AI testing. Build unique evaluation standards beyond defaults. - [Auto Prompt Optimization](https://docs.raga.ai/ragaai-catalyst/auto-prompt-optimization.md): Automatically optimize prompts for better results. Enhance AI performance with adaptive tuning. - [Human Feedback & Annotations](https://docs.raga.ai/ragaai-catalyst/human-feedback-and-annotations.md): Collect human feedback on AI responses. Use annotations to refine and improve model performance. - [Thumbs Up/Down](https://docs.raga.ai/ragaai-catalyst/human-feedback-and-annotations/thumbs-up-down.md): Capture quick human feedback with thumbs up/down. Improve models using simple quality ratings. - [Add Metric Corrections](https://docs.raga.ai/ragaai-catalyst/human-feedback-and-annotations/add-metric-corrections.md): Correct metric evaluations manually. Ensure fairness and accuracy in AI testing. - [Corrections as Few-Shot Examples](https://docs.raga.ai/ragaai-catalyst/human-feedback-and-annotations/corrections-as-few-shot-examples.md): Apply corrections as few-shot examples. Guide models with feedback-driven improvements. - [Tagging](https://docs.raga.ai/ragaai-catalyst/human-feedback-and-annotations/tagging.md): Tag AI responses for easy categorization. Streamline analysis with structured annotations. - [On-Premise Deployment](https://docs.raga.ai/ragaai-catalyst/on-premise-deployment.md): RagaAI Catalyst can be used in both a SaaS mode as well as within the customer's environment (on-prem). This section contains details about hosting RagaAI Catalyst on-prem. - [Self-Hosting RagaAI Catalyst on Kubernetes](https://docs.raga.ai/ragaai-catalyst/on-premise-deployment/self-hosting-catalyst-on-k8s.md): Learn how to self-host Catalyst on Kubernetes. Scale AI testing with secure, containerized deployment. - [Enterprise Deployment Guide for AWS](https://docs.raga.ai/ragaai-catalyst/on-premise-deployment/on-premise-deployment-for-aws.md): Deploy Catalyst seamlessly on AWS. Follow enterprise-ready steps for secure and scalable setup. - [Enterprise Deployment Guide for Azure](https://docs.raga.ai/ragaai-catalyst/on-premise-deployment/on-premise-deployment-for-azure.md): Learn how to deploy Catalyst on Azure. Follow structured steps for enterprise-ready AI testing. - [Enterprise Deployment Guide for GCP](https://docs.raga.ai/ragaai-catalyst/on-premise-deployment/on-premise-deployment-for-gcp.md): Deploy Catalyst on Google Cloud. Follow enterprise instructions for scalable AI evaluation. - [Evaluation Deployment Guide](https://docs.raga.ai/ragaai-catalyst/on-premise-deployment/raga-catalyst-deployment-guide.md): Step-by-step guide for deploying Catalyst evaluation. Ensure smooth setup for AI testing. - [Evaluation Maintenance Guide](https://docs.raga.ai/ragaai-catalyst/on-premise-deployment/raga-catalyst-deployment-guide/evaluation-maintenance-guide.md): Maintain Catalyst deployments effectively. Follow best practices for stability and uptime. - [Fine Tuning (OpenAI)](https://docs.raga.ai/ragaai-catalyst/fine-tuning.md): Learn how to fine-tune OpenAI models within Catalyst. Improve accuracy with custom training data. - [Integration](https://docs.raga.ai/ragaai-catalyst/integration.md): Integrate Catalyst seamlessly into your workflows. Connect tools, data, and models in one platform. - [SDK Release Notes](https://docs.raga.ai/ragaai-catalyst/sdk-release-notes.md): Stay updated with the latest SDK changes. Review features, fixes, and improvements for Catalyst. - [ragaai-catalyst 2.1.7](https://docs.raga.ai/ragaai-catalyst/sdk-release-notes/ragaai-catalyst-2.1.7.md): Explore updates in Catalyst v2.1.7. Review new features, enhancements, and bug fixes. - [RagaAI Prism](https://docs.raga.ai/ragaai-prism.md): This page provides a high level introduction to the RagaAI Prism - An end to end testing platform for DiscriminativeAI models - Computer Vision, NLP and Tabular Data. - [Quickstart](https://docs.raga.ai/ragaai-prism/quickstart.md): Get started with Prism quickly. Follow steps to begin testing and evaluating AI models in minutes. - [Copy of Quickstart](https://docs.raga.ai/ragaai-prism/copy-of-quickstart.md): Begin your journey into AI testing with the RagaAI public sandbox in four easy steps - - [Sandbox Guide](https://docs.raga.ai/ragaai-prism/sandbox-guide.md): Learn how to use Prism sandbox. Run structured experiments and validate model performance. - [Object Detection](https://docs.raga.ai/ragaai-prism/sandbox-guide/object-detection.md): Evaluate object detection models with Prism. Identify errors, drift, and model weaknesses. - [LLM Summarization](https://docs.raga.ai/ragaai-prism/sandbox-guide/llm-summarization.md): Assess summarization performance of LLMs. Detect fluency, coherence, and coverage issues. - [Semantic Segmentation](https://docs.raga.ai/ragaai-prism/sandbox-guide/semantic-segmentation.md): Test semantic segmentation models in Prism. Spot class imbalance and analyze labeling quality. - [Tabular Data](https://docs.raga.ai/ragaai-prism/sandbox-guide/tabular-data.md): Evaluate models on tabular data. Detect drift, imbalance, and errors in structured predictions. - [Super Resolution](https://docs.raga.ai/ragaai-prism/sandbox-guide/super-resolution.md): Assess AI models for image super resolution. Identify drift, errors, and resolution quality. - [OCR](https://docs.raga.ai/ragaai-prism/sandbox-guide/ocr.md): Test OCR models in Prism. Detect missing values, outliers, and errors in extracted text. - [Image Classification](https://docs.raga.ai/ragaai-prism/sandbox-guide/image-classification.md): Evaluate classification models. Detect drift, imbalance, and labeling issues in outputs. - [Event Detection](https://docs.raga.ai/ragaai-prism/sandbox-guide/event-detection.md): Test event detection models for accuracy. Identify drift, bias, and classification errors. - [Test Inventory](https://docs.raga.ai/ragaai-prism/test-inventory.md): Access a library of tests for AI models. Use Prism’s inventory to validate diverse model behaviors. - [Object Detection](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection.md): Test object detection models thoroughly. Identify weaknesses, drift, and labeling issues. - [Failure Mode Analysis](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/failure-mode-analysis.md): Analyze failure modes in object detection. Spot recurring errors and refine models. - [Model Comparison Test](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/model-comparison-test.md): Compare object detection models side by side. Benchmark performance to find the most reliable. - [Drift Detection](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/drift-detection.md): Detect drift in object detection models. Track shifts in data and retrain before accuracy drops. - [Outlier Detection](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/outlier-detection.md): Identify anomalies in object detection outputs. Catch unusual predictions early. - [Data Leakage Test](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/data-leakage-test.md): Test datasets for leakage in object detection. Prevent training-test contamination. - [Labelling Quality Test](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/labelling-quality-test.md): Check annotation accuracy in datasets. Improve object detection results with cleaner labels. - [Scenario Imbalance](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/scenario-imbalance.md): Identify imbalanced scenarios in training data. Improve fairness and robustness. - [Class Imbalance](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/class-imbalance.md): Detect class imbalance issues. Improve model performance with balanced data. - [Active Learning](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/active-learning.md): Use active learning strategies to improve detection models. Retrain with the most informative samples. - [Image Property Drift Detection](https://docs.raga.ai/ragaai-prism/test-inventory/object-detection/image-property-drift-detection.md): Detect changes in image properties that affect detection. Keep models robust against drift. - [Large Language Model (LLM)](https://docs.raga.ai/ragaai-prism/test-inventory/large-language-model-llm.md): Test LLMs with Prism. Evaluate reasoning, hallucination, and output reliability. - [Failure Mode Analysis](https://docs.raga.ai/ragaai-prism/test-inventory/large-language-model-llm/failure-mode-analysis.md): Analyze where LLMs fail. Spot reasoning errors, hallucinations, and inconsistencies. - [Semantic Segmentation](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation.md): Test semantic segmentation models for robustness. Detect errors, drift, and imbalance issues. - [Failure Mode Analysis](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/failure-mode-analysis.md): Analyze segmentation model errors. Identify recurring weaknesses to improve performance. - [Labelling Quality Test](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/labelling-quality-test.md): Validate labeling quality in segmentation datasets. Improve accuracy with clean annotations. - [Active Learning](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/active-learning.md): Apply active learning to boost segmentation accuracy. Retrain with most informative samples. - [Drift Detection](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/drift-detection.md): Detect drift in semantic segmentation inputs. Catch data shifts early to preserve accuracy. - [Class Imbalance](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/class-imbalance.md): Identify and correct class imbalance. Improve fairness and accuracy of segmentation models. - [Scenario Imbalance](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/scenario-imbalance.md): Spot imbalance in dataset scenarios. Ensure diverse coverage for better segmentation outcomes. - [Data Leakage Test](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/data-leakage-test.md): Test for data leakage in segmentation tasks. Ensure clean splits between training and evaluation. - [Outlier Detection](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/outlier-detection.md): Identify unusual segmentation outputs. Detect anomalies before they harm performance. - [Label Drift](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/label-drift.md): Detect shifts in labels over time. Ensure annotation consistency in evolving datasets. - [Semantic Similarity](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/semantic-similarity.md): Evaluate similarity between segmentation outputs. Improve consistency with semantic checks. - [Near Duplicates Detection](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/near-duplicates-detection.md): Identify near-duplicate samples in datasets. Improve segmentation training with clean, diverse data. - [Cluster Imbalance Test](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/cluster-imbalance-test.md): Detect imbalance across data clusters. Ensure segmentation models train on fair, balanced distributions. - [Image Property Drift Detection](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/image-property-drift-detection.md): Detect changes in image properties that affect segmentation accuracy. Keep models robust with drift checks.. - [Spatio-Temporal Drift Detection](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/spatio-temporal-drift-detection.md): Detect drift across space and time dimensions. Maintain segmentation model reliability with ongoing checks. - [Spatio-Temporal Failure Mode Analysis](https://docs.raga.ai/ragaai-prism/test-inventory/semantic-segmentation/spatio-temporal-failure-mode-analysis.md): The Spatio-Temporal Failure Mode Analysis Test is designed to analyze the model's performance on the spatio-temporal dataset. - [Tabular Data](https://docs.raga.ai/ragaai-prism/test-inventory/tabular-data.md): RagaAI offers robust testing support across a spectrum of machine learning models and AI applications. - [Failure Mode Analysis](https://docs.raga.ai/ragaai-prism/test-inventory/tabular-data/failure-mode-analysis.md): Failure Mode Analysis is a test that allows users to deeply analyse their model's performance. - [Instance Segmentation](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation.md): Test instance segmentation models. Detect imbalance, drift, and labeling quality issues. - [Failure Mode Analysis](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/failure-mode-analysis.md): Failure Mode Analysis is a test that allows users to deeply analyse their model's performance. - [Labelling Quality Test](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/labelling-quality-test.md): Check annotation accuracy in segmentation datasets. Improve outcomes with reliable labeling. - [Drift Detection](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/drift-detection.md): The Drift Detection Test allows users to identify shifts between training and field/test datasets - [Class Imbalance](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/class-imbalance.md): Identify class imbalance issues in datasets. Train robust instance segmentation models with fair data. - [Scenario Imbalance](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/scenario-imbalance.md): The Scenario Imbalance Test evaluates the distribution of scenarios or contexts within a dataset, providing insights into potential imbalances that may affect model performance. - [Label Drift](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/label-drift.md): Monitor changes in dataset labels over time. Prevent quality drops in segmentation training. - [Data Leakage Test](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/data-leakage-test.md): Detect leakage between training and test data. Improve model fairness and trustworthiness. - [Outlier Detection](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/outlier-detection.md): The Outlier Detection Test in RagaAI is crucial for identifying anomalies in your dataset. - [Active Learning](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/active-learning.md): The Active Learning Test in RagaAI optimises dataset by selecting the most representative data points within a specified budget. - [Near Duplicates Detection](https://docs.raga.ai/ragaai-prism/test-inventory/instance-segmentation/near-duplicates-detection.md): The Near Duplicate Detection Test in RagaAI is designed to identify both exact and near duplicates within your image dataset. - [Super Resolution](https://docs.raga.ai/ragaai-prism/test-inventory/super-resolution.md): Evaluate super resolution models. Detect drift, anomalies, and output quality issues. - [Semantic Similarity](https://docs.raga.ai/ragaai-prism/test-inventory/super-resolution/semantic-similarity.md): The Semantic Similarity Test in RagaAI assesses the likeness between high-resolution (HR) and low-resolution (LR) images. - [Active Learning](https://docs.raga.ai/ragaai-prism/test-inventory/super-resolution/active-learning.md): The Active Learning Test in RagaAI optimises dataset by selecting the most representative data points within a specified budget. - [Near Duplicates Detection](https://docs.raga.ai/ragaai-prism/test-inventory/super-resolution/near-duplicates-detection.md): The Near Duplicate Detection Test in RagaAI is designed to identify both exact and near duplicates within your image dataset. - [Outlier Detection](https://docs.raga.ai/ragaai-prism/test-inventory/super-resolution/outlier-detection.md): The Outlier Detection Test in RagaAI is crucial for identifying anomalies in low-resolution and high-resolution datasets, separately. - [OCR](https://docs.raga.ai/ragaai-prism/test-inventory/ocr.md): RagaAI offers robust testing support across a spectrum of machine learning models and AI applications. - [Missing Value Test](https://docs.raga.ai/ragaai-prism/test-inventory/ocr/missing-value-test.md): The Missing Values Test enables you to identify data points with missing bounding boxes. Use filters to filter by class or by test results to directly look at data points of interest. - [Outlier Detection](https://docs.raga.ai/ragaai-prism/test-inventory/ocr/outlier-detection.md): The Outlier Detection Test in RagaAI is crucial for identifying anomalies in low-resolution and high-resolution datasets, separately. - [Image Classification](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification.md): Test image classification models. Detect class imbalance, drift, and labeling quality issues. - [Failure Mode Analysis](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/failure-mode-analysis.md): Analyze common errors in classification tasks. Refine training and improve model accuracy. - [Labelling Quality Test](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/labelling-quality-test.md) - [Class Imbalance](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/class-imbalance.md): The Class Imbalance Test is designed to assess the distribution of classes within a dataset, particularly in the context of machine learning tasks like object detection. - [Drift Detection](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/drift-detection.md): The Drift Detection Test allows users to identify shifts between training and field/test datasets - [Near Duplicates Test](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/near-duplicates-test.md): The Near Duplicate Detection Test in RagaAI is designed to identify both exact and near duplicates within your image dataset. - [Data Leakage Test](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/data-leakage-test.md): The Data Leakage Test results provide insights into the presence of data leakage from the training dataset to the test/validation dataset. - [Outlier Detection](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/outlier-detection.md): The Outlier Detection Test in RagaAI is crucial for identifying anomalies in your dataset. - [Active Learning](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/active-learning.md): The Active Learning Test in RagaAI optimises dataset by selecting the most representative data points within a specified budget. - [Image Property Drift Detection](https://docs.raga.ai/ragaai-prism/test-inventory/image-classification/image-property-drift-detection.md): The Image Property Drift Detection Test is designed to monitor the shifts in individual image properties over time. - [Event Detection](https://docs.raga.ai/ragaai-prism/test-inventory/event-detection.md): RagaAI offers robust testing support across a spectrum of machine learning models and AI applications. - [Failure Mode Analysis](https://docs.raga.ai/ragaai-prism/test-inventory/event-detection/failure-mode-analysis.md): Failure Mode Analysis is a test that allows users to deeply analyse the pipeline performance. - [A/B Test](https://docs.raga.ai/ragaai-prism/test-inventory/event-detection/a-b-test.md): A/B Test is a systematic method to compare two or more variations of a pipeline. - [Metric Glossary](https://docs.raga.ai/ragaai-prism/metric-glossary.md): Quick definitions and explanations of the various metrics used within the RagaAI Testing Platform - [Upload custom model](https://docs.raga.ai/ragaai-prism/upload-custom-model.md): Upload your own models into Prism. Run evaluations and validate performance on your datasets. - [Event Detection](https://docs.raga.ai/ragaai-prism/event-detection.md): Evaluate event detection with Prism tools. Identify drift, anomalies, and misclassifications. - [Upload Model](https://docs.raga.ai/ragaai-prism/event-detection/upload-model.md): Upload event detection models into Prism. Start evaluations and monitor real-world performance. - [Generate Inference](https://docs.raga.ai/ragaai-prism/event-detection/generate-inference.md): Run inferences with event detection models. Evaluate predictions and test outputs instantly. - [Run tests](https://docs.raga.ai/ragaai-prism/event-detection/run-tests.md): Execute structured tests on event detection models. Track performance across conditions. - [On-Premise Deployment](https://docs.raga.ai/ragaai-prism/on-premise-deployment.md): RagaAI Prism can be used in both a SaaS mode as well as within the customer's environment (on-prem). This section contains details about hosting RagaAI Prism on-prem. - [Enterprise Deployment Guide for AWS](https://docs.raga.ai/ragaai-prism/on-premise-deployment/on-premise-deployment-for-aws.md): Learn how to deploy Prism on AWS. Follow enterprise-ready instructions for secure AI testing. - [Enterprise Deployment Guide for Azure](https://docs.raga.ai/ragaai-prism/on-premise-deployment/on-premise-deployment-for-azure.md): Deploy Prism seamlessly on Azure. Follow best practices for scalable, compliant AI evaluation. - [Enterprise Deployment Guide for GCP](https://docs.raga.ai/ragaai-prism/on-premise-deployment/on-premise-deployment-for-gcp.md): Deploy Prism on Google Cloud. Ensure secure, enterprise-ready setup for AI testing. - [Support](https://docs.raga.ai/support.md): Access RagaAI support resources. Get help with Catalyst, Prism, integrations, and deployment. - [AgentNeo](https://docs.raga.ai/agentneo.md) - [Installation](https://docs.raga.ai/agentneo/installation.md) - [Getting Started](https://docs.raga.ai/agentneo/getting-started.md) - [Metric Library](https://docs.raga.ai/agentneo/metric-library.md) - [Tool Selection Accuracy](https://docs.raga.ai/agentneo/metric-library/tool-selection-accuracy.md): Assess if the tool selections in your agentic pipeline is optimal - [Tool Usage Efficiency](https://docs.raga.ai/agentneo/metric-library/tool-usage-efficiency.md): Check how efficiently the tools are used in your agent pipeline - [Goal Decomposition Efficiency](https://docs.raga.ai/agentneo/metric-library/goal-decomposition-efficiency.md): Find out how well the agent pipeline is able to divide the complex task into manageable chunks and how well the task is delegated - [Plan Adaptability](https://docs.raga.ai/agentneo/metric-library/plan-adaptability.md): Evaluate how well your agentic pipeline responds to any unforeseen events like errors, new information, etc. - [Plan Execution Error Rate](https://docs.raga.ai/agentneo/metric-library/plan-execution-error-rate.md) - [Examples](https://docs.raga.ai/agentneo/examples.md) - [RagaAI AAEF (Agentic Application Evaluation Framework)](https://docs.raga.ai/ragaai-aaef-agentic-application-evaluation-framework.md) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on a page URL with the `ask` query parameter: ``` GET https://docs.raga.ai/readme.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.