Router

Also referred to as "Gateway" interchangeably.

Exclusive to enterprise customers. Contact us to activate this feature.

The Catalyst Router is a cutting-edge solution designed for enterprises that leverage multiple Large Language Models (LLMs) in their operations. By intelligently managing API keys and routing requests based on a set of dynamic variables, this software ensures that users receive the most accurate and timely responses while maximising the efficiency and reliability of their AI systems.

Key Features:

1. Smart Routing

  • Prompt Compexity Analysis: The router dynamically evaluates the complexity of each user prompt, directing it to the most suitable LLM based on its specialized capabilities (e.g., general knowledge, technical expertise, creative generation).

  • Contextual Parameter Evaluation: Takes into account additional parameters such as prompt length, domain specificity, and response latency, ensuring the selected model aligns with the user's immediate needs.

2. Load Balancing and Fallback Mechanisms

  • Timeout Mitigation: The software generates a prioritized fallback list of LLMs for each request, rerouting to alternative models in case of timeouts or extended response times.

  • Redundancy Assurance: Ensures high availability by automatically switching to backup models if the primary choice fails, maintaining service continuity.

3. Performance Analytics

  • Real-time Monitoring: Provides live tracking of model performance metrics, including response times, accuracy, and cost-effectiveness.

  • Data-Driven Optimization: Uses historical data to refine routing algorithms, continuously enhancing decision-making processes.

4. Multi-API Key Management

  • Unified Access Control: Seamlessly integrates multiple LLM API keys from various providers, offering a single interface to manage and deploy requests.

  • API Usage Optimization: Balances the workload across different models to prevent overuse of any single API, reducing costs and improving response consistency.

Benefits

  • Increased Accuracy and Relevance: By leveraging the strengths of multiple LLMs, users receive more accurate and contextually relevant responses, enhancing overall productivity.

  • Cost Efficiency: Optimises API usage, reducing unnecessary expenditures by selecting the most cost-effective model for each request.

  • Enhanced Reliability: Minimises downtime and service disruptions through intelligent load balancing and failover strategies, ensuring consistent performance.

  • Scalability: Supports enterprise-level operations, allowing for seamless expansion as new LLMs become available or as usage demands increase.

Last updated