Logging traces

RagaAI Catalyst currently supports tracing only for Langchain applications. Tracing with custom decorators and for LLamaIndex pipelines is coming soon.

Users can log traces from their real-time (production) applications by invoking the Catalyst Tracer from their SDK. This allows Catalyst to read inferences in real-time and log them for instant evaluations. This way, users can avoid downloading and uploading in batches.

The Catalyst Tracer can be enabled with the following commands:

from ragaai_catalyst import Tracer
tracer_dataset_name = "your-dataset-name"

tracer = Tracer(
    project_name="your-project-name",
    dataset_name="your-dataset-name",
    metadata={"key1": "value1", "key2": "value2"},
    tracer_type="langchain",
    pipeline={
        "llm_model": "gpt-3.5-turbo",
        "vector_store": "faiss",
        "embed_model": "text-embedding-ada-002",
    }
).start()

• In case you see an error around tenacity v9.0.0 on executing Tracer.start(), retry the same after using the following command (ignore in absence of error):

pip install tenacity==8.3.0

• Once the tracer has successfully started, execute your RAG Langchain code. You can find a sample RAG code in this Colab project. Your OpenAI API key will be required to run this example, as with a real-time application.

• Once traces have been logged using your code, stop the tracer and check its status:

tracer.stop()
tracer.get_upload_status()

• If successful, you can view the logged dataset by navigating to Project_Name -> Dataset_Name

• Metrics can be run on logged datasets in the same fashion as for CSV uploads.