Logging traces (LlamaIndex, Langchain)

RagaAI Catalyst currently supports tracing only for Langchain applications. Tracing with custom decorators and for LlamaIndex pipelines is coming soon.

Users can log traces from their real-time (production) applications by invoking the Catalyst Tracer from their SDK. This allows Catalyst to read inferences in real-time and log them for instant evaluations. This way, users can avoid downloading and uploading in batches.

The Catalyst Tracer can be enabled with the following commands:

from ragaai_catalyst import Tracer
tracer_dataset_name = "your-dataset-name"

tracer = Tracer(
    project_name="your-project-name",
    dataset_name="your-dataset-name",
    metadata={"key1": "value1", "key2": "value2"},
    tracer_type="langchain",
    pipeline={
        "llm_model": "gpt-3.5-turbo",
        "vector_store": "faiss",
        "embed_model": "text-embedding-ada-002",
    }
).start()

• In case you see an error around tenacity v9.0.0 on executing Tracer.start(), retry the same after using the following command (ignore in absence of error):

pip install tenacity==8.3.0

• Once the tracer has successfully started, execute your RAG Langchain code. You can find a sample RAG code in this Colab project. Your OpenAI API key will be required to run this example, as with a real-time application.

• Once traces have been logged using your code, stop the tracer and check its status:

tracer.stop()
tracer.get_upload_status()

• If successful, you can view the logged dataset by navigating to Project_Name -> Dataset_Name

• Metrics can be run on logged datasets in the same fashion as for CSV uploads.