Observability Settings And Safety

A service sets Settings.llm to a powerful expensive model globally, then a high-volume summarization path silently inherits it instead of using a cheaper per-call LLM, blowing the budget.

Understand Settings as global defaults that every unconfigured component inherits; override llm/embed_model locally on the components that need a different (cheaper/specialized) model. Audit which paths inherit the global LLM so cost-sensitive flows do not accidentally use the premium model.

A RAG pipeline gives wrong answers and the team has no visibility into which nodes were retrieved or what prompt the LLM saw, because no instrumentation/callback handler is attached.

Attach instrumentation (the event/span API) or a CallbackManager / observability integration so retrieval, rerank, and LLM events are traced — capturing retrieved node ids/scores and the synthesized prompt — to debug grounding. Verify traces show the retrieve→synthesize path end to end.

The team estimates LLM cost by counting characters in answers rather than using the TokenCountingHandler / callback token usage, and their cost dashboard is off by a large factor.

Use LlamaIndex token-counting instrumentation (TokenCountingHandler or callback token-usage events) to attribute embedding, prompt, and completion tokens per query, rather than estimating from text length. Roll these up against a budget.