There's a pattern I've seen more than once. A team builds a ChatGPT wrapper, it works in demos, and six months later I'm being asked to figure out why it hallucinates every third response and the inference bill is four times the estimate.
The tutorial version of LLM integration is honest about the capabilities. It's misleading about everything else.
Latency is the first real problem
GPT-4o averages 1-3 seconds for a short completion. For a RAG query that hits your vector store, runs a retrieval step, and then calls the LLM with 6-8K tokens of context, you're looking at 4-8 seconds wall time. For a customer-facing UI, that's borderline unusable.
The fix isn't faster prompts. It's streaming — start rendering the response token-by-token so the user sees something in under a second. Most LLM APIs support it. Surprisingly few integrations actually implement it, because the tutorial example uses a synchronous call and nobody questions it.
For agents with multiple tool calls, latency compounds. A three-step agent runs 15-20 seconds synchronously. Run independent steps in parallel. Use progress indicators. For long-running jobs, decouple the trigger from the result delivery — fire and return a job ID, then poll or webhook when done.
RAG is harder than the demos suggest
Retrieval-augmented generation looks simple in a notebook: embed documents, store in a vector database, retrieve relevant chunks, pass to LLM. The demo works because the demo uses three clean documents.
In production, your documents are PDFs with misaligned columns, HTML scraped with noise, inconsistent formatting, and versions that haven't been cleaned since 2019. The retrieval step returns chunks that are semantically similar but contextually wrong. The LLM answers confidently — based on bad context.
The work is in the pipeline before the vector store: chunking strategy, metadata tagging, deduplication, handling document updates cleanly. I've spent as much engineering time on ingestion pipelines as on the actual retrieval logic. Nobody blogs about the ingestion pipeline.
Evaluation is the hidden cost
You need to know when retrieval is getting worse. That means a ground-truth dataset — questions with expected answers drawn from your document set — and a retrieval accuracy metric you compute on every deploy. Most teams skip this until they get a production incident and realize they have no way to tell if the last model upgrade improved or degraded quality.
Building that ground-truth dataset takes time. It's not glamorous. It's also the only way to run the system like an engineer rather than hoping for the best.
Token costs compound fast
GPT-4o is roughly $5 per million input tokens and $15 per million output tokens. A RAG query with 8K tokens of context and a 500-token response: about $0.048 per query. If you have 5,000 active users running 10 queries a day, that's $2,400 a day — ₹2 lakh. A month of that is ₹60 lakh. That math sneaks up on you, especially when you've been testing at low volume.
Three places to recover cost: use smaller models for simpler queries (GPT-4o Mini is 15-20x cheaper and handles a large share of real-world queries without quality loss), cache responses for repeated or near-repeated queries (surprisingly effective in support and FAQ contexts), and reduce context size by improving retrieval precision rather than padding with more chunks hoping to get lucky.
We've reduced client inference costs by 60-70% by doing precision work on retrieval and routing instead of throwing GPT-4 at everything. The model is rarely the bottleneck.
Graceful degradation is non-negotiable
LLM APIs go down. They throttle under load. Latency spikes at 2am for reasons you can't diagnose. OpenAI had three significant outages in the first half of 2025. If your product has no fallback path, your product breaks with the vendor.
The naive fallback is an error message. The useful fallback is a degraded experience — a static response, a cached result, a handoff to a human — that the user can work with. In a customer support context, that means routing to a human queue when AI confidence is low, not throwing a 500 error.
Build the fallback path before launch. Retrofitting it after an outage is miserable and rushed.
A note on infrastructure for Indian teams
For teams cost-sensitive about inference and working with non-sensitive data, Groq is worth a look — fast, cheap on LLaMA and Mixtral variants, and the latency is noticeably better than OpenAI for streaming applications. For data that can't leave your environment, running Ollama on a GPU instance (AWS g4dn, or a Hetzner GPU server at roughly a third of the AWS cost) closes the data-residency question entirely. We've done this for a healthcare client and it's more manageable than it sounds once you pick your model and size it correctly.