A vendor-neutral reference architecture for LLM applications, the stack choices at each layer, a dated comparison of model providers and how build and run costs are structured, with a production readiness checklist.
Calling a large language model takes ten lines of code. Building an LLM application that answers correctly from your own data, respects who may see what, stays within budget, survives a provider outage and can be shown to work after every change takes rather more. The gap between the demo and the product is where most of the engineering, and most of the cost, sits.
This guide sets out a vendor-neutral reference architecture for LLM applications, the stack choices at each layer, a dated comparison of the main model providers, how build and run costs are structured, and a production readiness checklist. It reflects how we build our own AI products and client systems, from contact centre copilots to voice agents.
Key takeaways
- Treat the model as one replaceable component behind a gateway. Providers, models and prices change several times a year.
- Most answer quality comes from retrieval and context, not from the model: what you give the model to read decides what it can say.
- Build an evaluation set of real questions early and run it on every change. Without it you cannot tell an improvement from a regression.
- Run cost is driven by tokens per request, model choice and caching. In our worked example the same assistant costs from about $36 to $720 a month at list prices.
- Production needs the dull parts: timeouts and retries, permissions, logging, monitoring, spending limits and a fallback when the model is unavailable.
What counts as an LLM application
An LLM application is any product where a large language model does part of the work at run time: reading, writing, classifying, extracting or deciding what to do next. Common business examples:
- an assistant that answers customer or staff questions from your own documents;
- a copilot inside an existing tool that drafts replies, summaries or notes for a person to approve;
- a document pipeline that extracts fields from contracts, invoices or forms into a system;
- an agent that completes multi-step tasks through your APIs, covered in our guide to AI agents for business;
- a voice assistant that understands callers and answers or routes them.
They share one architecture, with layers switched on or off depending on the job.
Reference architecture for an LLM application
The layers below appear, in some form, in every production LLM system we build. Thinking of them separately keeps each one testable and replaceable.
| Layer | What it does | Typical choices | What goes wrong without it |
|---|---|---|---|
| Interface | Chat widget, copilot panel, API, WhatsApp, phone line | Web app, mobile app, messaging and telephony APIs | No feedback, no handover to people |
| Application and orchestration | Builds prompts, manages conversation state, calls tools, enforces rules | Python (FastAPI) or Node.js (NestJS) services | Logic scattered in prompts, untestable |
| Model gateway | One internal API in front of all model providers: routing, retries, timeouts, quotas, caching, cost tracking | Own service or an open-source gateway | Vendor lock-in, no cost control, outages take the product down |
| Retrieval | Finds the passages the model needs from your content, filtered by the user's permissions | PostgreSQL with pgvector, a search engine, or a vector database | Confident answers that are wrong or leak data |
| Tools | Typed functions the model can ask to call: look up an order, create a ticket | Your APIs, MCP servers | Model guesses instead of checking |
| Guardrails | Input and output checks, permissions, limits | Validation code, policy checks, approval steps | Prompt injection, unsafe output, runaway spend |
| Evaluation | Test sets, scoring and release gates | Own harness, Ragas for retrieval quality | Regressions ship unnoticed |
| Observability | Traces, logs, metrics, cost per request, user feedback | OpenTelemetry, a log store, dashboards | Problems found by customers first |
Source: AcmaSoft reference architecture; risks from the OWASP Top 10 for LLM applications (2025). Checked 13 Nov 2026.
The application layer owns the logic
Keep business rules in code, not in prompts. The prompt tells the model how to behave; the application decides who the user is, what they may see, which tools exist, how many steps are allowed and what happens with the answer. In our attpro CollectOS collections platform, for example, the Python FastAPI back end and its Celery workers, not the model, enforce conduct rules such as quiet hours, consent and do-not-call lists, and every call adds its recording, transcript and outcome to a hash-chained audit trail.
The model gateway keeps you portable
A gateway is one internal endpoint that every part of your product calls instead of calling providers directly. It handles authentication with each provider, timeouts and retries, routing by task (a small model for classification, a larger one for reasoning), fallback to a second provider, caching, per-tenant quotas and cost logging. When a better or cheaper model arrives, you change configuration, not application code. Given how often models change (the pricing pages we checked for this article list several model generations side by side), this layer pays for itself quickly.
Retrieval decides answer quality
Retrieval-augmented generation (RAG) means finding the relevant passages in your content and putting them in front of the model with the question. Its quality depends on decisions that have nothing to do with the model: how documents are split, which metadata is kept, how results are filtered by permission and how often the index is refreshed. For many business systems, pgvector in an existing PostgreSQL database is enough and avoids running another data store. Our explainer on how retrieval-augmented generation works covers these decisions step by step.
Retrieval must respect permissions. If a user cannot open a document in your intranet, the assistant must not quote it to them. Filter by access rights at query time, not after the answer is written.
Tools turn answers into actions
Tool calling lets the model request a function call, such as checking an order status, which your application executes and returns. It is supported across the major provider APIs. The Model Context Protocol is an open-source standard for exposing tools and data to AI applications, useful when several agents or models need the same systems. Expose only the actions a feature needs.
Guardrails and security
The OWASP Top 10 for LLM applications (2025) is the most practical checklist of what to design against. The entries we see most in client systems:
- Prompt injection (LLM01): instructions hidden in user input, documents or web pages. Keep system instructions separate, treat retrieved text as data and limit what tools can do.
- Sensitive information disclosure (LLM02): personal data or secrets reaching the wrong user. Filter retrieval by permission and minimise what you send to the model.
- Improper output handling (LLM05): passing model output straight into HTML, SQL or another system. Validate and escape it like any user input.
- Excessive agency (LLM06): tools with more power than the feature needs.
- System prompt leakage (LLM07): never put credentials or rules you rely on for security inside a prompt.
- Unbounded consumption (LLM10): no limits on request size, steps or spend.
Evaluation and observability
An LLM application needs its own test suite: a set of real questions or cases with expected answers or criteria, scored automatically where possible and by people where not. Run it before release and after every change to prompts, retrieval, tools or model. For retrieval quality, open-source tools such as Ragas measure whether the right passages were found and whether the answer sticks to them.
In production, trace each request from the user’s message through retrieval, model calls and tools to the answer, with tokens, latency and cost on every span. OpenTelemetry gives you a standard format so traces land in whichever monitoring tool you use. Add a feedback button; it is the cheapest source of new test cases.
Want a second opinion on your plans?
Tell us what you are working on. A specialist will look at it and reply with practical next steps.
Get in touchChoosing the stack
Languages and frameworks
Python has the deepest set of AI libraries and is our default for retrieval pipelines, evaluation and model-heavy services; see our Python development page. TypeScript on Node.js is a strong choice when the LLM features live inside an existing JavaScript product: our Acmafone CRM, which includes a copilot panel that suggests replies and finds answers in the knowledge base, runs on a NestJS API. Use whichever your team will maintain. Thin libraries that wrap provider APIs are fine; heavy frameworks that hide prompts and retries make debugging harder.
Data stores
Start with the database you already run. PostgreSQL with pgvector handles vector search alongside your normal data, with the same backups and access controls. Move to a dedicated vector database when volume, latency or filtering needs justify it; a planned comparison of vector databases will cover where that point lies. Keep conversation state, approvals and audit logs in your main database.
Hosting
Host the application wherever the rest of your product runs. The decision that matters more is where model inference happens, which the provider comparison below addresses.
Voice adds its own layer
Voice applications put speech recognition before the model and speech synthesis after it, with a strict latency budget for each hop. In our Acmafone PBX, a voice gateway streams calls to speech recognition and AI voice bots; in CollectOS, voice runs on LiveKit with Indian-language speech and language models. The same gateway, evaluation and logging rules apply.
Model providers compared
The table compares the main ways to buy model access as of 13 November 2026, with list prices from each provider’s own pricing page. Prices are per million tokens, standard tier, in US dollars, for short prompts. They change often; check the linked pages before budgeting.
| Provider | Example models and list price (input / output per 1M tokens) | Where inference can run | Notes |
|---|---|---|---|
| Anthropic Claude API | Claude Haiku 5.5 $0.10 / $0.50; Claude Sonnet 5.5 $2 / $10; Claude Opus 5.5 $4 / $20 | Global by default; US-only inference at a 1.1x multiplier on newer models | Batch API 50% off; prompt cache hits from $0.01 per 1M on Haiku 5.5 |
| OpenAI API | GPT-6 Luna $0.10 / $0.50; GPT-6.1 Sol $2 / $10; GPT-6 Astra $10 / $50 | Regional processing endpoints, 10% uplift for models released on or after 5 Mar 2026 | Batch and Flex listed at about half the standard price; cached input priced lower |
| Google Gemini API | Gemini 3.5 Flash-Lite $0.30 / $2.50; Gemini 4 Hydrogen $1.50 / $7.50; Gemini 4 Carbon $2 / $10 until 30 Nov 2026, then $4 / $20 | Gemini API, or Google Cloud through Vertex AI | Batch and Flex listed at half the standard price; Gemini 4 Carbon and Argon list prices double from 1 Dec 2026 |
| Microsoft Foundry (Azure) | OpenAI and other models | Global, Data Zone (US, EU or Asia Pacific) or a chosen Azure geography | Batch deployments at 50% less than Global Standard |
| Amazon Bedrock | Models from several providers | AWS regions | AWS states Bedrock never stores or uses your data to train models |
| Open-weight models, self-hosted | Llama and similar models | Your own cloud account or servers | You pay for compute and operations instead of tokens |
Source: Claude API pricing (platform.claude.com/docs); OpenAI API pricing (developers.openai.com); Gemini API pricing (Google AI Studio docs); Microsoft Foundry deployment types (learn.microsoft.com); Amazon Bedrock (aws.amazon.com). Checked 13 Nov 2026.
How to choose in practice:
- Start with quality on your own test set, not with benchmarks. Run your evaluation set on two or three candidate models and compare accuracy, latency and cost per request.
- Match residency to your markets. If customer data must stay in the EU, the US or a specific geography, use an option that processes it there, and check the provider’s data terms for each market you serve. Our AI adoption roadmap lists the main data laws in the eight markets we work in.
- Use more than one model. Route easy steps to small models and keep large ones for hard steps. The gateway makes this a configuration choice.
- Consider the cloud you already use. Buying model access through Azure, AWS or Google Cloud can simplify contracts, networking and billing, at the cost of slower access to the newest models in some cases.
Our comparison of GPT, Claude, Gemini, Llama and Mistral for business use goes model by model; the pricing pages linked above remain the primary source for prices.
Calling a model reliably
Provider APIs fail in ordinary ways: rate limits, overloaded servers, network timeouts. Every call needs a timeout, a bounded number of retries with backoff and jitter, respect for the provider’s retry hint, and no retries on errors that will never succeed (an invalid request or a bad key). The example below calls the Claude Messages API directly with the httpx library; the same pattern applies to any provider and belongs inside your gateway.
import os
import random
import time
import httpx
API_URL = "https://api.anthropic.com/v1/messages"
RETRYABLE = {408, 409, 429, 500, 502, 503, 504, 529}
client = httpx.Client(timeout=httpx.Timeout(60.0, connect=5.0))
def ask(question: str, context: str, max_attempts: int = 4) -> str:
"""Answer a question from the given context, retrying transient failures."""
payload = {
"model": os.environ["LLM_MODEL"],
"max_tokens": 800,
"system": "Answer only from the context. If the answer is not there, say so.",
"messages": [
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
],
}
headers = {
"x-api-key": os.environ["ANTHROPIC_API_KEY"],
"anthropic-version": "2023-06-01",
}
for attempt in range(1, max_attempts + 1):
retry_after = 0.0
try:
response = client.post(API_URL, json=payload, headers=headers)
except httpx.TransportError: # Timeouts and network errors.
if attempt == max_attempts:
raise
else:
if response.status_code == 200:
blocks = response.json()["content"]
return "".join(b["text"] for b in blocks if b["type"] == "text")
if response.status_code not in RETRYABLE or attempt == max_attempts:
response.raise_for_status()
retry_after = float(response.headers.get("retry-after", 0) or 0)
# Exponential backoff with jitter, capped, never shorter than the server's hint.
time.sleep(max(retry_after, min(2 ** (attempt - 1), 20) + random.random()))
raise RuntimeError("unreachable")In production, add three things around this function: log the model, token counts, latency and cost of every call; record which request and user it belongs to; and fall back to a second model or a polite “please try again” when retries run out. Provider SDKs include their own retry logic; whichever you use, set the timeout and retry count explicitly rather than relying on defaults.
What an LLM application costs
Costs fall into three budgets: building it, running it and improving it. Our general guide to AI development cost explains the build budget across AI project types; the table below shows what drives each line for an LLM application specifically.
| Budget | Line item | What drives it |
|---|---|---|
| Build | Discovery and design | Number of use cases, users, channels and languages |
| Build | Data and retrieval | State of your content: how scattered, outdated or duplicated it is |
| Build | Integrations and tools | Number of systems, API quality, permission model |
| Build | Evaluation set and harness | How many case types, how hard correct answers are to judge |
| Build | Interface and admin | New app versus a widget in an existing product; review and content tools for staff |
| Run | Model usage | Requests per month, tokens per request, model choice, caching |
| Run | Hosting and data stores | Application servers, database, vector index, logs |
| Run | Channel fees | WhatsApp, SMS and telephony charges where used |
| Improve | Review and upkeep | Hours to review conversations, update content and prompts, re-run tests, adopt new models |
Source: AcmaSoft delivery practice. No prices: effort depends on scope.
Worked example: monthly model cost of a support assistant
To show how much the run cost moves with model choice and caching, take a support assistant handling 20,000 conversations a month. Assume four turns per conversation, and on each turn 3,000 input tokens (instructions, retrieved passages and history) and 300 output tokens. At list prices on 13 November 2026:
| Model | 1,000 chats | 20,000 chats |
|---|---|---|
| Claude Haiku 5.5 | $1.80 | $36 |
| Claude Sonnet 5.5 with prompt caching | $17 | $340 |
| Claude Sonnet 5.5 without caching | $36 | $720 |
Source: Calculated by AcmaSoft from Claude API list prices (platform.claude.com/docs), standard tier, USD, checked 13 Nov 2026. 4 turns per conversation; 3,000 input and 300 output tokens per turn. Cached case: 2,500 input tokens per turn read from the prompt cache; cache writes excluded.
The same calculation with other providers’ list prices lands in similar bands for models of similar size. Two lessons generalise. The model bill for a well-scoped assistant is often smaller than the cost of the people who review and improve it. And the gap between a small model and a large one, or between caching on and off, is larger than the gap between providers, so routing and caching deserve design time.
Indexing your content is usually cheap. OpenAI lists text-embedding-3-small at $0.02 per million tokens, so embedding a knowledge base of 2,500 pages at roughly 1,000 tokens a page costs about five US cents at list price. The cost of retrieval is the work of cleaning and structuring the content, not the embeddings.
Production readiness checklist
We do not put an LLM feature in front of customers until each of these is true.
Ready for production
- An evaluation set of real cases passes the agreed threshold, and it runs automatically on every change
- All model calls go through one gateway with timeouts, bounded retries, fallback and cost logging
- Retrieval filters by the user's permissions at query time, and the index refresh schedule is set
- Model output is validated before it reaches HTML, a database or another system
- Tools have the narrowest permissions possible, and sensitive actions need approval
- Request size, steps per task and spend per user or tenant are capped
- Personal data sent to the model is minimised, and provider data terms and residency suit each market served
- Every request is traced with tokens, latency, cost and outcome, and dashboards alert on failures and spend
- Users can flag a bad answer, and flags feed the evaluation set
- The assistant hands over to a person when unsure or when asked, with the conversation attached
- An owner, a run budget and a review schedule are agreed
How we build LLM applications
Our generative AI development projects start with a short discovery: the questions or cases the system must handle, the content and systems behind them, and the markets and data rules involved. We then build a working slice end to end (interface, gateway, retrieval, evaluation and logging) on a small set of real cases, so cost per request and accuracy are measured early instead of estimated.
Depending on the product, the work runs under our AI app development service for new web and mobile apps, our ChatGPT and LLM integration service for adding features to existing software, or AI SaaS development for multi-tenant products, where per-tenant quotas, data separation and cost tracking matter most. If you have your own engineering team and need extra hands, you can hire AI and LLM developers to work in it.
Delivery runs remotely from India with one named lead, sprint demos and a staging environment. Office hours are 9:30 am to 6:30 pm India time, Monday to Saturday, with evening hours for North American clients by arrangement. Clients choose a fixed price for a defined first release, a dedicated team for an ongoing product, or time and materials while the scope is still being learned.
Planning an LLM feature?
Tell us what the system must answer or do and which systems hold the data. Our generative AI development team will suggest an architecture, a model shortlist to test and an estimate of the cost per request.
Planned articles in this cluster go deeper on single topics: RAG compared with fine-tuning and prompt engineering, LLM security against the OWASP Top 10, evaluating LLM output quality, running generative AI in production at a predictable cost, vector databases, AI content workflows and data preparation for AI.
Get new guides by email
Articles and guides from our team, and company news. Unsubscribe at any time.
About AcmaSoft
AcmaSoft designs and builds websites, mobile apps and custom software, and runs SEO and digital marketing for its clients.
- Years of experience
- 17+
- Clients served
- 650+
- Projects delivered
- 2,000+
- Products shipped
- 300+


