Skip to content

This site is shown in English. Switch language if you prefer.

Menu

Monday to Saturday, 9:30 am to 6:30 pm India time (IST)

Contact us

How to build an LLM application: architecture, stack and costs in 2026

Chief Technology and AI Officer

  • Published
  • 15 min read
A chat window sits on a stack of six layers: gateway, retrieval, tools, model, evaluation and observability. Three model providers plug into the model layer, a code card shows an API call with retries and a cost card splits build and run costs.
Table of contentsContents
  1. What counts as an LLM application
  2. Reference architecture for an LLM application
    1. The application layer owns the logic
    2. The model gateway keeps you portable
    3. Retrieval decides answer quality
    4. Tools turn answers into actions
    5. Guardrails and security
    6. Evaluation and observability
  3. Choosing the stack
    1. Languages and frameworks
    2. Data stores
    3. Hosting
    4. Voice adds its own layer
  4. Model providers compared
  5. Calling a model reliably
  6. What an LLM application costs
    1. Worked example: monthly model cost of a support assistant
  7. Production readiness checklist
  8. How we build LLM applications
  9. Frequently asked questions

Share this article

A vendor-neutral reference architecture for LLM applications, the stack choices at each layer, a dated comparison of model providers and how build and run costs are structured, with a production readiness checklist.

Calling a large language model takes ten lines of code. Building an LLM application that answers correctly from your own data, respects who may see what, stays within budget, survives a provider outage and can be shown to work after every change takes rather more. The gap between the demo and the product is where most of the engineering, and most of the cost, sits.

This guide sets out a vendor-neutral reference architecture for LLM applications, the stack choices at each layer, a dated comparison of the main model providers, how build and run costs are structured, and a production readiness checklist. It reflects how we build our own AI products and client systems, from contact centre copilots to voice agents.

Key takeaways

  • Treat the model as one replaceable component behind a gateway. Providers, models and prices change several times a year.
  • Most answer quality comes from retrieval and context, not from the model: what you give the model to read decides what it can say.
  • Build an evaluation set of real questions early and run it on every change. Without it you cannot tell an improvement from a regression.
  • Run cost is driven by tokens per request, model choice and caching. In our worked example the same assistant costs from about $36 to $720 a month at list prices.
  • Production needs the dull parts: timeouts and retries, permissions, logging, monitoring, spending limits and a fallback when the model is unavailable.

What counts as an LLM application

An LLM application is any product where a large language model does part of the work at run time: reading, writing, classifying, extracting or deciding what to do next. Common business examples:

  • an assistant that answers customer or staff questions from your own documents;
  • a copilot inside an existing tool that drafts replies, summaries or notes for a person to approve;
  • a document pipeline that extracts fields from contracts, invoices or forms into a system;
  • an agent that completes multi-step tasks through your APIs, covered in our guide to AI agents for business;
  • a voice assistant that understands callers and answers or routes them.

They share one architecture, with layers switched on or off depending on the job.

Reference architecture for an LLM application

The layers below appear, in some form, in every production LLM system we build. Thinking of them separately keeps each one testable and replaceable.

The layers of an LLM application
LayerWhat it doesTypical choicesWhat goes wrong without it
InterfaceChat widget, copilot panel, API, WhatsApp, phone lineWeb app, mobile app, messaging and telephony APIsNo feedback, no handover to people
Application and orchestrationBuilds prompts, manages conversation state, calls tools, enforces rulesPython (FastAPI) or Node.js (NestJS) servicesLogic scattered in prompts, untestable
Model gatewayOne internal API in front of all model providers: routing, retries, timeouts, quotas, caching, cost trackingOwn service or an open-source gatewayVendor lock-in, no cost control, outages take the product down
RetrievalFinds the passages the model needs from your content, filtered by the user's permissionsPostgreSQL with pgvector, a search engine, or a vector databaseConfident answers that are wrong or leak data
ToolsTyped functions the model can ask to call: look up an order, create a ticketYour APIs, MCP serversModel guesses instead of checking
GuardrailsInput and output checks, permissions, limitsValidation code, policy checks, approval stepsPrompt injection, unsafe output, runaway spend
EvaluationTest sets, scoring and release gatesOwn harness, Ragas for retrieval qualityRegressions ship unnoticed
ObservabilityTraces, logs, metrics, cost per request, user feedbackOpenTelemetry, a log store, dashboardsProblems found by customers first

Source: AcmaSoft reference architecture; risks from the OWASP Top 10 for LLM applications (2025). Checked 13 Nov 2026.

The application layer owns the logic

Keep business rules in code, not in prompts. The prompt tells the model how to behave; the application decides who the user is, what they may see, which tools exist, how many steps are allowed and what happens with the answer. In our attpro CollectOS collections platform, for example, the Python FastAPI back end and its Celery workers, not the model, enforce conduct rules such as quiet hours, consent and do-not-call lists, and every call adds its recording, transcript and outcome to a hash-chained audit trail.

The model gateway keeps you portable

A gateway is one internal endpoint that every part of your product calls instead of calling providers directly. It handles authentication with each provider, timeouts and retries, routing by task (a small model for classification, a larger one for reasoning), fallback to a second provider, caching, per-tenant quotas and cost logging. When a better or cheaper model arrives, you change configuration, not application code. Given how often models change (the pricing pages we checked for this article list several model generations side by side), this layer pays for itself quickly.

Retrieval decides answer quality

Retrieval-augmented generation (RAG) means finding the relevant passages in your content and putting them in front of the model with the question. Its quality depends on decisions that have nothing to do with the model: how documents are split, which metadata is kept, how results are filtered by permission and how often the index is refreshed. For many business systems, pgvector in an existing PostgreSQL database is enough and avoids running another data store. Our explainer on how retrieval-augmented generation works covers these decisions step by step.

Retrieval must respect permissions. If a user cannot open a document in your intranet, the assistant must not quote it to them. Filter by access rights at query time, not after the answer is written.

Tools turn answers into actions

Tool calling lets the model request a function call, such as checking an order status, which your application executes and returns. It is supported across the major provider APIs. The Model Context Protocol is an open-source standard for exposing tools and data to AI applications, useful when several agents or models need the same systems. Expose only the actions a feature needs.

Guardrails and security

The OWASP Top 10 for LLM applications (2025) is the most practical checklist of what to design against. The entries we see most in client systems:

  • Prompt injection (LLM01): instructions hidden in user input, documents or web pages. Keep system instructions separate, treat retrieved text as data and limit what tools can do.
  • Sensitive information disclosure (LLM02): personal data or secrets reaching the wrong user. Filter retrieval by permission and minimise what you send to the model.
  • Improper output handling (LLM05): passing model output straight into HTML, SQL or another system. Validate and escape it like any user input.
  • Excessive agency (LLM06): tools with more power than the feature needs.
  • System prompt leakage (LLM07): never put credentials or rules you rely on for security inside a prompt.
  • Unbounded consumption (LLM10): no limits on request size, steps or spend.

Evaluation and observability

An LLM application needs its own test suite: a set of real questions or cases with expected answers or criteria, scored automatically where possible and by people where not. Run it before release and after every change to prompts, retrieval, tools or model. For retrieval quality, open-source tools such as Ragas measure whether the right passages were found and whether the answer sticks to them.

In production, trace each request from the user’s message through retrieval, model calls and tools to the answer, with tokens, latency and cost on every span. OpenTelemetry gives you a standard format so traces land in whichever monitoring tool you use. Add a feedback button; it is the cheapest source of new test cases.

Want a second opinion on your plans?

Tell us what you are working on. A specialist will look at it and reply with practical next steps.

Get in touch

Choosing the stack

Languages and frameworks

Python has the deepest set of AI libraries and is our default for retrieval pipelines, evaluation and model-heavy services; see our Python development page. TypeScript on Node.js is a strong choice when the LLM features live inside an existing JavaScript product: our Acmafone CRM, which includes a copilot panel that suggests replies and finds answers in the knowledge base, runs on a NestJS API. Use whichever your team will maintain. Thin libraries that wrap provider APIs are fine; heavy frameworks that hide prompts and retries make debugging harder.

Data stores

Start with the database you already run. PostgreSQL with pgvector handles vector search alongside your normal data, with the same backups and access controls. Move to a dedicated vector database when volume, latency or filtering needs justify it; a planned comparison of vector databases will cover where that point lies. Keep conversation state, approvals and audit logs in your main database.

Hosting

Host the application wherever the rest of your product runs. The decision that matters more is where model inference happens, which the provider comparison below addresses.

Voice adds its own layer

Voice applications put speech recognition before the model and speech synthesis after it, with a strict latency budget for each hop. In our Acmafone PBX, a voice gateway streams calls to speech recognition and AI voice bots; in CollectOS, voice runs on LiveKit with Indian-language speech and language models. The same gateway, evaluation and logging rules apply.

Model providers compared

The table compares the main ways to buy model access as of 13 November 2026, with list prices from each provider’s own pricing page. Prices are per million tokens, standard tier, in US dollars, for short prompts. They change often; check the linked pages before budgeting.

Model providers and hosting options, as of 13 Nov 2026
ProviderExample models and list price (input / output per 1M tokens)Where inference can runNotes
Anthropic Claude APIClaude Haiku 5.5 $0.10 / $0.50; Claude Sonnet 5.5 $2 / $10; Claude Opus 5.5 $4 / $20Global by default; US-only inference at a 1.1x multiplier on newer modelsBatch API 50% off; prompt cache hits from $0.01 per 1M on Haiku 5.5
OpenAI APIGPT-6 Luna $0.10 / $0.50; GPT-6.1 Sol $2 / $10; GPT-6 Astra $10 / $50Regional processing endpoints, 10% uplift for models released on or after 5 Mar 2026Batch and Flex listed at about half the standard price; cached input priced lower
Google Gemini APIGemini 3.5 Flash-Lite $0.30 / $2.50; Gemini 4 Hydrogen $1.50 / $7.50; Gemini 4 Carbon $2 / $10 until 30 Nov 2026, then $4 / $20Gemini API, or Google Cloud through Vertex AIBatch and Flex listed at half the standard price; Gemini 4 Carbon and Argon list prices double from 1 Dec 2026
Microsoft Foundry (Azure)OpenAI and other modelsGlobal, Data Zone (US, EU or Asia Pacific) or a chosen Azure geographyBatch deployments at 50% less than Global Standard
Amazon BedrockModels from several providersAWS regionsAWS states Bedrock never stores or uses your data to train models
Open-weight models, self-hostedLlama and similar modelsYour own cloud account or serversYou pay for compute and operations instead of tokens

Source: Claude API pricing (platform.claude.com/docs); OpenAI API pricing (developers.openai.com); Gemini API pricing (Google AI Studio docs); Microsoft Foundry deployment types (learn.microsoft.com); Amazon Bedrock (aws.amazon.com). Checked 13 Nov 2026.

How to choose in practice:

  • Start with quality on your own test set, not with benchmarks. Run your evaluation set on two or three candidate models and compare accuracy, latency and cost per request.
  • Match residency to your markets. If customer data must stay in the EU, the US or a specific geography, use an option that processes it there, and check the provider’s data terms for each market you serve. Our AI adoption roadmap lists the main data laws in the eight markets we work in.
  • Use more than one model. Route easy steps to small models and keep large ones for hard steps. The gateway makes this a configuration choice.
  • Consider the cloud you already use. Buying model access through Azure, AWS or Google Cloud can simplify contracts, networking and billing, at the cost of slower access to the newest models in some cases.

Our comparison of GPT, Claude, Gemini, Llama and Mistral for business use goes model by model; the pricing pages linked above remain the primary source for prices.

Calling a model reliably

Provider APIs fail in ordinary ways: rate limits, overloaded servers, network timeouts. Every call needs a timeout, a bounded number of retries with backoff and jitter, respect for the provider’s retry hint, and no retries on errors that will never succeed (an invalid request or a bad key). The example below calls the Claude Messages API directly with the httpx library; the same pattern applies to any provider and belongs inside your gateway.

llm_call.py
import os
import random
import time

import httpx

API_URL = "https://api.anthropic.com/v1/messages"
RETRYABLE = {408, 409, 429, 500, 502, 503, 504, 529}

client = httpx.Client(timeout=httpx.Timeout(60.0, connect=5.0))


def ask(question: str, context: str, max_attempts: int = 4) -> str:
    """Answer a question from the given context, retrying transient failures."""
    payload = {
        "model": os.environ["LLM_MODEL"],
        "max_tokens": 800,
        "system": "Answer only from the context. If the answer is not there, say so.",
        "messages": [
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
        ],
    }
    headers = {
        "x-api-key": os.environ["ANTHROPIC_API_KEY"],
        "anthropic-version": "2023-06-01",
    }

    for attempt in range(1, max_attempts + 1):
        retry_after = 0.0
        try:
            response = client.post(API_URL, json=payload, headers=headers)
        except httpx.TransportError:  # Timeouts and network errors.
            if attempt == max_attempts:
                raise
        else:
            if response.status_code == 200:
                blocks = response.json()["content"]
                return "".join(b["text"] for b in blocks if b["type"] == "text")
            if response.status_code not in RETRYABLE or attempt == max_attempts:
                response.raise_for_status()
            retry_after = float(response.headers.get("retry-after", 0) or 0)

        # Exponential backoff with jitter, capped, never shorter than the server's hint.
        time.sleep(max(retry_after, min(2 ** (attempt - 1), 20) + random.random()))

    raise RuntimeError("unreachable")

In production, add three things around this function: log the model, token counts, latency and cost of every call; record which request and user it belongs to; and fall back to a second model or a polite “please try again” when retries run out. Provider SDKs include their own retry logic; whichever you use, set the timeout and retry count explicitly rather than relying on defaults.

What an LLM application costs

Costs fall into three budgets: building it, running it and improving it. Our general guide to AI development cost explains the build budget across AI project types; the table below shows what drives each line for an LLM application specifically.

Cost structure of an LLM application
BudgetLine itemWhat drives it
BuildDiscovery and designNumber of use cases, users, channels and languages
BuildData and retrievalState of your content: how scattered, outdated or duplicated it is
BuildIntegrations and toolsNumber of systems, API quality, permission model
BuildEvaluation set and harnessHow many case types, how hard correct answers are to judge
BuildInterface and adminNew app versus a widget in an existing product; review and content tools for staff
RunModel usageRequests per month, tokens per request, model choice, caching
RunHosting and data storesApplication servers, database, vector index, logs
RunChannel feesWhatsApp, SMS and telephony charges where used
ImproveReview and upkeepHours to review conversations, update content and prompts, re-run tests, adopt new models

Source: AcmaSoft delivery practice. No prices: effort depends on scope.

Worked example: monthly model cost of a support assistant

To show how much the run cost moves with model choice and caching, take a support assistant handling 20,000 conversations a month. Assume four turns per conversation, and on each turn 3,000 input tokens (instructions, retrieved passages and history) and 300 output tokens. At list prices on 13 November 2026:

Monthly model cost of 20,000 conversations, worked example
Model1,000 chats20,000 chats
Claude Haiku 5.5$1.80$36
Claude Sonnet 5.5 with prompt caching$17$340
Claude Sonnet 5.5 without caching$36$720

Source: Calculated by AcmaSoft from Claude API list prices (platform.claude.com/docs), standard tier, USD, checked 13 Nov 2026. 4 turns per conversation; 3,000 input and 300 output tokens per turn. Cached case: 2,500 input tokens per turn read from the prompt cache; cache writes excluded.

The same calculation with other providers’ list prices lands in similar bands for models of similar size. Two lessons generalise. The model bill for a well-scoped assistant is often smaller than the cost of the people who review and improve it. And the gap between a small model and a large one, or between caching on and off, is larger than the gap between providers, so routing and caching deserve design time.

Indexing your content is usually cheap. OpenAI lists text-embedding-3-small at $0.02 per million tokens, so embedding a knowledge base of 2,500 pages at roughly 1,000 tokens a page costs about five US cents at list price. The cost of retrieval is the work of cleaning and structuring the content, not the embeddings.

Production readiness checklist

We do not put an LLM feature in front of customers until each of these is true.

Ready for production

  • An evaluation set of real cases passes the agreed threshold, and it runs automatically on every change
  • All model calls go through one gateway with timeouts, bounded retries, fallback and cost logging
  • Retrieval filters by the user's permissions at query time, and the index refresh schedule is set
  • Model output is validated before it reaches HTML, a database or another system
  • Tools have the narrowest permissions possible, and sensitive actions need approval
  • Request size, steps per task and spend per user or tenant are capped
  • Personal data sent to the model is minimised, and provider data terms and residency suit each market served
  • Every request is traced with tokens, latency, cost and outcome, and dashboards alert on failures and spend
  • Users can flag a bad answer, and flags feed the evaluation set
  • The assistant hands over to a person when unsure or when asked, with the conversation attached
  • An owner, a run budget and a review schedule are agreed

How we build LLM applications

Our generative AI development projects start with a short discovery: the questions or cases the system must handle, the content and systems behind them, and the markets and data rules involved. We then build a working slice end to end (interface, gateway, retrieval, evaluation and logging) on a small set of real cases, so cost per request and accuracy are measured early instead of estimated.

Depending on the product, the work runs under our AI app development service for new web and mobile apps, our ChatGPT and LLM integration service for adding features to existing software, or AI SaaS development for multi-tenant products, where per-tenant quotas, data separation and cost tracking matter most. If you have your own engineering team and need extra hands, you can hire AI and LLM developers to work in it.

Delivery runs remotely from India with one named lead, sprint demos and a staging environment. Office hours are 9:30 am to 6:30 pm India time, Monday to Saturday, with evening hours for North American clients by arrangement. Clients choose a fixed price for a defined first release, a dedicated team for an ongoing product, or time and materials while the scope is still being learned.

Planning an LLM feature?

Tell us what the system must answer or do and which systems hold the data. Our generative AI development team will suggest an architecture, a model shortlist to test and an estimate of the cost per request.

Planned articles in this cluster go deeper on single topics: RAG compared with fine-tuning and prompt engineering, LLM security against the OWASP Top 10, evaluating LLM output quality, running generative AI in production at a predictable cost, vector databases, AI content workflows and data preparation for AI.

Share this article

About the author:

Chief Technology and AI Officer

Amit leads technology, engineering and the AI practice at AcmaSoft. He sets the technical direction for client projects, from architecture and code quality to the AI and automation work we deliver.

  • Technology and engineering
  • Architecture and code quality
  • AI practice
  • Automation

Get new guides by email

Articles and guides from our team, and company news. Unsubscribe at any time.

About AcmaSoft

AcmaSoft designs and builds websites, mobile apps and custom software, and runs SEO and digital marketing for its clients.

Years of experience
17+
Clients served
650+
Projects delivered
2,000+
Products shipped
300+

Frequently asked questions

Short answers to the questions readers ask most about this topic.

Ask a question
How long does it take to build an LLM application?

A focused first release, such as an assistant that answers from one body of content on one channel, can reach a tested pilot in weeks rather than months when the content is in reasonable shape. Integrations with several systems, multiple channels or languages, and regulated data add time. Content preparation is the item most often underestimated.

Should we use one model provider or several?

Design for several, even if you start with one. Put a gateway in front of the models so you can route easy tasks to small, cheap models, fall back when a provider has an outage, and switch when a better or cheaper model arrives without changing application code.

Do we need to fine-tune a model?

Usually not for a first application. Retrieval from your own content and well-written instructions solve most business use cases, and they are easier to update. Fine-tuning can help with narrow, high-volume tasks that need a fixed format or style. Test retrieval and prompting first.

How do we keep customer data safe in an LLM application?

Send the model only the data each request needs, filter retrieval by the user’s permissions, validate output before it reaches other systems, log access, and choose a provider and region whose data terms suit your markets. Read each provider’s terms on whether API data is stored or used for training.

What does an LLM application cost to run each month?

It depends on requests, tokens per request, model choice and caching. In our worked example of 20,000 support conversations a month at list prices on 13 November 2026, model usage ranged from about $36 on a small model to $720 on a larger one without caching. Add hosting, monitoring, channel fees and review time.

Which programming language is best for LLM applications?

Python has the widest choice of AI, retrieval and evaluation libraries, so it is our default for model-heavy services. TypeScript on Node.js works well when the features live in an existing JavaScript product. The language matters less than having a gateway, tests and observability.

Discuss your project with a specialist

Share what you are planning or trying to fix. We reply with practical advice, a rough timeline and the questions we need answered.

Estimated budget (optional)

Goals, users, key features and any deadline.0 / 2,000

Attachment (optional)

Protected against spam without puzzles. We never share your details.

Choose your region & language

Americas

Asia-Pacific

Europe, Middle East and Africa

Translations are produced automatically by Google Translate.