How AI assistants work: models, memory, tools, and permissions

Every AI assistant — from a S$10/month personal helper to a five-figure AI employee — is the same four layers: a model that reasons over language, memory that carries context across time, tools that let it touch the world, and permissions that decide what it may do without you. Vendors differentiate on marketing; systems differentiate on how honestly these four layers are built.

The language model (from providers like Anthropic, OpenAI, or Google) supplies raw capability: understanding requests, reasoning, producing text. It's also the layer buyers overweight most. Frontier models are increasingly comparable for everyday assistant work, and every serious product can swap models as they improve.

What the model alone cannot supply: knowledge of your business, memory of your preferences, access to your systems, or judgment about what it's allowed to do. A brilliant model with none of those is an eloquent stranger.

Models forget everything between conversations by default. Useful assistants add memory deliberately: your preferences and standing instructions; the state of open tasks ('waiting on the contractor's reply since Tuesday'); and your factual context — pulled in through what engineers call retrieval, where relevant documents are found and shown to the model when a question needs them.

Memory is also where privacy lives or dies. Ask any vendor: what exactly is stored, where, for how long, who can read it, and how do I delete it? Vague answers here disqualify a product regardless of how good the demo felt.

A model that can only produce text can only advise. Tools let an assistant act: search the web, read your calendar, send email, query a database, update a CRM. Each tool is a capability and a risk surface at once — which is why tool lists, not model names, are what to scrutinise when comparing products.

The engineering that matters: does the assistant check tool results (or trust blindly)? Does it recover when a tool fails mid-task? Can it explain, afterwards, which tools it used and why? Systems that log every tool call are auditable; systems that don't are a black box wearing a smile.

The difference between an assistant you can trust with real work and a liability is almost entirely here. Well-built systems express autonomy in stages: advise only; prepare drafts for approval; take specific low-risk actions within written rules; operate autonomously in narrow lanes, escalating exceptions. Each stage is earned when the previous one runs clean.

Concretely, permissions look like: send limits and approval thresholds, allowlists of who it may contact and which systems it may write to, audit logs of everything it did, and hard rules that need no judgment ('never quote below cost', 'never reply to a lawyer'). When we publish assistant concepts on this site, each one states its autonomy boundary explicitly — because a concept that won't state one isn't finished thinking.