Execution (3): Adding AI Features to Your Product#

If the previous chapter was about “building a product with agents,” this chapter is about “putting AI inside the product.” They are different problems. The former is about the tools I use; the latter is about the structure and cost of my product calling a model on behalf of its users. In a one-person startup, mixing the two breaks your cost structure, and separating them simplifies your design.

Separate the Two Kinds of AI Cost#

As the API chapter in Part 3 said, a coding agent’s monthly subscription is so much cheaper than consuming the same volume through the API that it is tempting to think “what if I connect my subscription account to my service’s backend and use it cheaply?” I looked into the technically possible routes myself, but the conclusion was clear. An AI subscription is not a discounted API product. A subscription carries time-based limits grounded in compute rather than exact token counts, and its economics hold only inside the provider’s own product, where the provider can control the workload. Routes where a third party calls the API with subscription credentials are blocked or billed as separate credits, and pooling the limits of multiple accounts to use them like server capacity violates the terms of service. When your business’s cost structure depends on a provider looking the other way, that is not optimization; it is technical debt.

So the principle is simple. Use subscriptions for development and conversation that a person performs directly, and from day one treat API cost as cost of goods for everything the product calls on behalf of users. The rest of this chapter is about how to design and reduce that API cost.

Architecture: Calls on the Server, Models as Parts#

The basic structure for putting AI in a product is the “user app ↔ my server ↔ model API” we saw in Part 3. On top of it, there are three design principles a one-person startup should follow from the start.

First, API keys live only on the server. Calling the model directly from the client exposes the key, makes usage impossible to control, and reveals your prompts and logic as-is. Every call goes through the server, and the server handles usage limits, logging, cost aggregation, and caching.

Second, wrap model calls in a single adapter layer. Models change every few months, and so do prices. If your business logic is bound directly to a particular SDK’s response format, switching vendors becomes expensive. Wrap the call sites thinly, and “which model is cheap and good this month” becomes a single setting. Details such as how prompt caching works and the tool-call response format differ by vendor, though, so the adapter has to absorb those differences.

Third, decide a tier for each call site. Even within one product, classifying user queries, summarizing documents, and generating the final answer require different levels of intelligence. Use the top-tier model for every call and your costs multiply; use a lightweight model for every call and quality collapses. Decide “the minimum tier this task needs” at each call site, and confirm the choice with the evals covered in the next chapter. When designing a single feature, I fill in the table below first.

The Heart of Cost Design: Prompt Caching#

Chapter 1 said that a token is both a unit of length and a unit of currency. Once you put AI in a product, you learn that most of that currency goes to input tokens, and in particular to the leading part that repeats on every request: system instructions, tool definitions, fixed documents. Agent features are worse, because they resend the entire history every turn. Prompt caching is a feature where the provider stores that repetition and charges a much lower rate for it, and used properly it removes a large share of input cost.

The most important fact I learned from digging deep into caching is that caching works on prefixes. The request has to be byte-for-byte identical to a previous request from the start up to some point for that span to be reused. Change even one character in the middle and everything after it is recomputed. So caching optimization is not a question of “what to cache” but of “what to put in front,” and the answer is to stack the prompt in layers (strata) with different rates of change.

Ranked by impact, the practical rules come out as follows.

1. Treat system instructions as a build-time constant. The moment you put values like the current time, the user’s name, or a session ID into the system instructions, your cache hit rate becomes permanently zero. This one mistake was the most common thing separating 0% from 90%.
2. Fix the tool list so it is shared by all users. Tool definitions sit at the very front, so if you split the list by permission there is no way to recover. Give everyone the full set, and check permissions in the execution handler.
3. Make serialization deterministic. The cache key is the rendered bytes. If JSON key order wobbles, the same content becomes a different prompt.
4. Only append to conversation history. Editing or trimming the middle invalidates everything after that point.
5. Put search results at the very end. If something that changes with every query sits in front, everything behind it is recomputed every time.

And there is one trap. The cache hit rate is not the goal. A hit rate is a ratio; cost is an absolute. Carry a long history around and the hit rate approaches 99%, but the cost per request keeps rising. The cache discount applies equally to both sides, so a context ten times larger is still ten times more expensive after the discount. Use the hit rate as a diagnostic that catches “is the prefix wobbling,” and judge the final cost by total context volume. Whether to summarize and compress a long conversation is decided not by the hit rate but by “compression cost versus the cumulative savings afterward.” Cache reads also count in full against per-minute throughput limits, so a long history eats throughput as well as cost.

Beyond caching, there are two more. Work that does not need a real-time response (overnight aggregation, bulk classification) can go through the batch API, which often cuts the rate roughly in half. And every call gets a budget cap. The server enforces per-user and per-task caps, and a task that hits its cap is treated as a signal that “something is going wrong” before it is treated as a cost problem.

Where to Run It: Public APIs and Managed Platforms#

Most one-person startups can start with a vendor’s public API. It is the easiest, and you get the latest models immediately. But once your customers are companies and they start asking “does our data leave the building,” you have to move toward the right side of the deployment spectrum we saw in Part 3.

Knowing precisely what a managed platform (AWS Bedrock, Google Vertex AI, Azure, and so on) means helps with sales here. I too initially thought Bedrock was “a dedicated line to the model vendor,” but in reality it is a structure where the cloud provider loads the model weights onto its own infrastructure and runs inference directly. That is why it can guarantee that inputs are not sent to the model vendor, never leave the region you specify, and are not used for training. The weights of closed models are delivered encrypted and decrypted only inside a verified execution environment, and by design there is no path by which even the cloud provider’s operators can reach them. It is not a “trust us” policy but a structure where “the access path itself has been removed.” When pitching security-sensitive customers, prepare a checklist: no training on data, region pinning, network isolation, access control, audit logs, and compliance certifications.

Managed platform pricing is usually the same as or close to the vendor’s direct API. You get the security guarantees at no extra cost, so if you have B2B in mind, it is worth opening this route in the adapter layer from the start.

Adding Agent Features: Borrow the Loop, Own the Control Plane#

To put “a feature that does work” rather than “a feature that answers” into your product, you need an agent loop, a sandbox, and tool-execution infrastructure. As the previous chapter said, building this from scratch is irrational for a solo developer. There are two options.

One is to use a coding agent runtime as the execution engine. Run the official runtime in an isolated work environment, and have the product own the control plane that handles task creation, approval, cost tracking, and result logging. The other is a hosted agent runtime. The vendor provides the agent loop, sandbox, session state, context compaction, failure recovery, and token tracking all built in; the developer declares an agent (model + instructions + tools) and an environment (packages, network, secrets), then creates a session and runs it. The brain (model and harness), the hands (sandbox and tools), and the session (an event log persisted outside the context) are separated, so even if the container dies the session survives and a new harness reads the log and resumes.

I compared the multi-agent system we built ourselves at my company with a hosted runtime. Functionally they were interchangeable, but there were trade-offs. Tool calls can change from in-process function calls to network round trips, some conditional branching in the pipeline has to be controlled by prompts rather than code, and advanced features are sometimes still in beta. On the other hand, deployment finishes with a single API call and you do not have to implement state management or failure recovery yourself, which is a big advantage for a one-person startup. Either way, what the product must own is the user experience, approval structure, work history, cost policy, security policy, and domain data, not the loop itself.

Distribution: Bringing Your Product into the User’s Agent#

Finally, there is an option that runs in the opposite direction from putting AI into your product: bringing your product into the AI. The MCP server from Chapter 2 is the channel. Expose your service’s data and features as an MCP server, and users call your product from inside the agents they already use, such as Claude, ChatGPT, and Cursor. The user does not need to open a new app, and you do not need to build a UI.

This direction is attractive for two reasons. First, distribution cost. Instead of acquiring users from an app store or search, your feature sits inside the agent where the user spends the whole day. Second, the shift in AI cost. The user’s agent calls the model, so your product only has to provide data and features, with no model cost. It is a structure where the user’s subscription pays for inference. Whether this opens up more widely remains to be seen (providers have begun officially supporting “sign in with our account” for third parties, but the scope is still limited), but an MCP server itself can be built right now at almost no cost.

Of course, opening an MCP server brings security responsibilities. Who reads and writes your data with what permissions, how to prevent mistaken calls by agents, and how to limit usage are covered in the next chapter.

Checklist#

1. Track subscriptions (my tools) and the API (product cost of goods) in separate cost accounts.
2. Keep API keys only on the server, and route every call through the server.
3. Wrap model calls in a single adapter layer, and decide a tier and a budget cap for each call site.
4. Stack the prompt in order of rate of change. Put no variable values in the system instructions, and put search results at the very end.
5. Judge cost by total context volume, not by cache hit rate.
6. Send non-real-time work through the batch API.
7. Once B2B customers appear, open the managed platform route in the adapter.
8. Borrow the agent loop, own the control plane.
9. Consider exposing your data and features as an MCP server as a distribution channel.