AI Services and Tools (2): APIs and Model Selection#

This chapter has two parts. The first uses the OpenAI API as an example to give you a feel for what you can do with an LLM API, and the second lays out the criteria for choosing what fits your service among the many vendors and models. Even if you are not a developer, knowing just the concept of an API will help you understand “what I can do” and set a direction.

What It Means to Use AI Through an API#

AI providers such as OpenAI run user-facing services like ChatGPT themselves, but they also expand the AI service ecosystem by offering APIs to outside developers. Because an API lets you deliver a high-quality AI service quickly instead of building your own model, it is a very sensible choice for a startup.

And (on a somewhat different note) when non-developers ask me where to start learning programming, I used to recommend small goals such as building a dynamic page with HTML and JavaScript or scraping a web page with Python. After I integrated an AI API, my ranking changed. Wiring up an AI API is the thing I now recommend most, because it is easy, fun, and motivating. The sample code below is short and intuitive, and that is exactly why.

What You Can Do with the OpenAI API#

Text generation#

The most basic API takes text input and generates a text response. In the OpenAI Python SDK, the Responses API(client.responses.create) is the main entry point. The older Chat Completions API is still available, but the Responses API is recommended for new development. instructions corresponds to “setting the scene (system instructions),” and input holds the user input (or the conversation history).

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
  model="gpt-5-mini",
  instructions="You are a helpful assistant.",
  input=[
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"},
  ],
)

print(response.output_text)

The thing to notice here is role. A role of user means “something the user said,” and assistant means “an answer the model gave earlier (conversation history).” What people usually call system instructions are the shared rules you give the AI before the conversation starts (role / tone / prohibitions / output format), and in this example they are managed separately in instructions. The five layers of context we saw in Chapter 3 split in code like this, into instructions (fixed guidance) and input (conversation history plus the current request).

Function calling#

Seen at the API level, the Function Calling described in Chapter 2 works as follows. Assume a [user (client) ↔ server ↔ model] structure.

1. When the server receives the user’s query and passes it to the model, it also sends the names, descriptions, and parameter specifications of its functions. There can be several functions, and all of them are implemented on the server.
2. When the model generates its response, if the situation calls for a function call, it includes “which function to call with which arguments” in the response and sends that to the server.
3. The server calls that function. The function contains whatever algorithms and external API calls it needs.
4. The server sends the function’s result back to the model, and only then does the model generate the final response, which the server receives and passes on to the user.

from openai import OpenAI
import json

client = OpenAI()

tools = [{
  "type": "function",
  "name": "get_weather",
  "description": "Get current weather for a city.",
  "parameters": {
    "type": "object",
    "properties": {"city": {"type": "string"}},
    "required": ["city"],
  },
}]

response = client.responses.create(
  model="gpt-5-mini",
  input=[{"role": "user", "content": "What's the weather in Seoul?"}],
  tools=tools,
)

# Check whether the model requested a function call, then have the server run the real function.
for item in response.output:
  if item.type == "function_call" and item.name == "get_weather":
    args = json.loads(item.arguments)
    result = {"city": args["city"], "temp_c": 27, "sky": "clear"}  # in practice, call a weather API
    final = client.responses.create(
      model="gpt-5-mini",
      previous_response_id=response.id,
      input=[{"type": "function_call_output", "call_id": item.call_id, "output": json.dumps(result)}],
    )
    print(final.output_text)

Adding Actions to a custom GPT so it can call external APIs is, from the server’s point of view, the same concept: “defining the tools (functions) the model may call when it needs them.”

Image generation#

There is an API that generates images from text (a prompt). Image generation uses a dedicated image model separate from the text generation models. The gpt-image family returns the generated image only as a Base64 string (URL responses apply only to the older dall-e family), so you need to decode the data and save it to a file.

import base64
from openai import OpenAI

client = OpenAI()

response = client.images.generate(
  model="gpt-image-1.5",
  prompt="a white siamese cat",
  size="1024x1024",
  quality="medium",  # low / medium / high
  n=1,
)

image_bytes = base64.b64decode(response.data[0].b64_json)
with open("siamese_cat.png", "wb") as f:
  f.write(image_bytes)

Running the code above saves the generated image as a PNG file. Prompt engineering matters if you want specific details. Besides generating from a text prompt, you can also supply an existing image and a masked region to produce an edited image for that region (client.images.edit()). The chapter illustrations in this book were made the same way (with a different vendor’s image model), by supplying one style-reference image and directing a different scene for each chapter.

Vision#

You can supply image data together with a request about it (a text prompt) and generate a text response. You can ask for a description of the image, ask whether a particular object appears in it, compare several images with one another, and so on.

from openai import OpenAI

client = OpenAI()

img_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

response = client.responses.create(
  model="gpt-5-mini",
  input=[
    {
      "role": "user",
      "content": [
        {"type": "input_text", "text": "What's in this image?"},
        {"type": "input_image", "image_url": img_url},
      ],
    }
  ],
)
print(response.output_text)

[Example response]
This image depicts a picturesque scene of a wooden boardwalk path stretching through a grassy field. The sky is blue with some scattered clouds, and the surrounding vegetation is lush and green, suggesting a summer or late spring setting.

Text to speech#

There is an API that takes text and generates an audio file for it. This feature uses dedicated TTS models (gpt-4o-mini-tts, plus the previous-generation tts-1 and tts-1-hd) rather than a text generation model. The newest model accepts an instructions parameter for directing the speaking style, such as tone or emotion. You can pick one of several preset voices, choose from several audio formats (MP3, Opus, AAC, FLAC, WAV, PCM), and it supports many languages. Generating a file for a long text can take a while, so streaming is also supported so you can respond to the user quickly.

from pathlib import Path
from openai import OpenAI

client = OpenAI()

speech_file_path = Path(__file__).parent / "text_to_speech_result.mp3"
with client.audio.speech.with_streaming_response.create(
  model="gpt-4o-mini-tts",
  voice="alloy",
  input="Today is a wonderful day to build something people love!",
) as response:
  response.stream_to_file(speech_file_path)

Speech to text#

There is an API that takes audio data and produces the corresponding text, with support for many languages. This feature uses dedicated transcription models (for example gpt-4o-transcribe, gpt-4o-mini-transcribe, and the previous-generation whisper-1).

from pathlib import Path
from openai import OpenAI

client = OpenAI()

audio_file = Path("text_to_speech_result.mp3")
transcription = client.audio.transcriptions.create(
  model="gpt-4o-transcribe",
  file=audio_file,
)

print(transcription.text)

With the whisper-1 model you can also receive timing information (timestamps) for when each word is spoken, so you can jump straight from a point in the audio file to the matching point in the text (the newer transcription models return only json / text formats, so use whisper-1 when you need word-level timestamps). You can also pass a text prompt along with the audio to improve transcription quality. For example, when the model hears “dah-lee,” it is not obvious whether to write “DALI” or “DALL-E,” but a prompt like “this audio is a talk about artificial intelligence and mentions DALL-E and GPT models” reduces such mistakes.

Embeddings#

The embeddings described in Chapter 1 are also offered as a dedicated API. You put in text and get out a numeric vector; you store those vectors, then compute the distance to a query vector to find “documents that are close in meaning.” The indexing and retrieval stages of RAG are built on top of this API.

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
  model="text-embedding-3-small",
  input="Refunds can be requested within 7 days of purchase.",
)
vector = response.data[0].embedding  # a list of 1536 floats

Moderation#

There is an API that takes text and checks it for harmful content. It scores the text across categories, such as whether it contains sexual content or hateful content. You use it as a safeguard before feeding user input to a model, or before showing model output to a user.

from openai import OpenAI

client = OpenAI()

response = client.moderations.create(input="Hello world!!")
print(response.results[0].flagged)  # True / False

Conversation state and built-in tools#

So far we have looked at APIs that handle one-off requests. To build a ChatGPT-like chatbot inside your own app, you also need “remembering the previous conversation” and “built-in tools such as file search or code execution.” In the past a separate Assistants API was offered for this, but that API was shut down in August 2026. Today the Responses API alone does the same job.

• Conversation state: chain onto the previous response with previous_response_id, or create a conversation on the server with the Conversations API and continue it through the conversation parameter. The client no longer has to resend the conversation history on every turn.
• Built-in tools: specify OpenAI-hosted tools such as file_search (search over uploaded files), code_interpreter (run Python), or web_search in tools, and the model calls them on its own when needed. Unlike the Function calling described above, OpenAI’s servers execute the tools for you.

from openai import OpenAI

client = OpenAI()

# Create a Conversation that stores the dialogue state on the server.
conversation = client.conversations.create()

while True:
  user_input = input("\nuser > ")
  if user_input.lower() in ["exit", "quit"]:
    break

  response = client.responses.create(
    model="gpt-5-mini",
    instructions="You are a personal math tutor. Write and run code to answer math questions.",
    tools=[{"type": "code_interpreter", "container": {"type": "auto"}}],
    conversation=conversation.id,
    input=user_input,
  )
  print("assistant >", response.output_text)

Parameter names and tool types can change, so check the official documentation when you actually implement this.

Other Vendors’ APIs Share the Same Structure#

APIs from other vendors such as Anthropic (Claude) and Google (Gemini) share the same broad outline of “text generation / vision / tool calling / embeddings / built-in tools.” Only the parameter names and response formats differ. For example, the same text generation written with the Anthropic SDK looks like this.

import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
  model="claude-sonnet-5-5",
  max_tokens=1024,
  system="You are a helpful assistant.",
  messages=[{"role": "user", "content": "Where was the 2020 World Series played?"}],
)
print(message.content[0].text)

instructions has become system and input has become messages, but the structure is identical. Once you learn one vendor’s API, the cost of moving to another is small. Details such as how prompt caching works (covered later) and the format of tool-call responses do differ by vendor, though, so when you actually switch vendors it is safer to keep a thin adapter layer that absorbs those differences.

Criteria for Choosing a Model#

Once you know what an API can do, the next question is “which vendor’s model should I use?” The decision criteria a solo startup actually runs into are summarized below.

There are three principles I apply in practice.

First, split the work and mix tiers. Even within a single product, classifying user queries, summarizing documents, and generating the final answer require different levels of intelligence. Use the top-tier model for every call and your costs multiply; use a lightweight model for every call and quality collapses. Decide “the minimum tier this task needs” at each call site, and use evals to confirm the choice.

Second, block vendor lock-in with a single adapter layer. Models change every few months, and so do prices. If your prompts and business logic are bound directly to one SDK’s response format, switching vendors becomes expensive. Wrap the call sites thinly, and “which model is cheap and good this month” becomes a single setting.

Third, let security requirements decide the deployment form. Choosing where to run the model means picking a point on the spectrum below.

One misconception about managed platforms is worth clearing up: a service like AWS Bedrock is not “a dedicated line to the model vendor.” The model itself runs inside the cloud provider’s infrastructure, so the core value is that your data is never sent directly to the model vendor, never leaves the region you specify, and your inputs and outputs are not used for training. If you serve security-sensitive B2B customers, that one sentence can decide whether a deal closes.

Subscriptions and APIs Are Different Products#

Finally, one trap developers often fall into. Because a monthly subscription to a coding agent is far cheaper than consuming the same volume through the API, it is tempting to think “what if I hook my subscription account up to my service’s backend and use it cheaply?” The answer is “you can’t.” A subscription is an individual’s right to use a product the provider controls; having your product call a model on behalf of its users must be built on the separate economy called the API. The terms of service say so, and attempts to spend subscription quotas through third-party routes have been blocked.

So a solo startup’s AI costs fall into two buckets: the subscriptions I use to do my own work (coding agents, chat services) and the API calls my product makes on behalf of its users. Keeping the two separate rather than mixing them is the starting point of cost design, and the concrete ways to reduce the API side (caching, batching, tier routing) are covered in “Adding AI Features to a Product” in the implementation chapters.

Summary#

• An LLM API consists of text generation, tool calling, images, vision, audio, embeddings, moderation, and built-in tools, and the structure is the same across vendors.
• Choose a model on seven criteria: tier, cost structure, context, tool-calling reliability, latency, data policy, and ecosystem.
• Mix tiers per task, block vendor lock-in with one adapter layer, and match the deployment form to security requirements.
• The subscriptions I use and the API my product calls are different products, and their costs are managed separately.