LangChain on an OpenAI-compatible API
The two fields ChatOpenAI needs to reach this endpoint, why invoke, stream and tool calls pass through unchanged, and how cache reads keep an agent loop cheap.
LangChain does not have to know it is talking to us. Its OpenAI chat model reads a base URL out of its own constructor, and this gateway answers in the OpenAI wire format, so a chain written against OpenAI runs here once two arguments change. Nothing below is a fork of langchain-openai, a wrapper around it or a patched copy of it: it is the package you already have, configured.
The two fields that matter
ChatOpenAI takes a long list of arguments and nearly all of them stay where they are. Two do not — the address it sends to and the key it sends with:
import os
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model="terra",
base_url="https://aifromu.com/v1",
api_key=os.environ["AIFROMU_API_KEY"],
)
print(llm.invoke("Name three uses for a cache read.").content)
That base URL is printed by this page rather than typed into it. It is the address this deployment actually serves, so the snippet works against whichever host you are reading it on instead of against a hostname somebody compiled into an example once. The key comes out of the environment for the reason keys always do: a key written into a constructor is a key committed to a repository, and rotating it afterwards means editing code rather than changing a variable.
Everything else about the object is the library's. invoke returns the finished message and .content is the text; a list of messages works where a bare string does; ChatPromptTemplate | llm | StrOutputParser() composes exactly as it did. None of that is ours to change, which is the whole argument for serving somebody else's wire format rather than inventing one.
Streaming, tools and structured output
llm.stream("...") hands you the answer in pieces as the model writes it. The server-sent events pass through this gateway to your process, so the loop you write around them is the library's own:
for chunk in llm.stream("Explain a cache read in two sentences."):
print(chunk.content, end="", flush=True)
Reach for it on anything long. A generation that runs for minutes has to put something on the wire before an idle connection is closed somewhere between your process and the model, and streaming is what puts it there — the streaming guide covers where that edge sits and what to do about it.
Tool calling and structured output get no section of their own here, and that is the point. bind_tools and with_structured_output build the same request bodies they always built, the gateway serves the OpenAI wire format, and the answers come back in the shape LangChain already parses. There is nothing to port and nothing to work around.
Which model to name
model= takes one of four aliases, chosen by what the work needs rather than by whose model it is: astra for frontier reasoning on a 1M-token window, sol for deep reasoning and hard coding, terra for everyday assistant work — RAG, agents, tools and coding, which is why it is the one in the snippet above — and luna for fast drafts, classification and high-volume chat.
A chain that already sends an OpenAI-style model name does not have to be edited call site by call site. Those names are accepted on the wire and read as the weight class they belong to: a mini or a nano is somebody's cheap high-volume call and is served on luna, a gpt-6 name is this year's frontier and is served on astra, and the ordinary 4-series sits between them on terra. The answer echoes back the string you sent, so your client still recognises it as its own request, and a name that matches none of the rules is refused rather than guessed at — a guess would misbill in both directions at once, and neither direction is visible until somebody reads the ledger.
Chains that loop, and the cache read
An agent is a loop, and a loop re-sends its prefix. Every turn of an agent executor carries the system prompt, the tool definitions and the transcript so far back up the wire; a RAG chain that puts the same instructions in front of every retrieved passage does the same thing on a smaller scale. That prefix, not the answer, is the bulk of what a long-running chain pays for.
It is also the cheap part to re-send. A token read back out of a cached prefix is charged at a tenth of its own model's input rate, so a system prompt that stays byte-for-byte identical from turn to turn is what keeps an agent loop cheap. Order the messages accordingly: the parts that never change first, the retrieved documents and the user's turn after them.
None of that has to be taken on faith. The portal itemises every request, so the saving arrives where you can watch it — turn by turn, against the run that spent it.
A token read back out of a cached prefix is charged at a tenth of its own model's input rate, so a system prompt that stays byte-for-byte identical from turn to turn is what keeps an agent loop cheap.