Long-form Output Practices

How to reliably get results for long-output workloads — drama scripts, fiction, 10k-word articles: stream, size the read timeout to the inter-event gap, give max_tokens room, check stop_reason, split very long outputs, and resume dropped streams. With a Claude native example and per-language timeout settings.

In one line: when you ask a model to produce tens of thousands of characters in a single call (episode outlines, long fiction, large translations, big code), stream the response, do not use non-streaming; set the client read timeout to the gap between data events (tens of seconds — 90–120s is a safe value), not to the total generation time; give max_tokens room; and check stop_reason before using the text. Do these four things and long-output calls stop “returning nothing.” Beyond 20k–30k characters, add one more: split the work into several calls instead of holding one connection open for 20 minutes.

This page applies to all large models (OpenAI, Claude, Gemini, Grok, and others). Code examples use Claude’s native /v1/messages; differences for the OpenAI-compatible format are called out separately.

Three things to know first

1. For a 10k-word output, 10–20 minutes of real generation is normal. The model emits tens of thousands of characters token by token, plus a reasoning/thinking phase — end-to-end latency is genuinely long. This is not the gateway being slow; generation itself is slow.

2. Non-streaming buffers the whole thing before sending. With non-streaming (stream omitted or false), the server must wait for the model to finish the entire generation, then send the whole body back at once. During those minutes your client read timeout is racing against it, and the longer the generation the more likely you disconnect before the result arrives — and the exception is often empty (httpx.ReadError’s str(e) is blank), so you cannot see the cause.

3. A dropped connection is still billed, so blind retries double-charge. Once the server has generated the output, the call is billed even if the result never reached you. Retrying after you have already received part of the body means the model runs again and you pay again.

Stream, do not use non-streaming

With streaming (stream: true), the first byte arrives within seconds, and after that a data event arrives every few tens of seconds. Your read timeout only has to cover the gap between events, not a generation that runs for many minutes — that is the whole reason streaming reliably delivers long output.

The two protocols have different terminators — do not mix them up:

Protocol	Terminator	How to read the text
Claude native /v1/messages	event: message_stop (no [DONE])	the delta.text of a content_block_delta whose delta.type == "text_delta"
OpenAI-compatible /v1/chat/completions	data: [DONE]	choices[0].delta.content

With adaptive thinking on, Claude native emits a type: "thinking" block first (its increments are thinking_delta), then the text block. When rendering, route thinking_delta and text_delta separately and do not concatenate thinking into the body text.

Minimal Claude native /v1/messages streaming example (plain httpx, line-by-line SSE parsing):

import json
import httpx

def generate_long_text(prompt, api_key, model="claude-opus-5", max_tokens=64000):
    url = "https://api.apiyi.com/v1/messages"
    headers = {
        "x-api-key": api_key,
        "anthropic-version": "2023-06-01",
        "content-type": "application/json",
        "accept": "text/event-stream",
    }
    payload = {
        "model": model,
        "max_tokens": max_tokens,
        "stream": True,                        # key: long output must stream
        "thinking": {"type": "adaptive"},      # adaptive thinking; the model decides depth
        "messages": [{"role": "user", "content": prompt}],
    }
    # Three-part timeout: read only covers the inter-event gap, not the whole generation
    timeout = httpx.Timeout(connect=30, write=120, read=120, pool=30)

    text, stop_reason = [], None
    with httpx.Client(timeout=timeout) as c:
        with c.stream("POST", url, json=payload, headers=headers) as r:
            if r.status_code != 200:
                raise RuntimeError(f"HTTP {r.status_code}: {r.read()[:400]}")
            event, data = None, []
            for line in r.iter_lines():
                if line == "":                 # events are separated by a blank line
                    if data:
                        d = json.loads("\n".join(data))
                        t = d.get("type") or event
                        if t == "content_block_delta" and d.get("delta", {}).get("type") == "text_delta":
                            text.append(d["delta"]["text"])
                        elif t == "message_delta":
                            stop_reason = d.get("delta", {}).get("stop_reason") or stop_reason
                    event, data = None, []
                    continue
                if line.startswith("event:"):
                    event = line[6:].strip()
                elif line.startswith("data:"):
                    data.append(line[5:].strip())
    # A Claude stream ends on message_stop, never [DONE]
    if stop_reason == "max_tokens":
        raise RuntimeError(f"truncated by max_tokens after {len(''.join(text))} chars; raise max_tokens and retry")
    return "".join(text).strip()

If you use the official anthropic SDK, point base_url at https://api.apiyi.com and use client.messages.stream(...).get_final_message() — the SDK handles SSE parsing, timeouts, and stop_reason for you. The httpx version above is for when you would rather not pull in the SDK.

Size the read timeout to the inter-event gap, not the total time

Many people set the read timeout to one huge value meant to cover the whole generation (say 1800 seconds) and still time out — because with non-streaming that value has to race the entire generation and any hiccup breaks it. The right approach is streaming plus a read timeout sized to the inter-event gap.

Measured reference (claude-opus-5 producing a ~20k-character episode outline from a ~15k-character input):

Metric	Measured
Time to first byte	3 – 130 s
Max silent gap during thinking	~42 s (with keepalive pings in between)
End-to-end total	9 – 12 minutes

So a read timeout of 90–120 seconds covers the largest inter-event gap with headroom — no need for a value of many minutes. A three-part timeout splits the phases and sizes each one:

# connect: establish; write: upload the request body; read: max gap between reads
timeout = httpx.Timeout(connect=30, write=120, read=120, pool=30)

Setting it in each client

Many client libraries have both a per-read timeout and a total timeout, often with similar names. For long output you want the former; leave the latter unset or unlimited:

Client	Set this (max gap between reads, 120 s recommended)	Avoid
Python httpx / OpenAI SDK / anthropic SDK	httpx.Timeout(connect=30, write=120, read=120, pool=30)	a single big number like timeout=3600, which also makes the read timeout one hour
Python requests	timeout=(30, 120) — the second value is the read timeout	timeout=3600
Java Hutool	setConnectionTimeout(30000) + setReadTimeout(120000)	setReadTimeout(3600000) or timeout(3600000) (the latter sets both connect and read)
Java OkHttp	readTimeout(120, TimeUnit.SECONDS)	keeping the 10 s default (silence during thinking trips it); callTimeout is a total limit — leave it unset
Node.js fetch	undici’s new Agent({ bodyTimeout: 120_000 }), passed as the dispatcher	AbortSignal.timeout(...) is a total limit and will cut long output off
Go net/http	your own idle timer on the body: reset it on every chunk, cancel the context after 120 s without data	http.Client{Timeout: ...} is a total limit that includes reading the whole body

Streams can stall too: the connection stops receiving mid-way

Streaming fixes the “buffer everything before sending” problem, but there is another failure mode to guard against: a connection that stalls mid-stream.

• Symptom: text arrives normally at first, then nothing more comes — yet the connection neither errors nor closes. The client hangs until its own timeout: if that is one hour, it waits a full hour, and ends up with only the first small piece.
• Cause: on cross-region and cross-border network paths, a connection held open for a long time can get stuck at an intermediate network device. The server finished generating and closed the connection long ago, but the close never reaches you. The longer a connection lasts, the more likely this is — a 20-minute connection is hit more often than a 3-minute one.
• Billing: the server finishes generating and bills as usual (see point 3 of “Three things to know first”).

What to do:

1. Size the read timeout to the inter-event gap (previous section). With a 120 s read timeout a stall costs you two minutes before you can retry; with a one-hour read timeout it costs a full hour. This is the most important one.
2. Shorten each connection: split very long outputs into several calls — see “Split very long outputs into several calls” below.
3. Resume a dropped stream from where it stopped — see “Retry strategy” below.

Do not set the read timeout to “total generation time plus headroom” (1800 s, 3600 s, and so on). It makes no difference while things work, but when a connection stalls you wait that long before you even learn it failed.

Give max_tokens room, and check stop_reason

Long output easily hits the max_tokens ceiling and gets truncated. This is especially true for models like Claude with thinking on — thinking itself consumes the max_tokens budget, and a long piece can exhaust it.

• Start max_tokens at 64000 (give even more at high effort / deep thinking; claude-opus-5 supports 128K output).
• Check stop_reason before using the response:
  • end_turn — finished normally, the text is complete; this is the only success.
  • max_tokens — truncated, the text may be incomplete or even empty. This is truncation, not an “empty result” — raise max_tokens and retry.
  • refusal — declined by a safety policy; handle separately.

Judging success by str(e) alone or by “the text is empty” is misleading — an empty body is usually max_tokens truncation.

Split very long outputs into several calls

Asking for 50k characters in one call that runs for 20 minutes is the most fragile pattern, even with streaming: any network hiccup along the way voids the whole piece, and the whole piece is billed. Once the output exceeds 20k–30k characters, split it:

1. Generate the outline / episode list / chapter list first — a short call.
2. Then generate one chapter or episode per call, keeping each under 10k–20k characters. Include the outline and the context each part needs (character settings, how the previous part ended) so the pieces stay consistent.
3. Once the outline is fixed, the parts are independent and can run in parallel.

	One call	Split
Duration of each connection	10 – 20 minutes	3 – 5 minutes
Cost of a stall or drop	redo and pay for the whole piece	retry one part
Total time	one long serial run	parts run in parallel, usually faster
Risk of max_tokens truncation	high	low

Retry strategy

Be conservative with retries on long output, so “retry on failure” does not become “double billing plus a second long run”:

• Retry only on failures before the response headers arrive, and on 5xx / 429 (with backoff, at most twice). Those are connection or transient issues where a retry makes sense.
• If a stream drops after you already received part of the body, resume from where it stopped instead of starting over. The server has generated and billed it; restarting runs it again and pays again. Keep the text you received, include it in a new call as what has been written so far, and let the model continue — you pay only for the remainder.
• Treat a read timeout (a stalled connection) the same way: if you already have a good amount of text, resume; if you received almost nothing, retry the whole call.
• Log the request id from the response headers for reconciliation and troubleshooting.

A resume request looks like this (works with any model):

messages = [{
    "role": "user",
    "content": (
        f"{original_prompt}\n\n"
        f"Here is what has been written so far:\n\n{partial_text}\n\n"
        "Continue directly from the last sentence above. Do not repeat existing content or add any commentary."
    ),
}]

Scenario cheat sheet

Scenario	Typical output	max_tokens	read timeout
Drama / short-drama episode outline	10k – 30k chars	64000	90 – 120 s
Fiction (novels / chapters)	10k – 50k chars	64000 – 128000	90 – 120 s
Long translation	scales with source	~1.5x source tokens	90 – 120 s
Large code generation	thousands of lines	32000 – 64000	90 – 120 s

For a single piece over 30k characters, split it by chapter or episode (see “Split very long outputs into several calls”). Always stream. Use api.apiyi.com (or the backup b.apiyi.com), and not api-cf.apiyi.com (the CDN node returns 524 at ~100 seconds and cannot carry long requests).

Related

How to avoid API timeouts

Timeout values by scenario

Streaming vs non-streaming

The tradeoffs and how to choose

Claude thinking & effort

Adaptive thinking, effort tiers, max_tokens and truncation

Claude response handling

Native response shape, SSE events, stop_reason