Tool-Call Streaming Behaves Differently Across OpenAI, SGLang, and llama.cpp

local models
tool calling
inference
Author

Yann Dupis

Published

July 17, 2026

I ran the same OpenAI SDK tool-calling request against OpenAI, SGLang, and llama.cpp. Despite exposing OpenAI-compatible APIs, they streamed tool-call arguments differently. OpenAI returned small argument deltas, SGLang emitted larger chunks, and llama.cpp’s behavior varied by model. In one setup, it even returned the tool call as plain text instead of a structured tool_calls event.

This matters for applications that display, validate, or process arguments before generation finishes. In practice, an “OpenAI-compatible” API does not guarantee identical streaming semantics.

In this post, I reproduce those differences and investigate how the model, chat template, parser, and inference engine contribute to them.

I got interested in this after reading Armin Ronacher’s post, Pushing Local Models With Focus and Polish. One detail that stood out to me was tool-call streaming. It sounds minor, but it can make a big difference to how responsive an agent feels.

I wanted to see how well it actually works with local inference servers. So I kept the client code the same and compared OpenAI, SGLang, and llama.cpp.

What is tool-call streaming?

Tool calling (or function calling) enables LLMs to interface with external tools and data. It can be any tool: get weather, SQL execution, bash, read files, write files. You just need to define the function signature (name, input parameters, and descriptions) through a JSON schema. The LLM doesn’t call the function directly; it emits a tool-call message containing the function name and arguments. You parse this message, call the actual function, and return the response back to the LLM.

Streaming means returning output incrementally as it is generated instead of waiting for the complete response. For tool calls, this lets you see whether the LLM is actively generating an answer or stuck, and lets you interrupt generation early if it is going in the wrong direction or emitting an undesirable tool call like bash(rm -rf /User/oopsy). You’re laughing, but this happened recently to this user with gpt-5.6-sol…

The experiment

The idea is simple: write one piece of client code that makes a streaming tool-call request, then point it at different inference engines by swapping the base_url. For each one, we’ll watch whether the arguments field of the tool call arrives as small deltas or in big chunks (whole key/value pairs at once).

We used the OpenAI Python SDK throughout, since all three targets (OpenAI’s API, SGLang, and llama.cpp) expose an OpenAI-compatible /v1/chat/completions endpoint.

First, we need a tool. We defined a simple write_file function with type hints and inline comments (using the docments convention), then used toolslm to auto-generate the JSON schema from the function signature:

from toolslm.funccall import get_schema
from pathlib import Path

def write_file(
    content:str, # content to be written to disk
    path:str,    # path to the file, e.g. /data/foo.txt
):
    "Write some text to disk"
    Path(path).write_text(content)

get_schema(write_file)
{'name': 'write_file',
 'description': 'Write some text to disk',
 'input_schema': {'type': 'object',
  'properties': {'content': {'description': 'content to be written to disk',
    'type': 'string'},
   'path': {'description': 'path to the file, e.g. /data/foo.txt',
    'type': 'string'}},
  'required': ['content', 'path']}}

This produces an Anthropic-style schema with input_schema. Since we’re targeting the OpenAI format, we just need to rename that key to parameters and nest everything under a function key:

def mk_openai_fmt(anth_schema):
    anth_schema["parameters"] = anth_schema.pop("input_schema")
    return {'type': 'function', 'function': anth_schema}

Now the core of the experiment is a streaming chat completion with the tool attached, accumulating the arguments fragments as they arrive:

from openai import OpenAI

client = OpenAI()  # defaults to OpenAI's API

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user",
               "content": "Write 'Local model will win' to data.txt"}],
    tools=[mk_openai_fmt(get_schema(write_file))],
    stream=True,
)

args = ""
for chunk in response:
    tcs = chunk.choices[0].delta.tool_calls
    if tcs:
        args += tcs[0].function.arguments
        print(repr(args))

print("\nFinal:", args)
''
'{"'
'{"path'
'{"path":"'
'{"path":"data'
'{"path":"data.txt'
'{"path":"data.txt","'
'{"path":"data.txt","content'
'{"path":"data.txt","content":"'
'{"path":"data.txt","content":"Local'
'{"path":"data.txt","content":"Local model'
'{"path":"data.txt","content":"Local model will'
'{"path":"data.txt","content":"Local model will win'
'{"path":"data.txt","content":"Local model will win"}'

Final: {"path":"data.txt","content":"Local model will win"}

In this run, OpenAI returned the tool-call arguments as small deltas. Now let’s see if the local engines match it.

SGLang

For the first local engine, I deployed Qwen 3.6 27B (FP8) on Modal using SGLang. I used a repo I’d previously built pi-modal-deploy which deploys SGLang on Modal GPUs and exposes an OpenAI-compatible /v1 endpoint. It took care of the model weights, SGLang server config, and speculative decoding, so I could just point the OpenAI SDK at the Modal URL.

For reproducibility, the deployment used the SGLang v0.5.12.post1-cu130-runtime image and pinned the Qwen model checkpoint to commit e89b16ebf1988b3d6befa7de50abc2d76f26eb09.

Same client code, just a different base_url:

client = OpenAI(
    base_url="https://yanndupis--pi-modal-qwen3-6-27b-fp8-sglangserver.us-east.modal.direct/v1",
    api_key=os.environ["PI_MODAL_API_KEY"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B-FP8",
    messages=[{"role": "user",
               "content": "Write 'Local model will win' to data.txt"}],
    tools=[mk_openai_fmt(get_schema(write_file))],
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

args = ""
for chunk in response:
    tcs = chunk.choices[0].delta.tool_calls
    if tcs:
        args += tcs[0].function.arguments
        print(repr(args))

print("\nFinal:", args)

Here’s what came back:

'{"content": "Local model will win"'
'{"content": "Local model will win","path": "data.txt"}'

Final: {"content": "Local model will win","path": "data.txt"}

The entire "content" value arrived in one chunk. Unlike OpenAI’s small argument deltas, it arrived in whole key/value pairs. SGLang is technically streaming (we got multiple chunks), but the granularity is dramatically coarser.

Why? Digging into SGLang’s source in base_format_detector.py, the streaming parser uses a partial_json_parser library with this flag:

flags = Allow.ALL if self.current_tool_name_sent else Allow.ALL & ~Allow.STR

Before the tool name has been sent, ~Allow.STR means partial strings are not allowed. After the name is sent, the parser switches to Allow.ALL. The emission is then diff-based: argument_diff comes from json.dumps(cur_arguments) via _find_common_prefix, so the chunks returned by the API are governed by what _partial_json_loads returns, not directly by the model’s raw token slices.

This makes the parser a plausible source of the coarser chunks, but this experiment doesn’t prove that this specific flag caused the complete "content" value to arrive at once. To confirm that, I would need to instrument the parser and inspect what _partial_json_loads returns for each model token.

llama.cpp: it works (when it works)

For the second local engine, I tried llama.cpp. My first attempt was building it from source on a Linux instance. After installing dependencies, compiling, and downloading a model, the output was gibberish, likely a build or quantization issue. So I switched to running llama-server on my Mac, and it worked right away.

mac_client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="none",
)

I used the same streaming request for both models:

resp = mac_client.chat.completions.create(
    model="qwen",
    stream=True,
    messages=[{
        "role": "user",
        "content": "Write 'Local model will win' to data.txt",
    }],
    tools=[mk_openai_fmt(get_schema(write_file))],
)

args = ""
for chunk in resp:
    tcs = chunk.choices[0].delta.tool_calls
    if tcs:
        args += tcs[0].function.arguments
        print(repr(args))

print("\nFinal:", args)

I started with Qwen/Qwen2.5-Coder-14B-Instruct-GGUF. But the tool call never arrived as a native tool_calls delta. Instead, it leaked into the response as plain text, wrapped in <function> tags:

<function=write_file>{"path": "data.txt", "content": "Local model will win"}</function>

No error, no warning, just silently wrong. I restarted with --jinja (which enables the model’s chat template and tool-call parser). The wrapper changed to <tools>, but the result was the same.

So I switched to Qwen2.5-3B-Instruct. For this successful run, I started the server with the following command from my shell history:

llama-server \
  -hf Qwen/Qwen2.5-3B-Instruct-GGUF:Q4_K_M \
  -c 2048 \
  -np 1 \
  --cache-ram 0 \
  -ngl 99 \
  --host 127.0.0.1 \
  --port 8080

I was using llama.cpp version 9670 (02810c7aa). This time the arguments arrived as small deltas, much like OpenAI:

''
'{"'
'{"path'
'{"path":"'
'{"path":"data'
'{"path":"data.txt'
'{"path":"data.txt","'
'{"path":"data.txt","content'
'{"path":"data.txt","content":"'
'{"path":"data.txt","content":"Local'
'{"path":"data.txt","content":"Local model'
'{"path":"data.txt","content":"Local model will'
'{"path":"data.txt","content":"Local model will win'
'{"path":"data.txt","content":"Local model will win"}'

Final: {"path":"data.txt","content":"Local model will win"}

And it worked without --jinja too. Yet the 14B Coder failed silently regardless of flags.

It turns out llama.cpp has had tool-call argument streaming since PR #12379, merged in May 2025, roughly a year before Armin’s post.

So based on this experiment it seems that the capability isn’t missing. The roughness is about which model you pick: a user reaching for the bigger Coder variant gets silent failure, while the 3B Instruct works fine.

Conclusion

All right, I think it’s the end of this exploration. Honestly, it’s really amazing to have access to all these models for free and inference engines for local development or in the cloud. Even with Modal, I don’t have to worry about the infrastructure or Kubernetes nightmare.

But I agree with Armin that because the space is evolving so quickly, it’s fragmented. The critical mass isn’t always there to support all the options, and the tech isn’t always as polished as we’d hope. The capability to stream tool-call arguments as small deltas exists in llama.cpp today, but finding the right model and flags is still trial and error. But it feels like it’s getting closer every day.