Testing Tool-Call Streaming with SGLang and llama.cpp

local models
tool calling
inference
Author

Yann Dupis

Published

July 17, 2026

Intro

A few weeks ago, I was really inspired by Armin’s blog post about local models: Pushing Local Models With Focus And Polish. It does an amazing job of stating the current state, challenges, and why having access to efficient local models matters:

“I really, really want local models to work.

I want them to work in the very practical sense that I can open my coding agent, pick a local model, and get something that feels competitive enough that I do not immediately switch back to a hosted API after five minutes. There are a lot of reasons why I want this, but the biggest quite frankly is that we’re so early with this stuff, and the thought of locking all the experimentation away from the average developer really upsets me.” — Armin Ronacher

He argues that the main reason local models aren’t competitive yet is that the space is too fragmented. There are many inference engines (llama.cpp, vLLM, MLX), many models to choose from (Qwen 3.6, DeepSeek V4, GLM 5.2, etc.), and many parameters to select such as quantization and context size. The critical mass isn’t big enough to make the entire ecosystem competitive with closed-source options. Instead, he suggests we should pick one model, one inference engine, and a series of defaults—and make that combination very, very good, so we don’t want to switch back to closed-source options right away. He advocates for DeepSeek V4 Flash running on a beefy Mac with the inference engine ds4. I’ll let you read the blog to find out why he’s betting on this combination.

Since Armin wrote that, events have only reinforced the case for local alternatives—most notably when the US government sent a directive to suspend access to Fable 5 and Mythos 5.

I’ve been wanting to blog again for a long time. I worked for several years at the intersection of privacy and AI, so exploring local models seemed a natural fit. In his post, to illustrate that local solutions are lacking, Armin mentions that local options often don’t support streaming function calls. It might seem like a small detail, but I thought deploying several LLMs using different inference engines and checking whether they stream function calls could be a good way to get a feel for where the local ecosystem actually stands today.

What is tool-call streaming?

Tool calling (or function calling) enables LLMs to interface with external tools and data. It can be any tool: get weather, SQL execution, bash, read files, write files. You just need to define the function signature—name, input parameters, and descriptions—through a JSON schema. The LLM doesn’t call the function directly; it emits a tool-call message containing the function name and arguments. You parse this message, call the actual function, and return the response back to the LLM.

Streaming consists of emitting one token at a time instead of the entire message at once. Armin argues that the benefit of streaming for tool calls is being able to see if the LLM is actively generating an answer or stuck. But also to interrupt the function call generation early if it’s going in the wrong direction or emitting an undesirable tool call like bash(rm -rf /User/oopsy). You’re laughing, but this happened recently to this user with gpt-5.6-sol…

The experiment

The idea is simple: write one piece of client code that makes a streaming tool-call request, then point it at different inference engines by swapping the base_url. For each one, we’ll watch whether the arguments field of the tool call arrives in small fragments (token-by-token) or in big chunks (whole key/value pairs at once).

We used the OpenAI Python SDK throughout, since all three targets—OpenAI’s API, SGLang, and llama.cpp—expose an OpenAI-compatible /v1/chat/completions endpoint.

First, we need a tool. We defined a simple write_file function with type hints and inline comments (using the docments convention), then used toolslm to auto-generate the JSON schema from the function signature:

from toolslm.funccall import get_schema
from pathlib import Path

def write_file(
    content:str, # content to be written to disk
    path:str,    # path to the file, e.g. /data/foo.txt
):
    "Write some text to disk"
    Path(path).write_text(content)

get_schema(write_file)
{'name': 'write_file',
 'description': 'Write some text to disk',
 'input_schema': {'type': 'object',
  'properties': {'content': {'description': 'content to be written to disk',
    'type': 'string'},
   'path': {'description': 'path to the file, e.g. /data/foo.txt',
    'type': 'string'}},
  'required': ['content', 'path']}}

This produces an Anthropic-style schema with input_schema. Since we’re targeting the OpenAI format, we just need to rename that key to parameters and nest everything under a function key:

def mk_openai_fmt(anth_schema):
    anth_schema["parameters"] = anth_schema.pop("input_schema")
    return {'type': 'function', 'function': anth_schema}

Now the core of the experiment is a streaming chat completion with the tool attached, accumulating the arguments fragments as they arrive:

from openai import OpenAI

client = OpenAI()  # defaults to OpenAI's API

response = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user",
               "content": "Write 'Local model will win' to data.txt"}],
    tools=[mk_openai_fmt(get_schema(write_file))],
    stream=True,
)

args = ""
for chunk in response:
    tcs = chunk.choices[0].delta.tool_calls
    if tcs:
        args += tcs[0].function.arguments
        print(repr(args))

print("\nFinal:", args)
''
'{"'
'{"path'
'{"path":"'
'{"path":"data'
'{"path":"data.txt'
'{"path":"data.txt","'
'{"path":"data.txt","content'
'{"path":"data.txt","content":"'
'{"path":"data.txt","content":"Local'
'{"path":"data.txt","content":"Local model'
'{"path":"data.txt","content":"Local model will'
'{"path":"data.txt","content":"Local model will win'
'{"path":"data.txt","content":"Local model will win"}'

Final: {"path":"data.txt","content":"Local model will win"}

Here, we can see that OpenAI is generating one token at a time for the tool call. Now let’s see if the local engines match it.

SGLang

For the first local engine, I deployed Qwen 3.6 27B (FP8) on Modal using SGLang. I used a repo I’d previously built pi-modal-deploy which deploys SGLang on Modal GPUs and exposes an OpenAI-compatible /v1 endpoint. It took care of the model weights, SGLang server config, and speculative decoding, so I could just point the OpenAI SDK at the Modal URL.

Same client code, just a different base_url:

client = OpenAI(
    base_url="https://yanndupis--pi-modal-qwen3-6-27b-fp8-sglangserver.us-east.modal.direct/v1",
    api_key=os.environ["PI_MODAL_API_KEY"],
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-27B-FP8",
    messages=[{"role": "user",
               "content": "Write 'Local model will win' to data.txt"}],
    tools=[mk_openai_fmt(get_schema(write_file))],
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

args = ""
for chunk in response:
    tcs = chunk.choices[0].delta.tool_calls
    if tcs:
        args += tcs[0].function.arguments
        print(repr(args))

print("\nFinal:", args)

Here’s what came back:

'{"content": "Local model will win"'
'{"content": "Local model will win","path": "data.txt"}'

Final: {"content": "Local model will win","path": "data.txt"}

The entire "content" value arrived in one chunk. Unlike OpenAI, it arrived in whole key/value pairs rather than token-by-token. SGLang is technically streaming (we got multiple chunks), but the granularity is dramatically coarser.

Why? Digging into SGLang’s source in base_format_detector.py, the streaming parser uses a partial_json_parser library with this flag:

flags = Allow.ALL if self.current_tool_name_sent else Allow.ALL & ~Allow.STR

Before the tool name has been sent, ~Allow.STR means partial strings are not allowed. After the name is sent, the parser switches to Allow.ALL. The emission is then diff-based: argument_diff comes from json.dumps(cur_arguments) via _find_common_prefix, so the chunks returned by the API are governed by what _partial_json_loads returns, not directly by the model’s raw token slices.

This makes the parser a plausible source of the coarser chunks, but this experiment doesn’t prove that this specific flag caused the complete "content" value to arrive at once. To confirm that, I would need to instrument the parser and inspect what _partial_json_loads returns for each model token.

llama.cpp: it works (when it works)

For the second local engine, I tried llama.cpp. My first attempt was building it from source on a Linux instance—after installing dependencies, compiling, and downloading a model, the output was gibberish, likely a build or quantization issue. So I switched to running llama-server on my Mac, and it worked right away.

mac_client = OpenAI(
    base_url="http://127.0.0.1:8080/v1",
    api_key="none",
)

I used the same streaming request for both models:

resp = mac_client.chat.completions.create(
    model="qwen",
    stream=True,
    messages=[{
        "role": "user",
        "content": "Write 'Local model will win' to data.txt",
    }],
    tools=[mk_openai_fmt(get_schema(write_file))],
)

args = ""
for chunk in resp:
    tcs = chunk.choices[0].delta.tool_calls
    if tcs:
        args += tcs[0].function.arguments
        print(repr(args))

print("\nFinal:", args)

I started with Qwen/Qwen2.5-Coder-14B-Instruct-GGUF. But the tool call never arrived as a native tool_calls delta. Instead, it leaked into the response as plain text, wrapped in <function> tags:

<function=write_file>{"path": "data.txt", "content": "Local model will win"}</function>

No error, no warning—just silently wrong. I restarted with --jinja (which enables the model’s chat template and tool-call parser). The wrapper changed to <tools>, but the result was the same.

So I switched to Qwen2.5-3B-Instruct. For this successful run, I started the server with the following command from my shell history:

llama-server \
  -hf Qwen/Qwen2.5-3B-Instruct-GGUF:Q4_K_M \
  -c 2048 \
  -np 1 \
  --cache-ram 0 \
  -ngl 99 \
  --host 127.0.0.1 \
  --port 8080

I was using llama.cpp version 9670 (02810c7aa). This time the arguments streamed token-by-token, exactly like OpenAI:

''
'{"'
'{"path'
'{"path":"'
'{"path":"data'
'{"path":"data.txt'
'{"path":"data.txt","'
'{"path":"data.txt","content'
'{"path":"data.txt","content":"'
'{"path":"data.txt","content":"Local'
'{"path":"data.txt","content":"Local model'
'{"path":"data.txt","content":"Local model will'
'{"path":"data.txt","content":"Local model will win'
'{"path":"data.txt","content":"Local model will win"}'

Final: {"path":"data.txt","content":"Local model will win"}

And it worked without --jinja too. Yet the 14B Coder failed silently regardless of flags.

It turns out llama.cpp has had tool-call argument streaming since PR #12379, merged in May 2025—roughly a year before Armin’s post.

So based on this experiment it seems that the capability isn’t missing. The roughness is about which model you pick: a user reaching for the bigger Coder variant gets silent failure, while the 3B Instruct works fine.

Conclusion

All right, I think it’s the end of this exploration. Honestly, it’s really amazing to have access to all these models for free and inference engines for local development or in the cloud. Even with Modal, I don’t have to worry about the infrastructure or Kubernetes nightmare.

But I agree with Armin that because the space is evolving so quickly, it’s fragmented. The critical mass isn’t always there to support all the options, and the tech isn’t always as polished as we’d hope. The capability for fine-grained tool-call streaming exists in llama.cpp today, but finding the right model and flags is still trial and error. But it feels like it’s getting closer every day.