Response Streaming

1. What exactly is Response Streaming?

Your ASP.NET Core support chatbot calls Azure OpenAI's chat completions API, and buffers the full response inside ASP.NET Core before sending it to the customer's screen as one SignalR message.

For a short answer, like a shipping estimate, that's under a second, and nobody notices any delay.

For a longer troubleshooting answer, though, Azure OpenAI takes 6 to 8 seconds to finish generating the full response, and the customer's chat bubble sits there completely empty the whole time. Support tickets start coming in, customers saying the chatbot isn't responding.

Azure OpenAI has the answer ready to stream out the whole time. ASP.NET Core just isn't forwarding any of it to the browser until the whole thing is done. That's exactly what response streaming fixes.

2. How would you fix it in .NET/Azure?

Response streaming sends each piece the moment it exists: call Azure OpenAI's chat completions endpoint with streaming turned on, so it returns the response as a sequence of chunks instead of one final block. As you read each chunk in ASP.NET Core, push it straight to the customer's screen over a SignalR connection, so the reply builds up continuously, the same way Azure OpenAI is writing it.

Use: Azure.AI.OpenAI SDK (CompleteChatStreamingAsync), Microsoft.AspNetCore.SignalR

Show the words as Azure OpenAI makes them, not all at once at the end.

3. How can you visualize it in everyday terms?

Ask a friend a question in person, and they answer you sentence by sentence as they think it through, not by standing silently for a minute and then reciting the whole answer at once.

You start hearing, and understanding, the answer immediately, well before they've finished speaking.

You start reading before the model is done writing.