Per-Tenant Rate Limiting

Your support chatbot serves many companies from one shared Azure OpenAI deployment, with one combined tokens-per-minute quota covering all of them together.

Now one company's customers send a sudden burst of questions, using up most of that shared quota by themselves. Every other company's customers start seeing the chatbot respond slowly, or fail outright, even though their own traffic never changed at all.

The quota was never divided between companies in the first place. Whoever sends the most traffic in a given minute gets to spend almost all of it. That's exactly what per-tenant rate limiting fixes.

Per-tenant rate limiting splits that quota up front: give each company its own fixed share of the shared tokens-per-minute quota, and track each company's usage against its own share, separately from every other company's usage. Once a company reaches its own share, slow down or queue its requests, and leave every other company's share untouched.

Use: ASP.NET Core Rate Limiting Middleware (Partitioned Limiter)

A shared limit with no per-tenant split lets the loudest customer starve every quiet one.

A shared water tank supplies an entire building, with no meter or valve per household.

One household running every tap at once can drain the tank dry for everyone else, even though their own usage never changed.

Give each household its own valve, so one household's excess can never starve the rest.