Tree News

IO · 2026-11-19 09:21 UTC

How we cut our LLM streaming bandwidth by 98% with two small changes

The IO.NET team found their daily cost alert fired on November 11 after network cost on their API path rose more than 5x in two days, with no errors or latency problems. Metrics showed reply traffic from GPU servers had more than tripled, and output tokens grew almost 7x as one model shifted from short to long generations. Each streamed token was about 4 bytes of text but measured 364 bytes on the wire, roughly 99% JSON packaging, and every token crosses a metered cloud boundary.

Loading the story on Tree News…