Updates on Inference Optimization

Every AI story we track on Inference Optimization: 22 stories so far, each summarized in our own words and linked to the publisher that reported it.

Pulled from 125 sources

This week

Politics5 min read

Latest open artifacts (#25): DeepSeek V4.1 Flash 1017, American-made models, and new regulatory controls in the US & China

Interconnects5d ago

LLM & Text Generation10 min read

Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM

AWS Blog6d ago

This month

Business & Enterprise7 min read

Tensormesh Launches Platform to Cut LLM Inference Costs Through KV-Cache Reuse

Unite.AI11d ago

Business & Enterprise12 min read

Turning Opportunistic AI Inference Capacity Into Better Ads Recommendations

Meta AI News13d ago

Chips & Compute5 min read

Cerebras Reports 5X Inference Throughput Gain From Disaggregation

Unite.AI24d ago

Dev8 min read

Introducing Amazon SageMaker HyperPod Inference Gateway

Covered by 2 sourcesSep 18

Machine Learning9 min read

Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference

AWS BlogSep 10

Machine Learning9 min read

Reduce inference cold starts on Amazon SageMaker HyperPod with model caching

AWS BlogSep 10

LLM & Text Generation7 min read

Speed Up LLM Inference with DSpark Speculative Decoding

KDnuggetsAug 31

Machine Learning16 min read

Reduce ASR inference costs by 75% with NVIDIA MPS on Amazon EC2

AWS BlogAug 27

AI Research5 min read

Z.AI's Use of Chinese Chips for New Model is About Optimization

AI BusinessAug 27

Machine Learning6 min read

Up to 3.2x Faster Inference with LFM2.5-DSpark

Covered by 3 sourcesAug 20

LLM & Text Generation18 min read

Introducing cross-Region inference for OpenAI GPT-5.6 models on Amazon Bedrock

AWS BlogAug 20

Dev4 min read

NVIDIA Releases TensorRT Model Connect in Public Preview: Hugging Face Checkpoint to Native C++ Inference in Two Commands

MarkTechPostAug 18

AI Research4 min read

IBM bets $240m on cheap, open-source inference to take on the hyperscalers

Covered by 2 sourcesAug 11

Dev29 min read

Tiered KV cache for large LLMs on Amazon SageMaker HyperPod with Curvine

AWS BlogAug 12

LLM & Text Generation2 min read

Arbitrage: Efficient Reasoning via Advantage-Aware Speculation

Apple Research BlogAug 7

LLM & Text Generation

Predictive Speculative KV Replication for Bursty LLM Inference

Hacker NewsJul 31

Dev10 min read

Launching UI for generative AI inference recommendations in Amazon SageMaker AI

AWS BlogJul 13

Machine Learning11 min read

Streaming benchmark and recommendation results to MLflow with Amazon SageMaker AI

AWS BlogJul 6

LLM & Text Generation

Parallelize speculative decoding with P-EAGLE on Amazon SageMaker AI

AWS BlogJun 16

Machine Learning

Amazon SageMaker AI now supports optimized generative AI inference recommendations

AWS BlogApr 22

That's everything we have on Inference Optimization right now