AWS Trainium
Hugging Face adds a lookup for which Hub models run on AWS Trainium
Hugging Face has published a blog post titled "Does Your Model Run on AWS Trainium? Look It Up", pointing developers to a Space that shows which Hub models run on AWS…
Push-based KV transfer in vLLM cuts TTFT up to 3x in AWS benchmark
AWS has contributed a push-based NIXL connector to vLLM that moves KV cache transfer coordination off the critical path in disaggregated prefill-decode inference. In…
TorchNeuron brings native PyTorch support to AWS Trainium chips
PyTorch has introduced TorchNeuron, an open source native backend that lets developers run existing PyTorch code on AWS Trainium by treating the chip as a standard…
TorchNeuron brings native PyTorch support for HF models to AWS Trainium
Hugging Face says its main libraries now run natively on AWS Trainium through TorchNeuron, a new PyTorch backend that exposes Trainium as a standard device. TorchNeuron…
vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads
vLLM has added bi-directional KV cache transfer for prefill-decode disaggregated serving, letting decode nodes send previously computed context back to prefill nodes on…