PyTorch Consolidates Media Processing Around TorchCodec, TorchVision and TorchAudio

TorchCodec becomes the single place for media I/O

According to the PyTorch team, decoding and encoding used to be scattered and partly duplicated across TorchVision and TorchAudio, with each library building its own stack. TorchVision offered several entry points, io.read_video() and io.VideoReader(), spread over three backends: PyAV in Python, a C++ FFmpeg backend, and a CUDA/NVCUVID backend. Only the PyAV backend worked out of the box. The other two had to be built from source and were tied to a specific FFmpeg version.

TorchAudio had its own StreamReader and StreamWriter for video and audio, plus extra audio decoding utilities with backends such as FFmpeg, libsoundfile and libsox. Image decoding lived in TorchVision.

All of this now sits in TorchCodec, covering images, video and audio on CPU and CUDA. The corresponding APIs in TorchVision and TorchAudio are deprecated or removed.

Three reasons for the move

The team gives three reasons for the consolidation.

• One location for users. Users no longer need to work out which library offers the decoder they need, and get a library designed as a whole.
• Performance work in one place. With several implementations, it was never obvious which deserved optimization effort. According to the team, TorchCodec is generally more performant than the earlier implementations in TorchVision and TorchAudio, particularly for CUDA video decoding.
• Simpler maintenance. Media I/O depends on six major FFmpeg versions (4 through 9 at the time of writing), NVIDIA's codec SDK for GPU decoding, and one library per image format such as libjpeg and libpng, each with its own licensing rules. None are pure Python dependencies, which makes builds, CI and releases harder. That complexity is now contained in TorchCodec.

TorchVision and TorchAudio narrow their scope

Both libraries once covered models, datasets, pipelines, transforms and I/O. They are now focused on transforms, and the rest is not under active development. The team says transforms are the most active usage area in the community, while models, datasets and pipelines have lagged in usage as alternatives such as the HuggingFace libraries have emerged.

The change was particularly disruptive for TorchAudio, where many APIs were deprecated and eventually removed. The team thanks the community for its patience and feedback, which led it to reconsider its initial plan: several popular APIs originally slated for removal were kept. In the team's words, "A smaller TorchAudio is one that can actually be maintained."

How the pieces fit together

TorchCodec turns media files or encoded bytes into tensors, and tensors back into media files. TorchVision and TorchAudio transform the tensors in between.

For video, the post shows TorchCodec's VideoDecoder feeding TorchVision's v2 transforms such as RandomResizedCrop and ToDtype. For audio, AudioDecoder output goes into TorchAudio's MelSpectrogram. Images work the same way: decode_image and related functions return plain tensors that go straight into torchvision.transforms.v2. For encoding, TorchCodec provides VideoEncoder, AudioEncoder, JpegEncoder and PngEncoder.

All three libraries are now ABI stable. A given version is no longer tied to a single PyTorch version and keeps working with later releases, so the libraries no longer need rebuilding for each PyTorch release and their release cadence no longer matches PyTorch's.

What it means for users

Anyone still using decoding or encoding APIs in TorchVision or TorchAudio is advised to move to TorchCodec, starting with the migration guide. The team asks users to report missing functionality on GitHub. A follow-up post is planned on what is new in each library and how to get the most performance from a full decoding and transform pipeline.