The reasoning performance of H100 has been improved up to 8 times, Nvidia released TensorRT-LLM

Thanks to CTOnews.com netizen OC_Formula for the clue delivery! CTOnews.com Sept. 9, Nvidia today announced the release of TensorRT-LLM, a deeply optimized open source library that accelerates the reasoning performance of all large language models on AI GPU such as Hopper.

Nvidia has worked with the open source community to implement the AI kernel to optimize its GPU using cutting-edge technologies such as SmoothQuant, FlashAttention and fMHA, which can accelerate GPT-3 (175B), Llama, Falcom (180B) and Bloom models.

The highlight of TensorRT-LLM is the introduction of a scheduling scheme called In-Flight batching, which allows work to enter and exit GPU independently of other tasks.

This scheme allows the same GPU to process multiple smaller queries dynamically when dealing with large computing-intensive requests, which improves the processing performance of GPU and speeds up the throughput of H100 by two times.

In the performance test, Nvidia compares H100 with H100 and H100 with TensorRT-LLM enabled. In GPT-J 6B reasoning, H100 reasoning performance is 4 times higher than A100, while H100 with TensorRT-LLM is 8 times higher than A100.

In Llama 2, the reasoning performance of H100 is 2.6 times that of A100, while the performance of H100 with TensorRT-LLM enabled is 4.6 times that of A100.

The original text of the report is attached to CTOnews.com, which can be read in depth by interested users.