It can run 128LLaMA 27B streams in parallel, and developers can test the reasoning performance of Apple M2 Ultra chip model.

CTOnews.com, October 13, according to the Y Combinator Forum, a developer named Georgi Gerganov recently conducted a series of LLaMA model reasoning tests using Apple M2 Ultra chip. The tests show that the chip works better when running 128Llama 27B streams in parallel.

According to the test performance disclosed by ▲ source Georgi Gerganov in GitHub, it is reported that LLaMA model reasoning tests emphasize memory bandwidth rather than computing power, while Apple's M2 Ultra chip is known as "memory bandwidth up to 800GB/s", so reasoning works well.

According to the test performance of ▲ source Georgi Gerganov in GitHub, although the memory bandwidth of M2 Ultra chip has advantages, users can only get 192GB RAM at most. Users spend $5600 (CTOnews.com note: bank of China 44999 yuan) to buy Mac Studio with M2 Ultra chip and 192GB running memory, the price is enough to buy a "PC with 1TB RAM and RTX 4090".

▲ drawing Source Y Combinator

According to this, Georgi Gerganov, the ▲ source Apple's official website, believes that if the user's workload (such as reasoning) does not need to exceed 192 GB, then you can pay more attention to the "memory bandwidth" performance when selecting the relevant machines, but if you have other work requirements, you can take a look at the corresponding price PC.

Referenc

M2 Ultra can run 128 streams of Llama 2 7B in parallel

Test results submitted by developers on GitHub