MosaicML of AI Company launched the 30 billion parameter model MPT-30B, which claims that the training cost is only a fraction of the competitive product.

CTOnews.com June 25 news, AI startup MosaicML recently released its language model MPT-30B, this model has 30 billion parameters, training cost "only a fraction of other similar competitor models," is expected to promote the industry to gradually reduce the training cost of such models, expand the application of AI models in a wider range of fields.

Naveen Rao, CEO and co-founder of MosaicML, said the MPT-30B costs $700,000 to train, far less than the tens of millions of dollars needed for similar products such as GPT-3. In addition, due to its lower cost and smaller size, the MPT-30B can also be trained more quickly and is more suitable for deployment on local hardware.

MosaicML uses Alibi and FlashAttention technologies to optimize models for longer text lengths and higher utilization of GPU computing. MosaicML is also one of the few labs to be able to use Nvidia H100 GPUs, with throughput per GPU increased by more than 2.4 times compared to previous efforts, resulting in faster completion times.

In addition to making AI technology more accessible, MosaicML focuses on improving data quality and improving model performance. They are currently developing a tool that will help users layer in domain-specific data during pre-training to ensure diverse and high-quality data combinations during training.

MosaicML says expanding the model to 30 billion parameters is only the first step, and they will then launch larger, higher-quality models with lower costs.

CTOnews.com notes that developers can now download and use the open-source MPT-30B base model from Hugging Face and fine-tune the model using their own data on local hardware.