Shulou(Shulou.com)11/24 Report--

CTOnews.com June 14, a few days ago, Microsoft researchers demonstrated the LLaVA-Med model, which is mainly used in biomedical research, and can infer the pathological condition of patients according to CT, X-ray pictures and so on.

It is reported that Microsoft researchers in cooperation with a number of hospitals have obtained the use of biomedical image text corresponding to large data sets to train multimodal AI models. The data set includes chest X-ray, MRI, histology, pathology and CT images, covering a relatively comprehensive range.

▲ Image Source Microsoft uses GPT-4, based on Vision Transformer and Vicuna language models, to train LLaVA-Med on eight Nvidia A100 GPU, which contains "all pre-analysis information for each image" to generate questions and answers about images to meet the assistant vision of "answering questions about biomedical images in natural language".

In the process of learning, the model mainly focuses on "describing the content of this kind of image" and "expounding the biomedical concept (CTOnews.com Note: judging from the picture)". According to Microsoft, the model ultimately has "excellent multimodal dialogue capabilities" and "LLaVA-Med leads other advanced models in the industry on some of the three standard biomedical data sets used to answer visual questions."

▲ Picture Source Microsoft research team said: "although we believe that the LLaVA-Med model represents an important step towards building a useful biomedical visual assistant, there are still some shortcomings in the current LLaVA-Med model, that is, the common false examples and poor accuracy of large models. The research team will focus on improving the quality and reliability of the model in the future. So that the model can one day be used in commercial biomedicine. "

CTOnews.com notes that the model is currently open source, and you can find relevant information at GitHub.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.