MIT calls third-party Twitter robot detection tools "inaccurate": data sets are too simple and poor generality

CTOnews.com June 26, the Massachusetts Institute of Technology (MIT) research team recently published a paper pointing out that the existing third-party Twitter (Twitter) robot account automatic detection tool is not accurate because its data set is too simple and lack of generality.

Earlier, it was reported that too many robot accounts were one of the reasons to prevent Musk from buying Twitter. Twitter claimed at the time that 5% of its daily active users were robot accounts, but Musk said that number was much higher than 5%.

Twitter has its own robot account identification system, but it has not been made public. Therefore, for the general public, the third-party tool is a more feasible detection method. These third-party tools use data sets and machine learning models collected from Twitter to detect suspicious signs of robots. Many tools and models have been used to study robot activities on social media, and there have been thousands of related papers.

▲ 's public benchmark data set for Twitter robot detection most of the benchmark data sets in these papers are data sets collected in different tweets, many of which are collected in specific tweets (such as tweets with specific topic tags), each of which is manually marked as a robot or human. However, this specially trained robot detection model does well in this professional field, does not cover all areas, and relies heavily on specific data, rather than the fundamental differences between robots and humans.

When these models are tested on data sets in other areas, their accuracy is very poor, almost equal to the level of random prediction. At the same time, in many data sets, even the relatively simple model is as accurate as the most advanced machine learning model (SOTA).

Comparison of the performance of ▲ simple model and SOTA model on basic data sets in other words, the model trained on one data set can not be extended to other data sets, and the existing robot detection data sets are less universal because of simple data collection.

Finally, the researchers warn that when using existing robots to detect data sets, users should carefully consider what types of deviations may exist. The researchers believe that a fundamental solution is that social media such as Twitter itself should provide researchers with rich and reliable data and high-quality real tags.

CTOnews.com enclose the address of the paper: click here to