Shulou(Shulou.com)06/01 Report--

This article mainly introduces the "Python load data from different formats in the speed of example analysis", in the daily operation, I believe that many people in Python from different formats of files in the speed of loading data for example analysis of the problem, Xiaobian consulted all kinds of information, sorted out a simple and easy to use method of operation, hope to answer the "Python load data from different formats in the speed of example analysis" of the doubt! Next, please follow the editor to study!

Anyone familiar with Python data analysis and dealing with various data storage files may know that even with the same data content, the speed of reading data from different formats with Python is not the same. To verify this conclusion, I specifically tested the efficiency of reading data from Excel,MySQL and CSV using Python, respectively.

Tools:

Jupyter notebook (Python)

Navicat (MySQL)

Microsoft Excel 2016

CSV (Excel saved as by F12)

Let's get down to business. Here are my test results:

Disclaimer: all use the same data source (excluding the header, there are a total of 159727 data)!

① directly uses Python to load data (29m) from Excel, which takes 48 seconds for the first time and 46.1 seconds for the second time. In addition, the calculation results only take tens to hundreds of milliseconds.

It took dozens to hundreds of milliseconds to process the data:

It took ② 28.9 seconds to import Excel data (29m) into Navicat (an editor that can write MySQL statements).

It took 0.134s to process the data.

③ uses Python to get the data just imported from Excel directly from MySQL. It took 1.91 seconds for the first time, 2.01 seconds for the second time, 1.99 seconds for the third time, and 1.79 seconds for the fourth time.

The time of processing data took 59ms.

④ used Python to read the 29m Excel and saved it as a 32.1m CSV file (the data content is the same). It only took 1.3 seconds (instant! Surprise!)

It took 19ms to process the data.

At this point, the study on the "Python load data from different formats of the speed of example analysis" is over, I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.