Shulou(Shulou.com)06/02 Report--

In this issue, the editor will bring you some tips about the use of Pandas. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

For tens or hundreds of gigabytes of data, when reading such big data, is there any way to randomly select a small part of the data and then read it into memory to quickly understand the data and carry out EDA?

Using Pandas's skiprows and probability knowledge, you can do it.

Let's explain how to do it.

Read some 100G big_data.csv data as shown below

Use the skiprows parameter

X > 0 make sure the first line is read

Np.random.rand () > 0.01means 99% of the data will be randomly filtered out.

The implication is that only 1% of all data has a chance of being selected into memory.

Import pandas as pd

Import numpy as np

Df = pd.read_csv ("big_data.csv"

Skiprows =

Lambda x: X > 0 and np.random.rand ()

Print ("The shape of the df is {}.

It has been reduced 100 times! ".format (df.shape))

Using this method, the amount of data read is rapidly reduced to 1% of the original, which is helpful for the rapid development of data analysis.

These are the tips that the editor shares with Pandas. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.