How to execute website crawler regularly in Python
How to execute website crawler regularly in Python? in view of this problem, this article introduces the corresponding analysis and answer in detail, hoping to help more partners who want to solve this problem to find a more simple and easy way.
Write crawler code
Write a crawler that crawls and parses Yahoo using requests and beautifulsoup4 packages! Stock market-listed transaction price ranking with Yahoo! Stock market-the information of OTC transaction price ranking, and then use pandas package to display the parsed data.
Import datetime
Import requests
From bs4 import BeautifulSoup
Import pandas as pd
Def get_price_ranks ():
Current_dt = datetime.datetime.now () .strftime ("% Y-%m-%d% X")
Current_dts = [current_dt for _ in range]
Stock_types = ["tse", "otc"]
Price_rank_urls = ["https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e={}&n=100".format(st) for st in stock_types]
Tickers = []
Stocks = []
Prices = []
Volumes = []
Mkt_values = []
Ttl_steps = 100.100
Each_step = 10
For pr_url in price_rank_urls:
R = requests.get (pr_url)
Soup = BeautifulSoup (r.text, 'html.parser')
Ticker = [i.text.split () [0] for i in soup.select (".name a")]
Tickers + = ticker
Stock = [i.text.split () [1] for i in soup.select (".name a")]
Stocks + = stock
Price = [float (soup.find_all ("td") [2] .find _ all ("td") [I] .text) for i in range (5, 5+ttl_steps, each_step)]
Prices + = price
Volume = [int (soup.find_all ("td") [2] .find _ all ("td") [I] .text.replace (",", ") for i in range (11, 11+ttl_steps, each_step)]
Volumes + = volume
Mkt_value = [float (soup.find_all ("td") [2] .find _ all ("td") [I] .text) * 100000000 for i in range (12, 12+ttl_steps, each_step)]
Mkt_values + = mkt_value
Types = ["listed" for _ in range] + ["OTC" for _ in range (100)]
Ky_registered = [True if "KY" in st else False for st in stocks]
Df = pd.DataFrame ()
Df ["scrapingTime"] = current_dts
Df ["type"] = types
Df ["kyRegistered"] = ky_registered
Df ["ticker"] = tickers
Df ["stock"] = stocks
Df ["price"] = prices
Df ["volume"] = volumes
Df ["mktValue"] = mkt_values
Return df
Price_ranks = get_price_ranks ()
Print (price_ranks.shape)
The result of this is shown as
# # (200,8)
Next, we use pandas to show the first few lines.
Price_ranks.head ()
Price_ranks.tail ()
Then we start to deploy to the server.
For the choice of the server, the environment configuration is outside the scope of this lesson, we will mainly talk about how to set up scheduled tasks.
Next, let's modify the code so that the results are stored in sqlite.
Import datetime
Import requests
From bs4 import BeautifulSoup
Import pandas as pd
Import sqlite3
Def get_price_ranks ():
Current_dt = datetime.datetime.now () .strftime ("% Y-%m-%d% X")
Current_dts = [current_dt for _ in range]
Stock_types = ["tse", "otc"]
Price_rank_urls = ["https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e={}&n=100".format(st) for st in stock_types]
Tickers = []
Stocks = []
Prices = []
Volumes = []
Mkt_values = []
Ttl_steps = 100.100
Each_step = 10
For pr_url in price_rank_urls:
R = requests.get (pr_url)
Soup = BeautifulSoup (r.text, 'html.parser')
Ticker = [i.text.split () [0] for i in soup.select (".name a")]
Tickers + = ticker
Stock = [i.text.split () [1] for i in soup.select (".name a")]
Stocks + = stock
Price = [float (soup.find_all ("td") [2] .find _ all ("td") [I] .text) for i in range (5, 5+ttl_steps, each_step)]
Prices + = price
Volume = [int (soup.find_all ("td") [2] .find _ all ("td") [I] .text.replace (",", ") for i in range (11, 11+ttl_steps, each_step)]
Volumes + = volume
Mkt_value = [float (soup.find_all ("td") [2] .find _ all ("td") [I] .text) * 100000000 for i in range (12, 12+ttl_steps, each_step)]
Mkt_values + = mkt_value
Types = ["listed" for _ in range] + ["listed" for _ in range (100)]
Ky_registered = [True if "KY" in st else False for st in stocks]
Df = pd.DataFrame ()
Df ["scrapingTime"] = current_dts
Df ["type"] = types
Df ["kyRegistered"] = ky_registered
Df ["ticker"] = tickers
Df ["stock"] = stocks
Df ["price"] = prices
Df ["volume"] = volumes
Df ["mktValue"] = mkt_values
Return df
Price_ranks = get_price_ranks ()
Conn = sqlite3.connect ('/ home/ubuntu/yahoo_stock.db')
Price_ranks.to_sql (price_ranks, conn, if_exists= "append", index=False)
Next, if we ask him to start on a regular basis, we need the linux crontab command:
If we want to set it to be executed every hour between 9:30 and 16:30 every day
So we just need to name the file price_rank_scraper.py first.
This is the answer to the question about how to regularly execute the website crawler in Python. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.
