Qfyilyi

I want to find all the links on the page, this code is only getting the links which starts with http://, however most of the links are https:// how can i edit below to find both.

for link in soup.find_all('a',attrs={'href':re.compile("^http://")}):



import requests,bs4,re

res=requests.get('https://www.nytimes.com/2018/11/21/nyregion/president-trump-immigration-law-firms.html?action=click&module=Top%20Stories&pgtype=Homepage')

soup=bs4.BeautifulSoup(res.text,'html.parser')

x=

y=

z=

for link in soup.find_all('a',attrs={'href':re.compile("^http://")}):

    print(link.get('href'))

    x=link.get('href')

I know i can simply do to get all the links.

for i in soup.select('a'):

    print(i.get('href'))

You can use this regular expression to match http or https:

^(http|https)://.*

Regular expression (a|b) means : match pattern a or b.

you want to categorize your link into http and https? find it using .startswith() or re.match()

http = 

https = 

for link in soup.find_all('a'):

    url = link.get('href')

    if url.startswith('http://'): # or: if re.match("^http://", url)

      http.append(url)

    else:

      # should be https://

      https.append(url)



print(https)

print(http)