Breaking News: Grepper is joining You.com. Read the official announcement!

not getting full data of a site when using python requests

Obnoxious Oystercatcher answered on December 8, 2022 Popularity 2/10 Helpfulness 3/10

answer not getting full data of a site when using python requests

not getting full data of a site when using python requests

Comment

The page is rendered with JavaScript making more requests to fetch additional data. You can fetch the complete page with selenium.

```
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome()
url = "https://shop.nordstrom.com/c/womens-dresses-shop?origin=topnav&cm_sp=Top%20Navigation-_-Women-_-Dresses&offset=11&page=3&top=72"
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()
print(soup.prettify())
```

For other solutions see my answer to https://stackoverflow.com/questions/45259232/scraping-google-finance-beautifulsoup/45259523#45259523

Popularity 2/10 Helpfulness 3/10 Language python

Source: Grepper

Tags: python python-requests using when

Link to this answer
Share Copy Link

Contributed on Dec 08 2022

Obnoxious Oystercatcher

0 Answers Avg Quality 2/10

Continue with Google

By continuing, I agree that I have read and agree to Greppers's Terms of Service and Privacy Policy.

not getting full data of a site when using python requests

Contents

More Related Answers

not getting full data of a site when using python requests

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.