Breaking News: Grepper is joining You.com. Read the official announcement!

scrapy startproject crawler

Sparkling Sheep answered on August 17, 2022 Popularity 8/10 Helpfulness 1/10

answer scrapy startproject crawler

scrapy startproject crawler

Comment

import scrapy


class AuthorSpider(scrapy.Spider):
    name = 'author'

    start_urls = ['https://quotes.toscrape.com/']

    def parse(self, response):
        author_page_links = response.css('.author + a')
        yield from response.follow_all(author_page_links, self.parse_author)

        pagination_links = response.css('li.next a')
        yield from response.follow_all(pagination_links, self.parse)

    def parse_author(self, response):
        def extract_with_css(query):
            return response.css(query).get(default='').strip()

        yield {
            'name': extract_with_css('h3.author-title::text'),
            'birthdate': extract_with_css('.author-born-date::text'),
            'bio': extract_with_css('.author-description::text'),
        }

Popularity 8/10 Helpfulness 1/10 Language python

Source: docs.scrapy.org

Tags: python scrapy

Link to this answer
Share Copy Link

Contributed on Aug 17 2022

Sparkling Sheep

0 Answers Avg Quality 2/10

scrapy startproject crawler

Contents

More Related Answers

scrapy startproject crawler

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.