Python crawling comments


Answers ( 1 )

  1. Specifically, it pertains to web scraping or web crawling in Python, a technique used to extract data (in this case, comments) from websites.

    To perform web scraping in Python, there are several libraries and approaches you can use:

    1. Using requests and BeautifulSoup

    • requests is a Python library used for making HTTP requests.
    • BeautifulSoup is a library for parsing HTML and XML documents. It creates parse trees that are helpful to extract data easily.


    import requests
    from bs4 import BeautifulSoup
    # URL of the page where comments are
    url = ''
    # Send a GET request
    response = requests.get(url)
    # Parse the page content
    soup = BeautifulSoup(response.text, 'html.parser')
    # Extract comments assuming they are in <div> tags with class 'comment'
    comments = soup.find_all('div', class_='comment')
    for comment in comments:

    2. Using Scrapy

    • Scrapy is an open-source web crawling framework for Python. It is used to write spiders, which are classes that define how a certain site (or a group of sites) will be scraped.


    import scrapy
    class CommentSpider(scrapy.Spider):
        name = 'comment_spider'
        start_urls = ['']
        def parse(self, response):
            for comment in response.css('div.comment'):
                yield {'comment': comment.css('::text').get()}

    3. Using Selenium for JavaScript-Rendered Pages

    • If the comments are loaded dynamically with JavaScript, Selenium can be used. Selenium automates web browsers, allowing you to interact with JavaScript-rendered content.


    from selenium import webdriver
    # Path to chromedriver or another browser driver
    driver = webdriver.Chrome('/path/to/chromedriver')
    # URL of the page
    # Extract comments
    comments = driver.find_elements_by_class_name('comment')
    for comment in comments:


    • Respect Robots.txt: Before scraping any website, ensure you are allowed to scrape it by checking the site's robots.txt file.
    • Handle Rate Limiting: Do not send too many requests in a short period to avoid getting blocked.
    • Check Website's Terms of Service: Some websites explicitly forbid web scraping in their terms of service.

    These are the primary methods to scrape comments from websites using Python. The specific implementation might vary depending on the website's structure and the nature of the data you're trying to extract.

Leave an answer