Google News Scraper

Google News is a powerful news aggregator that collects and organizes news articles from various news sources worldwide. You can browse articles directly on the Google News web page or access specific categories, like business, technology, or sports, based on your interests.

This Python scraper collects article listings from a selected Google News category or subcategory. It extracts headlines, Google News URLs, publishers, publication dates, and authors when available. It can also capture related-story group information.

The script uses Requests and BeautifulSoup to process the HTML returned by a single request. It does not search Google News by keyword, scroll for more results, or visit individual publishers to retrieve full articles.

• Category and subcategory selection. Paste a Google News URL under /topics/, including /topics/.../sections/... subcategory URLs.
• Residential proxy support. Send requests through Decodo proxies using credentials stored in .env.
• Article metadata extraction. Collect titles, links, publishers, publication timestamps, authors, and related-story information when available.
• Configurable output. Choose a result limit, filter articles by publication age, and optionally keep only the lead article from each identified story group.
• CSV export. Save the results to scraped_articles.csv for further analysis.

1. Install Python. Download Python 3.9 or later from the official website.

2. Install dependencies:

   pip install requests beautifulsoup4 python-dotenv

3. Configure your Decodo residential proxy. Create a .env file in the same directory as google-news-topic-with-proxies.py and provide your proxy credentials:

   PROXY_SERVER=http://gate.decodo.com:7000
   PROXY_USERNAME=your_username
   PROXY_PASSWORD=your_password

   Alternatively, set a single PROXY_URL with the credentials included. Never commit your .env file to GitHub.

4. Choose a category or subcategory. Open Google News, select a category or subcategory, and paste its full URL into TOPIC_URL in google-news-topic-with-proxies.py. Subcategory URLs include /sections/.

5. Adjust the optional settings:

   MAX_ITEMS = 50                   # Set to None for all articles in the initial HTML.
   MAX_AGE_DAYS = None              # Set to 1, 7, or 30 to filter by publication age.
   INCLUDE_RELATED_ARTICLES = True  # False keeps only the first article per story group.

6. Run the script:

   python google-news-topic-with-proxies.py

1. Fetches the selected Google News topic page through the configured proxy.
2. Parses the returned HTML using BeautifulSoup and finds article links and their metadata.
3. Applies publication-date and related-story filters, followed by the result limit.
4. Saves the selected records to scraped_articles.csv and prints a summary of the extracted fields and output path.

It uses only the initial HTML response, so increasing MAX_ITEMS won't trigger infinite scrolling or retrieve further pages. Some articles do not include author or related-story details.

The output CSV includes these columns:

Access restrictions: As checked in October 2026, Google News' robots.txt allowed /topics/ pages but disallowed its search and RSS search endpoints. Check the current robots.txt and Google's Terms of Service before scraping. Proxies do not override access restrictions.

For a more in-depth tutorial on how to create your own Google News scraper with Python, read the full blog post.

Google Maps scraper

Python scraper tutorial