Sitemap Crawler Example

A sitemap crawler is a tool that reads XML sitemap files and extracts URLs for indexing, analysis, or monitoring purposes. Whether you're building an SEO auditing tool, a web scraper, or a search engine component, understanding how to parse and traverse sitemaps programmatically is essential. This guide walks through a practical sitemap crawler example with executable code that you can adapt for your own projects.

What Is a Sitemap Crawler and Why You Need One

A sitemap is an XML file that lists all URLs on a website, along with metadata like last modified date, change frequency, and priority. A sitemap crawler automatically fetches this file, parses its contents, and extracts the URLs for further processing.

Common use cases include:

• Validating that all website pages are listed in the sitemap
• Checking HTTP status codes for each URL
• Monitoring changes to page priority or update frequency
• Building a database of website structure for SEO analysis
• Feeding URLs into a web crawler for deep site analysis

Rather than manually visiting each URL or writing site-specific parsing logic, a generic sitemap crawler standardizes the process. The XML structure is consistent across most websites, so the same crawling logic works everywhere.

Basic Sitemap Crawler Example in Python

Here is a working example of a sitemap crawler in Python using the built-in libraries:

import urllib.request
import xml.etree.ElementTree as ET

def crawl_sitemap(sitemap_url):
    """Fetch and parse a sitemap, returning a list of URLs."""
    try:
        # Fetch the sitemap file
        with urllib.request.urlopen(sitemap_url) as response:
            sitemap_content = response.read()
        
        # Parse XML
        root = ET.fromstring(sitemap_content)
        
        # Extract all URL elements
        # Namespace is required for proper parsing
        namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
        urls = []
        
        for url_element in root.findall('ns:url', namespace):
            loc = url_element.find('ns:loc', namespace).text
            lastmod = url_element.find('ns:lastmod', namespace)
            lastmod_text = lastmod.text if lastmod is not None else 'N/A'
            
            urls.append({
                'loc': loc,
                'lastmod': lastmod_text
            })
        
        return urls
    
    except Exception as e:
        print(f"Error crawling sitemap: {e}")
        return []

# Example usage
if __name__ == '__main__':
    sitemap = 'https://example.com/sitemap.xml'
    results = crawl_sitemap(sitemap)
    
    for item in results:
        print(f"URL: {item['loc']}")
        print(f"Last Modified: {item['lastmod']}")
        print()

This basic example fetches the sitemap, parses the XML, and extracts the loc (URL) and lastmod (last modified date) fields. The namespace parameter is crucial because sitemaps use XML namespaces.

Handling Sitemap Indexes

Many large websites use sitemap indexes instead of a single sitemap file. A sitemap index is an XML file that contains references to multiple individual sitemap files. Your crawler needs to detect and handle both formats.

def crawl_sitemap_recursive(sitemap_url):
    """Crawl sitemap or sitemap index, handling both formats."""
    namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
    all_urls = []
    
    try:
        with urllib.request.urlopen(sitemap_url) as response:
            content = response.read()
        
        root = ET.fromstring(content)
        
        # Check if this is a sitemap index
        sitemaps = root.findall('ns:sitemap', namespace)
        if sitemaps:
            # This is a sitemap index; crawl each referenced sitemap
            for sitemap_element in sitemaps:
                sitemap_loc = sitemap_element.find('ns:loc', namespace).text
                print(f"Found sitemap index, crawling: {sitemap_loc}")
                all_urls.extend(crawl_sitemap_recursive(sitemap_loc))
        else:
            # This is a regular sitemap; extract URLs
            for url_element in root.findall('ns:url', namespace):
                loc = url_element.find('ns:loc', namespace).text
                all_urls.append(loc)
    
    except Exception as e:
        print(f"Error: {e}")
    
    return all_urls

# Example usage
if __name__ == '__main__':
    sitemap = 'https://example.com/sitemap.xml'
    urls = crawl_sitemap_recursive(sitemap)
    print(f"Found {len(urls)} URLs")
    for url in urls[:10]:  # Print first 10
        print(url)

This version recursively traverses sitemap indexes, so it works with both single and multi-file sitemap structures. This is important for real-world websites that publish hundreds of thousands of pages.

Adding HTTP Status Checking

A useful extension is to verify that each extracted URL actually exists by checking its HTTP status code. This helps identify broken links or removed pages:

import urllib.request
import urllib.error

def check_url_status(url, timeout=5):
    """Return HTTP status code for a URL."""
    try:
        request = urllib.request.Request(url, method='HEAD')
        response = urllib.request.urlopen(request, timeout=timeout)
        return response.status
    except urllib.error.HTTPError as e:
        return e.code
    except Exception as e:
        return None

def crawl_and_validate(sitemap_url):
    """Crawl sitemap and check status of each URL."""
    namespace = {'ns': 'http://www.sitemaps.org/schemas/sitemap/0.9'}
    results = []
    
    try:
        with urllib.request.urlopen(sitemap_url) as response:
            content = response.read()
        
        root = ET.fromstring(content)
        
        for url_element in root.findall('ns:url', namespace):
            loc = url_element.find('ns:loc', namespace).text
            status = check_url_status(loc)
            
            results.append({
                'url': loc,
                'status': status
            })
    
    except Exception as e:
        print(f"Error: {e}")
    
    return results

# Example usage
if __name__ == '__main__':
    sitemap = 'https://example.com/sitemap.xml'
    results = crawl_and_validate(sitemap)
    
    for item in results:
        print(f"{item['url']} - Status: {item['status']}")

Using HEAD requests instead of GET requests is more efficient because it retrieves only headers, not the full page content. This speeds up validation for large sitemaps significantly.

Best Practices for Sitemap Crawlers

When building a production sitemap crawler, follow these guidelines:

• Respect robots.txt: Check if the website's robots.txt allows crawling before fetching the sitemap.
• Add delays between requests: Space out HTTP requests to avoid overloading the server. Use time.sleep() between requests.
• Set a User-Agent: Identify your crawler with a descriptive User-Agent header so website owners know what's accessing their site.
• Handle timeouts: Always set timeout values on network requests to prevent hanging indefinitely.
• Log errors separately: Track which URLs failed validation and why for debugging.
• Cache results: Store sitemap data temporarily to avoid repeated fetches within a short time window.

These practices ensure your crawler is respectful, reliable, and maintainable over time.

Common Challenges and Solutions

XML namespace handling is the most common issue when parsing sitemaps. Many developers forget the namespace parameter and get empty results. Always include the namespace when using findall() or find() methods.

Gzipped sitemaps are also common. Some websites serve sitemap.xml.gz instead of plain XML. Python's urllib automatically handles gzip decompression if the server sends the correct Content-Encoding header, but you can also manually decompress if needed:

import gzip

with urllib.request.urlopen(sitemap_url) as response:
    if response.headers.get('Content-Encoding') == 'gzip':
        content = gzip.decompress(response.read())
    else:
        content = response.read()

Another challenge is handling very large sitemaps that contain millions of URLs. Parsing the entire file into memory can exhaust system resources. For large files, use iterative XML parsing with an event-driven approach instead of loading the entire document.

Next Steps

Once you understand the basics of sitemap crawling, you can extend the examples here to build more sophisticated tools. Store results in a database, generate reports on URL health, or integrate the crawler into an automated SEO monitoring system.

The code examples in this guide use only standard library modules, so they run on any Python installation without external dependencies. For production systems with advanced requirements, libraries like requests and lxml offer additional convenience and performance, but the fundamentals remain the same.