UCI Web Crawler
A polite, multithreaded crawler that swept the UCI ICS web domains and reported what it found without falling into traps.
A crawler's real enemy isn't scale. It's the pages that never end.
The task was to crawl four UCI departmental domains end to end and report on them: how many unique pages exist, which is the longest, the fifty most common words, and how many pages sit under each subdomain. The fetching itself was solved for us: requests go through a shared cache server, and the framework hands back a response object.
What isn't solved is knowing when to stop. Departmental sites are full of calendars that generate a fresh URL for every day into the next century, wiki histories that link to every revision of themselves, and paths that recurse into each other forever. A crawler with no defenses spends its entire run inside one of those and never reaches the rest of the domain, while a crawler that's too aggressive about filtering quietly skips real content and undercounts the report it was written to produce.
What the crawl has to get right.
The frontier must terminate. Calendar generators, date-stamped archives, and self-recursive paths have to be recognized as infinite before the crawler walks into them.
No host may be hit more often than the configured delay allows, and that has to hold with several worker threads running at once, because the limit is per host, not per thread.
Every statistic is computed over unique pages. URLs that differ only by fragment are the same page, and a page must never be counted twice no matter which thread reached it.
A four-person course project for UCI's Information Retrieval course; the fetch layer and worker skeleton ship with the assignment. The crawl policy, parsing, statistics, and the threading changes to the frontier are ours.
Decide before you queue, not after you fetch.
Every URL is validated at the moment it's extracted, before it ever enters the frontier. A link that fails the domain check or trips a trap heuristic is dropped where it's found, so the queue only ever holds URLs the crawler intends to fetch, and the expensive part of the pipeline never sees them.
Statistics are accumulated during the crawl rather than in a second pass over saved pages. Each page is parsed once: its text feeds the word counts and the longest-page record, its links feed the frontier, and the page itself is never stored. The run produces its report as a side effect of crawling, so a crawl that has to be stopped early still has usable numbers.
Four threads, one shared frontier.
Queue with a politeness clock
A thread-safe queue backed by a shelve file, plus a per-host last-access timestamp. A URL whose host was hit too recently is set aside and retried, not blocked on.
Fetch through the cache
Four workers pull URLs and download them through the course cache server, so the crawl never hits the real hosts directly.
Parse, filter, count
BeautifulSoup extracts text and links; the text updates the running statistics under a lock, and each link is validated before it's returned.
What happens to one page.
Every URL a worker pulls off the frontier runs this path. Anything that fails a check is dropped immediately rather than queued for later.
Take a URL the politeness clock allows
Workers pull from a shared queue, and each URL's host is checked against a last-access timestamp guarded by its own lock. If the host was contacted within the delay window, the URL is set aside and the worker moves on to another host instead of sleeping, so four threads crawl four hosts in parallel while each individual host still sees a well-spaced request stream.
Refuse the response before parsing it
Anything that isn't a 200, has no body, or exceeds 5 MB is dropped without being parsed. The size cap matters more than it looks: a handful of enormous documents can otherwise dominate both the parse time and the word-frequency totals for the whole run.
Count the page exactly once
The URL is defragmented, since two links to the same page with different anchors are one page. Under a lock, the crawler checks whether it has seen that canonical form before; only the first sighting updates the unique-page set, the longest-page record, the word frequencies, and the per-subdomain tally. Pages under 50 content words are excluded from the statistics, since navigation shells and stubs would otherwise dominate the word counts without being real content.
Validate every extracted link at the source
Links are resolved to absolute URLs and then filtered: HTTPS or HTTP only, hostname inside the four allowed domains or a subdomain of them, no binary or media extensions, no path over 400 characters. Only survivors are returned to the frontier, deduplicated in discovery order.
Recognize the traps by shape
Three heuristics catch the infinite regions. Calendar and event URLs are matched on path and query keywords plus embedded date patterns like /2024/03/, which kills the generators that emit a new page per day forever. Repeated path segments flag recursive routes, with a tolerance so that a legitimately repetitive path isn't discarded on one coincidence. And known wiki endpoints are excluded outright, since their revision and diff views link to every version of every page.
What I chose, and what it cost.
Filter aggressively on URL shape
CostThe date and calendar heuristics will drop legitimate pages that happen to live under a dated path, a news post at /2024/03/title being the obvious case. Losing a few real pages was worth not losing the run to a calendar.
Track politeness per host, not globally
CostMore bookkeeping and a second lock, and a URL can be pulled and requeued several times before its host is ready. In exchange, the threads actually run in parallel instead of serializing behind one global delay.
The crawler was written single-threaded, and the report was correct. Turning on four workers broke it in a way that looked like nothing at all: the unique-page count was off, and the longest page changed between runs on the same corpus. Two threads finishing the same URL would both see it missing from the unique set and both count it, and the longest-page record was a read-then-write that either could lose. The fix was to make the whole statistics update (membership test, insert, and every counter it feeds) one critical section under a single lock, with the politeness timestamps kept under a separate lock so page bookkeeping and host scheduling never wait on each other.
Where I'd take it.
Near-duplicate detection. Exact URL deduplication misses the same content served under different paths. Simhash over the extracted text would catch the mirrored pages the current counts treat as distinct.
Write statistics out incrementally instead of only at the end, so a crawl killed mid-run doesn't lose its report.
Report why a URL was rejected rather than just that it was. The filters are tuned by staring at logs, and a per-rule rejection count would show which heuristic is being over-eager.