How Search Engines Work: Crawl, Index, Rank
How a query becomes 1.5 million results in 0.4 seconds
How a query becomes 1.5 million results in 0.4 seconds
A search engine starts with a crawler: a program that visits a web page, reads it, and follows every link it finds. Then it visits those pages and follows their links. The web becomes a giant graph the crawler walks. Click "Crawl" to send the spider out.
Click a starting page to send the crawler out.
Reading every page for every search would take forever. Instead, search engines build an inverted index: a map from each word to the list of pages it appears on. When you search "robot", the engine doesn't read pages, it just looks up "robot" in this map. Type a query to see it work.
Type a word and click Search to look it up in the index.
Many pages might match your query. Which one shows up first? PageRank treats every link as a vote: a page is important if other important pages link to it. Click pages to add new links and watch the rankings shift.
Click and drag from one page to another to add a link. Watch the larger circles get bigger as more pages link to them.
PageRank was the breakthrough, but today's search uses far more. Each of these factors adjusts where a page lands. Modern engines also use AI models that read your query and the page text to figure out what you actually mean.
Does the page contain your exact words? In the title? In headings? Repeated naturally? Older techniques like BM25 and TF-IDF measure this. Newer ones use neural embeddings to match meaning, not just words.
How many sites link here, and how trusted are those sites? A link from a major news site counts more than a link from a random blog. Spam links are detected and discounted.
For news and trending topics, recent pages rank higher. For "how do volcanoes work", a 10-year-old page might be just fine. The engine guesses how much freshness matters per query.
If everyone who searches "best pizza near me" clicks the third result and ignores the first two, the engine learns to promote that third result. This feedback loop is huge.
Slow pages drop in ranking. Pages that aren't mobile-friendly drop on phones. Google has used these as ranking factors since 2018.
Your location, language, and past searches all tweak results. "Football" means different sports in the US vs Europe. The engine quietly adapts.
The first web search engine, Archie (1990), didn't even read web pages. It just listed file names on FTP servers. The full text of pages wasn't searchable until 1993, when WebCrawler launched. Google didn't arrive until 1998 and changed everything by ranking with links instead of just keywords.
You've seen how a search engine turns the entire web into instant answers. Three big ideas: a crawler walks every link to find pages, an inverted index maps words to pages so lookups are fast, and PageRank uses links as votes to rank the best results first. Every Google search uses these same ideas at planet scale.
A crawler (also called a spider or bot) starts on a few known pages, follows every link, then follows every link on those pages. Google's crawlers visit billions of pages a day.
Instead of reading every page for every query, search engines build an "inverted index": a giant map from each word to the list of pages it appears on. Looking up a word becomes a single hash table lookup.
A page is important if other important pages link to it. PageRank treats every link like a vote, and votes from highly-ranked pages count more. Larry Page and Sergey Brin used this idea to build Google in 1998.
Modern search engines use far more than PageRank: text relevance, page speed, freshness, user clicks, mobile-friendliness, and increasingly AI models that understand meaning, not just keywords.
Put your new knowledge into practice!