Ask Willie

A closer look at what it is, what is in it, and what I built.

What it does

Loads 686 Wikipedia articles about programming languages, each with its text and the pages it links to.
Takes a query at a search> prompt and prints the ten best matches with their Wikipedia URLs, until you type :quit.
Ranks results by combining page importance and text match, each normalized to the range 0 to 1, with a harmonic mean.

Three page-importance measures: every page equal, in-degree (how many pages link to it), and a random-surfer PageRank.
The PageRank sends 10,000 random walkers for 100 steps each, with a 15 percent chance on each step of jumping to a random page instead of following a link, and weights each page by where the walkers finish: (visits + 1) / (walkers + pages).
Three text-match measures: raw term count, term frequency, and TF-IDF.
Arithmetic, geometric, and harmonic mean orderings for combining the two scores. The finished program uses PageRank, TF-IDF, and the harmonic mean.

The 10,000 random walks run as a Scala parallel collection, since every walker is independent.
TF-IDF computes each query term's document frequency in parallel, then each page's per-term scores in parallel before summing them.
Term frequency measures page lengths in parallel.

The data loading, page classes, normalization, and search loop were starter code; the assignment was to fill in the functions.
PageRank is random, so rankings shift slightly from run to run.
Term frequency divides by the page's character count rather than its word count.
The ranking method is fixed in the code, with no command-line options to switch it, and there are no tests.