Python & data / SAMPLE ARTICLE
Finding related news with n-grams
An explainable search experiment across Reuters-21578.
Reuters Similarity Lab compares news articles through shared sequences of words. It is an experiment in lexical overlap: the score describes shared text, not whether two articles mean the same thing.
From words to sets
A unigram is one word, a bigram is two consecutive words, and a trigram is three. Each article becomes a set of these sequences. Jaccard similarity divides the number of shared sequences by the number of distinct sequences across both articles.
The full corpus
The downloader verifies the complete 22-file Reuters-21578 collection. Its 21,578 records include 19,043 nonempty article bodies that can be indexed. Records without a body remain accounted for.
Doing less work per query
An inverted index records which articles contain each sequence. Search considers articles sharing query sequences instead of comparing every article from scratch. In the recorded benchmark, all 30 top-five rankings matched an exhaustive reference.
Where this approach stops
Word overlap does not understand synonyms or paraphrases. Larger n-grams preserve more phrase structure but become more sensitive to wording. The notebook makes those tradeoffs visible through queries, plots, and reproducible measurements.