20 Newsgroups - Topic Map
18,846 postsNLPembeddingscosine
A TMAP of the 20 Newsgroups dataset (18,846 posts), embedded with all-MiniLM-L6-v2 (cosine metric). 73.5% of tree edges connect posts from the same newsgroup (mean subtree purity 0.619), rising to 78.2% mean purity when grouped into supercategories (comp.*, rec.*, sci.*, talk.*, etc). The most-crossed topic pairs make intuitive sense: soc.religion.christian <-> talk.religion.misc (203 edges), comp.os.ms-windows.misc <-> comp.sys.ibm.pc.hardware (188 edges), and rec.autos <-> rec.motorcycles (150 edges). Post length also tracks tree structure, with a mean edge-to-edge word-count delta of 183 words. Coherence varies sharply by topic: comp.windows.x is the tightest (31.3% boundary edges) while talk.religion.misc is the most scattered (73.9%).
20 Newsgroups - Topic MapOpen full page
Loading interactive demo…
How it was made
generate.pypython
from sentence_transformers import SentenceTransformer
from tmap import TMAP
from tmap.graph.analysis import boundary_edges, subtree_purity
encoder = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = encoder.encode(posts, show_progress_bar=True)
model = TMAP(metric="cosine", n_neighbors=20, seed=42).fit(embeddings)
purity = subtree_purity(model.tree_, newsgroup_labels, min_size=20)
print(f"Mean subtree purity: {purity.mean():.3f}")
viz = model.to_tmapviz()
viz.add_label("newsgroup", newsgroup_labels)
viz.add_color_layout("newsgroup", newsgroup_labels, categorical=True, color="tab20")
viz.write_html("newsgroups.html")