Select Page

DataCamp’s Chief Data Scientist David Robinson set out to use his tidytext package (co-authored with Julia Silge) to identify the author of the anonymous op-ed that appeared in the New York Times last week.

Robinson catalogues the difficult problems he faced in his analysis — how to get samples of text written by senior administration officials? — and the uncertainty of any result.

Robinson writes all about the process on Variance Explained.

This is a useful opportunity to demonstrate how to use the tidytext package that Julia Silge and I developed, and in particular to apply three methods:

  • Using TF-IDF to find words specific to each document (examined in more detail in Chapter 3 of our book)
  • Using widyr to compute pairwise cosine similarity
  • How to make similarity interpretable by breaking it down by word

Since my goal is R education more than it is political analysis, I show all the code in the post.

Robinson explains that he would still be skeptical about the results of the text mining exercise because it was such a small sample. Nevertheless, Robinson could not help but conclude that the anonymous writer came from the State Department — but we won’t spoil the results here.