Topic Modeling Definition of Nature

Topic Modeling

https://cran.r-project.org/web/packages/topicmodels/vignettes/topicmodels.pdf

These are “mixed-membership models” (see [1]). This assumes that definitions (in the case of this analysis) can belong to several topics and that the topic distribution can vary across definitions. This differs to other models (such as Semantic Analysis) which assumes that each word comes from the same distribution - that is, it is just as likely for a given word to appear in any one definition.

Two models can be used:

1. Latent Dirichlet Allocation (LDA) - this is a Bayesian mixture model when it is assumed that topics are uncorrelated (this would not be the case in this analysis)

2. Correlated Topics model (CTM) - extends upon LDA by allowing correlations between the topics

(see [2] and [3] for intros to these models)

Correlated Topics Model

This analysis uses the topicmodels package which uses the VEM (variation expectation-maximization) algorithm.

Preprocessing

• Each row = participant
• Each column = word
• convert to lower case
• remove punctuation
• remove numbers
• stemming (removing prefixes and suffixes)
• removing stop words
• removing words below a certain length minimum
• Optional: select only words that occur in a minimum number of definitions (see [4])
• Optional: select terms which highest term-frequency inverse document frequency (tf-idf) scores (see [3]) - this is only used for selecting the vocabulary in the corpus

Fitting the Model

In the CTM you fix the number of topics (k) a-priori. In this analysis I ran the model fixed at both 14 (the number of categories identified during manual coding) and 7 (the higher-order topics that each of these categories fit into).

“Additionally, estimation using Gibbs sampling requires specification of values for the parameters of the prior distributions. [4] suggest a value of 50/k for α and 0.1 for δ. Because the number of topics is in general not known, models with several different numbers of topics are fitted and the optimal number is determined in a data-driven way.”

Selecting the number of topics can be done by splitting the data into training and testing sets. ” The likelihood for the test data is then approximated using the lower bound for VEM estimation.”

“Another possibility for model selection is to use hierarchical Dirichlet processes as suggested [5]”.

Topic modelling uses a probabilistic algorithm to calculate the probability of a word appearing given a particular topic (category of definition)

After going through all of this work, I do not think that this is a useful way of categorizing definitions. There is too much complexity in the way that people use synonyms in order for the algorithm to detect probabilistically what their definition should be categorized as.

1. Airoldi EM, Blei DM, Fienberg SE, Xing EP (2008). “Mixed Membership Stochastic Block- models.” Journal of Machine Learning Research, 9, 1981–2014.

2. Steyvers M, Griffiths T (2007). “Probabilistic Topic Models.” In TK Landauer, DS McNamara,S Dennis, W Kintsch (eds.), Handbook of Latent Semantic Analysis. Lawrence Erlbaum Associates.

3. Blei DM, Lafferty JD (2009). “Topic Models.” In A Srivastava, M Sahami (eds.), Text Mining: Classification, Clustering, and Applications. Chapman & Hall/CRC Press.

4. Griffiths TL, Steyvers M (2004). “Finding Scientific Topics.” Proceedings of the National Academy of Sciences of the United States of America, 101, 5228–5235.

5. Teh YW, Jordan MI, Beal MJ, Blei DM (2006). “Hierarchical Dirichlet Processes.” Journal of the American Statistical Association, 101(476), 1566–1581

https://heartbeat.comet.ml/text-classification-using-machine-learning-algorithm-in-r-ba763117c8aa