.. _liwc:

Linguistic Inquiry and Word Count (LIWC) and Other Lexicons
============================================================

High-Level Intuition
*********************
The Linguistic Inquiry and Word Count (LIWC) is a series of dictionaries representing various psychologically relevant concepts (for example, "Home" includes words such as "family," "apartmentment," and "kitchen;" "Exclusive" language refers to words such as "but," "without," and "exclude.") These create a very simple proxy for the content of language.

In addition to LIWC, we also incorporate additional lexicons where relevant, such as Hu and Liu's `Positive Word Lexicon <http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html>`_.

Citation
*********
By default, we use LIWC-2007; `Pennebaker et al. (2007) <https://www.liwc.net/LIWC2007LanguageManual.pdf>`_

Positive Word Lexicon: `Hu and Liu (2004) <https://www.cs.uic.edu/~liub/publications/kdd04-revSummary.pdf>`_

`NLTK's List of English Stowords <https://gist.github.com/sebleier/554280>`_

Implementation 
****************
For each word in the LIWC lexicon, we use a regular expression to count the number of times the word appears. The regular expression captures word stems where relevant; for example, "certain*" would capture "certainty," "certainly," etc.
In each regex pattern, the lexicons are separated by `|` and wrapped by word boundaries `\b` to ensure that we are capturing whole words, except for the following exceptions:

1. We preserve 8 emojis from LIWC-2015: `"(:", "(;", "):", "/:", ":(", ":)", ":/", ";)"` so that they can be counted even when they are not separated by spaces.

2. Words end with non-alphanumeric characters, such as `bachelor'` are wrapped by `(?<!\w)` and `(?!\w)` because `\b` doesn't work properly.

Furthermore, all parentheses are escaped to ensure that they are treated as literals. After processing these, we put hyphenated words at the beginning of the list and sort it based on the raw length of each lexicon.
As a caveat, words containing underscores are not recognized by the LIWC dictionary, and are not counted.

**Note:** 

- **In v.1.0.3 and earlier:**  
  Word counts were presented as a scaled rate of **number of words per 100 words**, computed using the following formula:

  .. code-block:: text

      Rate of word use per 100 words = (count / utterance length) * (utterance length / 100)

- **In v.1.0.4 and later:**  
  Lexical values are represented as a **raw count** of the number of times they appear in the utterance.

New in v.1.0.5: "Bring Your Own LIWC" Custom Lexicon
******************************************************

v.1.0.5 introduces a new feature, **"Bring Your Own LIWC"**. We are incredibly grateful to `Ryan Boyd <https://www.ryanboyd.io/>`_ and `Jamie Pennebaker <https://liberalarts.utexas.edu/psychology/faculty/pennebak>`_ for their close support in creating this feature.

The Team Communication Toolkit's built-in version of LIWC is the 2007 version of the lexicon. However, academic users can reach out to Ryan and Jamie for a more up-to-date version of the lexicon. The lexicon is formatted in a standard `.dic` file format.

If you have your own local copy of the LIWC lexicon, you can use the Team Communication Toolkit to generate category wordcounts for your "custom" dictionary. We expect the format to be the LIWC standard format from Boyd and Pennebaker.

To use "Bring Your Own LIWC," simply point the **custom_liwc_dictionary_path** in your FeatureBuilder to a local copy of the lexicon:

.. code-block:: python
  
  feature_builder = FeatureBuilder(
        input_df = your_data, # This is your input dataframe
        output_file_base = "liwc_test_output",
        custom_liwc_dictionary_path = "./LIWC2015.Dictionary.English.2024.08.27.95450.dic" # Obtain this file from Ryan and Jamie
  )
  feature_builder.featurize()

Outputted columns from a custom lexicon will have the name of the lexicon appended by ``_lexical_wordcount_custom`` to denote that they come from a custom lexicon. The names of columns generated by the custom lexicon will appear in ``FeatureBuilder.chat_features``.


Interpreting the Feature 
*************************
In general, the higher the value, the more that an utterance displays the concept "represented" by the lexicon. For example, a high value of the Home lexicon suggests that a speaker is discussing topics related to the home.

We note, however, that the lexicon-based approach to measuring concepts has several limitations. **Lexicons use a bag-of-words-based approach**, which means that it does not take the ordering of words or the context into account. This can be particularly signifiant for issues such as positive language; the sentence "I am not happy" contains the word "happy," which would count towards the positive lexicon --- even though the statement is negative!

Finally, **LIWC typically recommends having an utterance of at least 100 words**, as measures for shorter pieces of text may not be reliable.

Related Features 
*****************
LIWC and related lexicons capture a variety of concepts that re-occur in other parts of the toolkit; for example, positivity is also measured using :ref:`positivity_bert`; measures of certainty appear both here and in :ref:`certainty`; :ref:`proportion_of_first_person_pronouns` uses first-person pronouns, which also appear in LIWC; :ref:`politeness_strategies` and :ref:`politeness_receptiveness_markers` also contain redundant measures (e.g., first person, second person).