Supplementary Results

Home / Supplementary Results

Opinion mining tools for sentiment polarity/emotion/politeness/trust analysis

Emotion Detection

Paper: The Development and Psychometric Properties of LIWC2015
Implementation: https://liwc.wpengine.com
Input: text
Output: word count, summary variable (Analytic, Clout, Authentic, Tone), approximately 90 output variables
Core technique: dictionary-based
Advantage: LIWC2015 is accurate, easier to use, and provides a broader range of social and psychological insights.
Limitations: Commercial software.

Paper: https://arxiv.org/ftp/arxiv/papers/1607/1607.00139.pdf
Implementation: http://sentistrength.wlv.ac.uk/TensiStrength.html
Input: text
Output: relaxation strength (1 to 5) and stress strength (-1 to -5)
Core technique: dictionary-based
Advantage: It is able to accurately estimate the strength of stress and relaxation including general tweets, stress-rich tweets, emotion-rich tweets, tweets with insults, opinionated tweets, and transport-related tweets.
Limitations: A collection of stress/relaxation texts may be the most difficult for a stress/relaxation classification system to process because there will be few obviously neutral texts

Paper: https://doi.org/10.1145/3167132.3167296
Implementation: https://figshare.com/s/277026f0686f7685b79e
Input: text
Output: detected emotions (excitement, stress, depression, relaxation and neutrality)
Core technique: dictionary-based
Advantage: (1) DEVA is especially crafted for software engineering text. (2) DEVA is capable of detecting both valence and arousal in text and mapping them for capturing individual emotional states (e.g., excitement, stress, depression, relaxation and neutrality)
Limitations: It falls short in handling complex structures of negations. The tool cannot distinguish irony and sarcasm in text.

Paper: https://doi.org/10.1145/3297280.3297455
Implementation: https://figshare.com/s/a3308b7087df910db38f
Input: text
Output: detected emotions (excitement, stress, depression, relaxation and neutrality)
Core technique: Support Vector Machine classifier leveraging dictionaries
Advantage: MarValous demonstrates 83.37% precision and 79.33% recall, which outperforms other state-of-the-art tools.
Limitations: Trained on a unbalanced dataset where relaxation only accounts for 4.43% of all data.

Paper: https://doi.org/10.1109/ACIIW.2017.8272591
Implementation: https://github.com/collab-uniba/Emotion_and_Polarity_SO
Input: text
Output: detected emotions (joy, love, surprise, anger, sadness, and fear)
Core technique: Support Vector Machine
Advantage: It achieves good precisions (0.77-0.92) and recalls (0.77-0.92) for all emotions except surprise.
Limitations: Relatively low precision and recall for surprise.

Paper: https://doi.org/10.18653/v1/s18-1037
Implementation: https://github.com/cbaziotis/ntua-slp-semeval2018
Input: text
Output: detected emotions (anger, anticipation, disgust, fear, joy, love, optimism, pessimism, sadness, surprise, trust)
Core technique: Bidirectional LSTM
Advantage: It achieves excellent results in single and multi-label emotion classification tasks.
Limitations: When it comes to the emotion and valence intensity. the results are mixed.


Politeness Detection

Paper: https://www.aclweb.org/anthology/P13-1025/
Implementation: http://www.cs.cornell.edu/~cristian/Politeness.html
Input: text
Output: polite/impolite
Core technique: Support Vector Machine
Advantage: It achieves near-human performance.
Limitations: Not mentioned in the paper.


Trust Estimation

Paper: https://www.scitepress.org/Papers/2016/58306/58306.pdf
Implementation: https://github.com/Tulivick/Trust-Framework
Input: GitHub repository
Output: trust graph (trust score between members)
Core technique: mathematical model based on sentiment analysis scores
Advantage: It extracts trust evidences observed in member interactions inside versioning systems, without human intervention.
Limitations: Not verified with trust information about team members in real projects.