Tutorial

In this tutorial, first, we will evaluate linguistic agency in a list of sentences (each sentence will be evaluated separately). Then we will demonstrate how to use BERTAgent on a dataframe that contains a column of textual data (each entry in this column will also be evaluated separately).

Note

See Installation section for installation instructions.

Note

For a detailed documentation of BERTAgent’s functionality please refer to package docs section.

Process a list of sentences

First, we need to import and initialize BERTAgent.

>>> from bertagent import BERTAgent
>>> ba0 = BERTAgent()

Then we need to provide example sentences as a list of strings.

>>> sents = [
>>>    ...:     "hardly working individual",
>>>    ...:     "hard working individual",
>>>    ...:     "striving to achieve my goals",
>>>    ...:     "struggling to achieve my goals",
>>>    ...:     "struggling to survive",
>>>    ...:     "unable to survive",
>>>    ...:     "this car runs on gasoline with lead",
>>>    ...:     "this car runs on gasoline and it will lead us",
>>>    ...:     "this politician runs for office and he will lead us",
>>>    ...: ]

We get the linguistic agency evaluated for each utterance by running the predict method. bertagent.BERTAgent.predict()

Check the predict documentation for more advanced usage.

>>> vals = ba0.predict(sents)

Print results

>>> for item in zip(sents, vals):
>>>     print(f"  {item[0]!r} : {item[1]:.2f}")
#
#  'hardly working individual' : -0.57
#  'hard working individual' : 0.44
#  'striving to achieve my goals' : 0.73
#  'struggling to achieve my goals' : -0.67
#  'struggling to survive' : -0.52
#  'unable to survive' : -0.57
#  'this car runs on gasoline with lead' : -0.03
#  'this car runs on gasoline and it will lead us' : 0.09
#  'this politician runs for office and he will lead us' : 0.58
#
# NOTE: exact values may differ slightly from the above
# depending on the BERTAgent model and version used.

Process a texts in pandas dataframe

Note

Here we use EXAMPLE_SENTENCES data that is provided with BERTAgent. The code below provides an example use of BERTAgent that can be uses as a template to analyze any other textual data provided by user. Importantly BERTAgent should be used to quantify agency in short utterances (preferably a single sentence). This is why we need to provide BERTAgnet with a list of sentences. If the user has longer texts they can be “chunkified” using brute force approach or (prefferabely) using natural language processing libraries such as SpaCy for more information.

Imports

>>> import pathlib
>>> import pandas as pd
>>> from tqdm import tqdm
>>> from bertagent import BERTAgent
>>> from bertagent import EXAMPLE_SENTENCES as sents
>>> tqdm.pandas()
>>>

Load BERTAgent

>>> ba0 = BERTAgent()

Prepare dataframe.

>>> df0 = pd.DataFrame(dict(text=sents))

Extract sentences from text.

>>> # NOTE: This is not an optimal method to get sentences from real data!
>>> df0["sents"] = df0.text.str.split(".")

Check input dataframe

>>> print(df0.head(n=4))

Input data (pandas dataframe containing lists of sentences)

	sents
0	[‘She is a hard working individual’]
1	[‘She is a hardly working individual’]
2	[‘This thing was made of lead’]
3	[‘This is a car, it runs on gas’]
4	[‘This is a Karen, she runs for office’]
5	[‘This is a Jane, she runs for office’]
6	[‘Striving to achieve my goals’]
7	[‘Struggling to achieve my goals’]
8	[‘Striving to make it’]
9	[‘Struggling to make it’]
10	[‘Striving to survive’]
11	[‘Struggling to survive’]
12	[‘Well planned and well executed’]
13	[‘Coordinated activity’]
14	[‘Uncoordinated activity’]
15	[‘Not coordinated activity’]
16	[‘Everything is messy and uncoordinated’]
17	[‘A bad decisionmaker’]
18	[‘A marvelous decisionmaker’]
19	[‘They are submissive’]
20	[‘They submitted to his will’]
21	[‘They submitted a paper’]
22	[‘They submitted a request’]
23	[‘We are winners’]
24	[‘We are losers’]
25	[‘motivated’]
26	[‘We are motivated’]
27	[‘We are not motivated’]
28	[‘We are unmotivated’]
29	[‘Lazy and unmotivated’]
30	[‘I want to give up’]
31	[‘lost all hope’]
32	[“We’ll lose anyway”]
33	[‘We should give up and say nothing’]
34	[‘We must win’]
35	[‘We will lead our way out of trouble’]
36	[‘We must fight for our rights’]
37	[‘We should take control and assert our position’]
38	[‘We should take control’]
39	[‘We shoud take controll’]
40	[‘Hard working individual’, ‘ Hardly working individual’]

Evaluate agency

>>> model_id = "ba0"
>>> df0[model_id] = df0.sents.progress_apply(ba0.predict)

Compute more specific indices of agency (tot = total = sum af all values for all sentences, pos = only positive, neg = only negative, abs = sum of absolute values)

>>> df0["BATot"] = df0[model_id].apply(ba0.tot)
>>> df0["BAPos"] = df0[model_id].apply(ba0.pos)
>>> df0["BANeg"] = df0[model_id].apply(ba0.neg)
>>> df0["BAAbs"] = df0[model_id].apply(ba0.abs)
>>>
>>> cols0 = [
>>>     "sents",
>>>     "ba0",
>>>     "BATot",
>>>     "BAPos",
>>>     "BANeg",
>>>     "BAAbs",
>>> ]
>>>

Check output

>>> df0[cols0].tail(n=8)

Output data (pandas dataframe with agency evaluation)

	sents	ba0	BATot	BAPos	BANeg	BAAbs
0	[‘She is a hard working individual’]	[0.48]	0.48	0.48	0.0	0.48
1	[‘She is a hardly working individual’]	[-0.62]	-0.62	0.0	0.62	0.62
2	[‘This thing was made of lead’]	[-0.04]	-0.04	0.0	0.04	0.04
3	[‘This is a car, it runs on gas’]	[0.01]	0.01	0.01	0.0	0.01
4	[‘This is a Karen, she runs for office’]	[0.33]	0.33	0.33	0.0	0.33
5	[‘This is a Jane, she runs for office’]	[0.35]	0.35	0.35	0.0	0.35
6	[‘Striving to achieve my goals’]	[0.72]	0.72	0.72	0.0	0.72
7	[‘Struggling to achieve my goals’]	[-0.6]	-0.6	0.0	0.6	0.6
8	[‘Striving to make it’]	[0.13]	0.13	0.13	0.0	0.13
9	[‘Struggling to make it’]	[-0.52]	-0.52	0.0	0.52	0.52
10	[‘Striving to survive’]	[-0.03]	-0.03	0.0	0.03	0.03
11	[‘Struggling to survive’]	[-0.41]	-0.41	0.0	0.41	0.41
12	[‘Well planned and well executed’]	[0.51]	0.51	0.51	0.0	0.51
13	[‘Coordinated activity’]	[0.5]	0.5	0.5	0.0	0.5
14	[‘Uncoordinated activity’]	[-0.61]	-0.61	0.0	0.61	0.61
15	[‘Not coordinated activity’]	[-0.72]	-0.72	0.0	0.72	0.72
16	[‘Everything is messy and uncoordinated’]	[-0.52]	-0.52	0.0	0.52	0.52
17	[‘A bad decisionmaker’]	[-0.45]	-0.45	0.0	0.45	0.45
18	[‘A marvelous decisionmaker’]	[0.45]	0.45	0.45	0.0	0.45
19	[‘They are submissive’]	[-0.68]	-0.68	0.0	0.68	0.68
20	[‘They submitted to his will’]	[-0.08]	-0.08	0.0	0.08	0.08
21	[‘They submitted a paper’]	[0.04]	0.04	0.04	0.0	0.04
22	[‘They submitted a request’]	[0.05]	0.05	0.05	0.0	0.05
23	[‘We are winners’]	[0.51]	0.51	0.51	0.0	0.51
24	[‘We are losers’]	[-0.41]	-0.41	0.0	0.41	0.41
25	[‘motivated’]	[0.72]	0.72	0.72	0.0	0.72
26	[‘We are motivated’]	[0.7]	0.7	0.7	0.0	0.7
27	[‘We are not motivated’]	[-0.8]	-0.8	0.0	0.8	0.8
28	[‘We are unmotivated’]	[-0.48]	-0.48	0.0	0.48	0.48
29	[‘Lazy and unmotivated’]	[-0.85]	-0.85	0.0	0.85	0.85
30	[‘I want to give up’]	[-0.49]	-0.49	0.0	0.49	0.49
31	[‘lost all hope’]	[-0.57]	-0.57	0.0	0.57	0.57
32	[“We’ll lose anyway”]	[-0.43]	-0.43	0.0	0.43	0.43
33	[‘We should give up and say nothing’]	[-0.41]	-0.41	0.0	0.41	0.41
34	[‘We must win’]	[0.65]	0.65	0.65	0.0	0.65
35	[‘We will lead our way out of trouble’]	[0.49]	0.49	0.49	0.0	0.49
36	[‘We must fight for our rights’]	[0.29]	0.29	0.29	0.0	0.29
37	[‘We should take control and assert our position’]	[0.71]	0.71	0.71	0.0	0.71
38	[‘We should take control’]	[0.57]	0.57	0.57	0.0	0.57
39	[‘We shoud take controll’]	[0.37]	0.37	0.37	0.0	0.37
40	[‘Hard working individual’, ‘ Hardly working individual’]	[ 0.42 -0.41]	0.0	0.21	0.21	0.42

Note

The last row demonstrates how a text that contains multiple sentences is handled, each sentence is assigned a separate agency score.