The effects of automated prompts on the responses and self-efficacy of human communicators

AI-Mediated Communication (AI-MC) is a pervasive form of assistive AI that is changing how millions of people converse online every day. AI-MC enables “interpersonal communication that is not simply transmitted by technology, but modified, augmented, or even generated by a computational agent to achieve communication goals” (Hancock et al., 2020, p. 90; Jakesch et al., 2019). Google's Smart Reply (SR) is a prime example of an AI-MC system, offering short reply suggestions that are generated by machines to its 1.5 billion Gmail users (Bullock, 2017; Petrova & Elias, 2019). We study how AI assistance influences the recipient and the replies they ultimately write in the presence of AI-generated prompts. We experimentally study the causal link between the presence of AI-generated prompts and three phenomena: recipient knowledge, response complexity, and perception of self-efficacy. In the experiment, English-speakers based in the United States recruited through an online pool (Prolific) will be asked to reply to five messages sent from a friend or work colleague asking for their opinion regarding a common dilemma. Participants will be randomly assigned to one of three primary groups: those who receive brief (1-4 words) suggested responses, those who receive longer (full-sentence) suggested responses, and those assigned to a control condition in which no suggested responses are provided. To examine the effect of nudging (Kozyreva et al., 2020; Lorenz-Spreen et al., 2020), a portion of the participants will be asked if they would like to be aided by suggested responses; Those who answer yes will be randomly assigned to either the short or long suggested responses condition, while those who elect not to be aided, will be assigned to a condition similar to the control group in which no suggested responses are provided. Those who are assigned directly to a suggested response condition, and those who elect to, will be able to select a smart reply (or not) for their response and then add or edit their response before submitting their response. To measure between group differences, the following measures will be collected: (1) Reply word count, excluding words from response suggestions (should they be used by participants), and (2) response complexity: measured by the LIWC Analytic library, excluding the words from suggested responses (should they be used by participants). Word count and analytic measurement will be used to assess response length and complexity. Prompt knowledge will be assessed using (3) response duration to each email prompt, and (4) response to knowledge items (4 multiselect questions), that will follow each email reply about the content of the mail. For the measurement of self-efficacy, we will use (5) a social self-efficacy scale (Smith & Betz, 2000); and (6) a writing self-efficacy ideation scale (Bruning et al., 2013). A short form personality inventory will also be administered for the potential moderating effect of certain personality constructs for future exploratory purposes.