Opportunities and Risks of Conversational AI for Credit Unions: Empathy and Intimacy in Automated Financial Customer Service

by Scott Mainwaring, UCI and Melissa Wrapp, UCI, Filene's Center for Emerging Technology

As the use of digital channels continues to grow for credit unions, conversational artificial intelligence (AI) technologies provide an opportunity for improved service delivery and the potential for new service offerings such as financial advice.

EXECUTIVE SUMMARY

Conversational AI technologies create new ways for credit unions to serve their members, from providing alternatives to interacting with human agents to creating new channels for more tailored financial services. They provide opportunities to build upon the trust and appreciation members place in credit unions as more human-centered, nonpredatory, and community based. But conversational AI technologies risk invading members’ privacy and being frustrating and opaque.

WHAT IS THE RESEARCH ABOUT?

This exploratory study looks at existing consumer relationships with conversational AI and digital assistants, on one hand; and with credit unions, banks, and other businesses, on the other, to begin to sketch the dimensions of, and provide examples of, points within a “design space” of possible financial digital assistants. While operational hurdles remain high for credit unions to deploy these new technologies, the opportunity will continue to grow in coming years. 

Through ethnographic research with consumers, this report anticipates how credit union members might come to value, or reject, digital assistants. For this exploratory study, we focused on one main question: What are the implications of digital assistant technologies for how members and credit unions could relate to one another in the next five years?

Interviews covered three broad topics: experiences using banks and credit unions; experiences using digital assistant technologies; and reflections on the idea of a financial digital assistant and issues of privacy, trust, and potential bias. This report summarizes findings on these themes and provides insight into how credit unions could take advantage of digital assistants to improve service delivery and differentiate offerings by incorporating elements from their mission and value proposition into their digital assistants. The way forward is to develop particular product proposals and related data transparency policies that can provide members with a new understanding of what they could achieve by relating with their credit unions through “talking computers.”

WHAT ARE THE CREDIT UNION IMPLICATIONS? 

Credit unions have an opportunity to deploy digital assistants in ways that improve service delivery and member experience and provide new types of service offerings. In thinking about what types of digital assistants would provide the best fit for your credit union and member needs, keep the following research findings in mind: 

• People like the promise of bots as part of a modern, organized, and simplified life.
• The realities of existing bots fall short of expectations and can limit imagination.
• People are resigned to the constant advance of technology without transparency or the ability to meaningfully opt out.
• Relations with credit unions are valued for their human element and trustworthiness, even if this means older, clunkier tech.
• The design space is complex, including diverse combinations of technologies, member needs, and business opportunities worth considering.
• The idea of talking with/through bots is becoming mundane, but credit unions could pleasantly surprise members with unique service features.
• Credit unions could tailor these technologies to show their strengths and to educate members not just about finances but also about data. 

In order to create a competitive advantage, credit union digital assistants would have to not only be useful and usable but also embody and express the core values of the credit union system. By building upon these core values of empathy and respect, credit unions could focus their development of digital assistant technologies in a way that creates differentiation, even with fewer resources than are available to larger financial services providers. 

We use findings from our research to generate design ideas that are meant to illustrate pathways worth exploring, developing, and evaluating: 

• Build a helpful, always-accessible agent. This kind of digital assistant could serve as the voice of the specific credit union and provide basic support but also demonstrate the “members not customers” ethos of the credit union value proposition.
• Provide an assistant to help members maintain, augment, and monitor their personal financial support systems.
• Provide robot counsel. This financial digital assistant could serve as a “second pair of eyes” as members conduct transactions with any financial services provider, intervening if necessary but always being available for reassurance or advice.
• Connect members to each other. This assistant would embody the credit union as a member cooperative, helping connect members to each other.

Access complete report, summary slides, and design principles here.

A Survey of Fair and Responsible Machine Learning and Artificial Intelligence: Implications for Consumer Financial Services

by Stephen C. Rea, PhD, Research Assistant Professor at Colorado School of Mines and former IMTFI Research Assistant

Capital One Eno Chatbot

Stephen Rea recently published a white paper surveying literature in computer science, law, and the social sciences on developments in machine learning and artificial intelligence, with special focus on their implications for consumer financial services in the United States. This project grew out of a joint collaboration between IMTFI and Capital One's Responsible AI Program. We present here an excerpt from the introduction to the white paper, followed by some updates about recent developments in this space. 

Excerpt
Machine learning (ML) algorithms and the artificial intelligence (AI) systems that they enable are powerful technologies that have inspired a lot of excitement, especially within large business and governmental organizations. In an era when increasingly concentrated computing power enables the creation, collection, and storage of “big data,” ML algorithms have the capacity to identify non-intuitive correlations in massive datasets, and as such can theoretically be more efficient and effective than humans at using those correlations to make accurate predictions. What is more, AI systems powered by ML algorithms represent a means of removing human prejudices from decision-making processes; since an AI system renders its decisions based solely on the data available, it can avoid the conscious and unconscious biases that often influence human decision-makers.

Contrary to this rosy picture of ML and AI, though, decades of evidence demonstrate how these technologies are not as objective and unbiased as many perhaps wish they were. Biases can be encoded in the datasets on which ML algorithms are trained, arising from poor sampling strategies, incomplete or erroneous information, and the social inequalities that exist in the actual world. And since ML algorithms and AI systems cannot build themselves, the humans who construct them may, however unintentionally, introduce their own biases when deciding on a model’s goals, selecting features, identifying which attributes are relevant, and developing classifiers. Additionally, the inherent complexities of ML algorithms that defy explanation even for the most expert practitioners can make it difficult, if not impossible, to identify the root causes of unfair decisions. That same opacity also presents an obstacle for individuals who believe that they have been evaluated unfairly, want to challenge a decision, or try to determine who should—or even ​could​—be held accountable for mistakes.

Compared to other fields, the financial services industry has taken a relatively conservative approach to ML/AI integrations. Consumer-facing applications like robo-advisors for portfolio management, AI-powered banking assistants, algorithmic trading programs, and proactive marketing tools, as well as harnessing the power of ML to do sentiment analysis of social media feeds and news stories in search of trendlines, have garnered a lot of media attention. However, the visibility of initiatives like these in press releases and news items exaggerates their role in financial services today, as they represent less than one-tenth of the funding received in the financial technology, or “fintech,” vendor space. Thus far, financial institutions have primarily invested in ML and AI for automating routine, back-office tasks, improving fraud detection and cybersecurity, and making regulatory compliance easier. 

The current state of ML and AI in consumer financial services, then, is one in which there is still enormous opportunity for innovation, but also reasons to be cautious. To paraphrase the feminist geographer Doreen Massey, some individuals and groups are more on the “receiving end” of these technologies than others. In other words, ML and AI’s advantages and disadvantages are not equally distributed. Nor are the vulnerabilities entailed by digital surveillance techniques for data creation and collection, the sorts of harm that can occur from an erroneous data entry and the burden for correcting it, or the ability to affect how an algorithm interprets one’s individual attributes and characteristics. In many ways, ML/AI research’s most important contributions have been demonstrating the extent to which structural inequalities—that is, conditions by which one or more groups of people are afforded unequal status and/or opportunities in comparison to other groups—persist by providing quantifiable, documented evidence of social disparities. If an organization’s reason for integrating ML- and AI-powered systems is to improve its decision-making procedures so as to make them both more accurate and fairer, then it is imperative to understand and account for persistent inequalities in the social contexts where those systems are embedded. Furthermore, assessing how exactly an algorithmic and/or automated decision-making system could impact specific populations, the risk that it could violate legal standards prohibiting discrimination, and the extent to which the system could perpetuate structural inequalities are of the utmost importance when deciding whether or not to make the integration in the first place.

You can read the rest of the white paper on SSRN.

Updates

Work in ML and AI is fast-moving, and in the time since this paper was published, there have been a number of developments that will affect how these technologies are integrated with the consumer financial services industry and beyond. Two in particular merit attention here:

1) Congressional action: On February 12, 2020, the U.S. House Committee on Financial Services' Task Force on Artificial Intelligence heard testimony from experts on AI, ML, and race and inclusion in a panel titled “Equitable Algorithms: Examining Ways to Reduce AI Bias in Financial Services.” The Committee acknowledged the usefulness of standards for the fairness and accuracy of AI applications in financial services, while also noting that existing laws such as the Equal Credit Opportunity Act, the Fair Housing Act, and the Fair Credit Reporting Act are inadequate in many respects for regulating AI's impact. The panel of experts recommended drafting a definition of "fairness" that could be used for evaluating ML, developing audit and assessment methods for locating biases in data and models, and requiring ML/AI developers to implement and report upon continuous monitoring plans that can detect new biases as they emerge. They also voiced concern regarding the Department of Housing and Urban Development's plans to revise the Fair Housing Act's disparate impact standards, and how such action might exacerbate the discriminatory effects of AI in home lending. 

2) Sandvig v. Barr decision: In March 2020, the U.S. District Court for the District of Columbia delivered its ruling in Sandvig v. Barr, which challenged a provision in the Computer Fraud and Abuse Act (CFAA) that made it a crime for researchers and journalists to use "dummy" accounts for the purposes of auditing algorithms in order to identify possible discrimination. The American Civil Liberties Union had initially brought the lawsuit in 2016 on behalf of a group of academics and journalists led by Christian Sandvig of University of Michigan's School of Information. The plaintiffs argued that the CFAA violated their First Amendment rights, and noted that comparable research activities were not illegal in offline contexts. The Court ruled in favor of the plaintiffs, thereby opening the door for more independent review of ML/AI applications and scoring an important victory for researchers' ability to hold algorithms and the institutions that use them accountable.

Additional Resources

AI Now Institute: https://ainowinstitute.org
Data and Society's AI on the Ground initiative: https://datasociety.net/research/ai-on-the-ground/

Four Reasons to Keep Your Money at Home

The following is by Katherine Martineau, IMTFI Fellow and Ph.D. Candidate in Anthropology at the University of Michigan. Reach her at kbmartin@umich.edu. The research on which this post is based was conducted with Pradeep Baisakh and Nishita Trisal. Photos by Nishita Trisal, except where stated.

Purno's house is his bank.

Purno keeps his money at home. A low-caste man in rural eastern India, his family is connected to different aid and low-income finance programs. Purno has even taken an 8000 rupee business loan from the non-profit bank located in a nearby town. But when his household saves, Purno does not take the money to the bank. Instead, he tucks it into a metal box that he hides in the thatching of his roof.

The metal box in which he keeps money.

 Why do poor people like Purno continue to save their money at home? Why do they take loans from private moneylenders when there are Self-Help Groups, Grameen-style microfinance institutions, and low-rate bank loans designed for their demographic? These questions demand answers that are culturally and historically specific.

Our research in one urban and one rural low-income neighborhood was conducted through long-form open-ended conversations with ten households over six weeks. Most of our participants worked as laborers or as domestic servants and a few were small-scale entrepreneurs. These households self-identified as Below-Poverty-Line households. The households claimed monthly gross incomes between 3000 and 6000 rupees -- all made less than 25 USD per capita per month.

Below are four culturally and historically specific reasons that have emerged from our research to explain why Purno and other poor people in Odisha might choose to keep their money at home.

Banks cannot predict droughts.

Reason 1: Hardship is coming
Management of unpredictable hardship is essential to the livelihoods of our research participants. It is a temporal category, a phase that comes and goes. But it is seen as something that is likely to happen to everyone. It raises issues of liquidity, but it is not something that banks can always accommodate. We heard numerous stories of savings lost in the face of hardship -- flooding, drought, illness, and crop failures among the worst. When bad things happen, such as a terrible illness, access to money can mean life and death. The poor timing of hardship motivated many loans from private moneylenders in our study, and the expectation of hardship was repeatedly cited by our research participants as a reason to save money in their houses.

Responsibility is not a major concern in most cases, thus the occasion of hardship allows for requests for help. When Seema’s daughter contracted a high malarial fever, she borrowed the necessary amount from a neighbor who is also part of her caste group. That family had similarly received help from Seema’s family during a long illness. Hardship and its threat creates obligations and material interdependencies.

Reason 2: Liquidity is friendly

When in hardship, ask for help; when others are in
hardship, expect requests for help.
Photo by Pradeep Baisakh. 

Small amounts of money are constantly circulating among neighbors, friends, co-workers, and kin. This was especially true in our rural site. In the urban site it was more often confined to kin-groups and led to conflict more often. The basic principle was consistent: when in hardship, one can request help; when someone else is in hardship, one should give it. This means that knowledge about who has what circulates. There are strong moral feelings about the obligation to return assistance.

There is also moral ambivalence about removing money from social circulation as occurs publically when saving through banks. Though many of our research participants had used banks, most did not feel comfortable actually going to the bank to deposit money because everyone would know what they were doing. One participant explained that it would make people think that he thought he was rich. Of course this also becomes a risky strategy in the face of one’s own potential hardship: others are less likely to help out if they suspect you are not helping them like you could be.

Sometimes it pays to keep things from loved ones.

Reason 3: Nobody need know -- not even your husband
One’s own family members can be even more troublesome than other households when it comes to money management.

Sita takes care of the children’s expenses and that money is kept separately from the other household expenses managed by her husband Ram. A normal case of earmarking you might think. However, hidden money can foster household drama. For example one of our urban households shared by two brothers’ families had literally been split into two. Intense conflict arose from a dispute over a house loan. The brothers had resorted to building a wall separating their living spaces and cooking hearths.

Rashmi’s husband was a daily laborer who had worked his way up to headman. He drank away most of his income every night, claiming that this was necessary for job networking and that without it he wouldn’t get good jobs. He also gave money to his lover whom Rashmi believed he was supporting, along with her son. When Rashmi’s husband did come home, he’d beat her, leaving bruises that were visible during our interviews. Rashmi used to work as a domestic servant and had, back then, hidden some money from her husband and sons (who were also going out to booze at night). But at the time of our interviews she had been sick for months so the money she had hidden had run out. She faced an uncertain future. Deep shame prevented her from seeking help from others.

How do you know you can trust financial services?
Photo by Pradeep Baisakh.

Reason 4: Financial services will cheat you 
All of our research participants had at one point taken part in Self-Help Groups and many had bank accounts and formal bank loans. But the abundance of services had made things confusing; stories circulated about cheating and they were reinforced by the irregular appearance of itinerant financial services representatives.

Manoranjan is a father of two sons and a daughter of a marriageable age. He works as a laborer and together with his wife, a domestic servant, they save a little bit from everything they earn. They save it in a metal box in their house, which they keep locked within a locked cabinet and hidden behind some fabric. They do not have a specific idea of what they will do with their savings, but there will likely be high costs associated with their daughter’s marriage. They also hope to add a room to their house. They are ideal clients for financial services but several years ago Manoranjan had taken a “microfinance loan” only to discover that it demanded a very high interest rate. Since then, he has not trusted his money with financial services. Stories such as Manoranjan’s suggest that regulation and systems with local oversight would improve trustworthiness. However, government-led oversight and conflict resolution would face the same problems that private financial services seek to overcome -- the slowness of government and judicial action, corruption (e.g., demanding bribes from complainants), and the reproduction of entrenched caste/community inequalities in the structure of local institutions.

These four reasons for saving money at home shed light on some of the conditions affecting financial inclusion programs in Odisha, India