Algorithmic Transparency in Digital Spaces
Maria Bridge [moderator; Chief Communications Officer at Center for Humane Technology] in discussion with Irene Solaiman [AI Researcher and Public Policy Manager at Hugging Face], Natalia Domagala [Head of Data and AI Ethics at UK Cabinet Office] and Renée Cummings (@CummingsRenee), Data Activist in Residence at the University of Virginia]
Panel discussion from our Responsible Tech Summit: Improving Digital Spaces held at the Consulate General of Canada in New York on May 20th. Find the full event overview here.
Below are some of the questions and answers from this panel conversation. Notes taken by Lama Mohammed
Maria Bridge: What does algorithmic transparency look like in your work, and why are you interested in this topic?
Irene Solaiman: So my background is policy. I love a good framework and looking at what parts of a data set I can be evaluated, especially at this huge scale.
Natalia Domagala: From the government’s perspective, we remember that ultimately, at the end of any decision that we make, there is a real person, and we tend to automate many of those decisions these days. So I think algorithmic transparency boils down to communicating clearly about how, why, and when we automate those decisions.
Renée Cummings: Much of my work looks at justice, and if you're talking about transparency, then we're talking about a process, the duty of care, and due diligence — these are critical to ensuring that if we are dealing with data and we're building these algorithms based on data, what we want is accuracy in our decision making because we're using these algorithms to deploy high stakes decisions and in criminal justice.
Renée Cummings: I always say that criminal justice should be the conscience of AI. As a data activist, it comes back to justice and equity and to diversity and inclusion. Transparency is about accountability, about openness, and it's also about communicating that openness and accountability to the stakeholders in particular.
Maria Bridge: What do you mean by transparency? What’s beneficial about it in the context of your work?
Irene Solaiman: For context, I do some engineering work on evaluating and being able to document these more qualitative aspects of AI systems. Some things are not inherently quantifiable, and what's left for general-purpose systems is that they don't fit into a specific use case. Transparency is not just making sure that what we're documenting is accessible to engineers and technical people, but especially the people in this room, those watching this live stream, policymakers, and advocates in civil society. Concretely, the kinds of evaluations that I would run on an AI system would potentially be putting a quantifiable metric on something important, like toxicity.
Natalie Domagala: That's one of the big things that I spent the past two years thinking about and working on. In the UK, we have very strong transparency and open data movements, and there were a few lessons that we've kind of taken from that which are if you release a lot of information about what the government does, a very niche group of people that will be interested in that. But the truth is the majority — if you just walk down the street of London or any setting in the UK and just stop a person in the street and ask, “what do you think about algorithmic transparency in the public sector?” They will just look at you and say, “what?” So what we’re trying to achieve is meaningful transparency and making sure that whatever we're doing lands well not just with this fantastic audience that we have here today, but anyone, like my grandparents for example.
Renée Cummings: What we've got to always appreciate about transparency is public trust because it's about public confidence and legitimacy in the models that are being created. So if we want to deliver that then we've got to be accountable and ensure the things that we're doing are not only explainable and audible.
Renée Cummings: We need to come up with decisions to understand what is happening when gathering individuals’ data because some of the challenges we see from digital policing and digital surveillance, as well as the deployment of digital force against individuals, is a weaponization of their data. When we think about these algorithms, we've got to think beyond the algorithm — I speak about a legacy. What an algorithm does is create a legacy, and in a split second make a decision based on an outcome. It could change the trajectory of a life.
Irene Solaiman: I have this frustration with complaining about the pipeline problem and reaching out to marginalized groups that need to have a voice at the developer table. A lot of what I work on in this field of AI safety is generally said to mean reliability. An AI system is following the developer's intent, but if any of you have spent any time in Silicon Valley, I don't look like what a lot of developers will look like. The current community is not representative of all of the people that these AI systems and algorithms are impacting.
Irene Solaiman: How do we engage with the groups that are being impacted by these algorithms, especially the systems that are being held by high resource institutions?
Renée Cummings: I would say it's even more about just engaging with those groups because one of the things that I always say to my students (because I teach big data ethics) at UVA is about stretching your imagination
Renée Cummings: It is not so much about engaging with these groups, it’s more about engaging with our thinking and looking deep into our minds and seeing what creates that bias because that is the intention we bring to the design process — we bring our thinking.
Renée Cummings: If we're honest about our thinking, ask those uncomfortable questions, and be bold enough to deal with the answers, then we bring a new kind of thinking to the design — thinking that even brings more value to the process.
Renée Cummings: One of the things that we always say is the unintentional consequences of technology, and by doing that I think we let ourselves fall very easily because we just believe that that group is now collateral damage, so there is no unintended consequence. Each consequence feels very intended for the individual.
Irene Solaiman: What are some ways we can standardize, especially developers, their perspectives, and ensure we’re clear about where we’re coming from. How can we get these values from these underrepresented groups without exploiting them?
Natalia Domagala: Deliberation is part of the answer. As someone who works in the public sector, this is something I would like to see more in the private sector. The answer lies in deliberation and collaboration, especially with civil society organizations, and this is what we've been trying to do.
Natalia Domagala: In terms of setting expectations and making sure that this is not exploitative, I think what's key is to make people understand why they should engage. Things like citizen assemblies, where people are paid for their time, I think are fantastic because that's what gives people, who normally wouldn't be able to dedicate time, an opportunity to participate. In assemblies, people are being paid and appreciated for deliberating some of those big questions about the future of tech and algorithmic transparency
Natalia Domagala: We must also engage with civil society organizations, grassroots groups, activists, and everyone who essentially understands how underrepresented groups and individuals are discriminated against. I think this goes for governments, Big Tech companies, and the private sector to generally open up more and make it easier and more valuable for people to engage and take part in those processes.
Renée Cummings: I would say it begins with respect, and not just paying lip service to those with diverse perspectives. It's also about deconstructing knowledge and understanding that our communities are producers of knowledge because I think we always think of knowledge in the high and the low. It’s also about having an appreciation for each community that produces a unique set of knowledge that we use.
Renée Cummings: When thinking about data and about designing algorithms, models, systems, solutions, or products, I think it's really valuable having a unique amount of respect for all voices, particularly those vulnerable voices, but not just in simply designing or defining them. Our communities are under-resourced or have high needs, and we have to understand and know that those voices are the most critical to some of the work that we're doing.
Maria Bridge: How do we get people that are moving fast and maybe not taking the time to pause, listen, reach out, reconsider and ask the big questions? What thoughts do you have on that very big question — about what's happening in the private sector and in particular, where there's not this respect or reflecting deep questioning?
Renée Cummings: Well I think what's happening is what we're experiencing. We're experiencing many several challenges — the undermining of democracy, disinformation, violence, and more. We've got to also appreciate that moving fast doesn't necessarily mean that you're going anywhere. I think if we bring that level of understanding to the space, we bring that level of intention and think about understanding what we need to do.
Renée Cummings: I always say that we never want to get to the point where we've got to create algorithms to teach us how to be human again. If we think about what it means to be human, what that human-centered approach to AI and algorithms means, and think about each other and doing the right thing, then we are going to get the kinds of results that we want.
Irene Solaiman: It’s really hard to regulate these kinds of things, and we're creating this huge umbrella that's supposed to encompass systems from language models to facial recognition where the evaluations barely overlap, if at all. I do think that the regulation and having these many different approaches to being thoughtful and creating enforcement mechanisms, is helpful here.
Natalia Domagala: We don’t move particularly fast in the government. Where I sit is essentially focused on the public sector within the government. You could call it self-regulation, but it's about making sure that we are getting things right before we go off. What I do is go around the public sector in the UK and ask, “how do you use AI? How do you use algorithms in your work? Do they assist decisions? Do they support decisions? How and where?” And then I say, “Let me help you make this information more transparent. Let me help you put this into our standard. Let me help you communicate that to the members of the public.” Many private sector companies have started to do that as well.
Natalia Domagala: We've all heard of model cards and the kind of transparency statements they bring. I think the movement, the beginnings of this global algorithmic transparency movement are there. Perhaps, that's mainly what Big Tech companies that have the resources and time should focus on. When it comes to governments as well — there is a very small group of governments that are advanced in this space — I think we should be very careful not to reproduce some of the mistakes we've made in the past by monopolizing this conversation.
Maria Bridge: Explain the tool of model cards.
Irene Solaiman: So Mitchell et. al. is a more popular form of documentation. We're also very lucky that Dr. Mitchell is at Hugging Face. We gather the different components of a model we need to document and share with the world, but different companies will take different approaches to it. It gives you a more succinct overview of where the model is being trained. Whereas when did the data stop being scraped? Different sections just make it kind of like a checklist of what you want to share with folks and it's becoming more of a form of documentation.
Irene Solaiman: Model cards can help us understand what is out of scope? What are the use cases? What should we not use this model for? What are some of the risks and ethical considerations here? None of this is being enforced because there's no standardizing body and there's no enforcement mechanism.
Maria Bridge: What demands should we be placing on private sector Black Box algorithm-making machines?
Renée Cummings: What we want to see is regulation and we also want to hear the conversation, the voices, and the discussions about what needs to be done and regulated. So I think, from my perspective, we need to understand that what we do is fair and non-discriminatory, and then we remove and reduce the high levels of bias and discrimination that we're seeing, so it’s not just the regulation doing the work. We want to see regulation in action in an active space where we could understand that this is happening and being protected. We need to bring in not only that risk-based perspective but also bring that rights-based perspective.
Irene Solaiman: I would also be conscious about where this regulation is coming from. My heritage is from a developing country, so it is important to recognize the Brussels effect and how this regulation impacts the rest of the world, since this regulatory perspective is very privileged, and privileged people are the ones able to sit at the developer table. But having an active engagement with the more remote parts of the world that are being impacted by these kinds of systems and are developing their approach to regulation is important as well.
Natalie Domagala: When it comes to bringing the world together, the public sector, and the global algorithmic transparency movement, I think now is the right time to do that because this is in the early days. All of us here in this room need to do more to make this conversation more inclusive.
Maria Bridge: What are some trends in the space that you're excited about, or maybe even concerned about based on the things that we might be talking about in the future?
Irene Solaiman: I'm impressed with how much more community there is. On the research side, we have learned the norms and are being more conscious of communicating with each other about what we're evaluating, how we release things, and just generally sharing our lessons.
Irene Solaiman: I am looking forward to the ML Conference. There's a lot more attention and workshops on the ethics side and how we're building data sets - there's more acceptance of these kinds of papers that aren’t used to being called since they are not technical enough and have more sociology or an interdisciplinary lens that people are recognizing now.
Renée Cummings: This space and what's happening here today is a trend that makes me happy because we're seeing more of these kinds of communities coming together, and we're having more conversations around responsible technology, responsible AI, and ethics.
Renée Cummings: Just thinking about ethics in the AI and data science space in this way is critical because each of us represents so many communities, and if we're having these conversations, it means we're taking the knowledge back to those places that need to have it. I think there's a very vibrant movement of AI and data science, and people who are interested in doing things differently and responsibly are what excites me.
Natalie Domagala: I’ll be a little more epistemic here, but one trend I am a little bit worried about is transparency washing — it's essential when companies, and possibly some public sectors, pretend that they are being transparent, but there is no accountability there. I think one crucial point here is that we can't have meaningful transparency without good or functioning accountability mechanisms. I hope that through all this community work, conversation, and international knowledge-sharing, we can eliminate the warring trends.
Maria Bridge: Is there something you see from your perspective that others may not see from your work that you want others to see?
Irene Solaiman: “The Process for Adopting Language Models to Society” — that was a spotlight at one of the bigger ML Conferences in December. Not as a criticism of my work because I love what I do, but I don't think that that kind of work would have been in the spotlight 10 years ago, or even accepted. I think this is a concrete example of seeing more folks, especially on the technical side, really start to recognize and integrate these more social science perspectives and think about who we are impacting. I just really hope we don't get to the point of we're washing it so much that we have to hurt people to recognize that it's a problem.
Natalie Domagala: Around the world, there are a lot of well-meaning public servants who understand the need for greater transparency. But there are people in the space who want to make a difference, who want to bring it back to greater transparency and engage the public — that gives me a lot of hope.
Renée Cummings: My students. They are all very excited about our big data, ethics, and AI course. The many requests I get from high school students are to have this conversation with them around AI, algorithms, and ethics — bringing that justice-oriented and trauma-informed approach to the work we do gives me great hope because I know the future is definitely in good hands.
Bios for the speakers
Irene Solaiman is an AI safety expert. She is conducting novel AI research and leading public policy at Hugging Face and advising responsible AI initiatives at OECD and IEEE. Her research includes AI alignment, algorithmic fairness, and combating misuse and malicious use. Her recent speeches include guest lectures at Harvard University and technical talks at large AI labs.
She formerly built AI policy at Zillow Group. Irene also led public policy at OpenAI, where she initiated and led bias and social impact research. Notably, her research on adapting GPT-3 behavior received a spotlight at NeurIPS 2021. She previously advised policymakers on responsible autonomous decision-making and privacy as a fellow at Harvard’s Berkman Klein Center.
Natalia Domagala leads on data and AI ethics policy at the Central Digital and Data Office, Cabinet Office in the UK. She previously advised on open government and open data policies for the Department for Digital, Culture, Media and Sport in the UK, and implemented open data challenges for 360Giving. She has research experience in anthropology, gender, civic tech, and economic development, and she has recently co-edited a book: Situating Open Data: Global Trends in Local Contexts. Natalia is a Policy Fellow at the Centre for Science and Policy, University of Cambridge. She received her MSc in Local Economic Development from the London School of Economics and Political Science and her BA in Anthropology and Media from Goldsmiths, University of London.
Renée Cummings is an AI ethicist, Data Activist in Residence at the University of Virginia, and a co-lead for UVA's involvement in the Public Interest Technology University Network
Renee specializes in ethical AI, AI leadership, diversity, equity and inclusion in AI, algorithmic accountability, data integrity and sovereignty, algorithmic justice, AI policy and AI governance. Using AI and VR to reengineer and reimagine how we think about offending, police accountability, police use of force, police and community relations, public safety, sentencing and public health and AI and VR inspired approaches to homicide reduction, gun and gang violence prevention are all areas of specialization.
Maria Bridge is the Chief Communications Officer at Center for Humane Technology, a non-profit driving a comprehensive shift toward humane technology. In her role, she works to reframe the insidious effects of persuasive technology, expose the runaway systems beneath, and deepen the capacity of decision-makers to take wise action. She began her career in management consulting at Bain & Company and held operating roles at Warby Parker and sweetgreen. She received her MBA from Stanford Graduate School of Business. Maria was inspired to switch careers into ethical tech after understanding the malleability of the mind, as a result of extended insight meditation practice.