Key Questions Guide

Wherever you are in the process of looking at or fighting benefits tech, you will need to gather information to advocate effectively. This is a general roadmap for collecting that information. You do not need to know everything before you start advocating.

The questions below will help you start piecing together why and how benefits tech is being used and how it is impacting people. They can also guide you in advanced fact-finding efforts. You may not find or need answers to all these questions—start with what seems most relevant and know that you will continue learning as you go. You can use these questions to shape your advocacy goals and, ultimately, to help develop a vision for more just administration of benefits.

Note on terminology: In this document, we say “benefits tech” to refer to any software that is being used to administer benefits or make determinations about how much support someone receives, if any. These systems may be used to carry out basic administrative or logistical functions (like application processing or automating renewals). They can also be used to make decisions about eligibility or allocation of benefits by standardizing subjective measures, like care needs. Benefits tech that makes measurement decisions is often referred to as an automated decision system (ADS). ADSs are used to support decision-making processes, usually in conjunction with some human decision-making, like determining whether someone is qualified for benefits, how much support they get, or whether someone meets income or asset rules. For more information on the issues that arise with both logistical and measurement functions in benefits tech, check out the Making Sense of Technology Problems Framework.

The role of technology in benefits may not always be obvious. These questions help indicate that a benefits tech issue might be at play.

• Has a client lost benefits when nothing else has changed about their circumstances?
• Does a written notice mention some sort of system, algorithm, or logic?
• Have state employees said anything about a new system or new computer?

The questions below address the basic context and purpose for which the system is being used or considered.

• What benefits program(s) is it being used for?
• What’s the name of the system?
• Who developed it?
• What stage(s) of the application or renewal process is it being used in?
• Who is it being used on (everyone applying for or in the program, or a subset of people)?
• What decision(s) or process(es) is it being used for? (e.g., to decide eligibility, generate notices, determine the amount of benefits someone gets, detect fraud, or renew benefits)?
• Note: In many cases, one system will be used for multiple functions, stages, or decisions, including both logistical and measurement functions.

This is the most important question, and you will continue to learn about the impacts the more you research.

• What impacts have you observed that you know or suspect are due to the benefits tech (e.g., benefit cuts, termination or suspension of benefits, or application processing delays)?
• Who has been affected by these cuts? (Is it affecting everyone on the program negatively or only a portion of them? Can you quantify the negative impact? What do the affected people have in common? Are you seeing more cuts for people with certain conditions, or who live in certain areas? Do cuts seem to be tied to race, age, family status, or other factors?)
• Do the cuts suggest that some kind of discrimination is happening?
• What suggests that the benefits tech is causing these harms instead of something else? What role do you think the benefits tech is playing in these harms?

Governments operate benefits systems in an environment of scarcity, where budgets, agency administrative concerns, and client interests are often at odds. Finding out the government’s reasoning gives you a claim to try to confirm or disprove, which is a necessary step in any advocacy.

• What are the government’s stated goals of implementing the benefits tech (e.g., eliminating bias, promoting fairness, or increasing efficiency)?
• What do you think might be some of the unstated goals or desired impacts (e.g., benefit cuts, increased paperwork, false fraud accusations)?
• Are there federal requirements or incentives driving the adoption of the tech?

In some cases, the adoption of a new system is driven by an individual agency employee or team focused on modernization. It may also be driven by specific members of the state legislature focused on program cuts, enforcement, or modernization.

• Who are the actors driving the use of the system (e.g., a state agency, a vendor, policy organizations, or the state legislature)?
• Why are these actors interested in implementing or using the system?
• What are the power dynamics of the context in which the decision is being made? Are certain kinds of interests more influential? Do certain interests exercise more leverage, and in what form? Are certain actors within an agency more influential than others?
• Who is excluded from these decisions? It might be people getting the benefits, people with particular professions or types of expertise (especially the lived experience of people getting benefits through the program), or agency employees whose input and concerns are disregarded because of their age, race, gender identity, sexual orientation, disability, religion, etc.

Evaluating the scope of the agency’s thought process may show its true aims, inform advocacy goals or strategies, and point to helpful documents.

• How is the new system different from the way the state made decisions before?
• Do other government entities make similar decisions without using benefits tech?
• Did the government solicit bids from multiple vendors offering the benefits tech system? Did they only consider a particular vendor for the contract? Was it created in-house?

Technology projects have a lifecycle, spanning from the initial decision to create a new system, to that system operating in the real world. The phase of a system’s lifecycle affects what kind of information is available, how you can access the information, and what strategies you can use to advocate. See our Lifecycle Framework for more details on each of the following phases.

• Devising: During this phase, government officials are considering using technology to solve a problem with benefits administration or to implement policy changes. The government agency will start to define system requirements and other needs.
• Contracting: During this phase, the agency will put out a request for proposals (RFP) and start the bidding process, in which technology vendors submit proposals to build the project. They may also adapt an existing contract, depending on state vendor requirements.
• Building: During this phase, the vendor(s) builds the technology. This process can include technical testing, input from agency staff, or piloting the system to assess its functionality and impacts.
• Operating: During this phase, the agency uses the technology to make decisions about people’s benefits. This phase can also include revisions to the system, which potentially lead to a new contracting phase.

The system could be using information from a benefits application, an assessment, and/or a range of government or private databases. How the system obtains the information, and how it uses the information once obtained, is also important. See our case study on Missouri’s Medicaid LTSS eligibility algorithm for an example: the state publicized the logic of their proposed new algorithm and advocates used that to test and challenge it.

• Where does the system get the information it uses to make a decision (e.g., an application for benefits, a specialized assessment, or some database the government agency uses to check property records or bank accounts)?
• How does the system get information from other systems? (E.g., does the system reference eligibility for other programs to make new determinations?) Issues can arise when the connections between databases do not work properly.
• What information actually factors into each of the system’s decisions? (This information can be called a “variable,” a “factor,” or an “input.”)
• How does the system use the information it has to make a decision? (Note: This question can involve multiple variables working at the same time in complex ways. Are you able to get a copy of the logic that the system uses (it might be called an algorithm, rubric, or matrix) to process the input and produce an output or decision?)
• What information isn’t being used that should be?
• Is the system using information that feels inappropriate or absurd? Could the information being used have a discriminatory effect based on race, gender, disability, a specific medical condition, geography, family status, etc.? For example, Michigan used software to check SNAP recipients’ names against a law enforcement database of felony warrants and automatically cut off people’s food assistance. The use of law enforcement data has a disproportionately harmful effect on Black people, as well as Latinx and Indigenous people, who are targeted by police.

These systems often involve complex processes with several interdependent parts. A small error in code can produce glitches that result in improper decisions, incorrect notices being sent, or other errors.

• Has the state done any testing of the system’s performance? If so, what data and processes did it use?
• Does the system’s performance match the specifications from the RFP, vendor contract, or other documents?
• Did the state pilot the system before rolling it out to all beneficiaries? If so, are there reports, data, or results from that pilot?
• What kind of training and quality control does the state have for people using the system?
• Has the state established any benchmarks for the system’s performance? If so, how is the system performing against these benchmarks?

Governments often use these systems to replace human discretion in decisions about whether someone is getting the care they need or committing fraud. But the way the system actually decides may not make intuitive sense and could be based on bad data or faulty assumptions.

• What is the basis for believing that the system’s decision is an accurate reflection of whatever is being determined (e.g., care needs or fraud)?
• Is the system being used to make decisions? If so, is there a scoring rubric or other logic being used to inform the decision? Where did that logic come from (e.g., laws, a vendor, a tool used in another state or context, or the agency itself)?
• How is the state justifying the decisions made using the benefits tech?
• Does the developer of the tool have any validation studies that show it’s measuring what it claims to measure?
• Has the government tested the instruments or assessment tools? If so, what data and processes did it use?
• Has the government done any analysis of the impact of the system on people’s access to benefits?
• Have any of the validation studies, pilots, or audits looked at potential discrimination in the system’s measurement?

Usually, the state had some way of making the same decision before it started using this benefits tech system. Or, other government entities are making the same decisions without using automation.

• What might a good, non-tech decision-making system look like?
• Who is going to inform the vision for an alternative to this system in your state?
• How can you ensure meaningful participation of people and groups otherwise left out? (I.e., making opportunities for input “real”; rejecting mere promise of “access” or “opportunities.”)

There may not be an easy answer to “what would an alternative system look like” or “what would work better.” However, these questions should help you learn more about what kind of benefits tech is being used, its purpose, and how it works. Your learning process may involve asking these questions many times, but our goal is that answering them helps you limit the harms of benefits tech.