How might City departments create print and digital voter guides that clarify the vote-by-mail process and ballot, in partnership with immigrant-centered community groups?

IMPORTANT DISCLAIMER: The below project summary highlights my personal views of, and experience during, this project. This portfolio page is not an official City of Philadelphia communication document, nor does it represent the views of the CAO Service Design Studio.

All images and samples were carefully chosen to comply with our consent statements about protecting participant privacy.

PROJECT BACKGROUND

As the country moved towards a general election in the midst of a pandemic, cities and states were preparing for an unprecedented number of mail-in ballot voters. There were nationwide concerns about capacity and high number of spoiled votes, as well as how the more complicated mail-in processes might affect voters with low English literacy.

Since the City of Philadelphia had limited purview to change the ballot itself, some departments focused on designing supplemental materials to support Philadelphia’s voters. The Office of Immigrant Affairs (OIA) contacted the Service Design Studio to collaborate on a voter guide that residents could use to understand the mail-in ballot process. The guide would be initially produced in plain-language English, then translated into 12 languages. Immigrant-centered community groups (among others) would distribute these guides to registered voters to help them prepare to vote by mail in the general election.

The director of the Service Design Studio, Liana Dragoman, acted as the project lead for the overarching project. I led and managed the usability study work.

PREPARATION

DESIGNING the STUDY

To ensure the guide effectively supported residents who may not be comfortable reading English, we refined our demographics of focus to be residents who were more comfortable reading in a language other than English. We chose to conduct sessions with residents who were more comfortable reading Chinese, French, Spanish, and Vietnamese.

When designing this study, we had to work through several crucial components:

1. We chose to conduct in-person testing. This was our hardest decision during the usability study design. In the midst of a global pandemic, we knew this would require logistical hurdles that could be easily avoided by remote testing. However, we recognized that remote testing would automatically leave out the participants who may be most likely to use the voter guide—participants with low digital literacy who may not be able to confidently research answers to their questions. We chose to move forward with in-person testing, mandating a critical emphasis on safety measures.

2. We would use community partners as moderators. Since participants would likely be uncomfortable communicating in English, we needed to conduct sessions in their respective primary languages. If we used Service Design Studio moderators alongside translators, the sessions would have needed to be approximately two hours long in order to complete our study tasks. We therefore chose to empower our community partners to act as moderators, knowing this would require more pre-work and training.

Drafting materials

Like with any usability study, we needed to draft our study materials. However, since we would be relying on community partners to conduct moderation, we needed to build support into our materials whenever possible. In drafting the protocol, this looked like including periodic “moderator instructions,” which included information like how to probe on errors.

During this stage, I was the primary author of the following:

• Project summary for community partners: Explained the purpose of the project, general approach, and rationale for in-person sessions.

• Community partner agreement: Provided a breakdown of tasks community partners would be responsible for throughout the project.

• Informed consent: Worked with the City’s legal team to draft a comprehensive and accurate informed consent.

• Community partners recruiting packets: Included introductory recruiting communication, screening script, a translated informed consent, and a scheduling tracker.

• Protocol: Included a project overview, a COVID-oriented remote session contingency plan, recruiting materials, and the moderator’s guide (introduction, three activities, and a conclusion).

Training community moderators

We were fortunate to work with a dedicated group of community partners who cared deeply about helping their communities confidently vote during the upcoming election. In addition to recruiting and scheduling all of the participants, our community partners attended a 1.5 hour session to learn the fundamentals of moderating. I designed and conducted this training.

During this stage, I was the primary author of the following:

• Protocol dry run video example: Provided a video demonstration of the complete study design, speaking to potential challenges moderators might face.

• Moderation training: Conducted a 1.5-hour training that introduced our prospective moderators to usability studies, moderation best practices, example video clips of effective moderation, explanations of how to troubleshoot issues, and project next steps.

ASSEMBLING THE PACKETS AND KITS

We compiled kits for our community partners with a specific focus on COVID-19 safety precautions. We provided PPE and hand sanitizer for the moderators and participants. We included cleaning supplies to disinfect communal surfaces between participants.

We assembled packets for each participant that included everything they’d need for their entire session. This way, the moderator and participants wouldn’t have to exchange materials throughout the session. These packets included: A translated informed consent, writing utensils, Likert scales, a ballot with fake candidate names (Gritty would be a great senator!), and, of course, the mail-in ballot voter guide.

Once our kits were assembled, we drove around the city to distribute the materials.

INTRA-SESSION SUPPORT

We attended three of the community organizations pilot sessions to answer questions as they arose. We both helped moderators confidently respond to participant questions and checked in with moderators to ensure they were using the guide as we expected. This prevented us from receiving largely incomplete or inaccurate data at the end of the project.

SYNTHESIS

In order to reach our ideal number of participants, we extended the timeline for conducting sessions. This enabled us to conduct sessions with 39 participants, but shortened our synthesis timeline to a mere three days. I therefore entered all of the study data over a period of a day and a half, then worked with a colleague to synthesize findings over the next day and a half.

During this stage, I was the primary author of the following:

• Data sheet: Housed key parts of the interview guide and formulaic counts of key findings.

• Usability study summary: Communicated high-level study design and key findings to project stakeholders.

• Post-project partner survey: Assessed whether we worked with our community partners in accordance with our studio ethics.

PROJECT OUTCOME

Conducting the usability testing showed us where participants experienced disconnects in the overall mail-in ballot voting process. By quantifying challenges experienced during the sessions, we highlighted key areas that should be reinforced in both the guide and additional supplementary materials (e.g., social media campaigns). Examples included:

• Using the secrecy envelope: A high number of participants struggled to understand that they needed to put their ballot inside two envelopes—first the secrecy envelope and then the mailing envelope.

• Filling out the voter declaration: The majority of participants experienced some type of error when filling out the voter declaration. Some signed in the wrong spot, others didn’t sign, still others filled out information one line off from the directed.

In addition to these broader findings, testing revealed minor updates that the team could make to improve the usability of the guide. One example of this was including the English term next to translated key terms, such as the names of the different positions on the ballot.

You can find links to the final voter guides on the City’s website.

Project lead: Liana Dragoman (Service Design Studio, Director).

Project partner: Orlando Almonte (Office of Immigrant Affairs, Language Access Program Manager).