CDS design research handbook

Français

← Table of contents

Performing distributed research

What is distributed research?

Distributed or remote research is the act of performing research with participants in different physical locations in order to gather insights, usually with the help of digital or telecommunication technologies.

Various research methods

Keep in mind that by using these methods online (except by phone), you are most likely to sample those who are already online and have a degree of digital literacy.

Generative

  • Interviews (phone interviews, video conferencing)
  • Diary studies. Only valid if the product/service is not significantly impacted by changes in daily habits e.g. a service or activity that normally takes place at home, over time.
  • Participatory design (collaborative methods using mural or jamboard)
  • Surveys
  • Literature review
  • Artifact analysis

Evaluative

  • Usability tests (moderated/unmoderated)
  • Concept evaluations (moderated/unmoderated)
  • Card sorting (moderated/unmoderated)
  • Surveys (unmoderated)
  • Web Analytics. Keep in mind that web analytics are mainly useful for understanding people who are already using your product. If your research question is about people who aren’t using your product yet, this won’t be as helpful.

Tools and software

Video-conferencing platforms available to most GC departments

(All have voice, webcam screen sharing and voice recording capabilities)

  • Webex / for moderated research, good with Protected A information and the only tool accessible to all GC departments.
    • However, Webex is hard to use, clunky and not as intuitive as other newer platforms.
  • Zoom / for moderated research. Robust, but not accessible to all GC departments. We have a premium account. It’s only available one meeting at a time. It is not Protected A friendly, so it is encouraged not to use sensitive information on it. Chat with them to request its availability.
    • A free account will support meetings up to 40 mins.
    • Zoom seems to work with most screen readers, according to participants.

GC-friendly testing platforms

  • Qualtrics or Hosted in Canada Surveys, two surveys tools available for quantitative research.
  • User Testing / unmoderated, talk to your manager to request an account if needed.
    • Provide tasks and instructions for participants. It also records the participants’ screen and voice as they “think out loud” and use your site/platform.
  • Optimal Workshop / unmoderated, for card sorting, surveys, currently used by Digital Transformation Office in TBS. Chat with Antoine who used it in the past.

Testing prototypes

  • Invision. Chat with someone on the design team to learn more or chat with Eman to request your own account.
    • InVision’s LiveShare feature allows multiple parties to view and manipulate the same prototypes while discussing via voice.
  • Figma. Works the same as InVision, but is also a mockup/wireframing tool.
  • PDFs. They are simple enough to send over to your participants if they aren’t too big (single pages). However, some people won’t have the software to open a PDF, so JPEG or PNG might be better.
  • Coded prototypes.

Prepare and anticipate

As a reminder, prepare and anticipate that the technology setup (internet connection, screen sharing, audio, etc.) will go wrong at some point during a session. The best way to prevent and have strategies is to test in advance through pilot tests.

  • Recruit one extra participant and schedule a pilot test with them a few days before the “official” tests for example. Practice screen recording and audio.

  • Try your tool in various browsers and see what the interface looks like in each. You’ll have to guide people through their computer interface by guessing at what they’re seeing.

  • Write clear step-by-step instructions for the participant on how to access whatever tool is being used for the test. If they need to download something, let them know in advance.

  • Build in at least 10 minutes extra to each session for setup and instructions.

  • Have a backup plan (e.g. a way to speak on the phone if you’re having connectivity issues)

  • You can send the consent form and privacy statement in advance by email and ask them to send back a signature or an “I consent” statement. Make sure to document this process by keeping track of how the participant was discovered, the date and time, by which researcher, for which activity, etc.
  • Request a verbal consent at the beginning of the test if they forgot to sign the document. Again, make sure to keep track of the details of that research as in the first point.
  • You can decide to design an accessible consent and privacy statement PDF for participants with visual impairments. This means you send them an accessible consent form and ask them to confirm their consent via email, in advance of the research session. You must also review the consent verbally at the time of the research session and ask if they have any questions or concerns.

What about compensation?

  • We are now able to compensate participants! In order to learn more about the process for researchers, refer to this 5-step compensation process.
  • If it is not possible to compensate participants, know that many people are eager to participate in research without receiving compensation, because it is an opportunity for them to help improve government services for themselves and others. You don’t need to mention explicitly that you are not compensating, unless asked. As always, make sure to communicate the benefits of participating by explaining the goal of the project and how their input will be used.
  • Look for contracting platforms and third-party firms that offer incentives to participants as part of their operating model.
    • Researchers will need to work closely with policy advisors to determine which vendors we can work with, on what timelines.
    • Look to your policy team members for signals as to the viability of new ways of working: the TBS teams we work with (contracting, legal) may be more or less able to accommodate new research approaches depending on how the situation unfolds

What if we are conducting discovery research?

If you are currently starting a discovery with your team you might wonder how to perform primary research with remote participants.

  • Start with secondary research, like a literature review,or learning sessions with other teams who have conducted research in the problem space.

  • If you have clarity on the problem to be solved, now is a good time to start gathering baseline data: statistics on how the problem impacts the population you’re serving, departmental metrics related to the problem etc.

    • Gather all of the relevant statistics in one document for future reference. Even just understanding what is and is not currently measured will be helpful for later.
    • Good people to work with on this: PM, service owner, policy analyst
    • Pro-tip: Lucas has access to the Public Use Microdata File (PUMF) from statcan that has lots of useful census data in it

  • Go for snowball recruitment and work with community organizations to send out recruitment calls to potential participants by email to set up remote sessions.

  • Social media recruitment calls may also be a good option depending on your criteria

    • Make sure to check with Outreach first

What if they live in different timezones?

  • Depending on your research participants’ timezone, you might need to adjust your work schedule as you perform research. Make sure to check with other team members who will be present before scheduling sessions outside of normal working hours, and communicate with your manager if you decide to go ahead. Be mindful of your own time though.

  • Assess what times of the day you will be able to talk to them/test and enable asynchronous communication by scheduling emails to be sent at a specific time or using an unmoderated research method.

Great resources

What is needed to do this type of work distributed research? Blogs with tips and tools, phone interviews, community orgs (group phone sessions (Hillary), etc.

- Last updated by Anne-Marie on December 2, 2020