IDP Password Strength Testing

Overview

When: Dec 2019

Length: 2 weeks

Role: Primary researcher and designer

Team: 1 Product Designer (myself) and 2 Product Managers

Task: Research and design a new account creation user flow with a focus on encouraging users to create better passwords

1

Define

Vacasa had created an Identity Provider (IDP) team that was tasked with creating a backend tool to connect and manage guest and owner accounts.

Part of this process meant that new and existing users would need to make sure their passwords met certain standards. The IDP team approached me to design a new account creation flow that would encourage users to create stronger passwords without introducing too much friction.

2

Ideate

In brainstorming with the IDP team and the design team we came up with three scenarios to test:

  1. Standard password input with a list of password criteria and checking off that criteria as the user meets them.
  2. 'Strength meter' that indicates to the user the strength of their password through a bar filling and changing in color.
  3. Hybrid solution that includes both the criteria and the meter.

3

Build

In order to test these I created the designs in Sketch and then pulled the flow into FramerX. Using FramerX's React code component features I created the various password components.

The criteria component would update as the user typed and if their password met the criteria then the line would mark itself as complete.

For the strength meter, I found a npm package that checked passwords and returned a strength value. I then used that value to update a bar's color and length in the prototype.

For the purpose of the user testing I added tracking for certain actions:

  • Attempting to sign in without an account
  • Exiting the password input without a qualifying password or a password 'soft fail'
  • Attempting to advance to the next screen without a qualifying password or a password 'hard fail'
Screenshot of React code in FramerX. One can tell that the code checks password strength.
Screenshot of React code that checked password strength
We are interactive, you can go through the whole Sign Up flow and see your results!
Criteria Prototype
Iframe not loading? Make sure you're not viewing this in incognito.
Strength Meter Prototype
Iframe not loading? Make sure you're not viewing this in incognito.
Hybrid Prototype
This test included a change to the 'Sign Up' link
Iframe not loading? Make sure you're not viewing this in incognito.
4

Results

My assumption coming into these tests was that the strength meter would lead to longer and more complex passwords than the criteria and that the hybrid would lead to similarly strong passwords with less friction for the user.

After multiple rounds of testing on UserTesting.com, I found that the criteria only led to the strongest passwords and the least amount of user friction. Users entered longer and more complex passwords on the criteria prototype while experiencing less soft and hard fails. The average number of soft plus hard fails for the criteria was 1, compared to the average of 4 for the strength meter and 3 for the hybrid.

Prototype test results showing one soft fail
Screenshot of a criteria test result
5

Lookback

What I liked:

  • Data and test-driven decisions led to the implementation of a better user flow. Rather than designing on my assumptions around the strength meter, I allowed the data to lead the way and influence the final design.
  • Secondary test metrics like 'Sign In Attempts' notified me of other pain points outside of the password focus. This led to the implementation of a 'Create Account' button rather than a small 'Sign Up' link.

What I learned:

  • Creating concise and detailed scripts for user testing flows that usually require context.
  • Implementing complex code components in FramerX. Including adding an npm package for checking password strength.

What I would do differently:

  • Split the 'Create Account' button change into a different test. While it was a minor change, it could have affected other test metrics or user feedback about the whole flow.