"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (2024)

"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (2)

Advanced Search

chi

research-article

Open Access

  • Authors:
  • Hana Habib Carnegie Mellon University, Pittsburgh, PA, USA

    Carnegie Mellon University, Pittsburgh, PA, USA

    View Profile

    ,
  • Sarah Pearman Carnegie Mellon University, Pittsburgh, PA, USA

    Carnegie Mellon University, Pittsburgh, PA, USA

    View Profile

    ,
  • Jiamin Wang Carnegie Mellon University, Pittsburgh, PA, USA

    Carnegie Mellon University, Pittsburgh, PA, USA

    View Profile

    ,
  • Yixin Zou University of Michigan, Ann Arbor, MI, USA

    University of Michigan, Ann Arbor, MI, USA

    View Profile

    ,
  • Alessandro Acquisti Carnegie Mellon University, Pittsburgh, PA, USA

    Carnegie Mellon University, Pittsburgh, PA, USA

    View Profile

    ,
  • Lorrie Faith Cranor Carnegie Mellon University, Pittsburgh, PA, USA

    Carnegie Mellon University, Pittsburgh, PA, USA

    View Profile

    ,
  • Norman Sadeh Carnegie Mellon University, Pittsburgh, PA, USA

    Carnegie Mellon University, Pittsburgh, PA, USA

    View Profile

    ,
  • Florian Schaub University of Michigan, Ann Arbor, MI, USA

    University of Michigan, Ann Arbor, MI, USA

    View Profile

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsApril 2020Pages 1–12https://doi.org/10.1145/3313831.3376511

Published:23 April 2020Publication History"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (3)

  • 50citation
  • 2,664
  • Downloads

Metrics

Total Citations50Total Downloads2,664

Last 12 Months714

Last 6 weeks116

  • PDF

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices

Pages 1–12

PreviousChapterNextChapter

"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (4)

ABSTRACT

We conducted an in-lab user study with 24 participants to explore the usefulness and usability of privacy choices offered by websites. Participants were asked to find and use choices related to email marketing, targeted advertising, or data deletion on a set of nine websites that differed in terms of where and how these choices were presented. They struggled with several aspects of the interaction, such as selecting the correct page from a site's navigation menu and understanding what information to include in written opt-out requests. Participants found mechanisms located in account settings pages easier to use than options contained in privacy policies, but many still consulted help pages or sent email to request assistance. Our findings indicate that, despite their prevalence, privacy choices like those examined in this study are difficult for consumers to exercise in practice. We provide design and policy recommendations for making these website opt-out and deletion choices more useful and usable for consumers.

Skip Supplemental Material Section

Supplemental Material

a384-habib-presentation.mp4

mp4

46.8 MB

Play streamDownload

Available for Download

zip

pn5776aux.zip (5.5 MB)

Session_Script.pdf: Script used to conduct each participant session, including questions asked prior to and after completion of the study tasks

References

  1. Alessandro Acquisti, Idris Adjerid, Rebecca Balebako, Laura Brandimarte, Lorrie Faith Cranor, Saranga Komanduri, Pedro Giovanni Leon, Norman Sadeh, Florian Schaub, Manya Sleeper, and others. 2017. Nudges for Privacy and Security: Understanding and Assisting Users' Choices Online. ACM Computing Surveys (CSUR) 50, 3 (2017), 44.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (5)Digital Library
  2. Terence S Andre, H Rex Hartson, Steven M Belz, and Faith A McCreary. 2001. The User Action Framework: A Reliable Foundation for Usability Engineering Support Tools. International Journal of Human-Computer Studies 54, 1 (2001), 107--136.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (7)Digital Library
  3. BackgroundChecks.org. 2019. JustDelete.me. (2019). https://justdeleteme.xyz.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (9)
  4. California State Legislature Website. 2018. SB-1121 California Consumer Privacy Act of 2018. (2018). https://leginfo.legislature.ca.gov/faces/ billTextClient.xhtml?bill_id=201720180SB1121.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (10)
  5. Martin Degeling, Christine Utz, Christopher Lentzsch, Henry Hosseini, Florian Schaub, and Thorsten Holz. 2019. We Value Your Privacy... Now Take Some Cookies: Measuring the GDPR's Impact on Web Privacy. In Proceedings of Network and Distributed System Security Symposium (NDSS).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (11)Cross Ref
  6. Digital Advertising Alliance. 2009. Self-Regulatory Principles for Online Behavioral Advertising. (July 2009). http://digitaladvertisingalliance.org/principles.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (13)
  7. Pardis Emami-Naeini, Henry Dixon, Yuvraj Agarwal, and Lorrie Faith Cranor. 2019. Exploring How Privacy and Security Factor Into IoT Device Purchase Behavior. In Proceedings of the Conference on Human Factors in Computing Systems (CHI).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (14)Digital Library
  8. European Commission. 2018a. Article 29 Data Protection Working Party. Guidelines on Transparency under regulation 2016/679. (2018). http://europa.eu/rapid/press-release_SPEECH-11--461_en.htm.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (16)
  9. European Commission. 2018b. EU Data Protection Rules. (2018). https://ec.europa.eu/commission/priorities/ justice-and-fundamental-rights/data-protection/ 2018-reform-eu-data-protection-rules_en.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (17)
  10. Benjamin Fabian, Tatiana Ermakova, and Tino Lentz. 2017. Large-Scale Readability Analysis of Privacy Policies. In Proceedings of the International Conference on Web Intelligence (WI). 18--25.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (18)Digital Library
  11. Federal Trade Commission. 2009. CAN-SPAM Act: A Compliance Guide for Business. (2009). https://www.ftc.gov/tips-advice/business-center/guidance/can-spam-act-compliance-guide-business.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (20)
  12. Federal Trade Commission. 2017. Children's Online Privacy Protection Rule: A Six-Step Compliance Plan for Your Business. (2017). https://www.ftc.gov/tips-advice/business-center/guidance/childrens-online-privacy-protection-rule-six-step-compliance.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (21)
  13. Stacia Garlach and Daniel Suthers. 2018. 'I'm supposed to see that?' AdChoices Usability in the Mobile Environment. In Proceedings of the Hawaii International Conference on System Sciences (HICSS).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (22)Cross Ref
  14. Global Privacy Enforcement Network. 2017. GPEN Sweep 2017: User Controls over Personal information. (2017). https://www.privacyenforcement.net/system/files/2017% 20GPEN%20Sweep%20-%20International%20Report_0.pdf.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (24)
  15. Colin M Gray, Yubo Kou, Bryan Battles, Joseph Hoggatt, and Austin L Toombs. 2018. The Dark (Patterns) Side of UX Design. In Proceedings of the Conference on Human Factors in Computing Systems (CHI).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (25)Digital Library
  16. Hana Habib, Yixin Zou, Aditi Jannu, Neha Sridhar, Chelse Swoopes, Alessandro Acquisti, Lorrie Faith Cranor, Norman Sadeh, and Florian Schaub. 2019. An Empirical Analysis of Data Deletion and Opt-Out Choices on 150 Websites. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (27)
  17. Jovanni Hernandez, Akshay Jagadeesh, and Jonathan Mayer. 2011. Tracking the Trackers: The AdChoices Icon. (2011). http://cyberlaw.stanford.edu/blog/2011/08/tracking-trackers-adchoices-icon.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (28)
  18. IAB Europe. 2011. EU Framework for Online Behavioural Advertising. (2011). https://www.edaa.eu/wp-content/uploads/2012/10/ 2013--11--11-IAB-Europe-OBA-Framework_.pdf.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (29)
  19. IAB Europe. 2019. GDPR Transparency and Consent Framework. (2019). https://iabtechlab.com/standards/gdpr-transparency-and-consent-framework/.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (30)
  20. Saranga Komanduri, Richard Shay, Greg Norcie, and Blase Ur. 2011. AdChoices? Compliance with Online Behavioral Advertising Notice and Choice Requirements. A Journal of Law and Policy for the Information Society 7 (2011).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (31)
  21. Pedro Giovanni Leon, Justin Cranshaw, Lorrie Faith Cranor, Jim Graves, Manoj Hastak, Blase Ur, and Guzi Xu. 2012. What Do Online Behavioral Advertising Privacy Disclosures Communicate to Users?. In Proceedings of the Workshop on Privacy in the Electronic Society (WPES).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (32)Digital Library
  22. Thomas Linden, Hamza Harkous, and Kassem Fawaz. 2018. The Privacy Policy Landscape After the GDPR. arXiv:1809.08396 (2018).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (34)
  23. Mary Madden and Lee Rainie. 2015. Americans' Attitudes About Privacy, Security and Surveillance. (2015).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (35)
  24. Arunesh Mathur, Jessica Vitak, Arvind Narayanan, and Marshini Chetty. 2018. Characterizing the use of browser-based blocking extensions to prevent online tracking. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (36)
  25. Jonathan R Mayer and John C Mitchell. 2012. Third-Party Web Tracking: Policy and Technology. In Proceedings of the IEEE Symposium on Security and Privacy (S&P).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (37)Digital Library
  26. Aleecia M McDonald and Lorrie Faith Cranor. 2010. Americans' Attitudes About Internet Behavioral Advertising Practices. In Proceedings of the Workshop on Privacy in the Electronic Society (WPES).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (39)Digital Library
  27. William Melicher, Mahmood Sharif, Joshua Tan, Lujo Bauer, Mihai Christodorescu, and Pedro Giovanni Leon. 2016. (Do Not) Track Me Sometimes: Users' Contextual Preferences for Web Tracking. Proceedings on Privacy Enhancing Technologies 2016, 2 (2016), 135--154.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (41)Cross Ref
  28. Michael Morgan, Daniel Gottlieb, Matthew Cin, Jonathan Ende, Amy Pimentel, and Li Wang. 2018. California Enacts a Groundbreaking New Privacy Law. (2018). https://www.mwe.com/en/thought-leadership/publications/2018/06/california-enacts-groundbreaking-new-privacy-law.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (43)
  29. Ambar Murillo, Andreas Kramm, Sebastian Schnorf, and Alexander De Luca. 2018. "If I press delete, it's gone" - User Understanding of Online Data Deletion and Expiration. Proceedings of the Symposium on Usable Privacy and Security (SOUPS) (2018).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (44)
  30. Network Advertising Initiative. 2018. NAI Code of Conduct. (2018). https://www.networkadvertising.org/sites/default/files/nai_code2018.pdf.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (45)
  31. Nielsen Norman Group. 2018. Top 10 Design Mistakes in the Unsubscribe Experience. (2018). https://www.nngroup.com/articles/unsubscribe-mistakes/.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (46)
  32. Donald A. Norman. 1986. Cognitive Engineering. In User Centered System Design: New Perspectives on Human-Computer Interaction. Lawrence Erlbaum Associates, 31--61.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (47)
  33. Donald A. Norman. 1990. The Design of Everyday Things. Doubleday.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (48)Digital Library
  34. Norwegian Consumer Council. 2018. Deceived by Design: How Tech Companies Use Dark Patterns to Discourage Us from Exercising Our Rights to Privacy. (2018). https://fil.forbrukerradet.no/wp-content/uploads/2018/06/2018-06--27-deceived-by-design-final.pdf.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (50)
  35. Online Trust Alliance. 2018. Email Marketing & Unsubscribe Audit. (2018). https://www.internetsociety.org/resources/ota/2018/2018-email-marketing-unsubscribe-audit/.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (51)
  36. Enric Pujol, Oliver Hohlfeld, and Anja Feldmann. 2015. Annoyed Users: Ads and Ad-Block Usage in the Wild. In Proceedings of the Internet Measurement Conference.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (52)Digital Library
  37. Iskander Sanchez-Rola, Matteo Dell'Amico, Platon Kotzias, Davide Balzarotti, Leyla Bilge, Pierre-Antoine Vervier, and Igor Santos. 2019. Can I Opt Out Yet?: GDPR and the Global Illusion of Cookie Control. In Proceedings of the ACM Asia Conference on Computer and Communications Security.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (54)Digital Library
  38. Florian Schaub, Aditya Marella, Pranshu Kalvani, Blase Ur, Chao Pan, Emily Forney, and Lorrie Faith Cranor. 2016. Watching Them Watching Me: Browser Extensions' Impact on User Privacy Awareness and Concern. In Proceedings of NDSS Workshop on Usable Security (USEC).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (56)Cross Ref
  39. Fatemeh Shirazi and Melanie Volkamer. 2014. What Deters Jane from Preventing Identification and Tracking on the Web?. In Proceedings of the Workshop on Privacy in the Electronic Society (WPES).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (58)Digital Library
  40. United States Congress. 1999. S.900 Gramm-Leach-Bliley Act. (1999). https://www.congress.gov/bill/106th-congress/senate-bill/00900.Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (60)
  41. Blase Ur, Pedro Giovanni Leon, Lorrie Faith Cranor, Richard Shay, and Yang Wang. 2012. Smart, Useful, Scary, Creepy: Perceptions of Online Behavioral Advertising. In Proceedings of the Symposium on Usable Privacy and Security (SOUPS).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (61)Digital Library
  42. Christine Utz, Martin Degeling, Sascha Fahl, Florian Schaub, and Thorsten Holz. 2019. (Un)informed Consent: Studying GDPR Consent Notices in the Field. In Proceedings of Conference on Computer and Communications Security (CCS).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (63)Digital Library
  43. Ari Ezra Waldman. 2019. There is No Privacy Paradox: How Cognitive Biases and Design Dark Patterns Affect Online Disclosure. Current Opinion in Psychology (2019).Google Scholar"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (65)

Cited By

View all

"It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (66)

    Index Terms

    1. "It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices
      1. Human-centered computing

        1. Human computer interaction (HCI)

          1. Empirical studies in HCI

        2. Security and privacy

          1. Human and societal aspects of security and privacy

            1. Usability in security and privacy

          2. Social and professional topics

            1. Computing / technology policy

              1. Privacy policies

          Recommendations

          • Why Johnny can't opt out: a usability evaluation of tools to limit online behavioral advertising

            CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

            We present results of a 45-participant laboratory study investigating the usability of nine tools to limit online behavioral advertising (OBA). We interviewed participants about OBA and recorded their behavior and attitudes as they configured and used a ...

            Read More

          • What do online behavioral advertising privacy disclosures communicate to users?

            WPES '12: Proceedings of the 2012 ACM workshop on Privacy in the electronic society

            Online Behavioral Advertising (OBA), the practice of tailoring ads based on an individual's online activities, has led to privacy concerns. In an attempt to mitigate these privacy concerns, the online advertising industry has proposed the use of OBA ...

            Read More

          • An empirical analysis of data deletion and opt-out choices on 150 websites

            SOUPS'19: Proceedings of the Fifteenth USENIX Conference on Usable Privacy and Security

            Many websites offer visitors privacy controls and opt-out choices, either to comply with legal requirements or to address consumer privacy concerns. The way these control mechanisms are implemented can significantly affect individuals' choices and their ...

            Read More

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          Get this Publication

          • Information
          • Contributors
          • Published in

            "It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (67)

            CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

            April 2020

            10688 pages

            ISBN:9781450367080

            DOI:10.1145/3313831

            • General Chairs:
            • Regina Bernhaupt

              Eindhoven University of Technology, Netherlands

              ,
            • Florian 'Floyd' Mueller

              Monash University, Australia

              ,
            • David Verweij

              Newcastle University, UK

              ,
            • Josh Andres

              RMIT, Australia

              ,
            • Program Chairs:
            • Joanna McGrenere

              University of British Columbia, Canada

              ,
            • Andy co*ckburn

              University of Canterbury, New Zealand

              ,
            • Ignacio Avellino

              University of Maryland Baltimore County, USA

              ,
            • Alix Goguey

              Grenoble Alpes University, France

              ,
            • Pernille Bjørn

              University of Copenhagen, Denmark

              ,
            • Shengdong (Shen) Zhao

              National University of Singapore, Singapore

              ,
            • Briane Paul Samson

              Future University Hakodate, Japan & De La Salle University, Philippines

              ,
            • Rafal Kocielnik

              University of Washington, USA

            Copyright © 2020 Owner/Author

            This work is licensed under a Creative Commons Attribution International 4.0 License.

            Sponsors

              In-Cooperation

                Publisher

                Association for Computing Machinery

                New York, NY, United States

                Publication History

                • Published: 23 April 2020

                Check for updates

                "It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (80)

                Author Tags

                • data deletion
                • email marketing
                • privacy
                • privacy controls
                • targeted advertising
                • usability

                Qualifiers

                • research-article

                Conference

                Acceptance Rates

                Overall Acceptance Rate6,199of26,314submissions,24%

                Upcoming Conference

                CHI '24

                • Sponsor:
                • sigchi

                CHI Conference on Human Factors in Computing Systems

                May 11 - 16, 2024

                Honolulu , HI , USA

                Funding Sources

                • "It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (81)

                  Other Metrics

                  View Article Metrics

                • Bibliometrics
                • Citations50
                • Article Metrics

                  • 50

                    Total Citations

                    View Citations
                  • 2,664

                    Total Downloads

                  • Downloads (Last 12 months)714
                  • Downloads (Last 6 weeks)116

                  Other Metrics

                  View Author Metrics

                • Cited By

                  View all

                  PDF Format

                  View or Download as a PDF file.

                  PDF

                  eReader

                  View online with eReader.

                  eReader

                  Digital Edition

                  View this article in digital edition.

                  View Digital Edition

                  HTML Format

                  View this article in HTML Format .

                  View HTML Format

                  • Figures
                  • Other

                    Close Figure Viewer

                    Browse AllReturn

                    Caption

                    View Table of Contents

                    Export Citations

                      Your Search Results Download Request

                      We are preparing your search results for download ...

                      We will inform you here when the file is ready.

                      Download now!

                      Your Search Results Download Request

                      Your file of search results citations is now ready.

                      Download now!

                      Your Search Results Download Request

                      Your search export query has expired. Please try again.

                      "It's a scavenger hunt": Usability of Websites' Opt-Out and Data Deletion Choices (2024)
                      Top Articles
                      Latest Posts
                      Article information

                      Author: Carlyn Walter

                      Last Updated:

                      Views: 6625

                      Rating: 5 / 5 (50 voted)

                      Reviews: 89% of readers found this page helpful

                      Author information

                      Name: Carlyn Walter

                      Birthday: 1996-01-03

                      Address: Suite 452 40815 Denyse Extensions, Sengermouth, OR 42374

                      Phone: +8501809515404

                      Job: Manufacturing Technician

                      Hobby: Table tennis, Archery, Vacation, Metal detecting, Yo-yoing, Crocheting, Creative writing

                      Introduction: My name is Carlyn Walter, I am a lively, glamorous, healthy, clean, powerful, calm, combative person who loves writing and wants to share my knowledge and understanding with you.