Internet search analytics for shoulder arthroplasty: what questions are patients asking?

Article information

Clin Shoulder Elb. 2023;26(1):55-63
Publication date (electronic) : 2023 February 23
doi : https://doi.org/10.5397/cise.2022.01347
Department of Orthopedic Surgery, Rush University Medical Center, Chicago, IL, USA
Correspondence to: Johnathon R. McCormick Department of Orthopedic Surgery, Rush University Medical Center, 1620 W Harrison St, Chicago, IL 60612, USA Tel: +1-312-563-6306, Fax: +1-312-942-2040, Email: Johnathon_mccormick@rush.edu
Received 2022 November 3; Revised 2022 December 12; Accepted 2023 January 2.

Abstract

Background

Common questions about shoulder arthroplasty (SA) searched online by patients and the quality of this content are unknown. The purpose of this study is to uncover questions SA patients search online and determine types and quality of webpages encountered.

Methods

The “People also ask” section of Google Search was queried to return 900 questions and associated webpages for general, anatomic, and reverse SA. Questions and webpages were categorized using the Rothwell classification of questions and assessed for quality using the Journal of the American Medical Association (JAMA) benchmark criteria.

Results

According to Rothwell classification, the composition of questions was fact (54.0%), value (24.7%), and policy (21.3%). The most common webpage categories were medical practice (24.6%), academic (23.2%), and medical information sites (14.4%). Journal articles represented 8.9% of results. The average JAMA score for all webpages was 1.69. Journals had the highest average JAMA score (3.91), while medical practice sites had the lowest (0.89). The most common question was, “How long does it take to recover from shoulder replacement?”

Conclusions

The most common questions SA patients ask online involve specific postoperative activities and the timeline of recovery. Most information is from low-quality, non-peer-reviewed websites, highlighting the need for improvement in online resources. By understanding the questions patients are asking online, surgeons can tailor preoperative education to common patient concerns and improve postoperative outcomes.

Level of evidence

IV.

INTRODUCTION

Prior to surgery, orthopedic surgeons routinely counsel their patients on the risks, benefits, expectations, and rehabilitation associated with any proposed procedure. This understanding is necessary for informed consent on the part of patients, and studies have shown that appropriately aligned patient expectations are positively correlated with higher postoperative satisfaction [1-5]. Thorough preoperative education has been found to reduce anxiety, postoperative pain, and length of hospital stays for patients [3,6]. Furthermore, numerous studies have shown that preoperative expectations are predictive of postoperative outcomes [5-11]. These findings highlight the importance of establishing appropriate expectations in patients, which includes addressing patients’ questions and concerns regarding all aspects of surgery. Despite the advantages of in-depth patient education prior to surgery, providers may not be able to answer all patient questions due to time constraints around office visits. Furthermore, patients may seek answers to their questions elsewhere if and when additional questions come to them outside of their appointments, or if they do not feel comfortable asking their physicians in the moment.

Orthopedic patients seek answers to their surgical questions from many sources other than doctors. Today, most orthopedic patients use the internet to ask medical questions [12]. A 2013 Pew Research report finds that 72% of internet users and 59% of all US adults had looked online for health information in the preceding year, while 60% of respondents had obtained information from friends and family. The same report finds that 77% of all online health inquiries started on a search engine [13]. Queries using Google Search have become one of the most ubiquitous ways people seek answers to their questions in the internet age. Google has developed an incredibly successful search algorithm to answer users’ inquiries and commonly related questions. Google Search results feature a tool called “People also ask,” which uses machine learning to aid searchers by suggesting related questions based on data gathered from other internet users and natural language processing. These suggested questions and answers are frequently engaged by users and are highlighted near the top of the results page [14]. Each suggested question is followed by a brief snippet of text attempting to briefly answer the question, along with a hyperlink to the webpage from which the text originated. Unfortunately, information promoted through the Google Search algorithm is not verified by experts in pertinent fields as accurate or of high academic standards. Therefore, the most common answers to internet searches that patients receive—including with regard to shoulder arthroplasty—are of unknown quality and accuracy. Despite the vastness of its data, the internet has been shown, across numerous studies, to be a source of potentially poor quality information in matters of health [15-19]. Specifically, assessment of the quality of information provided online about common otolaryngology procedures revealed that webpages returned by Google Searches contain only 50%–65% of critical information that patients must know prior to undergoing surgery, and that nearly 3% of webpages contain false or inaccurate information [20,21]. These studies highlight the need for surgeons to identify and answer any questions that patients may have while reducing the risk of patients gathering false information online. At the same time, the research demonstrates the need for surgeons and health care institutions to publish high quality online resources for patients to easily access instead of relying on incomplete and possibly inaccurate information from non-surgical third parties promoted through the Google Search algorithm.

The purpose of this study is to investigate what questions shoulder arthroplasty patients search online and to determine the types and quality of webpages provided to patients from the top results of each query. By understanding what questions patients are asking online regarding shoulder arthroplasty, surgeons can improve communication in preoperative education and prepare patients for their experiences and outcomes prior to shoulder arthroplasty. These steps will, in turn, increase patient satisfaction and improve clinical outcomes.

METHODS

This study was conducted using publicly available, de-identified data to protect individual privacy, and as such, did not require Institutional Review Board approval or informed consent. Therefore, no additional ethical clearance was necessary for this research.

The methods of our study were adapted for shoulder arthroplasty from the work of Shen et al. [22]. A total of nine search strings were conducted under three categories of total shoulder arthroplasty using Google Search. The queries were chosen to capture general total shoulder arthroplasty questions, as well as those pertaining to anatomic and reverse total shoulder arthroplasty. For the general group, the search terms were as follows: “shoulder replacement,” “total shoulder replacement,” and “total shoulder arthroplasty.” For anatomic total shoulder arthroplasty, the search strings were “anatomic shoulder replacement,” “anatomic total shoulder replacement,” and “anatomic total shoulder arthroplasty.” For reverse total shoulder arthroplasty, the terms were “reverse shoulder replacement,” “reverse total shoulder replacement,” and “reverse total shoulder arthroplasty.”

Search queries were performed on internet browser Google Chrome, version 90.0.4430.212. Searches were performed independently on May 13, 2021, by authors (MCK and KDC), with the search location set to the United States. To avoid the bias of personalized search results influenced by prior search history, searches were run on a newly installed application with no prior queries. The following steps were performed according to instructions found on support.google.com to ensure no personalized search bias was present [23]. Browsing history was confirmed to be cleared, and any previously installed Google Chrome application was uninstalled from the hard drive. The hard drive was subsequently searched for any remaining files containing Google Chrome data, which were deleted if encountered. The Google Chrome application was then reinstalled using another browser.

For each search query, the “People also ask” tab was expanded until approximately 130 suggested searches appeared on the page. Each suggested question was paired with a single hyperlink to a webpage. The suggested questions and webpage hyperlinks were collected using automated Google Chrome extension Scraper, version 1.7. Questions that were clearly unrelated to the topic of shoulder arthroplasty were removed from the dataset. The final dataset was ultimately limited to the first 100 relevant “People also ask” questions per search string.

Based on previous studies in internet health information, each question was categorized using the Rothwell classification into one of three themes—fact, policy, or value [22,24,25]. The questions were also subcategorized based on content into one of the following categories: specific activities, timeline of recovery, restrictions, technical details, cost, indications/management, risks/complications, pain, longevity, and evaluation of surgery [22]. Descriptions and examples of classifications for each question can be found in Table 1.

Rothwell classifications, subcategorizations, and examples

Each website hyperlink was visited, and the website information source was categorized as academic, commercial, government, journal, lawyer/law-related, medical information site, medical practice, non-medical media site, or single-surgeon personal. Descriptions and examples of classifications for each website can be found in Table 2.

Website categorizations and examples

Each website was scored on a four-point scale for quality of source according to the established Journal of the American Medical Association (JAMA) benchmark criteria, which include authorship, attribution, currency, and disclosure [15,17,19,22,26,27]. Requirements to receive points for each criterion can be found in Table 3. Question classification, website classification, and JAMA benchmark scores were compiled independently by two authors (MCK and KDC) following consensus for categorical definitions. Discrepancies were reviewed by a third author (JRM) as a tiebreaker to decide final categorization.

JAMA benchmark criteria

Cohen’s kappa coefficient was used to evaluate interobserver reliability of question classification and website classification. Pearson’s chi-square tests and Student t-tests were used to evaluate the results for significance. Statistical significance was set to P-values <0.05.

RESULTS

Nine hundred questions (comprising 300 questions for each of the three categories of total shoulder arthroplasty—general, anatomic, and reverse) were generated with nine hundred associated webpages. Cohen’s kappa coefficient for interrater reliability was found to be 0.94 for question categorization and 0.98 for website categorization.

Most questions were categorized as fact questions by the Rothwell classification (54.0%), followed by value (24.7%) and policy (21.3%) questions (Table 4, Fig. 1). Fact questions were most commonly about specific activities (31.3%), followed by timeline of recovery (24.7%), technical details (23.3%), restrictions (16.5%), and costs (4.3%) of shoulder arthroplasty. The majority of value questions were categorized as questions about pain (62.2%), followed by evaluation of surgery (34.7%) and longevity (3.2%). Policy questions were most commonly categorized as indications/management (77.1%), with the remainder being questions about risks/complications (22.9%) (Table 4, Fig. 2). The most popular suggested question was, “How long does it take to recover from shoulder replacement?” This question, or a variation on the wording, was the top suggested question in five of nine search queries and was in the top three questions of all nine strings of queries.

Question results breakdown by classification and subclassification

Fig. 1.

Rothwell classification for general, anatomic, and reverse shoulder arthroplasty. Questions for each of the three search groups as categorized by Rothwell classification: fact, policy, and value.

Fig. 2.

Subcategorization of questions for general, anatomic, and reverse shoulder arthroplasty. Breakdown of questions into subcategorizations based on topical content.

Of the 900 total websites analyzed in this study, most were categorized as medical practice (24.6%), followed by academic (23.2%), medical information (14.4%), commercial (9.3%), journal (8.9%), single-surgeon personal (8.0%), government (6.2%), non-medical media (4.9%), and lawyer/law-related (0.4%) sites (Table 5, Fig. 3).

Website results breakdown by classification

Fig. 3.

Breakdown of websites classified based on host webpages.

The mean JAMA score for all 900 webpages was 1.69. The websites with the highest mean JAMA scores were journal websites (mean, 3.91) in comparison to all other websites (P<0.001). The websites with the lowest mean JAMA scores were medical practice websites (mean, 0.89) in comparison to all other websites (P<0.001). Means for the remaining categories were as follows: academic=1.36, commercial=1.32, government=1.91, lawyer/law-related=1.25, medical information sites=2.65, non-medical media sites=1.93, and single-surgeon practice=1.07.

There was a significant relationship between search category (general, anatomic, or reverse) and Rothwell question classification (fact, policy, or value). Pertinent values were χ2=27.80, df=4, and P<0.001. Most questions categorized herein as policy questions by the Rothwell classification (n=192) were retrieved from searches in reverse total shoulder arthroplasty (n=94, 49.0%), in comparison to general (n=52, 27.1%) or anatomic (n=46, 24.0%) searches. There was also a significant relationship between search category and question subclassification with pertinent values of χ2=40.83, df=18, and P=0.002. Questions categorized as pertaining to indications/management (n=148) were more frequently related to reverse searches (n=74, 50%) than general (n=42, 28.4%) or anatomic (n=32, 21.6%) searches (Table 4, Fig. 2).

DISCUSSION

The results of this study highlight the questions most commonly asked online by patients undergoing shoulder arthroplasty, free from the biases present in traditional clinical surveys. The key findings of our study are (1) the most commonly asked question is, “How long does it take to recover from shoulder surgery?”, (2) by Rothwell classification, the most common questions are those classified as fact (54.0%), (3) the most common subcategories of questions are related to specific activities (16.9%), followed by questions about pain (15.3%), (4) the majority of answers to commonly asked questions are found on webpages hosted by medical practices (24.6%), academic institutions (23.2%), and medical information sites (14.4%), and (5) overall, the majority of webpages score poorly according to JAMA benchmark criteria for source and quality (1.69), with journals scoring the highest (3.91).

Knowing what questions shoulder arthroplasty patients are seeking answers to online is an important aspect of shoulder surgeons’ practices, allowing for improved counseling and education prior to surgery. Most questions asked by patients on Google were fact-based questions (54.0%), with the majority of these relating to specific activities and timeline of recovery. Overall, the most popular suggested question was, “How long does it take to recover from shoulder replacement?” This question, or slight iterations thereof, was the top suggested question in five of nine search queries and was in the top three questions of all nine strings of queries. The most common subcategorization of questions was specific activities, representing 16.9% of all questions. This finding highlights patients’ concerns about their ability to perform and continue performing specific activities postoperatively. The activities in question varied, with most being related to basic activities of daily living such as bathing and dressing, but also included more personalized and high-functioning activities such as bowling, golfing, and swimming. This illustrates the vast differences in patients’ preoperative goals. Generally speaking, while some patients are simply looking to perform activities of daily living without pain, others are hoping to return to higher-level functionality and recreational athletics. Physicians can answer a range of questions by providing patients with basic information regarding expectations for postoperative daily activities, as well as by asking each patient what activities are important to them to customize their preoperative education. It is critically important that surgeons determine which activities a patient wishes or expects to return to postoperatively in order to best educate patients and set realistic expectations for them. Numerous studies have shown that expectations regarding functional outcomes are correlated with reported postoperative outcomes [1,2,5,11]. Pain was the second most common subcategorization of questions, representing 15.3% of all questions. It comes as no surprise that patients are concerned about pain following surgery. Knowing that preoperative expectations are correlated with postoperative experiences, pain must also be discussed with the goal of setting realistic expectations for patients and explaining how pain will be managed, particularly given today’s wide array of analgesics used for shoulder arthroplasty [3,5-11]. Preoperative risk factors for severe postoperative pain should be identified so that appropriate management and referrals can be provided insofar as severe pain is associated with higher costs and longer hospitalization stays [28]. With pain protocols continuing to improve and shoulder arthroplasty now being performed as an outpatient procedure, it is important to make patients aware of what to expect in terms of multimodal analgesia and a plan for adequate pain control upon being discharged home [7]. Similarly, a prospective study in 4,709 patients undergoing total joint arthroplasty finds the best predictors of postoperative satisfaction in patients to be the meeting of patients’ preoperative expectations and satisfactory pain relief [29].

Medical information sites scored highly on JAMA benchmark criteria because they nearly always listed authorship and date of publication. Medical practice, single-surgeon practice, and academic webpages often did not cite authors, references, or date of publication, and thus scored poorly according to JAMA benchmark criteria. All categories besides journal sites scored poorly for disclosure criteria. The most linked webpages were medical practice and academic pages, comprising 24.6% and 23.2% of total searches, respectively. When combined with single-surgeon practice sites (8.0%), most webpages were hosted by organizations employing orthopedic surgeons. However, each of these three categories scored poorly on the JAMA benchmark criteria, earning scores of 0.89, 1.36, and 1.07, respectively. These informational webpages were likely authored or reviewed and/or editorialized by surgeons. Nevertheless, these sites rarely listed authors, publication dates, disclosures, or references. This highlights the need for improvement in informational publishing practices from orthopedic sources.

We observed that 14.4% of webpages belonged to medical information sites such as verywellhealth.com and healthline.com. These webpages host topical articles written by healthcare journalists and undergo editorial review by physicians. Such medical information sites score very well according to JAMA benchmark criteria, with an average score of 2.65, insofar as they nearly always list authors, publication dates, and references. Non-medical media sites such as news articles, social media, and Wikipedia articles accounted for only 4.9% of all webpages returned by Google Search. As social media is generally regarded as a poor source of information, it is encouraging to see that social media websites are rarely provided as educational resources by the Google algorithm. Commercial and lawyer/law-related webpages only accounted for 9.3% and 0.4% of total webpages, respectively. While it is reassuring that most webpages visited by patients are hosted by organizations employing physicians, the quality of these published resources would benefit from increased academic rigor and source transparency.

The internet has been shown by numerous studies to be a source of poor quality health information for patients with specific needs [15-19]. Oftentimes, answers to medical questions are complicated and nuanced, thus not amenable to explanation by non-professionals. We believe these questions are best answered in conversations between patients and surgeons, with adequate time for questions and clarification, or via educational materials produced and provided directly to patients by their surgeons. The results of this study allow shoulder arthroplasty surgeons to better anticipate and address their patients’ concerns, with potential improvement in patient comprehension, expectations, and ultimately, outcomes. In addition, this study highlights the need for orthopedic surgeons to improve the quality of information available to patients published by their practices and institutions.

Questions about reverse shoulder replacements were more likely to be policy questions and subcategorized as indications/management according to our findings. Patients want to know what makes them a candidate for reverse shoulder replacement, as well as when and why reverse shoulders are used instead of anatomic shoulder replacements.

A recent study published by Sudah et al. [30] extracts 300 questions from Google pertaining to shoulder arthroplasty. The results of their analysis support our findings above. Beyond the preliminary analysis of questions and webpages in that previous research, our current study analyzes a three-times larger dataset of questions and websites, utilizes objective JAMA benchmark criteria to assess source quality, and additionally examines the topics of reverse and anatomic shoulder arthroplasty.

A limitation of this study is that it relies on Google-generated questions that people also ask when seeking online information about shoulder arthroplasty. These questions do not come directly from known orthopedic patients, and it is impossible to confirm who is searching for these questions. It is assumed these are questions posed by patients. It is also impossible to know whether these questions are being searched preoperatively or postoperatively. While there may be some uncertainty in who is asking these questions, there is a notable advantage in seeing what questions are being asked by patients in the reassuring context of online anonymity. The “People also ask” function generates results based on search trends within individuals’ regions of search. In this study, the United States was set as the search country, thus results are less applicable to nationalities and patient populations outside of the United States. Similarly, searches were conducted in 2021; results will change over time as patient populations and available information changes. Another limitation of this study is that JAMA benchmark criteria are a better measure of transparency and publishing practices than a measure of the accuracy of content. This is a known limitation of the criteria and was acknowledged in its original publication [22,25]. Despite this limitation, JAMA criteria remain a valuable tool for objectively evaluating sources of online health information [15,22,27]. Future studies should examine what types of questions patients ask their surgeons in offices and clinics in comparison to the questions to which patients seek answers online.

The most common questions asked online by SA patients are related to performing specific postoperative activities and the timeline of recovery. The majority of information is provided by low quality, non-peer-reviewed websites. This study highlights the need for improvement in online resources available to patients undergoing SA. By understanding which questions patients have and the quality of online information provided in response, surgeons can improve preoperative education and postoperative outcomes for patients of all types of shoulder arthroplasty.

Notes

Author contributions

Conceptualization: JRM, NM. Data curation: MCK, KDC. Formal Analysis: KDC. Methodology: JRM. Project administration: JRM. Writing – original draft: JRM, KDC. Writing – review & editing: JRM, NM, DD, NSH, GPN, NNV, GEG.

Conflict of interest

None.

Funding

None.

Data availability

Contact the corresponding author for data availability.

Acknowledgments

None.

References

1. Flood AB, Lorence DP, Ding J, McPherson K, Black NA. The role of expectations in patients’ reports of post-operative outcomes and improvement following therapy. Med Care 1993;31:1043–56.
2. Judge A, Cooper C, Arden NK, et al. Pre-operative expectation predicts 12-month post-operative outcome among patients undergoing primary total hip replacement in European orthopaedic centres. Osteoarthritis Cartilage 2011;19:659–67.
3. Klafta JM, Roizen MF. Current understanding of patients’ attitudes toward and preparation for anesthesia: a review. Anesth Analg 1996;83:1314–21.
4. Mondloch MV, Cole DC, Frank JW. Does how you do depend on how you think you’ll do?: a systematic review of the evidence for a relation between patients’ recovery expectations and health outcomes. CMAJ 2001;165:174–9.
5. Tashjian RZ, Bradley MP, Tocci S, Rey J, Henn RF, Green A. Factors influencing patient satisfaction after rotator cuff repair. J Shoulder Elbow Surg 2007;16:752–8.
6. Gandhi R, Davey JR, Mahomed N. Patient expectations predict greater pain relief with joint arthroplasty. J Arthroplasty 2009;24:716–21.
7. Cohn MR, Kunze KN, Wright-Chisem J, et al. Pain management after shoulder arthroplasty: a systematic review of randomized controlled trials. Semin Arthroplasty 2021;31:644–61.
8. Henn RF 3rd, Kang L, Tashjian RZ, Green A. Patients’ preoperative expectations predict the outcome of rotator cuff repair. J Bone Joint Surg Am 2007;89:1913–9.
9. Mahomed NN, Liang MH, Cook EF, et al. The importance of patient expectations in predicting functional outcomes after total joint arthroplasty. J Rheumatol 2002;29:1273–9.
10. Mancuso CA, Altchek DW, Craig EV, et al. Patients’ expectations of shoulder surgery. J Shoulder Elbow Surg 2002;11:541–9.
11. Swarup I, Henn CM, Nguyen JT, et al. Effect of pre-operative expectations on the outcomes following total shoulder arthroplasty. Bone Joint J 2017;99:1190–6.
12. Fraval A, Ming Chong Y, Holcdorf D, Plunkett V, Tran P. Internet use by orthopaedic outpatients: current trends and practices. Australas Med J 2012;5:633–8.
13. Fox S, Duggan M. Health online 2013 [Internet]. Pew Research Center; 2013 [cited 2023 Feb 1]. Available from: https://www.pewresearch.org/internet/wp-content/uploads/sites/9/media/Files/Reports/PIP_HealthOnline.pdf.
14. Andrienko O. How to maximize ‘people also ask’ SEO opportunities [Internet]. Semrush Blog; 2020 [cited 2023 Feb 1]. Available from: https://www.semrush.com/blog/how-to-maximize-people-also-ask-seo-opportunities-study/.
15. Cassidy JT, Baker JF. Orthopaedic patient information on the World Wide Web: an essential review. J Bone Joint Surg Am 2016;98:325–38.
16. De Martino I, D’Apolito R, McLawhorn AS, Fehring KA, Sculco PK, Gasparini G. Social media for patients: benefits and drawbacks. Curr Rev Musculoskelet Med 2017;10:141–5.
17. Kunze KN, Krivicich LM, Verma NN, Chahla J. Quality of online video resources concerning patient education for the meniscus: a YouTube-based quality-control study. Arthroscopy 2020;36:233–8.
18. Labovitch RS, Bozic KJ, Hansen E. An evaluation of information available on the internet regarding minimally invasive hip arthroplasty. J Arthroplasty 2006;21:1–5.
19. Saithna A, Ajayi OO, Davis ET. The quality of Internet sites providing information relating to hip resurfacing. Surgeon 2008;6:85–7.
20. Kulasegarah J, Harney M, Walsh M, Walsh RM. The quality of information on three common ENT procedures on the Internet. Ir J Med Sci 2012;181:221–4.
21. Kulasegarah J, McGregor K, Mahadevan M. Quality of information on the Internet-has a decade made a difference. Ir J Med Sci 2018;187:873–6.
22. Shen TS, Driscoll DA, Islam W, Bovonratwet P, Haas SB, Su EP. Modern Internet search analytics and total joint arthroplasty: what are patients asking and reading online. J Arthroplasty 2021;36:1224–31.
23. Google. Uninstall Google Chrome [Internet]. Google; 2021 [cited 2023 Feb 1]. Available from: https://support.google.com/chrome/answer/95319?co=GENIE.Platform%3DDesktop&hl=en .
24. Kanthawala S, Vermeesch A, Given B, Huh J. Answers to health questions: Internet search results versus online health community responses. J Med Internet Res 2016;18:e95.
25. Rothwell JD. In Mixed company: communicating in small groups and teams. Wadsworth Cengage Learning; 2018.
26. Bruce-Brand RA, Baker JF, Byrne DP, Hogan NA, McCarthy T. Assessment of the quality and content of information on anterior cruciate ligament reconstruction on the internet. Arthroscopy 2013;29:1095–100.
27. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: caveant lector et viewor: let the reader and viewer beware. JAMA 1997;277:1244–5.
28. Menendez ME, Lawler SM, Ring D, Jawa A. High pain intensity after total shoulder arthroplasty. J Shoulder Elbow Surg 2018;27:2113–9.
29. Hamilton DF, Lane JV, Gaston P, et al. What determines patient satisfaction with surgery?: a prospective cohort study of 4709 patients following total joint replacement. BMJ Open 2013;3:e002525.
30. Sudah SY, Pagani NR, Nasra MH, et al. What patients want to know about shoulder arthroplasty: a Google search analysis. Semin Arthroplasty 2022;32:162–8.

Article information Continued

Fig. 1.

Rothwell classification for general, anatomic, and reverse shoulder arthroplasty. Questions for each of the three search groups as categorized by Rothwell classification: fact, policy, and value.

Fig. 2.

Subcategorization of questions for general, anatomic, and reverse shoulder arthroplasty. Breakdown of questions into subcategorizations based on topical content.

Fig. 3.

Breakdown of websites classified based on host webpages.

Table 1.

Rothwell classifications, subcategorizations, and examples

Question categorization
Fact Ask whether something is true, and to what extent
e.g., Where is the incision for shoulder replacement?
Specific activities Can you throw a ball after shoulder replacement?
Timeline of recovery How long does it take to recover from shoulder replacement?
Technical details What muscles are cut during a total shoulder replacement?
Restrictions What can you not do after shoulder replacement?
Cost How much does Medicare pay for shoulder replacement?
Policy Ask whether a certain course of action should be taken to solve a problem
e.g., Is shoulder replacement an inpatient or outpatient procedure?
 Indications/management Are there alternatives to shoulder replacement?
 Risks/complications What is the chance of dying in surgery?
Value Ask for evaluation of an idea, object, or event
e.g., What is the success rate of shoulder replacement?
 Pain How painful is total shoulder replacement?
 Longevity How long does a shoulder replacement last?
 Evaluation of surgery Are shoulder replacements successful?

Table 2.

Website categorizations and examples

Website categorization
Academic Webpage hosted by an academic institution or organization
e.g., https://ucsfhealth.org, https://health.harvard.edu
Commercial Webpage hosted by a for-profit company
e.g., https://Zimmerbiomet.com, https://arthrosurface.com
Journal Academic journal publication, may be hosted by third party site
e.g., https://pubmed.com, https://springer.com
Government Governmental hosted webpage
e.g., https://myhealth.alberta.ca, https://medlineplus.gov
Lawyer Single attorney, law firm, or legal advice webpage
e.g., https://rosenfeldinjurylawyers.com, https://rossfellercasey.com
Medical information site Company or organization for the purpose of medical information reviewed by medical professionals
e.g., https://webmd.com, https://healthline.com
Medical practice Medical or surgical practice of physicians
e.g., https://clevelandshoulder.com, https://tusconortho.com
Non-medical media site Webpages not specializing in medical information such as general news and social media sites
e.g., https://wikipedia.com, https://abcnews.com
Single surgeon practice Single surgeon practice or personal webpage
e.g., https://howardluksmd.com, https://chalmersmd.com

Table 3.

JAMA benchmark criteria

JAMA benchmark criteria
Authorship Clearly identifiable author and contributors with affiliations and relevant credentials present
Attribution References and sources clearly listed with any copyright information disclosed
Currency Clearly identifiable posting date of any content as well as date of any revisions
Disclosure Website ownership clearly disclosed along with any sponsorship, advertising, underwriting, and financial support

Each criteria received 1 point for a maximum of 4 points.

JAMA: Journal of the American Medical Association.

Table 4.

Question results breakdown by classification and subclassification

Category General Anatomic Reverse No. (%)
Fact 179 175 132 486 (54.0)
 Specific activities 58 46 48 152 (16.9)
 Timeline of recovery 41 46 33 120 (13.3)
 Restrictions 29 29 22 80 (8.9)
 Technical details 42 47 24 113 (12.6)
 Cost 9 7 5 21 (2.3)
Policy 53 47 92 192 (21.3)
 Indications/management 42 32 74 148 (16.4)
 Risks/complications 11 15 18 44 (4.9)
Value 68 78 76 222 (24.7)
 Pain 41 45 52 138 (15.3)
 Longevity 1 2 4 7 (0.8)
 Evaluation of surgery 26 31 20 77 (8.6)
Total 300 300 300 900

Table 5.

Website results breakdown by classification

Website classification General Anatomic Reverse
Academic 70 82 57
Commercial 18 22 44
Government 25 18 13
Journal 27 27 26
Lawyer 2 1 1
Medical info site 46 42 42
Medical practice 74 65 82
Nonmedical media site 14 15 15
Single surgeon personal 24 28 20