INTRODUCTION
YouTube has become a popular source for health information among patients with over 50% of the population engaging with the internet platform [1]. In particular, sports medicine patients have significantly higher internet and social media usage relative to any other field in orthopedics [2,3]. While online orthopedic resources are becoming an increasingly popular avenue for patient education, videos on YouTube are not subject to institutional or peer review. Thus, despite the platform’s popularity, the accuracy, authenticity, or quality of the information it disseminates is not verified. This limitation has been identified in numerous studies that have found a high prevalence of inaccurate health information and low quality medical videos on YouTube [4-7].
Within orthopedic surgery, quality-based studies of YouTube videos have been conducted for topics such as kyphosis, the anterior cruciate ligament, and the meniscus [8-10]. However, the quality and accuracy of information on the ulnar collateral ligament (UCL) of the elbow has yet to be investigated. UCL injuries are especially applicable for YouTube due to the unique patient population. UCL injuries are common among athletes who throw or put significant pressure on their elbows, especially baseball pitchers, quarterbacks, and gymnasts [11]. Accumulation of microtrauma to the UCL—for example, through repeated ball pitching motions—can result in eventual injury and failure of the ligament [12]. Given the sports-related nature of the injury in a young patient population (average age of 21.6 years) [13], patients with UCL injuries may be more likely to access online resources for health information [2]. This trend of utilizing online resources related to healthcare information has only increased among young patients over the last decade [14], a trend evidenced by the drastic rise in UCL-related social media posts in recent years [15]. Previous findings have shown that nearly 70% of athletes utilize the internet for medical-based information with YouTube being the most frequently used platform. Furthermore, younger athletes use these video platforms for medical information significantly more often than older athletes [16]. The purpose of this study was to evaluate the quality of health information available on YouTube concerning UCL injuries. We hypothesized that most of the video content would be of low quality and low reliability for patient education.
METHODS
This study did not involve patients and thus did not require informed consent and does not require ethical approval of institutional review board.
YouTube Query and Video Characteristics
A YouTube search was performed on January 10, 2022 using four search terms: “ulnar collateral ligament,” “ulnar collateral ligament injury,” “medial collateral ligament,” and “medial collateral ligament injury.” The first 75 videos for each of the four search terms were reviewed. Inclusion criteria included any video pertaining to the UCL of the elbow. Exclusion criteria consisted of any duplicate videos, any videos focused on the UCL of the thumb, any videos focused on the medial collateral ligament of the knee, and any videos that were not in English. Of the 300 videos extracted from the YouTube search, 101 videos were duplicates, 78 videos were focused on the medial collateral ligament of the knee, and 38 videos were focused on the UCL of the thumb (Fig. 1). These videos were excluded from the analysis, and the remaining 83 videos were evaluated. Although the YouTube search using the four search terms produced over 2,600 videos, analysis of the remaining resources was deemed unnecessary as links on the first page (10–20 results) of an online search receive 70%–95% of web traffic [17,18].
Video characteristics extracted for each video included: (1) video title, (2) duration, (3) number of views, (4) video source, (5) type of content, (6) upload date, (7) days since upload, and (8) views per day. Video source was categorized into one of the following: (1) academic (pertaining to authors or uploaders affiliated with research groups, universities, or colleges), (2) physician (independent physician or physician group without research, university or college affiliation), (3) non-physician (health professional other than licensed medical doctors, including physical therapists), (4) fitness and health trainers (i.e., athletic trainers and strength coaches), (5) medical source (content or animators from health website), (6) patient, and (7) commercial. Content categories were grouped into: (1) exercise training, (2) disease-specific information, (3) patient experience, (4) surgical technique or approach, (5) non-surgical management, and (6) advertisement.
Evaluation of Video Reliability
Video reliability was assessed using two scoring systems: the Journal of American Medical Association (JAMA) benchmark criteria scored from 0 to 4 (Table 1) and a modified 5-point DISCERN tool. The JAMA reliability score is a grading system utilized in previous YouTube quality analyses [10,19-22]. This grading system assigns one point for the presence of each criterion, which are independent assessments of source reliability. A higher score indicates higher reliability. The modified DISCERN tool has previously been used in several YouTube patient education evaluation studies [23-25] and was adapted based on the original DISCERN tool [26]. The modified DISCERN evaluation system consists of five questions with the presence of each criterion given 1 point for a total of 5 points. A higher score indicates higher reliability. Furthermore, an accuracy score (AS) was utilized to specifically assess informational accuracy. The AS was scored from 1–3 with 3 being accurate information, 2 being misleading information, and 1 being inaccurate/wrong information. Informational accuracy of the videos was first evaluated by two independent raters and reviewed again for the final rating by one senior author, a sport -fellowship-trained orthopedic surgeon.
Evaluation of Video Quality
The video quality and educational value of the UCL videos were assessed using a 5-point Global Quality Score (GQS). The GQS provides a nonspecific assessment of the educational value for patients (Table 2), a grading system utilized in numerous previous YouTube quality analyses [8,10,20,27,28]. A higher score indicates higher educational quality of the video. Scores of 1–2 points indicated low quality, 3 points moderate quality, and 4–5 points high quality [29]. For a tailored evaluation of the YouTube videos, a UCL-focused scoring system was developed. This “Ulnar Collateral Ligament-Specific Score” (UCL-SS) for video content consisted of 16 items adapted from the Meniscus YouTube study scoring table utilized by Kunze et al. [10], a method shown to be effective with criteria from the American Academy of Orthopedic Surgeons [30]. The UCL-SS evaluates information concerning: (1) patient presentation/symptomology, (2) anatomy of the UCL, (3) diagnosis and evaluation, (4) treatment, and (5) postoperative course (Table 3). One point was awarded per item if the video included the relevant information visually or verbally with a total possible score of 16 points. All video scores—including JAMA score, modified DISCERN score, GQS, and UCL-SS—were evaluated by two independent raters. Discrepancies were resolved by a third independent reviewer.
Statistical Analysis
Descriptive statistics were used to analyze accuracy, quality, and reliability scores along with the qualitative characteristics of videos. One-way analysis of variance (ANOVA) and Kruskal-Wallis tests were used for normally distributed and non-normally distributed data, respectively. Post hoc pair-wise Tukey-Kramer multiple comparison tests were conducted to determine significant pairs [31]. Linear regression was used to determine associations between basic video characteristics (video duration, video views, and views per day) and video reliability and quality scores. The significance level was set at a two-tailed p<0.05. Data analysis was conducted using R ver. 3.5.3 (R Foundation for Statistical Computing, Vienna, Austria), and figures were made using GraphPad Prism 8 (GraphPad, San Diego, CA, USA). Inter-observer reliability was calculated as a kappa score; the following guidelines were used in the interpretation of the kappa score: no agreement less than 0.00, slight agreement between 0.00 and 0.20, fair agreement between 0.21 and 0.40, moderate agreement between 0.41 and 0.60, substantial agreement between 0.41 and 0.80, and almost perfect agreement between 0.81 and 1.00 [32].
RESULTS
Descriptive Statistics of Video Characteristics
Of the 300 videos extracted from the YouTube search, 101 videos were duplicates, 78 videos were focused on the medial collateral ligament of the knee, and 38 videos were focused on the UCL of the thumb (Fig. 1). These videos were excluded from the analysis and the remaining 83 videos were evaluated. All videos were in English. Based on video source, physician and commercial uploads accounted for the largest proportions of videos, 42% and 23%, respectively (Table 4). Academic posts accounted for the lowest proportion of videos at 17%. There were no videos whose source was directly from fitness/health trainers or patients. Based on video content, disease-specific information and surgical technique videos comprised the largest segments, 52% and 33%, respectively. The smallest segment was patient experience videos at 4%. There were no videos whose content was focused on advertisement. The mean video duration was 483.2±571.3 seconds (range, 20–3,549 seconds). The mean number of views was 17,294.4±51,232.0. Overall, the 50 videos were viewed 1,435,437 times. Videos received a mean views per day of 14.6 ± 31.3. The mean number of days since upload was 1,201.5±864.1 days.
Mean JAMA score was 1.8 ± 0.8; the mean modified DISCERN score was 2.4±1.0; the mean GQS was 1.9±1.2; the mean UCL-SS was 5.3±3.4; and the mean AS was 2.7±0.6. Assessed through Cohen’s kappa score, inter-observer reliability was 0.93 (95% confidence interval [CI], 0.86–1.00) for the JAMA score, 0.94 (95% CI, 0.88–1.00) for the mean modified DISCERN score, 0.85 (95% CI, 0.74–0.96) for the GQS, and 0.88 (95% CI, 0.77–0.99) for the UCL-SS. Overall, based on the GQS, 78% of videos were low quality in terms of patient education (rated with scores of 1–2). Based on the AS, 18% of videos were either misleading or inaccurate (rated with scores of 1–2).
Basic Video Characteristic Association Analysis
Video duration was significantly positively associated with UCL-SS (p=0.01) but was not significantly associated with JAMA score (p=0.55), modified DISCERN score (p=0.67), GQS (p=0.84), or AS (p=0.45). The number of video views was not significantly associated with JAMA score (p=0.78), modified DISCERN score (p=0.62), GQS (p=0.39), UCL-SS (p=0.96), or AS (p=0.61). Days since upload was not significantly associated with JAMA score (p=0.61), modified DISCERN score (p=0.33), GQS (p=0.66), UCL-SS (p=0.24), or AS (p=0.61). Views per day was not significantly associated with JAMA score (p=0.88), modified DISCERN score (p=0.69), GQS (p=0.39), UCL-SS (p=0.55), or AS (p=0.42).
ANOVA Analysis of Video Content and Video Source Evaluation Scores
For the JAMA score, ANOVA yielded significant variation across groups based on video content (p=0.02), with videos related to surgical technique and nonsurgical management having the highest mean JAMA scores. Post hoc pairwise Tukey-Kramer multiple comparison tests indicated that none of the video content groups differed significantly from one another (p>0.05). ANOVA also yielded significant variation across groups based on video upload source (p<0.001), with videos uploaded by non-physicians having the lowest mean JAMA scores. Post hoc Tukey-Kramer multiple comparison tests showed that the non-physician and physician groups differed significantly (p<0.01) and that the non-physician and academic groups differed significantly (p<0.01). None of the other video source groups differed significantly from one another (p>0.05). For the modified DISCERN score, ANOVA did not yield significant variation across groups based on video content (p=0.08). Post hoc pairwise Tukey-Kramer multiple comparison tests indicated that none of the video content groups differed significantly from one another (p>0.05). ANOVA yielded significant variation across groups based on video upload source (p=0.006), with videos uploaded by physicians having the highest mean modified DISCERN scores. Post hoc Tukey-Kramer multiple comparison tests showed that the physician and non-physician groups differed significantly (p<0.05) and that the physician and commercial groups differed significantly (p<0.05). None of the other video source groups differed significantly from one another (p>0.05).
For GQS, ANOVA yielded significant variation across groups based on video content (p<0.001), with surgical technique videos having the lowest mean GQSs. Post hoc pairwise Tukey-Kramer multiple comparison tests indicated that surgical technique and disease-specific information videos differed significantly (p<0.01) and that surgical technique and patient experience videos differed significantly (p<0.01). None of the other video content groups differed significantly from one another (p>0.05). ANOVA yielded significant variation across groups based on video upload source (p<0.001) with videos uploaded by physicians having the highest mean GQSs. Post hoc Tukey-Kramer multiple comparison tests showed that the physician and non-physician groups differed significantly (p<0.01) and that the physician and commercial groups differed significantly (p<0.05). None of the other video source groups differed significantly from one another (p>0.05). For the UCL-SS, ANOVA did not yield significant variation across groups based on video content (p=0.08). Post hoc pairwise Tukey-Kramer multiple comparison tests indicated that none of the video content groups differed significantly from one another (p>0.05). ANOVA yielded significant variation across groups based on video upload source (p<0.001) with videos uploaded by physicians and academic sources having the highest mean UCL-SSs. Post hoc Tukey-Kramer multiple comparison tests showed that the physician and non-physician groups differed significantly (p<0.01) and that the physician and commercial groups differed significantly (p<0.05). None of the other video source groups differed significantly from each other (p>0.05).
For AS, ANOVA yielded significant variation across groups based on video content (p<0.001), with disease-specific information and surgical technique videos having the highest mean AS. Post hoc pairwise Tukey-Kramer multiple comparison tests indicated that disease-specific information videos differed significantly from non-surgical management videos (p<0.01), exercise training videos (p<0.01), and patient experience videos (p<0.01) and that surgical technique videos differed significantly from non-surgical management videos (p<0.01), exercise training videos (p<0.01), and patient experience videos (p<0.01). None of the other video content groups differed significantly from one another (p>0.05). ANOVA yielded significant variation across groups based on video upload source (p=0.02) with videos uploaded from non-physicians having the lowest mean AS. Post hoc Tukey-Kramer multiple comparison tests showed that the non-physician and physician groups differed significantly (p<0.05) and that the non-physician and commercial groups differed significantly (p<0.05). None of the other video source groups differed significantly from one another (p>0.05).
DISCUSSION
With the increasing popularity of online platforms as a source for healthcare information among patients, tools such as YouTube have the potential to effectively communicate information outside of the traditional face-to-face physician consultation [33]. As patients with UCL injuries comprised a young patient cohort [13] that may be even more likely to turn to online resources for health information [2,14], investigating the reliability and quality of YouTube videos for UCL injuries is especially relevant. However, as with many online resources, YouTube is not peer reviewed. Thus, the lack of a review process creates the risk of propagating biased, misleading, and inaccurate information among patients [25,34,35]. To the best of our knowledge, this is the first study to evaluate YouTube videos regarding UCL injuries.
This study evaluated 83 YouTube videos related to injuries of the elbow UCL to gauge patient education quality. There were four main findings. First, among the 83 included videos, only 22% of videos were moderate to high quality patient education resources. The remaining 78% of videos were low quality. Second, video content was principally focused on disease-specific information and surgical technique while video source mostly comprised physician and commercial-created videos. Academic videos were the least represented. Third, the video source was significantly associated with reliability, educational quality, and accuracy. Non-physician videos not only had the lowest reliability and quality but were also the least accurate, and physician videos boasted the highest quality as patient education resources. Fourth, the type of video content was significantly associated with educational quality and accuracy. Along with disease-specific information videos, surgical technique videos had the highest accuracy. However, surgical technique videos also had the lowest quality as patient education resources.
Overall, there are three key issues with using YouTube as a resource for patient education of UCL injuries. First, the proportion of moderate to high quality patient education videos is low (22%) and is substantially lower than the proportions cited in other studies (51%–87%) [23-25,27,36]. The paucity of high quality videos, especially from reputable physician and academic sources, echoes the findings of other orthopedic YouTube media evaluation studies for the meniscus, kyphosis, disc herniation, anterior cruciate ACL, articular cartilage defects, and femoroacetabular impingement [8-10,37-39]. Second, due to related anatomical terminology, there is an excess amount of irrelevant information available online regarding elbow UCL injuries. Of the 199 unique (non-duplicate) videos reviewed from the YouTube search, 58% of videos focused on the medial collateral ligament of the knee or the UCL of the thumb. Patients must potentially filter through numerous unrelated videos to access relevant material. Third, the proportion of studies with misleading or inaccurate content was conspicuous (18%), which aligned with the proportion of misleading videos found in previous studies (14%–16%) [24,27,36].
YouTube’s lack of a peer review process produces the potential for inaccurate or biased information to be disseminated to patients [25,34,35]. This issue may only worsen as patient use of YouTube grows because of the platform’s ease of access and intuitive visual content. High-quality content produced by reputable sources is critical for patient education; however, authoritative and credible healthcare organizations produce only a small subset of high quality educational medical videos [40]. This study found that most relevant videos for UCL injuries lack reliability, patient education quality, and, occasionally, accuracy. While physician video sources produced the best reliability, quality, and accuracy, non-physician video sources were the opposite. Despite this, nearly half of all videos were created by non-physician or commercial sources. Among physician and academic sources, the type of content was crucial to the quality of patient education. Surgical technique videos—far too technical to be useful for the average patient—comprised 33% of total UCL injury videos. Although these videos may be highly educational for other orthopedic clinicians, these surgical technique learning resources often fail to explain the role of surgery in the broader context of UCL injury treatment. This failure to provide a view on the overall treatment of UCL injuries, paired with the highly technical vocabulary and content, creates a resource that is of little educational value to patients.
Based on the findings of this study, we offer three recommendations for orthopedic clinicians treating patients with UCL injuries. First, patients should be advised about potentially misleading online information as well as information provided by non-physicians. Clinicians can ameliorate these risks by guiding patients toward specific online resources and videos that are of high educational quality to facilitate an efficient and effective learning experience. Given the related anatomical terminology of the elbow UCL with the UCL of the thumb and the medial collateral ligament of the knee, working with patients to navigate these online resources can help patients avoid getting overwhelmed or confused by the excess, unrelated information available on YouTube. Second, recognizing the dearth of high-quality UCL injury videos available on YouTube, clinicians may consider producing their own information-rich, unbiased videos while citing references for their content. Third, clinicians may consider using separate platforms or specific keywords to share videos on surgical technique. Although these videos can be highly educational for other orthopedic clinicians, patients can quickly become confused while sifting through the excess information within these videos.
Strengths and Limitations
This study has multiple strengths. The specific UCL score (UCL-SS) allowed for a novel, in-depth analysis of UCL-specific content. By evaluating videos using distinct metrics, this study elucidated how video quality, reliability, and accuracy are associated with different sources and content. Specifically, high-quality and low-quality videos can be linked to specific video characteristics. Understanding these associations is essential for helping patients find appropriate online educational resources while also guiding clinicians in producing high-quality online educational material. Finally, the findings of this online patient education evaluation study are particularly applicable given the young patient population for UCL injuries.
The study also has several limitations. In evaluating YouTube videos, highly regarded but unvalidated quality assessment tools such as the JAMA score, modified DISCERN score, GQS, and AS were utilized. Despite the lack of validation of these measures, many previous studies have used these to provide a reproducible measure [5-7]. However, utilizing additional independent raters would bolster the validity of these measures. The analysis pulled from the first 75 videos of each of the four search terms for a total of 300 videos. However, the total video pool comprised over 2,600 videos. Thus, there may be more relevant videos among the total pool that were not captured in this analysis. Nonetheless, as users rarely search further than the first two pages when searching for information, focusing on the first set of videos that patients encounter as opposed to all possible information is likely to be the most applicable method [41]. Though this study evaluated videos through the search for “ulnar collateral ligament," “ulnar collateral ligament injury," “medial collateral ligament.” and “medial collateral ligament injury.” patients with UCL injuries may also search for videos using the term ‘Tommy John surgery." Even with substantial overlap between these two phrases, the analysis from this study may not capture all the possible videos from a search for “Tommy John surgery.”
Conclusion
Currently, YouTube is not a reliable or high-quality source of information for patients seeking information regarding elbow UCL injuries. Videos uploaded from physician sources have the highest reliability, quality, and accuracy while videos uploaded from non-physician sources have the lowest reliability, quality, and accuracy. Although some videos are robust educational resources, the multiplicity of low quality, low reliability, and irrelevant videos on YouTube can create a confusing and inaccurate learning experience for patients. Instead of patients independently traversing the low quality and low reliability landscape of YouTube, orthopedic clinicians should guide patients toward online resources and videos that are of high educational quality to facilitate an efficient and effective learning experience.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Full text via PMC
Download Citation
