Original Article

Do the Most Cited Web Sources Provide Proper Information on Influenza Vaccination?

Elif Mukime Sarıcaoğlu
  • Department of Infectious Diseases and Clinical Microbiology, Ankara University School of Medicine, Ankara, Turkey
Ayda Gholizadeh
  • Ankara University School of Medicine, Ankara, Turkey
Cemre Kırım
  • Department of Forensic Medicine, Başkent University School of Medicine, Ankara, Turkey
Elif Yunt
  • Department of Internal Medicine, Dışkapı Yıldırım Beyazıt Training and Research Hospital, Ankara, Turkey
İhsan Mert Türk
  • Gölbaşı Şehit Ahmet Özsoy Hospital, Ankara, Turkey
İlker Anıl Çelik
  • Department of General Surgery, Prof. Dr. Cemil Taşçıoğlu City Hospital, İstanbul, Turkey
Kardelennur Yıldırım
  • Ankara University School of Medicine, Ankara, Turkey
Pelin Özbakır
  • Ürgüp State Hospital, Nevşehir, Turkey
Alpay Azap
  • Department of Infectious Diseases and Clinical Microbiology, Ankara University School of Medicine, Ankara, Turkey


Objective: The Internet is increasingly preferred as a source of health-related information. However, several studies revealed deficiencies in the quality of web-based health information. Therefore, this study aimed to analyze the content and design of influenza vaccine-related websites.

Materials and Methods: This study was conducted using a search engine, Google Search, in the Turkish language. We identified 15 influenza-related search terms, and for each term, the top ten links on the first page were included in the study after removing duplicate links. Then, we evaluated identified websites in terms of content and design criteria.

Results: Seventy-one links related to influenza vaccination were evaluated in our study. The mean content criteria score of all selected websites was 11.5 points. While academic websites received the highest contents point with 22.5, news and hospital sites remained under the average with 10.5 and 8.9 points, respectively. Thirty-eight per cent of the websites had aesthetic/content professionalism, and 77.5% included contact information. References were shown only in 14.1% of websites. The websites with higher design criteria did not have higher content points.

Conclusion: The Internet remains a frequently referenced source of information. The accuracy of the information on health topics included on the Internet is important in terms of public health. Unfortunately, according to our study, the top websites giving information on influenza vaccination were not well designed and had insufficient content.

Keywords: influenza vaccine, information source, websites


  • The Internet, which is increasingly preferred as a source of health-related information, may have deficiencies in the quality of information.
  • Common network resources may not contain sufficient vaccine-related information.  
  • The contents and design of websites giving vaccine information must be planned and tested carefully.


Influenza constitutes one of the major threats to the health system, affecting up to 10% of the world’s population every year (1, 2). According to the Centers for Disease Control and Prevention (CDC) database, between 1999-2015, 291,000-646,000 deaths caused by influenza-associated respiratory disease were recorded worldwide (3). Seasonal influenza viruses can cause high morbidity and mortality, especially in people with underlying diseases, the elderly, pregnant women and children (4). Influenza vaccination is the primary prevention method of influenza infection (5).

Local policies, knowledge and perceptions can influence the vaccination rate in different communities. The rate of influenza vaccination and factors affecting infection control team (ICT) decision on getting vaccinated was assessed in a multinational study. In Turkey, the overall vaccination rate in the 2018-2019 season was only 39%, even among ICT members primarily responsible for the vaccination of health care professionals. Not being in a risk group and not believing the vaccine’s efficacy were the major reasons for not getting vaccinated (6). Therefore, it is important to find out the reasons affecting vaccination decision and provide reliable vaccine-related information to enhance the vaccination rates. 

The Internet remains a frequently referenced source of information, and the Internet usage rate has been augmented in Turkey as well as all over the world (7). According to the Turkish Statistics Institute Information and Communication Technology (ICT) Usage Survey on Households and Individuals, 2019, the Internet access rate throughout Turkey increased to 94.9% in 2019 (8). The Internet is increasingly preferred as a source of health-related information (9). However,  several studies revealed deficiencies in the quality of web-based health information (10). For example, some of the news on the websites can have positive/neutral/negative attitudes towards influenza vaccination (9). Despite the wealth of evidence that vaccines prevent disease and save millions of lives every year, vaccine hesitancy became an important challenge in public health (11). It causes delay, and vaccine refusal becomes a risk for vaccine-preventable disease outbreaks (11). Also, social media has allowed amplifying and spreading of antivaccination messages (12). Therefore, the contents and formats of websites giving vaccine information must be planned and tested carefully. Our study aimed to analyze the content and design of influenza vaccine-related websites.

Materials and Methods

Ankara University Faculty of Medicine Ethical Committee approved the study. This cross-sectional study was conducted using the most common search engine, Google Search, available in the Turkish language. The study was conducted between November 1, 2016, and December 1, 2016. We searched for 15 search terms related to influenza (H1N1, influenza vaccine, vaccine mandate, “who should be vaccinated”, “who should not be vaccinated”, elderly, childhood, malignancy, pregnancy, ingredients of the vaccine, vaccination at the pharmacy, the timing of vaccination, benefits of vaccination, harms of vaccine, vaccine marketing). The top ten links on the first page were included for each search term, and duplicated links were excluded from the study. The selected websites were evaluated according to a list of content and design criteria. Contents of websites on influenza vaccine were assessed according to key facts about influenza vaccination determined by CDC in 2016 (13). The information on the websites was graded from 1 to 3 according to the medical content (1: general information; 2: detailed information; 3: very detailed/elaborated information) (Table 1). Vaccine Safety Net (VSN), The Global Advisory Committee on Vaccine Safety (GACVS) initiative, developed four categories of criteria (credibility, content, accessibility and design) in order to assess information on vaccines publicly available on the Internet. World Health Organization (WHO) design standardization based on VSN criteria was used to detect the suitability of websites design (14). Presence of references, up-to-dateness, having vision/mission, presence of contact information, aesthetic/content professionalism, accessibility of websites were assessed based on this set of criteria. Sites were encoded as 1 or 0 depending on whether they met the evaluation criteria. 

Statistical analyses were performed by using Statistical Package for the Social Sciences (SPSS) software version 18.0. The results of descriptive statistics were reported as mean ± standard deviation for variables with normal distribution. Categorical data were reported as frequencies and percentages. The chi-square test was used to assess the significant association between categorical variables. Statistical significance was set at p < 0.05. 


Table 1. The significance level of content criteria of the website about influenza vaccination.

Figure 1. The distribution of website sources.

Figure 2. Average content points of different website categories.

Figure 3. The percentages of all websites meeting six design criteria.

Totally 150 links were obtained, 71 were evaluated after excluding duplication. Websites were categorized into academic, news, personal blog, hospital, portal sites and others (Figure 1). We applied 15 content criteria to determine the property of websites content (Table 1). The mean content criteria score of all selected websites was 11.5. The content points for each category are shown in Figure 2.  While academic websites received the highest content point with 22.5, news and hospital sites remained below average with 10.5 and 8.9 points, respectively. Less than fifty percent of all websites met criteria other than criteria 2, 4, and 6. Criteria 2 (vaccine should be repeated every year), 4 (risk groups should be vaccinated) and 6 (the recommended months for vaccination) were met by 53.5%, 74.7%, 59.2% of all websites, respectively. Criteria 1 (2.8%), 10 (5.6%), 11 (4.2%) were satisfied with the lowest rates in the websites.

Websites were evaluated according to WHO design criteria. Thirty-eight percent and 77.5% included aesthetic/content professionalism and contact information, respectively. References were found only in 14.1% of websites (Figure 3). Websites whose design criteria scores were 1, 2, 3 and 4 had 12, 11.4, 10.4 and 14 points for content criteria, respectively.  The websites with higher design criteria did not have higher content points.  


Influenza is an important viral respiratory disease with a significant burden on public health. Therefore, many interventions focus on the prevention of influenza. Preventive measures include vaccination, hand hygiene and surgical masks (15). The best method for controlling and preventing influenza disease is vaccination, reducing the incidence of the disease and its severity (16).

Vaccination is considered a milestone in public health because it significantly affects morbidity and mortality rates of infectious diseases (17). A wide range of factors can potentially influence the vaccination decision (18). Over the past few decades, vaccine hesitancy has become a significant threat to public health, resulting in outbreaks such as measles (19). The misinformation propagated by anti-vaccine groups and media augmented concerns about vaccines (20). Although many scientific data support the safety of the recommended vaccines, fighting against false information to persuade people with vaccine hesitation continues to be a major problem (19).

Most people search for health information online especially using search engines (21). However, there are concerns regarding the accuracy and reliability of the Internet besides it provides immediate and easily available information (22). Misleading and over-interpreted health news in social media can be a potential threat to public health. A study investigating the spread of fake medical news showed that the most misleading content was about vaccines (23). Anti-vaccine websites generally focus on the ineffectiveness and danger of vaccines with incorrect interpretations of scientific reports (21). People’s knowledge and attitude about influenza vaccination can be affected by the output of search engines (15). Inappropriate information is not only found on websites but also on other media platforms (24). For example, Frazier et al. showed that inaccurate information about hand hygiene was provided 76% from television, 53% from magazines, 12% from newspapers (25).

Seventy percent of studies concluded that the quality of health information on websites was an important problem (24). Significant differences can be observed between websites categories in terms of content (22). For example, scientific websites provided more appropriate information for idiopathic pulmonary fibrosis than news and personal blog websites (22).  Also, in our study, we showed that academic websites’ content points (22,5 points) were higher than news (10.5 points) and personal blogs (11.5 points). The scores of news websites, which constitute a significant part of the Internet, were below the average. There was not any website with total points based on content criteria.  

The websites with higher WHO design criteria did not have higher content points.  It was seen that websites meeting the WHO design criteria did not have appropriate content. Therefore, we conceived that the WHO design criteria did not warrant the appropriate information.  

Our study had some limitations. This research evaluated the content and design of vaccine-related websites in the Turkish language. Our results can differ from the average user’s results because Google Search uses the user’s location to personalize the results. Another limitation was the dynamic nature of the web. We limited our search to Turkish-language websites so that the results would differ in other languages. 

Websites of professional organizations such as The European Society for Clinical Microbiology and Infectious Disease (ESCMID), Infectious Diseases Society of America (IDSA), Turkish Clinical Microbiology and Infectious Disease Society (KLIMIK) are some of the most trustworthy information sources about this subject. However, the websites of these societies were not found among the first ten links during our search, and we could not evaluate their websites regarding design and content. Therefore, the reliability of these sources requires further investigation. 

The Internet remains a largely unregulated entity, so it is important to follow up on the impact of health-related information on public health (15). Several companies have developed different criteria lists for guiding and evaluating health-related websites’ content, but these criteria have not been applied routinely. Instead, these systems rely on voluntary self-assessment by web page developers, the acceptability and reliability of most of them is unknown (10). 

In conclusion, despite the varying quality, the Internet will remain a preferred resource for patients given the easy access to health-related topics. The ongoing use of the Internet necessitates a strategy to increase the accuracy of online information, highlighting and facilitating access to websites with appropriate content (22). We demonstrated that most cited network resources did not contain appropriate and satisfactory information about influenza vaccination.

Ethical Approval: Ankara University Faculty of Medicine Ethical Committee approved the study on December 23, 2016, with decision number 7525.

Informed Consent: N.A.

Peer-review: Externally peer-reviewed

Author Contributions: Concept – A.A., İ.M.T., A.G., C.K., K.Y., P.Ö., İ.A.Ç., E.Y., E.M.S.; Design – A.A., C.K., İ.A.Ç., E.Y., İ.M.T, K.Y., A.G., P.Ö; Supervision – A.A., E.M.S.; Fundings – A.A.; Materials – A.A., K.Y., İ.M.T, C.K., P.Ö., A.G; Data Collection and/or Processing – İ.M.T., A.G., C.K., K.Y., P.Ö., E.Y., İ.A.Ç.; Analysis and/or Interpretation – A.A., E.M.S., C.K.; Literature Review – A.A., E.M.S.; Writer –; A.A., E.M.S.; Critical Reviews – A.A., E.M.S.

Conflict of Interest: The authors declare no conflict of interest.

Financial Disclosure: The authors declared that this study has received no financial support. 

Show References


  1. Gerdil C. The annual production cycle for influenza vaccine. Vaccine. 2003;21(16):1776-9. [CrossRef]
  2. Couch RB. Prevention and treatment of influenza. N Engl J Med. 2000;343(24):1778-87. [CrossRef]
  3. GBD 2016 Lower Respiratory Infections Collaborators. Estimates of the global, regional, and national morbidity, mortality, and aetiologies of lower respiratory infections in 195 countries, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Infect Dis. 2018;18(11):1191-210.
  4. Global influenza strategy 2019-2030 [Internet]. World Health Organization; 2019. Licence: CC BY-NC-SA 3.0 IGO. (cited August 01, 2021). Available from: http://apps.who.int/iris.
  5. Simonsen L, Taylor RJ, Viboud C, Miller MA, Jackson LA. Mortality benefits of influenza vaccination in elderly people: an ongoing controversy. Lancet Infect Dis. 2007;7(10):658-66. [CrossRef]
  6. Keske Ş, Mutters NT, Tsioutis C, Ergönül Ö. Influenza vaccination among infection control teams: A EUCIC survey prior to COVID-19 pandemic. Vaccine. 2020;38(52):8357-61. [CrossRef]
  7. Mendi B. Sağlık iletişiminde sosyal medyanın kullanımı: Dünyadaki ve Türkiye’deki uygulamalar. Öneri Dergisi. 2015;11(44):275-90. Turkish. [CrossRef]
  8. Information and Communication Technology (ICT) Usage Survey on Households and Individuals, 2019 [Internet]. Turkish Statistics Institute. (August 27, 2019; cited August 01, 2021). Available from: https://data.tuik.gov.tr/Bulten/Index?p=Survey-on-Information-and-Communication-Technology-(ICT)-Usage-in-Households-and-by-Individuals-2019-30574
  9. Covolo L, Mascaretti S, Caruana A, Orizio G, Caimi L, Gelatti U. How has the flu virus infected the Web? 2010 influenza and vaccine information available on the Internet. BMC Public Health. 2013;13:83. [CrossRef]
  10. Berland GK, Elliott MN, Morales LS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001;285(20):2612-21. [CrossRef]
  11. Larson HJ, Jarrett C, Schulz WS, et al. Measuring vaccine hesitancy: the development of a survey tool. Vaccine. 2015;33(34):4165-75. [CrossRef]
  12. Burki T. Vaccine misinformation and social media. The Lancet Digital Health. 2019;1(6):e258-e9. [CrossRef]
  13. Key facts about seasonal flu vaccine [Internet]. Centers for Disease Control and Prevention. (updated November 18, 2021; cited August 01, 2021). Available from: https://www.cdc.gov/flu/prevent/keyfacts.htm
  14. . Criteria for assessing websites with vaccine safety content [Internet]. World Health Organization. (January 23, 2015; cited August 01, 2021). Available from: https://www.who.int/groups/global-advisory-committee-on-vaccine-safety/topics/vaccine-safety-information/criteria
  15. Maki A, Evans R, Ghezzi P. Bad news: analysis of the quality of information on influenza prevention returned by Google in English and Italian. Frontiers in immunology. 2015;6:616. [CrossRef]
  16. Houser K, Subbarao K. Influenza vaccines: challenges and solutions. Cell Host microbe. 2015;17(3):295-300. [CrossRef]
  17. Dubé E, Laberge C, Guay M, Bramadat P, Roy R, Bettinger JA. Vaccine hesitancy: an overview. Hum Vaccin Immunother. 2013;9(8):1763-73. [CrossRef]
  18. MacDonald NE; SAGE Working Group on Vaccine Hesitancy. Vaccine hesitancy: Definition, scope and determinants. Vaccine. 2015;33(34):4161-4. [CrossRef]
  19. Geoghegan S, O’Callaghan KP, Offit PA. Vaccine safety: myths and misinformation. Front Microbiol. 2020;11:372. [CrossRef]
  20. Smith MJ, Woods CR. On-time vaccine receipt in the first year does not adversely affect neuropsychological outcomes. Pediatrics. 2010;125(6):1134-41. [CrossRef]
  21. Madden K, Nan X, Briones R, Waks L. Sorting through search results: a content analysis of HPV vaccine information online. Vaccine. 2012;30(25):3741-6. [CrossRef]
  22. Fisher JH, O’Connor D, Flexman AM, Shapera S, Ryerson CJ. Accuracy and reliability of internet resources for information on idiopathic pulmonary fibrosis. Am J Respir Crit Care Med. 2016;194(2):218-25. [CrossRef]
  23. Waszak PM, Kasprzycka-Waszak W, Kubanek A. The spread of medical fake news in social media–The pilot quantitative study. Health policy and technology. 2018;7(2):115-8. [CrossRef]
  24. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA. 2002;287(20):2691-700. [CrossRef]
  25. Frazier PJ, Jenny J, Ostman R, Frenick C. Quality of information in mass media: a barrier to the dental health education of the public. J Public Health Dent. 1974;34(4):244-57. [CrossRef]