Jump to main content or area navigation.

Contact Us

Extramural Research

Bibliometrics

Research Project Search

Extramural Research Search

Bibliometric Analysis for the U.S. Environmental Protection Agency/Office of Research and Developments Water Quality Research Program
September 2008

This is a bibliometric analysis of the papers prepared by researchers of the U.S. Environmental Protection Agency (EPA) for the Water Quality Research Program. For this analysis, a total of 867 journal and 109 non-journal publications published from 1998 to 2008 were reviewed. The 867 journal publications were cited 8,697 times and the 109 non-journal publications were cited 279 times in the journals covered by Thomsons Web of Science1 and Scopus2 . Of the 867 journal publications, 738 (85.1%) have been cited at least once in a journal. Of the 109 non-journal publications, 50 (45.9%) have been cited at least once in a journal.

Searches of Thomson Scientifics Web of Science and Scopus were conducted to obtain times cited data for the journal publications. The analysis was completed using Thomsons Essential Science Indicators (ESI) and Journal Citation Reports (JCR) as benchmarks. ESI provides access to a unique and comprehensive compilation of essential science performance statistics and science trends data derived from Thomsons databases. For this analysis, the ESI highly cited papers thresholds as well as the hot papers thresholds were used to assess the influence and impact of the Water Quality publications. JCR is a recognized authority for evaluating journals. It presents quantifiable statistical data that provide a systematic, objective way to evaluate the worlds leading journals and their impact and influence in the global research community. The two key measures used in this analysis to assess the journals in which the EPA Water Quality papers were published are the Impact Factor and Immediacy Index. The Impact Factor is a measure of the frequency with which the average article in a journal has been cited in a particular year. The Impact Factor helps evaluate a journals relative importance, especially when compared to other journals in the same field. The Immediacy Index is a measure of how quickly the average article in a journal is cited. This index indicates how often articles published in a journal are cited within the same year and it is useful in comparing how quickly journals are cited.

The report includes a summary of the results of the bibliometric analysis, an assessment of the 867 Water Quality journal articles analyzed by ESI field (e.g., Chemistry, Environment/Ecology, Microbiology), an analysis of the journals in which the Water Quality papers were published, a table of the highly cited researchers in the Water Quality Research Program, information on the patents/patent applications that have resulted from the program, and an assessment of the 109 non-journal publications.

Summary of Results

  1. More than one-seventh of the 867 Water Quality journal publications are highly cited papers. 132 (15.2%) of the 867 Water Quality journal publications qualify as highly cited when using the ESI criteria for the top 10% of highly cited publications. This is 1.5 times the number expected. 9 (1.0%) of the Water Quality journal papers qualify as highly cited when using the ESI criteria for the top 1%, which is the number expected. 1 (0.1%) of the Water Quality publications qualifies as very highly cited when using the ESI criteria for the top 0.1% of highly cited publications, which is the number expected. 1 (0.1%) of the Water Quality journal papers qualifies as extremely highly cited when using the criteria for the top 0.01% threshold for the most highly cited papers, which is 10 times the number expected.
  2. The Water Quality journal publications are more highly cited than the average paper. Using the ESI average citation rates for papers published by field as the benchmark, in 9 of the 15 fields in which the 867 Water Quality journal papers were published, the ratio of actual to expected cites is greater than 1, indicating that the Water Quality journal publications are more highly cited than the average papers in those fields. For all 15 fields combined, the ratio of total number of cites to the total number of expected cites (8,697 to 6,826.3) is 1.3, indicating that the Water Quality journal papers are more highly cited than the average paper.
  3. Nearly one-seventh of the Water Quality journal papers are published in high impact journals. 120 of the 867 journal papers were published in the top 10% of journals ranked by JCR Impact Factor, representing 13.8% of the Water Quality journal publications. This number is 1.4 times higher than expected. 103 of the 867 papers appear in the top 10% of journals ranked by JCR Immediacy Index, representing 11.9% of EPAs Water Quality journal publications. This number is 1.2 times higher than expected.
  4. There were 4 hot papers among the 867 Water Quality publications. Using the hot paper thresholds established by ESI as a benchmark, 4 (0.5%) hot papers were identified in the analysis. This number is five times the number expected. Hot papers are publications that are highly cited shortly after they are published.
  5. The authors of the Water Quality journal publications cite themselves much less than the average author. 482 of the 8,697 total cites are author self-cites. This 5.5% author self-citation rate is well below the accepted range of 10-30% author self-citation rate.
  6. 25 of the 1,716 authors of the Water Quality journal publications are included in ISIHighlyCited.com,which is a database of the worlds most influential researchers who have made key contributions to science and technology during the period from 1981 to 1999.
  7. 1 patent was issued to investigators from 1998 to 2008 for research that was conducted under EPAs Water Quality Research Program. The patent was not cited by any other patents.
  8. The 109 non-journal publications were cited 279 times in journals and there were no author self cites. When applying the ESI benchmark for journal publications to these 109 non-journal publications, 2 (1.8%) of the publications are highly cited when using the ESI criteria for the top 10% of papers. None of the non-journal publications meet the criteria for highly cited when using the ESI thresholds for the top 1%, 0.1%, or 0.01%.

Highly Cited Water Quality Publications

All of the journals covered by ESI are assigned a field, and to compensate for varying citation rates across scientific fields, different thresholds are applied to each field. Thresholds are set to select highly cited papers to be listed in ESI. Different thresholds are set for both field and year of publication. Setting different thresholds for each year allows comparable representation for older and younger papers for each field.

The 867 Water Quality journal publications reviewed for this analysis were published in journals that were assigned to 15 of the 22 ESI fields. The distribution of the papers among these 15 fields and the number of citations by field are presented in Table 1.

Table 1. Water Quality Journal Publications by ESI Fields

ESI Field
No. of Citations
No. of EPA WQ Papers
Average Cites/Paper

Agricultural Sciences

25

6

4.2

Biology & Biochemistry

137

16

8.6

Chemistry

265

29

9.1

Clinical Medicine

211

13

16.2

Computer Science

1

3

0.3

Engineering

98

21

4.7

Environment/Ecology

4,492

439

10.2

Geosciences

231

18

12.8

Microbiology

467

50

9.3

Molecular Biology & Genetics

0

1

0

Multidisciplinary

147

4

36.8

Pharmacology & Toxicology

162

7

23.1

Physics

12

1

12.0

Plant & Animal Science

2,432

254

9.6

Social Sciences

17

5

3.4

Total = 15

Total = 8,697

Total = 867

10.0

There are 132 (15.2% of the 867 journal papers analyzed) highly cited Water Quality journal publications in 11 of the 15 fieldsAgricultural Sciences, Chemistry, Clinical Medicine, Engineering, Environment/Ecology, Geosciences, Microbiology, Multidisciplinary, Pharmacology & Toxicology, Plant & Animal Science, and Social Scienceswhen using the ESI criteria for the top 10% of papers. Table 2 shows the number of Water Quality journal publications in those 11 fields that meet the top 10% threshold in ESI.

Table 2. Number of Highly Cited Water Quality Journal Publications by Field (top 10%)

ESI Field
No. of Citations
No. of Papers
Average Cites/Paper
% of WQ Papers in Field

Agricultural Sciences

19

1

19.0

16.7%

Chemistry

92

3

30.7

10.3%

Clinical Medicine

98

1

98.0

7.7%

Engineering

45

5

9.0

23.8%

Environment/Ecology

2,082

51

40.8

11.6%

Geosciences

76

2

38.0

11.1%

Microbiology

253

7

36.1

14.3%

Multidisciplinary

147

4

36.8

100.0%

Pharmacology& Toxicology

134

2

67.0

28.6%

Plant & Animal Science

1,429

55

26.0

21.6%

Social Sciences

4

1

4.0

20.0%

TOTALS

Total = 4,379

Total = 132

33.2

15.3%

Nine (1.0%) of the journal publications analyzed qualify as highly cited when using the ESI criteria for the top 1% of papers. This is the number of papers expected to meet this threshold. These nine publications are in two of the ESI fieldsEnvironment/Ecology and Plant & Animal Science (see Table 3). The citations for these papers are provided in Tables 4 and 5. One (0.1%) of the Water Quality journal publications meets the top 0.1% ESI threshold for highly cited papers, which is the number expected to meet this threshold. This publication is listed in Table 6. One (0.1%) of the Water Quality journal publications actually meets the top 0.01% threshold in ESI, which is 10 times the expected number of publications to meet this threshold for this program.

Table 3. Number of Highly Cited Water Quality Journal Publications by Field (top 1%)


ESI Field

No. of Citations

No. of Papers

Average Cites/Paper

% of WQ Papers in Field

Environment/Ecology

957

8

119.6

1.8%

Plant & Animal Science

8

1

8.0

0.4%

TOTALS

Total = 965

Total = 9

107.2

1.0%

Table 4. Highly Cited Water Quality Journal Publications in the Field of Environment/Ecology (top 1%)


No. of Cites

First Author

Paper

141

Paerl HW

Ecosystem responses to internal and watershed organic matter loading: consequences for hypoxia in the eutrophying Neuse river estuary, North Carolina, USA. Marine Ecology-Progress Series 1998;166:17-25.

616

Daughton CJ

Pharmaceuticals and personal care products in the environment: agents of subtle change? Environmental Health Perspectives 1999;107(Suppl 6):907-938.

103

Simpson JM

Microbial source tracking: state of the science. Environmental Science & Technology 2002;36(24):5279-5288.

40

Kemp WM

Eutrophication of Chesapeake Bay: historical trends and ecological interactions. Marine Ecology-Progress Series 2005;303:1-29.

25

Stoddard JL

Setting expectations for the ecological condition of streams: the concept of reference condition. Ecological Applications 2006;16(4):1267-1276.

8

Lackey RT

Science, scientists, and policy advocacy. Conservation Biology 2007;21(1):12-17.

10

Peterson SA

Mercury concentration in fish from streams and rivers throughout the western united states. Environmental Science & Technology 2007;41(1):58-65.

14

Danz NP

Integrated measures of anthropogenic stress in the US Great Lakes basin. Environmental Management 2007;39(5):631-647.

Table 5. Highly Cited Water Quality Journal Publications in the Field of Plant & Animal Science (top 1%)


No. of Cites

First Author

Paper

8

Litaker RW

Recognizing dinoflagellate species using ITS rDNA sequences. Journal of Phycology 2007;43(2):344-355.

Table 6. Very Highly Cited Water Quality Journal Publications (top 0.1%)


No. of Cites

First Author

Paper

616

Daughton CJ

Pharmaceuticals and personal care products in the environment: agents of subtle change? Environmental Health Perspectives 1999;107(Suppl 6):907-938.

Table 7. Extremely Highly Cited Water Quality Journal Publications (top 0.01%)


No. of Cites

First Author

Paper

616

Daughton CJ

Pharmaceuticals and personal care products in the environment: agents of subtle change? Environmental Health Perspectives 1999;107(Suppl 6):907-938.

Ratio of Actual Cites to Expected Citation Rates

The expected citation rate is the average number of cites that a paper published in the same journal in the same year and of the same document type (article, review, editorial, etc.) has received from the year of publication to the present. Using the ESI average citation rates for papers published by field as the benchmark, in 9 of the 15 fields in which the EPA Water Quality journal papers were published, the ratio of actual to expected cites is greater than 1, indicating that the Water Quality journal publications are more highly cited than the average papers in those fields (see Table 8). For all 15 fields combined, the ratio of total number of cites to the total number of expected cites (8,697 to 6,826.3) is 1.3, indicating that the Water Quality journal publications are more highly cited than the average paper.

Table 8. Ratio of Actual Cites to Expected Cites for Water Quality Journal Publications by Field


ESI Field

Total Cites

Expected Cite Rate

Ratio

Agricultural Sciences

25

34.6

0.7

Biology & Biochemistry

137

276.9

0.5

Chemistry

265

253.4

>1.0

Clinical Medicine

211

224.3

0.9

Computer Science

1

2.5

0.4

Engineering

98

70.0

1.4

Environment/Ecology

4,492

3,719.7

1.2

Geosciences

231

139.6

1.6

Microbiology

467

449.6

>1.0

Molecular Biology & Genetics

0

35.6

0

Multidisciplinary

147

21.6

6.8

Pharmacology & Toxicology

162

85.5

1.9

Physics

12

8.1

1.5

Plant & Animal Science

2,432

1,487.0

1.6

Social Sciences

17

17.9

<1.0

TOTAL

8,697

6,826.3

1.3

JCR Benchmarks

Impact Factor. The JCR Impact Factor is a well known metric in citation analysis. It is a measure of the frequency with which the average article in a journal has been cited in a particular year. The Impact Factor helps evaluate a journals relative importance, especially when compared to others in the same field. The Impact Factor is calculated by dividing the number of citations in the current year to articles published in the 2 previous years by the total number of articles published in the 2 previous years.

Table 9 indicates the number of Water Quality journal publications published in the top 10% of journals, based on the JCR Impact Factor. One hundred twenty (120) of 867 journal papers were published in the top 10% of journals, representing 13.8% of EPAs Water Quality journal publications. This indicates that about one-seventh of the Water Quality journal publications are published in the highest quality journals as determined by the JCR Impact Factor, which is 1.4 times higher than the expected percentage.

Table 9. Water Quality Journal Publications in Top 10% of Journals by JCR Impact Factor


EPA WQ Papers in that Journal

Journal

Impact Factor
(IF)

JCR IF Rank

1

Nature

28.751

10

2

Lancet

28.638

11

1

Gastroenterology

11.673

81

2

Proceedings of the National Academy of Sciences of the United States of America

9.598

114

1

Journal of the American Chemical Society

7.885

156

1

Current Opinion in Microbiology

7.654

167

3

TRAC-Trends in Analytical Chemistry

5.827

261

11

Environmental Health Perspectives

5.636

279

5

Analytical Chemistry

5.287

309

1

Progress in Nuclear Magnetic Resonance Spectroscopy

5.205

324

1

Molecular Ecology

5.169

326

4

Ecology

4.822

370

3

Critical Reviews in Environmental Science and Technology

4.615

414

1

American Naturalist

4.543

423

19

Environmental Science & Technology

4.363

465

4

Frontiers in Ecology and the Environment

4.269

493

1

Proceedings of the Royal Society of London Series B-Biological Sciences

4.112

535

4

Bioscience

4.083

543

10

Applied and Environmental Microbiology

4.004

571

1

Journal of Organic Chemistry

3.959

586

2

Conservation Biology

3.934

587

4

Toxicological Sciences

3.814

622

1

Geochimica et Cosmochimica Acta

3.665

685

1

Journal of the American Society for Mass Spectrometry

3.664

686

6

Journal of Chromatography A

3.641

695

2

Electrophoresis

3.609

710

8

Ecological Applications

3.571

721

9

Water Research

3.427

777

1

Talanta

3.374

800

9

Limnology and Oceanography

3.277

836

1

Chemical Geology

3.231

859

Total = 120

 

 

Immediacy Index. The JCR Immediacy Index is a measure of how quickly the average article in a journal is cited. It indicates how often articles published in a journal are cited within the year they are published. The Immediacy Index is calculated by dividing the number of citations to articles published in a given year by the number of articles published in that year.

Table 10 indicates the number of Water Quality journal publications published in the top 10% of journals, based on the JCR Immediacy Index. One hundred three (103) of the 867 papers appear in the top 10% of journals, representing 11.9% of the Water Quality journal papers. This indicates that nearly one-eighth of the Water Quality journal papers are published in the highest quality journals as determined by the JCR Immediacy Index, which is 1.2 times higher than the expected percentage.

Table 10. Water Quality Journal Publications in Top 10% of Journals by JCR Immediacy Index


EPA WQ Papers in that Journal

Journal

Immediacy Index
(II)

JCR II Rank

2

Lancet

8.636

6

1

Nature

7.385

9

1

Gastroenterology

2.595

74

2

Proceedings of the National Academy of Sciences of the United States of America

1.724

136

1

Fisheries Management and Ecology

1.586

162

1

Journal of the American Chemical Society

1.397

212

1

Pedobiologia

1.290

245

1

Progress in Nuclear Magnetic Resonance Spectroscopy

1.250

253

2

Botanica Marina

1.231

268

1

Current Opinion in Microbiology

1.121

310

2

Environmental Modelling & Software

0.976

410

11

Environmental Health Perspectives

0.958

425

1

Proceedings of the Royal Society of London Series B-Biological Sciences

0.955

428

1

Agriculture Ecosystems & Environment

0.915

463

1

American Naturalist

0.914

465

5

Analytical Chemistry

0.911

471

4

Frontiers in Ecology and the Environment

0.907

476

1

Journal of Organic Chemistry

0.886

498

3

TRAC-Trends in Analytical Chemistry

0.863

524

5

Continental Shelf Research

0.780

605

1

Ambio

0.777

610

1

Journal of Statistical Software

0.767

623

5

Journal of Environmental Monitoring

0.763

625

4

Bioscience

0.761

626

2

Conservation Biology

0.745

651

1

Molecular Ecology

0.732

663

3

Critical Reviews in Environmental Science and Technology

0.727

670

1

Geochimica et Cosmochimica Acta

0.719

685

1

Journal of the American Society for Mass Spectrometry

0.700

713

1

Environmental Pollution

0.699

716

1

Vadose Zone Journal

0.689

730

1

Diversity and Distributions

0.663

783

2

FEMS Microbiology Ecology

0.643

819

4

Environmental Research

0.632

845

19

Environmental Science & Technology

0.615

876

2

Hydrological Sciences Journal-Journal des Sciences Hydrologiques

0.613

881

6

Journal of Geophysical Research

0.613

881

1

Talanta

0.611

886

Total = 103

Hot Papers

ESI establishes citation thresholds for hot papers, which are selected from the highly cited papers in different fields, but the time frame for citing and cited papers is much shorterpapers must be cited within 2 years of publication and the citations must occur in a 2-month time period. Papers are assigned to 2-month periods and thresholds are set for each period and field to select 0.1% of papers.

Using the hot paper thresholds established by ESI as a benchmark, 4 hot papers, representing 0.5% of the Water Quality papers, were identified in three fieldsEngineering, Environment/Ecology, and Plant & Animal Science. The number of Water Quality hot papers is 5 times higher than expected. The hot papers are listed in Table 11.

Table 11. Hot Papers Identified Using ESI Thresholds


Field

ESI Hot Papers Threshold

No. of Cites in 2-Month Period

Paper

Engineering

5

6 cites in August-September 2005

Cardoza LA, et al. Separations coupled with NMR detection. TRAC-Trends in Analytical Chemistry 2003;22(10):766-775.

Environment/ Ecology

7

11 cites in March-April 2008

Danz NP, et al. Integrated measures of anthropogenic stress in the US Great Lakes basin. Environmental Management 2007;39(5):631-647.

Plant & Animal Science

4

5 cites in October 2001

Burkholder JM, et al. Overview and present status of the toxic Pfiesteria complex (Dinophyceae). Phycologia 2001;40(3):186-214.

4

10 cites in October 2001

Glasgow HB, et al. A second species of ichthotoxic Pfiesteria (Dinamoebales, Dinophyceae). Phycologia 2001;40(3):234-245.

Author Self-Citation

Self-citations are journal article references to articles from that same author (i.e., the first author). Because higher author self-citation rates can inflate the number of citations, the author self-citation rate was calculated for the Water Quality papers. Of the 8,697 total cites of the 867 journal publications, 482 are author self-citesa 5.5% author self-citation rate. Garfield and Sher3 found that authors working in research-based disciplines tend to cite themselves on the average of 20% of the time. MacRoberts and MacRoberts4 claim that approximately 10-30% of all the citations listed fall into the category of author self-citation. Kovacic and Misak5 reported a 20% author self-citation rate for medical literature. Therefore, the 5.5% self-cite rate for the Water Quality papers is well below the range for author self-citation.

Highly Cited Researchers

A search of Thomsons ISIHighlyCited.com revealed that 25 (1.5%) of the 1,716 authors of the Water Quality papers are highly cited researchers. ISIHighlyCited.com is a database of the worlds most influential researchers who have made key contributions to science and technology during the period from 1981 to 1999. The highly cited researchers identified during this analysis of the Water Quality publications are presented in Table 12.

Table 12. Highly Cited Researchers Authoring Water Quality Journal Publications


Highly Cited Researcher

Affiliation

ESI Field

Anderson, Donald M.

Woods Hole Oceanographic Institution

Plant & Animal Science

Ankley, Gerald

U.S. Environmental Protection Agency

Environment/Ecology

Birnbaum, Linda S.

U.S. Environmental Protection Agency

Pharmacology

Burger, Joanna

Rutgers, The State University of New Jersey

Environment/Ecology

Campana, Steven E.

Bedford Institute of Oceanography

Plant & Animal Science

Caron, David A.

University of Southern California

Plant & Animal Science

Cole, Jonathan J.

Institute of Ecosystem Studies

Plant & Animal Science

DiToro, Dominic M.

University of Delaware

Environment/Ecology

Eisenreich, Steven J.

Rutgers, The State University of New Jersey

Engineering
Environment/Ecology

Giger, Walter

Eawag, the Swiss Federal Institute of Aquatic Science and Technology

Environment/Ecology

Gray Jr., Leon Earl

U.S. Environmental Protection Agency

Pharmacology

Groffman, Peter M.

Cary Institute of Ecosystem Studies

Environment/Ecology

Gschwend, Philip M.

Massachusetts Institute of Technology

Engineering
Environment/Ecology

Guillette, Louis J.

University of Florida

Environment/Ecology

Hansen, Dave J.

Formerly of the U.S. Environmental Protection Agency

Environment/Ecology

Hopkins, Theodore L.

Kansas State University

Plant & Animal Science

Jones, Kevin C.

Lancaster University

Engineering
Environment/Ecology

Landrum, Peter F.

U.S. Department of Commerce

Environment/Ecology

Morse, John W.

Texas A&M University

Geosciences

ONeill, Robert V.

Oak Ridge National Laboratory

Environment/Ecology

Paerl, Hans W.

University of North Carolina

Plant & Animal Science

Peterson, Richard E.

University of Wisconsin

Pharmacology

Prospero, Joseph M.

University of Miami

Geosciences

Thurman, E. Michael

U.S. Geological Survey

Engineering
Environment/Ecology

Stoecker, Diane K.

University of Maryland Center for Environmental Science

Plant & Animal Science

Total = 25

Patents

There was one patent issued to investigators from 1998 to 2008 for research that was conducted under EPAs Water Quality Research Program. The patent was not cited by any other patents (see Table 13).

Table 13. Patents from the Water Quality Research Program (1998-2008)


Patent/Patent Application No.

Inventor(s)

Title

Patent Date

Patents that Referenced This Patent

U.S. Patent No. 6,655,402

Fan, C-Y

System and method for vacuum flushing sewer solids

12/2/03

Referenced by 0 patents

Non-Journal Publications (Reports, Books, and Book Chapters)

One hundred eleven reports, book chapters, and other non-journal publications produced by the Water Quality Research Program from 1999 to 2008 were included in the analysis. The 109 non-journal publications were cited 279 times in journals and there were no author self cites. When applying the ESI benchmark for journal publications to these 109 non-journal publications, 2 (1.8%) EPA publications (i.e., The Stressor Identification Technical Guidance Document, EPA/822/B00/025; and Microbial Source Tracking Guide Document, EPA/600/R-05/064) were highly cited when using the ESI criteria for the top 10% of papers. None of the non-journal publications met the criteria for highly cited when using the ESI thresholds for the top 1%, 0.1%, or 0.01%.

1 Thomson Scientifics Web of Science provides access to current and retrospective multidisciplinary information from approximately 8,830 of the most prestigious, high impact research journals in the world. Web of Science also provides cited reference searching.

2 Scopus is a large abstract and citation database of research literature and quality Web sources designed to support the literature research process. Scopus offers access to 15,000 titles from 4,000 different publishers, more than 12,850 academic journals (including coverage of 535 Open Access journals, 750 conference proceedings, and 600 trade publications), 27 million abstracts, 245 million references, 200 million scientific Web pages, and 13 million patent records.

3Garfield E, Sher IH. New factors in the evaluation of scientific literature through citation indexing. American Documentation 1963;18(July):195-210.

4 MacRoberts MH, MacRoberts BR. Problems of citation analysis: a critical review. Journal of the American Society of Information Science 1989;40(5):342-349.

5 Kovacic N, Misak A. Author self-citation in medical literature. Canadian Medical Association Journal 2004;170(13):1929-1930.

You will need Adobe Reader to view some of the files on this page. See EPA's PDF page to learn more.

Top of page

Jump to main content.