Certain Doubts

[Update: The CENTRIA servers are down for maintenance. The links here should be back on 30 December.]

This link is to an excel spreadsheet of h- and g-scores of 75 philosophy journals that were recently classified by the European Science Foundation (ESF).

Some notes on interpretation. The ESF list classifies journals by grade: an A is better than a B, and B is better than a C. I’ve coded this as A=1, B=2, C=3. The list here is far from complete and comes with the caveats and warnings that have accrued in the last month. In addition, there are two things to keep in mind: the mean and median scores should be taken with a grain of salt, since they are for this list only. However, it is not likely that these numbers will be much greater on a more comprehensive list: nearly all of the major journals are listed here, so a more comprehensive list would (I think) only reveal a greater distance between scores for top journals from the mean and median. A graph of the spread for this list is included.

Another point that comes across clearly is that different fields have different citation practices: historians are far more reluctant than linguists to offer up a citation, and all of us pale in comparison to biochemists.

Finally, thanks much to Jon Kvanvig for crunching the data. (If anyone knows of a port for Publish or Perish to Mac OS X, I’d be keen to find out about it.) This exercise has helped me prepare for my center’s review next month. In our scheme, weight is given to publishing in “high-impact” journals, particularly for junior faculty. Perhaps this information will be useful to others as well.

Below the fold are summary tables for the top 25 journals.

(more…)

Here are some ideas about how philosophy programs might be evaluated more effectively than they now are. Right now, the most widely used evaluations are those of the Philosophical Gourmet Report. (Disclosure: I’m on the Advisory Board of the PGR, and have been and am a supporter and defender of that project. For my suggestions as to how a prospective student might best use the PGR in deciding which programs to apply to and attend, see this post.) My recommendations here will be presented in the form of potential modifications (what I would take to be improvements) of the PGR. But they could instead be implemented by some other evaluating project.

Below the Fold:
1. A Two-Stage Process
2. Informed Surveys
3. Citation Counts?
(more…)

Lots of emails from people at unrated programs, so I finally caved and gathered data from all programs. Info below the fold, but I want to reiterate the shortcomings of the data and the significance of the exercise once more. From what I know, it seems that no citation source is better than 25-30% accurate, and that the various sources (GS, ISI, Web of Science, Scopus) produce lists that have, at best, very weak correlations with each other. What makes the exercise interesting is that when the various sources are used for ordinal rankings, the correlations between the various sources is very high: over .9. Now, anyone mildly sophisticated about evidential import will not view this as strong evidence of reliability, much as we don’t take a sample of 5 positives as sufficient evidence to conclude anything much about the majority of a population of fairly substantial size. Administrators, however, must latch onto something, and these measures are already being used in other disciplines. So, it is a relatively safe prediction that they are going to be used on philosophy departments as well. It is thus worth seeing what they produce and what can be made of the data. So here goes…

(more…)

One might wonder what Hirsch numbers show about departments that are not Leiter-rated. I did, because I hear lots of howling about certain departments being excluded from that report, so I sampled some. Results below the fold, but first some foreshadowing: for the most part, as expected, Leiter wins…
(more…)

I’ve added one more page to the Page box in the right sidebar, a page using the data originally reported here (with further discussion of the value of the data here and here) to compare the rankings of departments from last year to this year, in light of the changes in faculty reported at Leiter’s blog here. The new page is labelled “Department Ranking Changes for 2007-8″, and it gives side-by-side comparisons of the measures used in this page, where the original data was presented.

As with the original data, the cautions urged in the posts linked above need to be taken seriously when assessing the significance of this data.

A few years ago, I published a paper in which I not only cited, but also discussed at considerable length, an epistemology paper by Robert Hambourger. I wrote that Hambourger’s paper was “fascinating, but largely overlooked.” Several months ago, I was working on revisions to that paper of mine (for inclusion in a book I’m working on). By then I had been alerted, by this post by Jason Stanley over at Leiter’s blog, of the possibility of “Using Google Scholar to Assess the Impact of Philosophical Work.” So I thought I’d use Google Scholar to check just how overlooked Hambourger’s paper (still) is. “I wonder who, other than me, has cited Hambourger’s paper?” I thought. The results? Well, as you can see, Google Scholar reports than 9 works have cited Hambourger’s paper, which isn’t bad. (And, hey, that’s just now gone up to 10.)

But it doesn’t count me as having cited it. Why not? Does anybody know? Some potentially relevant info:
(more…)

3/15–Moving to the top since the meeting is this week; hope to see you there!

(more…)

Here are some notes relevant to the table given in my last post.
(more…)

UPDATE: Because of high interest level, I’m moving this to the top. See also here for notes about the table below.

SECOND UPDATE: I’ve created a page with just the ranking tables, both the ones in this post and another one from the comments, for those who just want to see the results of the exercise. You can click on the “Department Hirsch Number Rankings” in the Pages box in the right column, or just click here.

I’ve used the Hirsch number to gather more data, this time on Leiter-rated departments, using the faculty lists used by those who ranked departments for Leiter last time. I’ll put the data below the fold, but first a word or two about the numbers. First, I think there are lots of problems with the data at present, but it is becoming clear to me that administrators will want such information as the methods of generating it become easier to use and more reliable. So it is worth knowing how philosophy departments fare on this score. There is also an advantage here, since a department that is slipping in the rankings might use such information as part of a case for new appointments; moreover, addressing such concerns in a department will require hiring senior and productive scholars, which is good for the profession as well. (A dominant concern of mine has been the immense power educational institutions have over faculty, and data that leads to greater mobility for senior faculty will force institutions to pay for quality departments.) There is an added benefit as well for departments not rated by the Leiter report: such departments can compare data for their own department to see how they fare compared to the departments that are ranked.

The data at present are not fully reliable, and I caution in advance about the problems. As before, I used Harzing’s Publish or Perish, which searches Google Scholar for the data to generate a Hirsch number for each scholar. Google Scholar has been subjected to some scrutiny by social scientists and the problems with it are discussed here, and Keith’s worries here are certainly legitimate and worth considering as well. The current wisdom seems to be that GS is far from accurate, but not inaccurate enough to render the data entirely without probative value.

There are also known issues with the h-index itself, as discussed here. It is, however, a measure that is growing in popularity and, I expect, will play an increasing role in administrative thinking about departments.

So, with all the limitations and caveats about the data, here are the results.

(more…)

In case you missed it, Ellen DeGeneres just visited Peter Kosso’s Epistemology Class at Northern Arizona. She wanted to know what justifies a belief, which was right on topic, and then proceeded to try to turn it into something else. Oh, well, great publicity for epistemology, with Plato referred to on the blackboard as well as Quine & Ullian. Nice job, PK!