I certainly think two changes should be made to the rating:

1. the rating we have now should be kind of a "higher standard" (review like), which one should only use if one feels one can judge the paper on significance etc. -> and it should be possible to rate only one or two of those if one, for example, is not able to judge the methods of the paper. This rating should also try to be an "objective" measure on that paper - independent of ones one interests etc...

2. the second rating category should be rather based on a quick impression - and this should/could fully reflect one's own interest in the subject. It could be used then to later cluster people with similar interests / suggest papers that might be of interest etc. And I think this rating should still have 5 categories (or so) - but not just two: I realized it is pretty neat to have your very own journal club and throwing papers of interest in there - and it would be nice if I could rank those - just for myself (and it would be fine if this is also used to compute general rating for the public)

3. I think it should be possible to change one own rating. I know that might generate non-linearities given what one uses those ratings for - but if that is really an issue one could always base it only on the primary rating. But it would for instance facilitate point 2 also..

Currently, I often hesitate to rate a paper, because I can't judge on all three categories - and if I do it I want to do it in a right way...

best,

ingmar