No, I cannot rank this student among their class (I have not taught classes at this institution!) & I cannot compare them as "2nd in 10"or similar among the master students I supervised: I supervised folks at three different institutions & systems in two different countries plus every one has their strength and weaknesses, there is no order from best to worst.
My reference letter will tell you nothing and you are wasting everyone's time because of your stupid form 😬
@vicgrinberg 👆
This.
Template for reference letters asking for such percentile are silly in many ways, and the one that bugs me the most is the statistical one: unless you have supervised more than 20 students, you cannot rank even one of them in the ("top 5%") etc. If you supervised 5 the best you can say is "top 20%", which however would look bad on this scale.
Whenever I could, I explicitly wrote ref. letters saying that I think that system is silly and I provided better evaluations!
@franco_vazza Worse yet, presumably they're really asking about percentile within some larger parent distribution (e.g. students with this degree/specialty), so even with 20 students you probably can't say this particular student is in the top 5% of the parent distribution with much confidence (unless you assume the parent distribution has a known functional form determined by a small number of parameters).
On the other hand, in a Bayesian mindset you could assume that recommenders already have (from being a student and other interactions with students) a prior distribution, and advising students just allows them to refine their posterior distribution. Of course, even under this assumption, then they should logically ask you some questions to determine how well informed your posterior distribution is.
Regardless of all that, I think @vicgrinberg hit the key objection, that the potential of a student isn't a one-dimensional distribution, so such a ranking doesn't even make sense. Also, i don't actually buy that people have an accurate prior for the distribution of student capabilities, and I think that all such fuzzy judgements are highly subject to unconscious bias.
Not that I have a magic prescription for how to evaluate people, but I do think that stating things in such mathematical terms probably just serves to mislead by creating the illusion of objectivity and precision.
@franco_vazza @vicgrinberg Yeah, I think a multidimensional space probably has a bit more correspondence to reality in terms of describing someone's skills, but now you'd need even more information to know what the percentiles look like in all those dimensions.
Someone I know was just filling out one of these, and one notable flaw was the ability to say basically "I think this person would be good on this dimension but this wasn't relevant to our past interactions, so I don't have a lot to go on." I guess the overarching theme here is that this is complex and doesn't make sense to try to shoehorn into a very prescriptive structure.
@internic @vicgrinberg
" but I do think that stating things in such mathematical terms probably just serves to mislead by creating the illusion of objectivity and precision."
yes indeed. And there is usually not just one percentile to use, but several; so an illusion of "multidimensionality" in the evaluation is there...but it's all flawed.