There’s that quote (usually attributed to a 1970s IBM presentation), “A computer can never be held accountable, therefore a computer must never make a management decision.”
Isn’t that the precise opposite of our lived experience of management though? Doesn’t the management consulting industry exist, paid billions primarily to relieve managers of accountability for their own decisions? Wouldn’t purchasing deniability by computerization instead be a tremendous cost savings?
@interfluidity That a computer cannot be held accountable does not mean it negates accountability... it just shifts it.
Those consultants relieve management of accountability by laying the blame on someone else (usually line workers).
If the computers screw up, then the one that needs to be held accountable is whoever made the decision to install them.
So, in the end, computerization would *increase* the liability of those on top, not decrease it.
@interfluidity Are you sure about that?
I don't think the EEOC would take "it was my computers that refused to hire black people and women" as an excuse.
@LouisIngenthron yes, i’m sure about it. in racial discrimination law, there’s unusually the idea of liability for disparate impact, as opposed to discriminatory intent, so you might have a chance. but that’s the exception, not the rule. 1/
@LouisIngenthron before the mcdonalds lady could famously win a settlement for getting burned by coffee, she had to present extraordinary evidence that mcdonalds was superheating water in ways likely to produce unusual burns and that mcdonalds had experienced this before. if it had been an unforeseeable outcome of an ordinary business process (as the public was misled to believe), there would have been no liability. 2/
@LouisIngenthron there are already large industries building tools to predict inmate recidivism, or purporting to measure teacher’s “value add” by black-box algorithms independent analysts are deeply skeptical of. (see Cathy O’Neill’s Weapons of Math Destruction.) people spend years in jail or lose their jobs because of these tools. no one is accountable for their processes. 3/
@LouisIngenthron when a self-driving car maker or operator injures or kills in ways that are deemed the “car’s” error, firms might be fined, but a fine as @pluralistic puts it is a price, just a cost of doing business. the level of accountability we apply to decisionmakers at these firms is incredibly attenuated compared to the level of accountability’s — often criminal — that we impose for similar infractions by human drivers. 4/
@LouisIngenthron @pluralistic we do this on purpose, pretty knowingly. the argument is “if we imposed strict liability on decisionmakers at these industries, so their liability would be comparable to that of an unmediated individual making the same choice and causing the same harm, we would strangle all progress, because firms and managers simply would not take the necessary risks.” and that’s not ridiculous! maybe we accept that some eggs need to be broken to make our solar punk omelette. 5/
@LouisIngenthron @pluralistic but we should at least acknowledge it for what it is, an informaly sociaized subsidy to those businesses in the form of tolerating means of neutralizing accountability that more traditional actors would have faced. if you pretend there’s nothing new here, just the same accountability shifted to different relevant decisionmakers, you are mistaken. /fin
@LouisIngenthron that would be an affirmative normative and ultimately legal choice that we could make, and should make. but we haven’t yet. we have not clearly stated that an identified party must accept responsibility and strict liability for outcomes that were arguably “unforeseeable” as a consequence of delegating decisionmaking to some automated system. we should do that, but we have not. yet.