Re: [projectvrm] Collectives and algorithms [was More from NYT]

[John Wunderlich < >]

I agree with the general point about it being a ‘bad thing’ for decision support systems to be closed algorithms. There’s a relatively new book out about this kind of thing, for those that are interested. Frank Pasquale, “The Black Box Society: The Secret Algorithms That Control Money and Information”, Harvard University Press, 2015, ISBN 978-0-674-36827-9. That being said, I’d like to parse this out a little. 

1. “Medicine, like law, or art rarely has a correct answer

But sometimes it does. This is why there are, for example, checklists in operating theatres and requirements around hand washing. Where the scientific evidence supports it, decision support software can make sense - with the proviso that the algorithm and the evidence used by the algorithm are open and peer reviewed. To the extent that medicine moves from art to science, the areas in which decision support systems improve health outcomes are likely to expand. This will be, I think, seriously disruptive. As a general commentary, whenever evidence based procedures are introduced the practitioners typically react with, “That makes sense so long as the system recognizes my expertise and allows me to override the system’s recommendations.” This makes sense until one realizes that statistically this statement is more often invalidated by the follow up data than otherwise. As a personal aside, I trained and worked, briefly, as an air traffic controller and one thing was made clear in the training about procedures in emergencies – 1) follow the procedures and don’t be inventive, 2) the pilot in command is the final authority. The corollary is that the pilot in command is also trained to follow their emergency procedures and checklists – no matter what their senses or instincts said. 

2. "Licensed profession agent of the individual”

The suggest that the agency rests with the individual, but the first point above is about the agency that resides with the physician. At the very least this is a tripartite relationship between the patient, the doctor, and the ominously named “collective”. It puts in play all kinds of power relationships including Ministries of Health (single payer countries) or HMO’s (U.S), medical practices guidelines, individual doctors’ ethical/cultural strictures and the patients’ wishes. It isn’t, nor should it be simple. What happens when the collective mandates empowering the individual against the beliefs of the doctor, for example? Transparency is needed here as well.

At the end of the day my bias is to empower the individual with the information that they need to make decisions about themselves and for themselves, but for those options to be bounded by their impact on others (whether expressed individually or collectively). 

JW

Sincerely,
John Wunderlich
@PrivacyCDN

Call: +1 (647) 669-4749
eMail: " target="_blank">

On 11 July 2015 at 18:18, Adrian Gropper < " target="_blank"> > wrote:

Explanations are not enough if nobody has the incentive to audit the explanation. This is the difference between "shared source" and "open source". Both can be audited in theory, but only the open source provides a meaningful incentive for the community to work with the source.

Our norms around intellectual property are going to need to adjust to the increasing importance of algorithms. I don't see this discussion anywhere much.

Adrian

On Saturday, July 11, 2015, Marc Lauritsen < " target="_blank"> > wrote:

Whether their algorithms include benign 'nudges' to bias us in directions that are probably good for us, or heavy thumbs on the scale that disserve individual autonomy, it seems reasonable to insist that decision support systems that purport to be fair include mechanisms for revealing their reasoning.  Transparency of rationale should be baked in wherever possible.  Rule-based systems can generally autogenerate explanations in non-technical terms. Weighted factor systems can be configured to output visualizations that shed light on the assessments and weightings behind conclusions.  As Brandeis said, sunshine is the best disinfectant.

How we secure transparent rationales in these times of shifting power balances is a key question.  And of course just knowing rationales is of little help if we can't challenge them.

On 7/10/2015 1:19 PM, Adrian Gropper wrote:

My point is not about correctness, it's about bias. Medicine, like law, or art rarely has a correct answer. The statistical interest of a society may be very different from the specific interest of an individual human participant. In medicine and law we deal with complexity by adding a licensed professional agent of the individual (doctor, lawyer) to balance the interest of the collective (hospital, government) because the collective has much better tools for dealing with complicated situations than the individual. All of this has evolved around the complicated stuff (medicine, law) being open source.

In the case of decision support software, the introduction of secret complicated stuff is changing this balance. It shifts the balance of power to the hospital, to the collective. It reduces the effectiveness of the licensed professional as an agent of the individual. The doctors increasingly become skilled technicians employed by the collective instead of the individual. The individual loses an ally in dealing with bias of the collective's algorithms.

Adrian

--

Adrian Gropper MD
Ensure Health Information Privacy. Support Patient Privacy Rights.
http://patientprivacyrights.org/donate-2/ 

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.