Week 3 Lecture: When Code Reaches People

Week 3 Lecture Companion · Ethics in IT

When Code Reaches People

Week 3's real cases (Therac-25, Boeing 737 MAX, COMPAS) dissected through the frameworks you met in Week 1. Software safety, commercial pressure, algorithmic fairness.

Three system failures. Engineers who could have said no. Names on every line of code that reached a human.

15 Slides ~50 min 3 cases × 2 lenses each In-class assignment included

Slide 2 of 15 · Recap from Week 2

The Five Frameworks, Back at Work

Same five frameworks. Harder cases. Software that kills, algorithms that judge.

Thanos

Utilitarianism

Greatest good for the greatest number.

Captain America

Deontology

Duty regardless of consequence.

Uncle Iroh

Virtue Ethics

What would a good person do?

Tony vs Steve

Social Contract

Negotiated rules for shared power.

Itachi / Ozymandias

Ends Justify Means

The math that wins, at terrible cost.

The shift this week

Week 2 was about rights. Whose property, whose privacy, whose voice. Week 3 is about responsibility. Who owns what software does at scale. Same protocol: run two frameworks, find the disagreement, defend the synthesis. The cases are harder because the harm is built into the code, not negotiated in a courtroom afterward.

Slide 3 of 15 · Three W3 Cases

Three Software Failures, Three Lessons

Every Week 3 topic has a defining case where engineers had options and the wrong one shipped.

Case 1 · Software Safety

Therac-25 (1985 to 1987). Medical linear accelerator. Race condition plus removed hardware interlocks. At least six patients delivered massive radiation overdoses. Three deaths attributed.

Lenses: Walter White vs Tony Stark

Case 2 · Commercial Pressure

Boeing 737 MAX MCAS (2018 to 2019). Single-sensor anti-stall system, hidden from airlines and regulators. Lion Air 610 and Ethiopian 302 crashes. 346 dead. Worldwide fleet grounded 20 months.

Lenses: Ozymandias vs Uncle Iroh

Case 3 · Algorithmic Fairness

COMPAS in criminal sentencing (2016 to present). Risk-assessment scoring used in pretrial, probation, sentencing across the United States. ProPublica showed disparate false-positive rates by race. The Wisconsin Supreme Court upheld its use.

Lenses: Light Yagami vs Captain America

The pattern, applied three times

For each case: a setup slide that gives you the facts. Then a dual-lens slide that runs two frameworks at each other. Then a synthesis slide for what the IT professional carries away.

Slide 4 of 15 · Case 1: Software Safety

Therac-25: When Software Killed

The first famous software fatality. The bug that defined safety engineering as a discipline.

The Therac-25 was a medical linear accelerator (used to treat cancer with high-energy X-ray or electron beams) built by Atomic Energy of Canada Limited starting in 1982. Between June 1985 and January 1987, at least six patients received radiation overdoses of approximately 100 times the intended dose. Three patients died from the injuries. A race condition in the operator console software, combined with the deliberate removal of hardware interlocks present on prior models (because the software "had been proven safe"), allowed the machine to fire its high-power electron beam without the metal target in place. AECL initially denied software involvement. The bug was eventually traced and documented by Nancy Leveson and Clark Turner in "An Investigation of the Therac-25 Accidents" (IEEE Computer, July 1993).

The technical failure

A race condition between the operator entering treatment parameters and the magnet positioning system. If parameters were edited within 8 seconds of a prior treatment, the software allowed a 25 MeV electron beam to fire without the metal target rotated into position. The patient received a full-power beam, not the attenuated therapeutic dose.

The institutional failure

Prior models (Therac-6, Therac-20) had hardware interlocks (independent circuits that physically prevented unsafe states). The Therac-25 designers removed them, relying instead on software checks. When the software failed, nothing else stopped the machine. The hardware was the redundancy. The redundancy was deleted.

The professional failure

After the first overdose at Kennestone (1985), AECL told other hospitals the machine "could not have done what was reported." Five more patients were injured before AECL acknowledged the bug. Reports were dismissed because the dosimeter readouts (driven by the same software) showed normal values. Engineers trusted their own output more than the patient's burn.

The framing question for the next slide

Two frameworks. Walter White says: the math justified shipping. The machine works most of the time. Recalls cost millions. Tony Stark says: I built this. It killed people. I own that. The redesign starts now. One engineer descends. One atones. Which arc is the working professional's posture?

Slide 5 of 15 · Therac-25 × Two Lenses

Walter White vs Tony Stark

Two engineers. One rationalizes the harm. One owns it and rebuilds.

Lens 1 · Utilitarian Drift

Walter White

"I am the one who knocks."

I did it for my family. I did it for the patients we DO help. The math works out if you don't get sentimental.

Walter's verdict on Therac-25: The machine cures thousands of cancer patients a year. A handful of incidents are tragic, but the aggregate good is enormous. Recall costs $40M and slows treatment for everyone in the queue. The rational, utilitarian move is to ship the software patch quietly, advise operators of the timing issue, and keep the machines running. He started as a chemist who took a small shortcut. He ended as the danger himself. The Therac engineers walked that path one rationalization at a time.

Lens 2 · Engineering Responsibility

Tony Stark

"I shouldn't be alive, unless it was for a reason."

I had my eyes opened. I came to realize I had more to offer this world than making things that blow up.

Tony's verdict on Therac-25: Stark Industries made weapons that killed Americans. When Tony saw what he had built, he shut down the weapons division, took the personal financial hit, and rebuilt from first principles. That is the engineering posture for the Therac-25 team after the first overdose at Kennestone: stop production, recall every unit, restore the hardware interlocks, publish the failure analysis, and rebuild trust through visible accountability. The cost is real. So is the cost of the next patient on the table.

The disagreement IS the lesson

Both arcs are real careers. The Therac team chose Walter White's path: keep operating, quietly patch, deny in public. By the time they pivoted to Tony Stark's path (full recall, redesign, hardware interlocks restored), three patients were dead. The cost of the late pivot was the cost of the harm that accrued during the denial. Source: Leveson and Turner, IEEE Computer 1993.

Slide 6 of 15 · Therac-25: Takeaway

Therac-25: Where This Leaves You

Two engineers, one defensible posture for the working IT professional.

Default Posture

Tony Stark wins. In safety-critical software (medical devices, aviation, industrial control, anything that can hurt a human if it misbehaves), the engineer's name is on every line of code that touches the human. Responsibility does not end at "QA signed off." It ends at "the patient walked out."

When Walter Has a Point

Triage contexts only. When the alternative is no treatment at all (third-world ICU with one machine, battlefield medicine), partial safety can beat zero coverage. But that bar is HIGH and the analysis must be explicit and visible to the patient. "Quietly patch and don't tell" is not triage. It is rationalization.

The Gray Zone

"Safe enough to ship" is the ethical work. Code review depth, regression coverage, fail-safe defaults, hardware interlocks, ALL of it is defining where "safe enough" sits. Defining that line BEFORE you ship is engineering. Re-defining it AFTER the incident is litigation.

Code Anchor

ACM Code 1.2: "Avoid harm." Direct and uncompromising. IEEE Code 1: "Hold paramount the safety, health, and welfare of the public." The Therac team violated both. The fact that the cancer patients DID need the treatment did not relieve the obligation to ensure the machine could not kill them.

Slide 7 of 15 · Case 2: Commercial Pressure

Boeing 737 MAX: 346 People

MCAS killed Lion Air 610 and Ethiopian 302. Boeing knew. The FAA was told what Boeing wanted them to be told.

The Boeing 737 MAX entered service in 2017. To compete with the Airbus A320neo, Boeing fitted larger, more fuel-efficient engines to a re-positioned mount that changed the aircraft's pitch behavior at high angle of attack. To compensate, Boeing added MCAS (Maneuvering Characteristics Augmentation System), a software function that automatically trimmed the horizontal stabilizer nose-down to prevent stall. MCAS was driven by a single angle-of-attack sensor (with no redundancy in the early implementation), could trim repeatedly without pilot acknowledgment, and was not disclosed to airlines as a system requiring pilot training, in order to preserve "common type rating" with the 737 NG and avoid recertification costs. Lion Air 610 crashed October 29, 2018 (189 dead) and Ethiopian Airlines 302 crashed March 10, 2019 (157 dead). The worldwide fleet was grounded for 20 months. Primary sources: the U.S. House Transportation & Infrastructure Committee's Final Committee Report on the Design, Development & Certification of the Boeing 737 MAX (September 2020) documents the MCAS single-sensor design, repeated trim authority, and non-disclosure to airlines via internal Boeing communications. NTSB Safety Recommendations A-19-010 through A-19-016 (September 19, 2019) address the post-crash certification process review.

The technical hazard

MCAS could trim the stabilizer the full available range (2.5 degrees per cycle), repeat the input, and resist pilot pushback on the yoke. With a single failed AoA sensor reading high, MCAS would aggressively command nose-down. Pilots had seconds to diagnose an undocumented system and find the cutout switches before the aircraft was unrecoverable.

The commercial deception

Boeing internal communications (released by the House Transportation Committee in 2020) showed engineers and test pilots privately criticizing MCAS. Boeing pressed the FAA to omit MCAS from pilot training requirements. Southwest's $1M-per-plane rebate clause depended on no new simulator training. The math behind "don't disclose" was: avoid recertification, preserve the order book, hit the delivery quota.

The regulatory rot

FAA had delegated significant safety certification to Boeing's own engineers under the Organization Designation Authorization program. The agency that should have caught MCAS was, on paper, Boeing employees wearing FAA hats. After the crashes, both Congress and DOT's Inspector General documented that the FAA had functionally outsourced the oversight that would have prevented this.

The framing question for the next slide

Two frameworks. Ozymandias says: the survival of Boeing as a US manufacturer was the larger plan; some deaths in the long tail were the price. Uncle Iroh says: deception is the harm. A person of good character does not hide MCAS from the pilots who fly the plane. One says the means justify the ends. The other says the means ARE the ends.

Slide 8 of 15 · 737 MAX × Two Lenses

Ozymandias vs Uncle Iroh

Utilitarian math when you hide the inputs. Virtue ethics when you cannot.

Lens 1 · Hidden Utilitarianism

Ozymandias / Adrian Veidt

"I did it thirty-five minutes ago."

The math is unforgiving. To save billions, sometimes one must accept the death of millions. The plan is everything.

Ozymandias's verdict on the 737 MAX: Boeing was losing market share to Airbus. The 737 MAX with a clean common-type rating preserved tens of thousands of US manufacturing jobs, the supplier ecosystem, the national strategic capability to build commercial airliners. Disclosing MCAS would have triggered Level D simulator training, killed the Southwest deal, and let Airbus take the decade. The 346 paid for the survival of an industry. This is the utilitarian argument AT ITS WORST: it ONLY works when the truth is hidden. Once the public knows what Boeing knew, the math collapses.

Lens 2 · Virtue Ethics

Uncle Iroh

"Pride is not the opposite of shame, but its source."

There is nothing wrong with letting people who love you help you.

Iroh's verdict on the 737 MAX: The character flaw IS the harm. A person of good character does not deceive the pilots who trust the airplane, regardless of what the deception buys or costs. The deception breaks who Boeing is as a company, who its engineers are as professionals, who the FAA is as a regulator. The 346 deaths are downstream of that character failure, not a separate utilitarian variable. The right move was: tell the FAA. Tell Southwest. Negotiate the training cost. Take the financial hit. Slow the rollout. Virtue ethics does not need to defeat Ozymandias by winning the outcome math (although it does also win the outcome math, since the deception failed on its own terms — Boeing nearly collapsed, lost the orders, lost the executives, AND people died). Virtue wins because cowardice was the harm, and the deaths were the cost of the cowardice.

The disagreement IS the lesson

The Ozymandias frame fails as soon as you remove the deception assumption. Hidden utilitarianism is the most common engineering rationalization, and it is the most fragile one. The minute the truth comes out (and in software, the truth always comes out, because the crash logs are public), the entire utilitarian argument inverts. The 346 deaths were not a price paid for value created. They were the cost of a decision that produced negative value on every dimension: regulatory, financial, reputational, human. Sources: House T&I Committee Final Report on the 737 MAX (September 2020) for the internal Boeing communications and MCAS design history; NTSB Recommendations A-19-010 through A-19-016 (September 2019) for the certification process review.

Slide 9 of 15 · 737 MAX: Takeaway

737 MAX: Where This Leaves You

When the org pressures you to ship something the engineering says is unsafe.

Default Posture

Iroh wins. Withholding safety-relevant information from regulators, customers, or end users is a violation of duty regardless of competitive pressure. The deception itself is the harm. The harm-to-people is a downstream consequence of the harm-to-truth.

When Ozymandias Has a Point

When the inputs are public. Utilitarian reasoning in safety engineering is legitimate when stakeholders can see the trade-offs and object: triage protocols, scarcity allocation, transparent cost-of-recall analysis, military procurement with disclosed risk. Boeing did not fail because cost-benefit reasoning is wrong. Boeing failed because the inputs to the cost-benefit reasoning were fraudulent. The test: if your utilitarian argument requires the public not to know, the utilitarian argument is broken (not utilitarianism itself).

The Gray Zone

Internal escalation is the first defense. Most engineers will never blow a whistle. They WILL be in design reviews where commercial pressure compresses safety analysis. Knowing how to escalate (in writing, to a named person, with a date) is the working skill. Whistleblowing is downstream of bureaucratic failure, not a replacement for it.

Code Anchor

ACM Code 1.3: "Be honest and trustworthy." IEEE Code 3: "Be honest and realistic in stating claims or estimates based on available data." Boeing's MCAS disclosure to the FAA failed both. The estimate of "no new pilot training required" was not honest, and the data was withheld from the people who would have caught the dishonesty.

Slide 10 of 15 · Case 3: Algorithmic Fairness

COMPAS: When Software Sentences

A risk-assessment algorithm used in pretrial, probation, and sentencing across the United States. ProPublica showed the bias. The Wisconsin Supreme Court upheld its use anyway.

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions), built by Northpointe (later Equivant), is a proprietary risk-assessment tool used by judges and probation officers across the United States. It produces scores from 1 to 10 estimating the defendant's risk of reoffending. In May 2016, ProPublica's "Machine Bias" investigation analyzed COMPAS scores for over 7,000 arrestees in Broward County, Florida. Among defendants who did not reoffend within two years, Black defendants were almost twice as likely as White defendants to be labeled high risk (the false-positive rate was 45% for Black, 23% for White). Northpointe disputed the methodology. Two months later, the Wisconsin Supreme Court decided State v. Loomis (881 N.W.2d 749, Wis. 2016), upholding the use of COMPAS in sentencing on the condition that judges receive written cautions about the tool's limitations. COMPAS remains in use across many jurisdictions.

The tool

COMPAS uses 137 questions to score defendants on risk of failure-to-appear, general recidivism, and violent recidivism. The full questionnaire and weighting algorithm are trade secrets. Race is not an input. Many of the inputs (employment history, neighborhood, family criminal history) are correlated with race in the source data.

The finding

ProPublica's analysis: Black defendants who did NOT reoffend were 45% likely to have been labeled high-risk. White defendants who did NOT reoffend were 23% likely to have been labeled high-risk. Equal error rate would have given equal false-positive rates. The algorithm was differentially wrong by race.

The persistence

Wisconsin v. Loomis upheld COMPAS use with judicial cautions. The defendant could not examine the algorithm (trade secret), could not contest individual feature weights, but could be sentenced partly on its output. Eric Loomis received six years; the trial judge cited COMPAS in the sentencing decision. COMPAS is still used widely.

The framing question for the next slide

Two frameworks. Light Yagami says: I built a system to make justice more consistent. Race is not an input, so the system cannot be racist. Disparate outcomes mean disparate underlying populations. Captain America says: outcomes are the measure of justice. If the algorithm produces different false-positive rates by race, the algorithm IS the injustice, no matter what inputs it used. One defines fairness as input-blindness. The other defines it as outcome-equality.

Slide 11 of 15 · COMPAS × Two Lenses

Light Yagami vs Captain America

"My system can't be unjust, race isn't an input" vs "outcomes are what we promised."

Lens 1 · Corrupted Idealism

Light Yagami / Kira

"I will create the new world."

I'll create a world where only people I deem righteous can live in peace.

Light's verdict on COMPAS: The system makes justice consistent. Judges varied wildly before, with race correlated to sentencing through human bias. An algorithm with structured inputs and validated weights removes that variance. Race is explicitly not an input. If outcomes differ across groups, that reflects the underlying populations differing on the risk-relevant factors. Critics demanding outcome-equality are demanding the algorithm encode race AS race, which is the actual discrimination. The algorithm is neutral. The world it measures is not. This is precisely how COMPAS designers defended it. It is the most internally consistent defense of algorithmic governance ever written. It is also wrong, in exactly the way Light is wrong: the SYSTEM that decides who is "righteous" is the system that becomes the injustice.

Lens 2 · Duty to Fairness

Captain America

"The price of freedom is high. It always has been."

When you can do the things that I can, but you don't, and then the bad things happen, they happen because of you.

Cap's verdict on COMPAS: Cap's deontology in Week 2 was "the rule is the rule, regardless of outcome." Here the rule itself IS about outcomes: equal treatment under the law. The DUTY is procedural fairness; outcome measurement is how you check compliance with that duty. A 45% false-positive rate for one group and 23% for another is the rule being violated, not a separate consequentialist concern. The defense that race was not an input is the algorithm's lie. The MATH the algorithm uses to defend itself IS the injustice. The duty is not to build an algorithm that satisfies the algorithm's own definition of fairness. The duty is to build one that satisfies the defendant's right to equal treatment, and the only honest measurement of that duty is the outcome audit.

The disagreement IS the lesson

This is the hard one. Light's argument is internally consistent and mathematically defensible. Cap's argument is internally consistent and morally defensible. They are NOT compatible: a deep result in fairness theory (Chouldechova 2017, Kleinberg-Mullainathan-Raghavan 2017) shows that "calibration" (Light's frame) and "equal false-positive rates" (Cap's frame) cannot both be satisfied when base rates differ across groups. You must CHOOSE which definition of fairness you are encoding, and you must DEFEND that choice. Source: ProPublica "Machine Bias" (May 2016) and State v. Loomis (Wis. 2016).

Slide 12 of 15 · COMPAS: Takeaway

COMPAS: Where This Leaves You

When the algorithm makes a decision that used to be a person's.

Default Posture

Cap wins on consequential decisions. When the algorithm makes a decision that materially affects a person's life (sentencing, hiring, lending, healthcare access), the deployment-level outcomes ARE the fairness measure. Input neutrality is necessary but not sufficient. Audit the outputs by group. Publish the audit.

When Light Has a Point

Calibration matters too. Light is not entirely wrong: a tool that is well-calibrated (its score predicts reoffense at the same rate regardless of group) IS doing real work. The error is treating calibration as the COMPLETE fairness story. Calibration plus disparate false-positive rates is exactly the COMPAS pattern. You have to pick which one you protect.

The Gray Zone

Fairness is mathematically multi-dimensional. Equalized odds, demographic parity, calibration, individual fairness, counterfactual fairness, ALL of these are valid definitions. They are mutually incompatible in most realistic distributions. Picking which to satisfy is the ethical choice that has to be made BEFORE training, in the open, with stakeholders who can object.

Code Anchor

ACM Code 1.4: "Be fair and take action not to discriminate." IEEE Code 8: "Treat all persons fairly, including avoiding discrimination based on race, religion, gender..." Note the word "treat" in both. The codes refer to outcomes, not inputs. The COMPAS defense (race wasn't an input) does not satisfy the codes' fairness requirement.

Slide 13 of 15 · Synthesis

When Frameworks Disagree About Software

The pattern across all three cases. Software changes the math because software changes the scale.

Pattern 1 · Scale changes the math

Therac-25 was six patients. The 737 MAX was 346. COMPAS is millions of sentencing decisions. Software-mediated harm is not a one-off bad decision. It is a bad decision REPLICATED. The framework that says "the math justifies it" must account for the replication. Usually it does not.

Pattern 2 · Hidden utilitarianism fails

All three cases share a structure: a utilitarian argument that ONLY works while the public does not know. Therac's bug was hidden. MCAS was hidden. COMPAS's weights are still hidden. When the truth surfaces (and software always leaves a paper trail), the utilitarian argument inverts. Whatever your framework, it must survive disclosure.

Pattern 3 · Outcomes over inputs

In all three cases, the engineering defense focused on intent (the inputs, the math, the QA process). The harm was measured in outcomes (patients killed, planes crashed, defendants mis-sentenced). The professional posture has to be: audit outcomes. The intent does not relieve the responsibility for the result.

The repeatable decision protocol (refined for software)

1. What is the actual decision, not the rationalization? 2. Run it through at least two frameworks. 3. Imagine the decision becomes public knowledge tomorrow. Does the utilitarian argument still hold? 4. What does the outcome audit (by group, by demographic, by user class) say? 5. If you cannot defend this to future you, your peers, the press, AND the family of the harmed person, redesign.

Slide 14 of 15 · In-Class Tribunal

The Framework Tribunal

One case on trial. Four lenses argue. A jury rules which lens should govern the engineer's decision.

Setup: The class is a tribunal. The instructor puts one case on trial. Four advocate groups each take one of the first four lenses below and argue the case from it, even if they personally disagree. The Social Contract group is the jury: it hears all four, names the disagreement, and rules which lens should govern and what professional rule that carries into next week.

Utilitarianism

Thanos. Greatest good for the greatest number. Sum the consequences.

Deontology

Cap. Duty regardless of outcome. Rules over results.

Virtue Ethics

Iroh. What would a person of good character do?

Ends-Justify-Means

Ozymandias / Itachi. The math that wins, paid for by the math-doer.

Social Contract

Civil War. Negotiated rules between people with shared stakes. This group is the jury.

Case Pool (instructor tries ONE with the whole class)

Software safety: Toyota unintended acceleration (2009 to 2010) · Knight Capital algorithm meltdown (2012) · Healthcare.gov rollout (2013).

Commercial pressure: Volkswagen Dieselgate emissions defeat device · Equifax breach (2017) · Tesla Autopilot fatality investigations.

Algorithmic fairness: Amazon AI hiring tool (scrapped 2018) · Apple Card credit limit allegations · Optum healthcare algorithm bias (Obermeyer 2019).

Privacy / autonomy: Cambridge Analytica (2018) · OPM data breach (2015) · IBM Watson for Oncology overpromise.

The Rounds

1. Prep. Each group builds its lens's strongest argument. 2. Opening verdicts. One ruling per advocate, no interruptions. 3. The Clash. The two most-opposed lenses rebut head to head; others may interject. 4. Jury ruling. The governing lens and the rule to carry forward. Timeboxes are instructor-set; Prep shortest, Clash longest.

What Each Side Delivers

Advocates: 1. a one-sentence verdict from your lens; 2. the fact that is decisive under your lens; 3. your rebuttal to the most-opposed lens; 4. what would change your verdict. Jury: name the precise ethical question the lenses disagree on, then which lens should govern and the rule the profession carries into next week's codes of ethics.

Facilitator anchor: The depth check is the Clash and the jury naming the disagreement, not whether advocates agree. Assign the most-opposed lenses to your most confident groups; the Ends/Ozymandias advocate is the hardest and most valuable. If groups converge, push them to name WHO BEARS THE COST: engineers, users, shareholders, future patients?

Slide 15 of 15 · Looking Ahead

Looking Ahead to Week 4

Week 3 was about what software does at scale. Week 4 is about who holds the people who build it accountable.

Codes of Ethics, Deep Dive

ACM Code 2018 revision. IEEE Code. Why both exist, where they differ, when each applies. The codes you cited in this week's takeaways become the textbook for Week 4. We will read them as ENGINEERING DOCUMENTS, not as moral aspirations.

IT Organizations and the Profession

Software does not have a licensing body the way medicine and civil engineering do. Why not? What replaces it? Professional associations, certifications, employer-side standards, voluntary codes. The institutional architecture of being a professional in an unlicensed profession.

Social Media Ethics

Platform-as-publisher, algorithmic amplification, content moderation at scale, attention economy. Builds on Section 230 (W2) plus the algorithmic-fairness frame from COMPAS (this week) plus the IT-impact subtopic deck's surveillance capitalism material. The synthesis question: what does responsible platform governance look like when the platform is the product, the user is the data, and the algorithm is the editor?

Before class next week

Complete the Week 3 Quiz. Review the topic decks (Software Ethics / IT Impact) for any case that felt thin in your pair work. Read the ACM Code of Ethics (2018 revision) and the IEEE Code of Ethics in full before Week 4. They are short. We will use them as the curriculum, not the citation.

Week 3 Lecture: When Code Reaches People | Ethics in IT