Ethics in IT — Week 3 · Checkpoint
Week 3 Checkpoint
Two questions: what you are responsible for when code fails, and what technology does to people who do not write it.
13 slides ~13 minutes ETH-11 · The Factionless
By the end of this checkpoint, you will recall software-quality obligations, recognize the Therac-25 / Uber ATG / Watson failure pattern, articulate the productivity paradox, and connect Week 3 to the synthesis ahead.
Slide 2 of 13
What You Covered Week 3
Two chapters that close the loop on builder-side obligations.
ETH-09: Software Development Ethics
The "good enough" problem. Four quality dimensions. Safety-critical systems. Therac-25. Methodology ethics. Liability standards. Uber ATG. The engineer's three decisions.
ETH-10: IT Impact on Society
The productivity paradox. Automation risk tiers. Healthcare IT trade. Algorithmic bias. The digital divide. IBM Watson for Oncology. What builders owe.
The shift in framing: Weeks 1-2 asked "what protects the individual?" Week 3 asks "what does the builder owe to the people who never agreed to the system?"
Slide 3 of 13
Software Quality & the Good-Enough Problem
Defect tolerance must scale with stakes. The same engineering culture often treats both contexts the same. That's the mistake.
Functionality
Does it do what it claims?
Reliability
Does it work consistently over time?
Safety
Does failure harm users or bystanders?
Maintainability
Can defects be found and corrected?
The shrink-wrap shield is thinner every year. EU AI Liability Directive. Strict liability moving into safety-critical software. The legal floor is rising toward the ethical ceiling.
Slide 4 of 13
Therac-25 (1985-87): The Defining Reference
Three dead. Three injured. The pattern that repeats in every safety-critical software failure since.
Root cause
Software reused from Therac-20 but removed hardware safety interlocks the previous version relied on. Replacement software controls were not adequately tested for the new configuration.
Pattern
Schedule over verification. Cryptic errors dismissed. Slow incident investigation. Pattern recognition delayed until external researchers connected the cases.
"The engineers who wrote the Therac-25 code were not malicious. They were rushed, they cut corners they thought were safe to cut, and they dismissed warning signs. That pattern repeats in every safety-critical failure."
Slide 5 of 13
Uber ATG Tempe (2018): Detection Suppression
The decision to suppress was made by engineers to improve ride comfort. That decision was made before anyone died.
The setup: Uber's autonomous vehicle struck and killed Elaine Herzberg. The vehicle's software detected the pedestrian 6 seconds before impact but failed to take corrective action. The system was designed to suppress false-positive detections to reduce phantom braking.
The ethical line: the engineering tradeoff — comfort vs. safety — was a value judgment dressed up as a technical decision. The harm was foreseeable from the suppression design choice. Made before anyone died.
"Every algorithm that touches the physical world encodes a value system. The engineer who writes it is making ethical choices whether they frame them that way or not."
Slide 6 of 13
The Engineer's Three Decisions
In every safety-critical system, three ethical decisions are made before any incident occurs. The engineer makes all three.
"I was just following orders" has no standing in professional ethics. Each of these three decisions is the engineer's call, even when management wants a different outcome.
Slide 7 of 13
The Productivity Paradox
"You can see the computer age everywhere except in the productivity statistics." — Robert Solow, 1987
The lesson
Technology alone does not create value. Productivity gains require complementary organizational change — process redesign, network effects, supply-chain optimization.
IT investment without organizational change underperforms the projection.
The ethics
Organizations justify layoffs and restructuring on productivity gains that may not materialize without the complementary investment.
The workers displaced carry the cost of a bet they did not make.
"The quarterly earnings report does not capture the career of a 47-year-old data entry clerk whose job was automated. The productivity gain is real. The cost is externalized."
Slide 8 of 13
AI & Cognitive Displacement
Earlier waves replaced manual labor. AI/ML targets cognitive work that previously felt automation-resistant.
Routine manual
Already automated. Assembly, data entry, toll collection.
Routine cognitive
Rapidly automating. Paralegals, radiology, loan processing.
Non-routine cognitive
Augmentation likely. Physicians, attorneys, engineers.
Social/relational
Difficult to automate. Teachers, therapists, caregivers.
The novel obligation: earlier automation displaced workers slowly enough for retraining to keep up. AI displacement is faster and broader. The professional debate — minimum positions through automation tax — is active and unresolved.
Slide 9 of 13
Healthcare IT & Watson for Oncology
EHRs reduce errors. Algorithmic bias is real. The marketing decision is sometimes the bigger ethical failure than the engineering decision.
Watson for Oncology: marketed as evidence-based AI. Trained primarily on hypothetical patient scenarios developed by physicians at MSK. Recommended "unsafe and incorrect" cancer treatments. Internal docs leaked to Stat News, 2018. Withdrawn 2022.
Algorithmic bias
ML diagnostics trained on non-representative data perform worse on underrepresented populations — including dark-skinned patients.
EHR trade
Reduced medication errors, better coordination — against ransomware risk and opaque data monetization.
The pattern
Marketing claim > methodology truth. The engineer who knew was obligated to make the marketing match the methodology.
Slide 10 of 13
Digital Divide: Structural Exclusion
When essential services move online, the divide stops being inconvenience and becomes exclusion from civic life.
Economic
Cost of devices, broadband. Below-poverty households have lower connectivity.
Geographic
Rural and tribal lands have lower availability + reliability.
Generational
Older adults face steeper learning curves; less institutional support.
Educational
Digital literacy correlates with formal education. Pandemic exposed how unevenly this is distributed.
Inclusive design is not charity. It is a professional obligation under ACM and IEEE codes. The app that works only on a 1Gbps connection and a current-gen smartphone is not neutral — it is a product that serves some people and excludes others.
Slide 11 of 13
The Week 3 Pattern
Three industries. Three case studies. The same mechanism every time.
Therac-25
Schedule pressure → removed safety interlocks → inadequate testing → dismissed warnings → patient deaths.
Uber ATG
Comfort optimization → detection suppression → degraded true-positive response → pedestrian death.
Watson for Oncology
Marketing pressure → methodology omission → physician deference → unsafe recommendations to patients.
The common mechanism: an engineering or communication decision made under cost / schedule / market pressure that produced foreseeable harm to people who were not in the room. The professional refusal — the refusal to make that tradeoff — is what each case wanted and didn't get.
Slide 12 of 13
Looking Ahead: Week 4
Synthesis week. Social media, organizational ethics, professional codes. The full picture.
ETH-12: Social Media Ethics
Personal vs business use. Section 230 revisited. Misinformation. Cambridge Analytica.
ETH-13: Ethics of IT Organizations
Contingent workers. Outsourcing. Gig economy. Whistleblowing. Green computing.
ETH-14: Codes of Ethics
ACM, IEEE, PMI, AITP — agreement, conflict, application. The SE Code.
The arc completes: Week 1 (foundations) → Week 2 (rights) → Week 3 (builder obligations) → Week 4 (organizational ethics + the codes that govern it all). Final assessment ties them together.
Slide 13 of 13
Week 3 Takeaways
Eight ideas to carry into Week 4 and beyond.
1The "good enough" problem: defect tolerance must scale with stakes. Safety-critical ≠ consumer software.
2Therac-25: the defining safety-critical software case. Removed hardware interlocks + inadequate testing + dismissed warnings.
3Uber ATG (2018): detection suppression for ride comfort killed Elaine Herzberg. The engineering tradeoff was made before anyone died.
4The engineer's three decisions: tradeoff, refusal, escalation. "I was just following orders" has no standing.
5The productivity paradox: IT alone doesn't create value — organizational change does. Workers carry the cost of unrealized gains.
6AI displacement targets cognitive work that previously felt automation-resistant. Novel obligations under active debate.
7Healthcare IT & Watson: the marketing decision is sometimes the bigger ethical failure than the engineering decision.
8Inclusive design is a professional obligation under ACM/IEEE codes. The "neutral product" that serves only some users is a choice.
Week 3 complete. Take the Week 3 quiz when you've finished this checkpoint. Move to Week 4: Social Media, IT Organizations, Codes of Ethics, Final Assessment.