Ethics in IT — Week 3 · Ch. 7
Software Development Ethics
"You shipped the code. It worked. Someone got hurt. Whose fault is it?"
13 slides ~16 minutes ETH-09 · The Factionless
By the end of this module, you will navigate the "good enough" problem, recognize safety-critical obligations, analyze methodology ethics, walk the Therac-25 and Uber ATG cases, and apply the developer's obligation to refuse.
Slide 2 of 13
The "Good Enough" Problem
Software is never perfect. The ethical question isn't whether defects exist — it's how many are acceptable given the stakes.
A spelling error in a word processor
Annoying. Trivial. Maybe a feature flag for autocorrect quality. Defect tolerance: high.
A rounding error in medical device dosage software
Can kill a patient. Defect tolerance: zero. Yet the same engineering culture often treats both with the same urgency.
The structural problem: the software industry has long resisted product-liability standards applied to physical goods. Shrink-wrap and click-through licenses routinely disclaim all warranties. The legal shield does not eliminate the ethical obligation to ship software that does not cause foreseeable harm.
"When the specification says 'best effort' and the deployment context is a hospital, the specification is the ethical problem — not just a legal one."
Slide 3 of 13
Four Quality Dimensions
Each dimension carries its own ethical weight. Skipping any one is a decision, not an oversight.
Functionality
Does it do what it claims?
Ethics: false advertising if not; breach of contract if specified.
Reliability
Does it work consistently over time?
Ethics: unreliable systems in critical infrastructure create foreseeable harm.
Safety
Does failure harm users or bystanders?
Ethics: the highest bar — safety failures can be life-threatening.
Maintainability
Can defects be found and corrected?
Ethics: technical debt deferred indefinitely becomes ethical liability.
The hierarchy in safety-critical work: safety > reliability > functionality > maintainability. In commercial software, the order often inverts — functionality and feature velocity dominate. The ethical question is whether the inversion is defensible for the deployment context.
Slide 4 of 13
Safety-Critical Systems
Where failure costs lives. The engineering standards are higher; the ethical stakes are dramatically higher.
Medical devices
Insulin pumps, infusion pumps, radiation therapy, pacemakers. FDA classes I/II/III with proportionate scrutiny.
Avionics
DO-178C standard. Five Design Assurance Levels (A-E). Level A failure = catastrophic = many lives.
Industrial control
Nuclear plant systems, chemical process control, grid management. IEC 61508 functional safety standard.
Air traffic management
FAA NextGen. Software failures can ground entire regions. Latency & correctness both safety-critical.
Autonomous vehicles
ISO 26262 functional safety. Tesla, Waymo, Uber ATG — the test environment is public roads.
The pattern
In each domain: rigorous standards, formal testing, regulatory oversight. The ethical floor is far above commercial software.
Slide 5 of 13
Case: Therac-25 (1985-1987)
The defining reference for software ethics in safety-critical systems. Three patients dead. Three severely injured.
What happened: Between 1985 and 1987, the Therac-25 radiation therapy machine delivered massive overdoses to at least six patients due to a race condition in the control software. Three patients died. Three were severely injured.
Root cause: the software reused code from an earlier model (Therac-20) but removed hardware safety interlocks that the previous version relied on. The software was not adequately tested for the new configuration. Engineers dismissed early error reports as user error.
3 dead
Documented patient deaths attributed to the overdose.
2 years undetected
Time between first incident and pattern recognition.
Hardware safety removed
Replaced with software interlocks that weren't independently verified.
Slide 6 of 13
The Therac-25 Failure Pattern
Not malice. Not incompetence. The pattern that repeats in every safety-critical failure since.
Schedule over verification
Development team prioritized shipping the new model over re-verifying that the inherited code worked in the new configuration.
Cryptic errors dismissed
Error codes were ambiguous. Operators were trained to interpret them as routine. Real warnings looked like noise.
Slow incident investigation
When patients were injured, investigations took months. Other patients continued to be treated on the same machine during the investigation.
Pattern recognition delayed
Each incident treated as unique. The cross-incident pattern (race condition triggered by specific keystroke sequence) wasn't seen until external researchers connected the cases.
"The engineers who wrote the Therac-25 code were not malicious. They were rushed, they cut corners they thought were safe to cut, and they dismissed warning signs. That pattern repeats in every safety-critical failure."
Slide 7 of 13
Methodology Ethics
Waterfall vs. Agile is a process question. The ethics shows up in what you do when the process gives permission to cut corners.
Waterfall
Sequential phases with defined requirements upfront.
Ethics benefit: clear documentation; explicit sign-offs at each gate.
Ethics risk: late testing — defects found after significant investment create pressure to ship anyway.
Agile
Iterative development with continuous delivery.
Ethics benefit: faster feedback; easier course correction.
Ethics risk: "move fast" culture deprioritizes security/safety; technical debt accumulates faster than it's serviced.
"The methodology is not the ethics. The ethics is what the team does when the methodology gives them permission to cut corners they know they should not cut."
Slide 8 of 13
Liability Standards
Software has historically been treated as a service to limit liability exposure. That distinction is collapsing in safety-critical contexts.
Negligence
Liability requires: (1) duty of care, (2) breach of that duty, (3) breach caused the harm.
In safety-critical software, duty of care is well-established. In consumer software, it's emerging through litigation.
Strict Liability
Manufacturer is liable regardless of fault. If the product is defective and causes harm, liability attaches.
Software is moving toward this in safety-critical contexts. The EU's AI Liability Directive is a leading example.
What this means for IT professionals: the legal regime is shifting from "we disclaimed warranty" to "you built a product that hurt someone." The shield is thinner every year. The professional question is whether you'd ship the same code if liability tracked you personally.
Slide 9 of 13
Autonomous Vehicle Ethics
Software making life-or-death decisions in milliseconds. The ethical decisions are embedded in algorithms before any incident occurs.
Occupant vs Bystander
When the system can save one or the other, who? The programmer encodes a moral position — not a technical one.
False-Positive Rate
Suppress detections to reduce false alarms. Each suppressed alarm is also a missed true positive. The tradeoff is a value judgment.
When to Deploy at Scale
Public roads are the live test environment. Other drivers and pedestrians did not consent to be participants in your beta.
"Every algorithm that touches the physical world encodes a value system. The engineer who writes it is making ethical choices whether they frame them that way or not."
Slide 10 of 13
Case: Uber ATG — Tempe, AZ (March 2018)
The decision to suppress detections was made by engineers to improve ride comfort. That decision was made before anyone died.
What happened: An Uber autonomous test vehicle struck and killed Elaine Herzberg, a pedestrian crossing outside a marked crosswalk. A safety driver was present but distracted. The vehicle's software detected the pedestrian 6 seconds before impact but failed to take corrective action.
The software factor: the system was designed to suppress false-positive detections to reduce unnecessary braking. The suppression prevented action on a true positive. The safety-driver alert system had been disabled.
The ethical dimension: the decision to suppress was made by engineers to improve ride comfort during testing — optimizing for one metric while degrading safety. That decision was made before anyone died. The harm was foreseeable from the engineering tradeoff.
Slide 11 of 13
The Engineer's Three Decisions
In every safety-critical system, the engineer makes three ethical decisions before any incident occurs.
1. The Tradeoff
Comfort vs. safety. Velocity vs. verification. Cost vs. controls. The engineer chooses the metric the system optimizes for.
The system you built will produce the outcomes the metric rewards.
2. The Refusal
When the request is to suppress, disable, or skip safety verification — the professional response is to document the request, identify the risk, and refuse if necessary.
"I was told to" is not a defense.
3. The Escalation
If management overrides safety judgment, the obligation is to escalate — to safety committees, regulatory liaisons, professional bodies, or external whistleblower channels.
Career cost is real. Lives at stake outweigh it.
The Uber ATG scenario: the decision to implement detection suppression was made by an engineer who could have refused. The decision to disable safety-driver alerts was made by someone who could have escalated. Each was a defensible-in-isolation engineering call. The aggregate killed someone.
Slide 12 of 13
Professional Responsibility
The ACM and Software Engineering codes both put it bluntly: software engineers shall act consistently with the public interest.
In practice
Discovers a vulnerability and is told to defer patching: document the decision, escalate if the risk is significant.
Asked to disable safety checks for a deadline: refuse, or formally record the override.
Asked to ship a system with known critical defects: the obligation runs to the user, not the schedule.
The defense that doesn't work
"I was just following orders" has no standing in professional ethics.
The engineer who builds the defeat device, the developer who implements surveillance software, and the programmer who writes the suppression logic all made a choice. Employment does not eliminate individual responsibility.
"The code you write is your signature. Years later, when the system fails, no one will remember who gave the order. They will ask who wrote the function."
Slide 13 of 13
Module 9 Summary
Software Development Ethics — key takeaways.
1The "good enough" problem — defect tolerance must scale with stakes. Spelling error in word processor ≠ rounding error in dosage software.
2Four quality dimensions: functionality, reliability, safety, maintainability. Skipping any is a decision, not an oversight.
3Safety-critical systems have higher engineering standards (DO-178C, IEC 61508, ISO 26262) and dramatically higher ethical stakes.
4Therac-25 (1985-87): 3 dead. Removed hardware interlocks + inadequate testing + dismissed warnings. The defining reference for safety-critical software ethics.
5Methodology is not ethics. Both Waterfall and Agile have ethical risks. The ethics is what the team does when the process permits cutting corners.
6Liability is shifting from negligence to strict in safety-critical contexts. The "we disclaimed warranty" shield is thinner every year.
7Uber ATG (2018): detection suppression for ride comfort killed Elaine Herzberg. The harm was foreseeable from the engineering tradeoff — made before anyone died.
8The engineer's three decisions: the tradeoff, the refusal, the escalation. "I was just following orders" has no standing in professional ethics.
Next up: Module 10 — IT Impact on Society. Productivity paradox, AI workforce displacement, healthcare IT, the digital divide, IBM Watson for Oncology.