FAILSAFE

Phase 1: Assessment
System Critical
0/47 Objectives

Domain Controller Status

DC01.hexworth.localOnline
Role:Domain Controller
IP Address:192.168.1.10
FSMO Roles:Schema, Domain Naming
Replication:Warning - Partner Offline
Last Backup:1/30/2026 2:00 AM
DC02.hexworth.localOffline
Role:Domain Controller
IP Address:192.168.1.11
FSMO Roles:RID, PDC, Infrastructure
Status:Storage Failure - Unrecoverable

Affected Services

DNSDegraded
Primary:DC01 - Online
Secondary:DC02 - Offline
DHCPDegraded
Failover:Partner Unreachable
Leases:Active (DC01 serving)
Windows PowerShell
FAILSAFE Disaster Recovery Environment
Server: DC01 | Domain: hexworth.local
WARNING: DC02 is offline. Replication is degraded.
Type 'help' for available commands.
PS C:\Users\Administrator>

Forward Lookup Zone: hexworth.local

NameTypeDataStatus
DC01A192.168.1.10Active
DC02A192.168.1.11STALE
@NSDC01.hexworth.localActive
@NSDC02.hexworth.localSTALE

SRV Records: _msdcs.hexworth.local

ServiceTypeTargetPortStatus
_ldap._tcpSRVDC01.hexworth.local389Active
_ldap._tcpSRVDC02.hexworth.local389STALE
_kerberos._tcpSRVDC01.hexworth.local88Active
_kerberos._tcpSRVDC02.hexworth.local88STALE
_gc._tcpSRVDC01.hexworth.local3268Active

Reverse Lookup Zone: 1.168.192.in-addr.arpa

NameTypeDataStatus
10PTRDC01.hexworth.localActive
11PTRDC02.hexworth.localSTALE

DNS Forwarders

47
Active Leases
54
Available
0
Expired
47%
Utilization

Scope: 192.168.1.100 - 192.168.1.200

Subnet Mask:255.255.255.0
Default Gateway:192.168.1.1
DNS Servers:192.168.1.10, 192.168.1.11
Lease Duration:8 hours

DHCP Failover Relationship

DC01
---
DC02 (OFFLINE)
Partner Down - DC01 serving all leases

Active Directory Sites and Services

>Sites
>Default-First-Site-Name
-DC01Online
-DC02Offline

Site Links

NameSitesCostIntervalTransport
DEFAULTIPSITELINKDefault-First-Site-Name100180 minIP

Group Policy Objects

GPO NameStatusReplication
Default Domain PolicyEnabledChecking...
Default Domain Controllers PolicyEnabledChecking...

SYSVOL Replication Status

SYSVOL replication status unknown. Run verification to check.
LevelDate/TimeSourceEvent IDMessage

Available Backups

  • DC02 Full Server Backup

    1/30/2026 2:00 AM - 45.2 GB

    Full
  • DC02 System State

    1/30/2026 2:00 AM - 8.3 GB

    System State
  • DC02 System State

    1/29/2026 2:00 AM - 8.1 GB

    System State

Recovery Process

After verifying backup integrity, configure and deploy the replacement server from the Server Status tab.

Initializing...

Incident Report - INC-2026-0131-001

1. Incident Timeline

Review the Event Viewer logs and document the sequence of events using specific timestamps and event sources. Must include at least 2 timestamps and 2 event sources. (min 150 characters)

Not saved

2. Recovery Actions Taken

Detail each recovery step performed and the specific commands or tools you used. Must mention at least 3 tools or commands. (min 150 characters)

Not saved

3. Root Cause Analysis

Identify the root cause including hardware details. Must mention the failure type. (min 100 characters)

Not saved

4. Preventive Measures

Recommend specific infrastructure improvements to prevent recurrence. Must reference at least 2 areas of improvement. (min 100 characters)

Not saved

MISSION COMPLETE

FAILSAFE Recovery Successful
100%

Outstanding work, engineer. You successfully executed a full disaster recovery across AD, DNS, DHCP, and Group Policy, verified infrastructure health, and documented the incident. Hexworth infrastructure is fully operational.

Return to CourseReview Briefing

OPERATION FAILSAFE

Disaster Recovery Briefing

SITUATION

At 02:47 AM, DC02.hexworth.local suffered a catastrophic RAID controller failure. The hardware is unrecoverable. DC02 held three critical FSMO roles: RID Master, PDC Emulator, and Infrastructure Master. DC02 also served as secondary DNS server and DHCP failover partner.

CURRENT STATE

  • DC01 is online but reporting replication warnings
  • DNS is degraded - secondary server offline
  • DHCP failover partner is unreachable
  • Three FSMO roles are orphaned on the failed DC
  • AD Sites topology is incomplete
  • Group Policy replication is impaired

NETWORK INFORMATION

  • Domain: hexworth.local
  • Subnet: 192.168.1.0/24 (255.255.255.0)
  • Default Gateway: 192.168.1.1
  • DC01: 192.168.1.10
  • DC02 (failed): 192.168.1.11
  • DC02-NEW (replacement): 192.168.1.12
  • DHCP Scope: 192.168.1.100 - 192.168.1.200

YOUR MISSION

  1. Assess the damage across all services
  2. Verify and select appropriate backup
  3. Deploy and configure replacement server (DC02-NEW)
  4. Recover Active Directory - seize FSMO roles, promote new DC
  5. Restore DNS service - clean stale records, verify zones
  6. Restore DHCP service - reconfigure failover
  7. Update AD Sites and verify Group Policy replication
  8. Verify all services and document the incident

RULES OF ENGAGEMENT

  • Work through objectives systematically
  • Use the PowerShell terminal for command-line operations
  • GUI tabs provide point-and-click management tools
  • Some actions require prerequisites - check error messages
  • Complete all 47 objectives for full credit
  • Document everything in the Incident Report

Previous Recovery In Progress

Saved progress detected from a previous session.

INCOMING ALERT

DEC 25, 2026 — 02:45 AM
PRIORITY: CRITICAL

DC02.hexworth.local not responding.
Multiple services degraded. Authentication failures rising.

You are the on-call engineer. Assess the situation remotely.