http://wiki.pchero21.com/index.php?action=history&feed=atom&title=Mail_writing 2026-04-18T22:14:41Z Revision history for this page on the wiki MediaWiki 1.38.2 http://wiki.pchero21.com/index.php?title=Mail_writing&diff=1508&oldid=prev 2016-08-16T14:27:05Z

<p>Created page with &quot;== Overview == 메일 예제 모음 == incident == === Report problem occurred === 문제의 내용과 문제를 어떻게 해결할 것인지를 같이 나타낸다. &lt;pre&gt; T...&quot;</p> <p><b>New page</b></p><div>== Overview ==<br /> 메일 예제 모음<br /> <br /> == incident ==<br /> === Report problem occurred ===<br /> 문제의 내용과 문제를 어떻게 해결할 것인지를 같이 나타낸다.<br /> &lt;pre&gt;<br /> Title: incident: earth01 disk array controller is dead<br /> <br /> The problem started ~ 08:20 and was rectified by removing earth01 from production ~08:50<br /> <br /> This is just a headsup: the disk controller has malfunctioned, and we now only process calls on earth02.<br /> <br /> <br /> Any faults before 08:50 should be attributed to earth01 failed, after this time, we want error reports and <br /> as soon as customers report problems with web. earth02 is _not_ expected to cope with a full maximum load of calls.<br /> <br /> <br /> <br /> Plan A is to make a virtual machine of earth01, restored from either earth02 and/or backup.<br /> <br /> Plan B is to try to recover the hardware earth01 runs on, pending PCHERO arrival at UNIVERSE hosting.<br /> <br /> <br /> more info to follow<br /> <br /> -- <br /> <br /> Re.<br /> <br /> Sungtae Kim<br /> &lt;/pre&gt;<br /> <br /> === Report problem solved ===<br /> 상세한 원인 분석 내용과 어떻게 처리했는지 내용을 나타낸다.<br /> &lt;pre&gt;<br /> Title: incident: earth01 was unavailable (Was: incident: earth01 disk array controller is dead)<br /> <br /> begin : 2016-08-16 08:17 end: 2016-08-16 11:30<br /> Area : test_server/earth<br /> Severity : Medium<br /> <br /> Noter / Notes:<br /> <br /> earth01 responded to ping and HTTP port, and ssh port, but login was not possible. <br /> The monitoring system did flag non-critical alarms, and the host was attempted powercycled.<br /> <br /> Web sevices where randomly affected since most queries where routed through earth02 (the customers request out on a random select between earth01 and earth02. <br /> Queries failed periodically up to 10% of the time, until earth01 was fully back into production.<br /> <br /> timeline:<br /> 08:20 Problems with earth01 was detected.<br /> 08:40 earth01 was attempted powercycled<br /> 09:10 fault with smart controller was identified, Network engineer was dispatched to UNIVERSE to perform hands-on fault correction.<br /> 09:20 alternative plan: rebuild earth01 as virtual machine and recover it, ETA to complete: 4-6 hours.<br /> 10:15 Engineer onsite<br /> 10:40 Hardware disassembled and reseated vital components, and after reboot controller was working again<br /> Checking hardware for faults and testing software + OS until ~11:25<br /> 11:30 earth was back in normal operation, rebuild earth01 as VM work was halted.<br /> <br /> <br /> Root cause: the most apparent cause was hung kernel on earth01 due to faults on storage IO subsystem:<br /> Slot 0 HP Smart Array P410i Controller<br /> 1783-Slot 0 Drive Array Controller Failure!<br /> [Command failure (cmd=11h, err=020h)]<br /> <br /> <br /> <br /> Corrective measures:<br /> a) Hardware has been disassembled and reassembled, thus reseating boards, disk, PSU and controller.<br /> <br /> <br /> b) We are currently working on replacing EARTH and TEST_SERVER functionality entirely with MARS so the impact of this will be resolved.<br /> <br /> c) We are evaluating if time invested in fully virtualizing earth01 would be mitigating downtime if it should occur again, <br /> however with regards to getting close to have b) done in short timespan, since the blocking customer for getting traction <br /> is being resolved this week, so b) can now progress.<br /> -o-o-o-<br /> <br /> <br /> <br /> -- <br /> <br /> Re.<br /> <br /> Sungtae Kim<br /> &lt;/pre&gt;<br /> <br /> <br /> [[category:etc]]</div>

Pchero