2008 server reboots after failure of LSASS.exe (ntdll.dll and RPC service)
i posted in 2008 general forum, ts2008 problem, post here..
i have following scenario, servers 2008 servers.
- 1 dc, native mode.
- 2 virtual , 2 fysical 2008 terminal servers.
- 1 ts gateway
- 1 session broker
- shim made application toolkit 5.5. have application has hklm key need redirect per user location in hkcu, created shim (redirect hklm\..\..\subkey -> hkcu\software\classes\virtualstore\..\..\..\subkey)
and installed "sdbinst act_application.sdb".
after install experienced spontaneous reboots on terminal servers. happens @ no specific time , happens quite lot. not find pattern.
the windows events are;
before restarting, following system log in event viewer
===============================
log name: system
source: user32
date: 18-5-2009 13:34:27
event id: 1074
task category: none
level: information
keywords: classic
user: systeem
computer: ts02.domain.loc
description:
the process wininit.exe has initiated opnieuw opstarten of computer ts02 on behalf of user following reason: er geen titel voor deze reden gevonden
reason code: 0x50006
shutdown type: opnieuw opstarten
comment: het systeemproces 'c:\windows\system32\lsass.exe' onverwacht afgesloten met statuscode 255. het systeem wordt afgesloten en opnieuw opgestart.
event xml:
<event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<system>
<provider name="user32" />
<eventid qualifiers="32768">1074</eventid>
<level>4</level>
<task>0</task>
<keywords>0x80000000000000</keywords>
<timecreated systemtime="2009-05-18t11:34:27.000z" />
<eventrecordid>257763</eventrecordid>
<channel>system</channel>
<computer>ts02.domain.loc</computer>
<security userid="s-1-5-18" />
</system>
<eventdata>
<data>wininit.exe</data>
<data>ts02</data>
<data>er geen titel voor deze reden gevonden</data>
<data>0x50006</data>
<data>opnieuw opstarten</data>
<data>het systeemproces 'c:\windows\system32\lsass.exe' onverwacht afgesloten met statuscode 255. het systeem wordt afgesloten en opnieuw opgestart.</data>
<data>
</data>
<binary>06000500</binary>
</eventdata>
</event>
===============================
there these 2 other errors in application log in event viewer
==========================
log name: application
source: application error
date: 18-5-2009 9:13:03
event id: 1000
task category: (100)
level: error
keywords: classic
user: n/a
computer: ts02.domain.loc
description:
faulting application lsass.exe, version 6.0.6001.18000, time stamp 0x47918d7c, faulting module kerberos.dll, version 6.0.6001.18000, time stamp 0x4791a76c, exception code 0xc0000005,
fault offset 0x00003d12, process id 0x278, application start time 0x01c9d5b2761c0cbf.
event xml:
<event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<system>
<provider name="application error" />
<eventid qualifiers="0">1000</eventid>
<level>2</level>
<task>100</task>
<keywords>0x80000000000000</keywords>
<timecreated systemtime="2009-05-18t07:13:03.000z" />
<eventrecordid>8526</eventrecordid>
<channel>application</channel>
<computer>ts02.domain.loc</computer>
<security />
</system>
<eventdata>
<data>lsass.exe</data>
<data>6.0.6001.18000</data>
<data>47918d7c</data>
<data>kerberos.dll</data>
<data>6.0.6001.18000</data>
<data>4791a76c</data>
<data>c0000005</data>
<data>00003d12</data>
<data>278</data>
<data>01c9d5b2761c0cbf</data>
</eventdata>
</event>
========================
========================
log name: application
source: microsoft-windows-wininit
date: 18-5-2009 9:13:07
event id: 1015
task category: none
level: error
keywords: classic
user: n/a
computer: ts02.domain.loc
description:
a critical system process, c:\windows\system32\lsass.exe, failed status code 255. machine must restarted.
event xml:
<event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<system>
<provider name="microsoft-windows-wininit" guid="{206f6dea-d3c5-4d10-bc72-989f03c8b84b}" eventsourcename="wininit" />
<eventid qualifiers="49152">1015</eventid>
<version>0</version>
<level>2</level>
<task>0</task>
<opcode>0</opcode>
<keywords>0x80000000000000</keywords>
<timecreated systemtime="2009-05-18t07:13:07.000z" />
<eventrecordid>8527</eventrecordid>
<correlation />
<execution processid="0" threadid="0" />
<channel>application</channel>
<computer>ts02.domain.loc</computer>
<security />
</system>
<eventdata>
<data>c:\windows\system32\lsass.exe</data>
<data>255</data>
</eventdata>
</event>
===============================
we dit not find memory dumps , expect crash of ntdll.dll causes service crash of rpc service, has tab "recovery options" says reboot on crashing of service, system account (user32 event) starts reboot cycle. did find files "problem reports , solutions",
-------------
product
local security authority process
problem
stopped working
date
13-5-2009 16:21
status
not reported
problem signature
problem event name: appcrash
application name: lsass.exe
application version: 6.0.6001.18000
application timestamp: 47918d7c
fault module name: stackhash_0e89
fault module version: 6.0.6001.18000
fault module timestamp: 4791a7a6
exception code: c0000374
exception offset: 000b015d
os version: 6.0.6001.2.1.0.16.36
locale id: 1043
additional information 1: 0e89
additional information 2: d96ebd0182612edc086757726eacf7e2
additional information 3: 46b4
additional information 4: 4ac0abacf80463ad5d81740e44bd5143
files describe problem
version.txt
appcompat.txt
memory.hdmp
minidump.mdmp
we tried;
- patched servers
- ran fullscan forefront client security on multiple servers.
- disable forefront client security.
- uninstalled software, except shim
- removed printer drivers (there printer errors during reboot).
- disabled uac feature, rule out conflict act shim , uac.
also found out;
- lsass crash leads system service crash , leads system initiated restart of server. need find cause of lsass crash.
- system initiated reboot system account, there no bluescreen memory dumps of os. there dump of application crash, didn't give info , ms support said nont extract useful info it.
- windows debugger activated monitor lsass process (by adplus script) , when closed cdb window (initiated through debugger), lsass proces crashes.
- might had "idle ts session limit". had set 3 hours, led logoff when user idle 3 hours. led logoff of account, led closing of window, led reboot of server. but.. if analyse previous reboots of server, there might user of administrator had idle session of 3 hours.
- found in de sp2 hotfix list;
"the dns client service , nla service crash intermittently in windows vista sp1 , in windows server 2008" hotfix http://support.microsoft.com/kb/956721 kb article isn’t public available though, can’t find more details on issue.
- helpfull turn kerberos logging on? explained in http://support.microsoft.com/kb/262177/en-us. , should in logging?
- http://support.microsoft.com/kb/911185/en-us: looks problem, don’t use mit domain + @ windows 2008 server, not 2003.
the redirection of key in shim handled through uac (registry redirect), turned off , still had reboots.
- uninstalled act shim , still.. reboots.
- found there still redirection active, though uac , shim deinstalled. has fact uac active in past , has redirections active in user profile. can't sure, because lack details of how uac works, can't seem find technical detail on uac.
- tried delete userprofiles on servers, resulted in users temp profiles being loaded. deletion of userprofile reg key of 1 of users led problem users couldn't log on entirely..
the application deinstall not option, because servers without users don't reboot (there no problem when there no users logged on). , application has been running since last year , has not given problem yet stability. application used on 2003 farm , has not given stability problems yet.
the other odd thing while running debug session on lsass process , closing dbc process, creates lsass.exe failure, causes server reboot (initiated system account). should run debugger on debugger itself? ;-)
so far disabled "ts idle time" policy, users won't logged off after long idle session. seems trick, seems "ts idle time" logoff policy might create authentication failure in lsass.exe. far (7 hours ongoing now), still no reboots.
currently case filed ms support, can think of nothing else do. post people or experience same behaviour (let's share..)
i have following scenario, servers 2008 servers.
- 1 dc, native mode.
- 2 virtual , 2 fysical 2008 terminal servers.
- 1 ts gateway
- 1 session broker
- shim made application toolkit 5.5. have application has hklm key need redirect per user location in hkcu, created shim (redirect hklm\..\..\subkey -> hkcu\software\classes\virtualstore\..\..\..\subkey)
and installed "sdbinst act_application.sdb".
after install experienced spontaneous reboots on terminal servers. happens @ no specific time , happens quite lot. not find pattern.
the windows events are;
before restarting, following system log in event viewer
===============================
log name: system
source: user32
date: 18-5-2009 13:34:27
event id: 1074
task category: none
level: information
keywords: classic
user: systeem
computer: ts02.domain.loc
description:
the process wininit.exe has initiated opnieuw opstarten of computer ts02 on behalf of user following reason: er geen titel voor deze reden gevonden
reason code: 0x50006
shutdown type: opnieuw opstarten
comment: het systeemproces 'c:\windows\system32\lsass.exe' onverwacht afgesloten met statuscode 255. het systeem wordt afgesloten en opnieuw opgestart.
event xml:
<event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<system>
<provider name="user32" />
<eventid qualifiers="32768">1074</eventid>
<level>4</level>
<task>0</task>
<keywords>0x80000000000000</keywords>
<timecreated systemtime="2009-05-18t11:34:27.000z" />
<eventrecordid>257763</eventrecordid>
<channel>system</channel>
<computer>ts02.domain.loc</computer>
<security userid="s-1-5-18" />
</system>
<eventdata>
<data>wininit.exe</data>
<data>ts02</data>
<data>er geen titel voor deze reden gevonden</data>
<data>0x50006</data>
<data>opnieuw opstarten</data>
<data>het systeemproces 'c:\windows\system32\lsass.exe' onverwacht afgesloten met statuscode 255. het systeem wordt afgesloten en opnieuw opgestart.</data>
<data>
</data>
<binary>06000500</binary>
</eventdata>
</event>
===============================
there these 2 other errors in application log in event viewer
==========================
log name: application
source: application error
date: 18-5-2009 9:13:03
event id: 1000
task category: (100)
level: error
keywords: classic
user: n/a
computer: ts02.domain.loc
description:
faulting application lsass.exe, version 6.0.6001.18000, time stamp 0x47918d7c, faulting module kerberos.dll, version 6.0.6001.18000, time stamp 0x4791a76c, exception code 0xc0000005,
fault offset 0x00003d12, process id 0x278, application start time 0x01c9d5b2761c0cbf.
event xml:
<event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<system>
<provider name="application error" />
<eventid qualifiers="0">1000</eventid>
<level>2</level>
<task>100</task>
<keywords>0x80000000000000</keywords>
<timecreated systemtime="2009-05-18t07:13:03.000z" />
<eventrecordid>8526</eventrecordid>
<channel>application</channel>
<computer>ts02.domain.loc</computer>
<security />
</system>
<eventdata>
<data>lsass.exe</data>
<data>6.0.6001.18000</data>
<data>47918d7c</data>
<data>kerberos.dll</data>
<data>6.0.6001.18000</data>
<data>4791a76c</data>
<data>c0000005</data>
<data>00003d12</data>
<data>278</data>
<data>01c9d5b2761c0cbf</data>
</eventdata>
</event>
========================
========================
log name: application
source: microsoft-windows-wininit
date: 18-5-2009 9:13:07
event id: 1015
task category: none
level: error
keywords: classic
user: n/a
computer: ts02.domain.loc
description:
a critical system process, c:\windows\system32\lsass.exe, failed status code 255. machine must restarted.
event xml:
<event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<system>
<provider name="microsoft-windows-wininit" guid="{206f6dea-d3c5-4d10-bc72-989f03c8b84b}" eventsourcename="wininit" />
<eventid qualifiers="49152">1015</eventid>
<version>0</version>
<level>2</level>
<task>0</task>
<opcode>0</opcode>
<keywords>0x80000000000000</keywords>
<timecreated systemtime="2009-05-18t07:13:07.000z" />
<eventrecordid>8527</eventrecordid>
<correlation />
<execution processid="0" threadid="0" />
<channel>application</channel>
<computer>ts02.domain.loc</computer>
<security />
</system>
<eventdata>
<data>c:\windows\system32\lsass.exe</data>
<data>255</data>
</eventdata>
</event>
===============================
we dit not find memory dumps , expect crash of ntdll.dll causes service crash of rpc service, has tab "recovery options" says reboot on crashing of service, system account (user32 event) starts reboot cycle. did find files "problem reports , solutions",
-------------
product
local security authority process
problem
stopped working
date
13-5-2009 16:21
status
not reported
problem signature
problem event name: appcrash
application name: lsass.exe
application version: 6.0.6001.18000
application timestamp: 47918d7c
fault module name: stackhash_0e89
fault module version: 6.0.6001.18000
fault module timestamp: 4791a7a6
exception code: c0000374
exception offset: 000b015d
os version: 6.0.6001.2.1.0.16.36
locale id: 1043
additional information 1: 0e89
additional information 2: d96ebd0182612edc086757726eacf7e2
additional information 3: 46b4
additional information 4: 4ac0abacf80463ad5d81740e44bd5143
files describe problem
version.txt
appcompat.txt
memory.hdmp
minidump.mdmp
we tried;
- patched servers
- ran fullscan forefront client security on multiple servers.
- disable forefront client security.
- uninstalled software, except shim
- removed printer drivers (there printer errors during reboot).
- disabled uac feature, rule out conflict act shim , uac.
also found out;
- lsass crash leads system service crash , leads system initiated restart of server. need find cause of lsass crash.
- system initiated reboot system account, there no bluescreen memory dumps of os. there dump of application crash, didn't give info , ms support said nont extract useful info it.
- windows debugger activated monitor lsass process (by adplus script) , when closed cdb window (initiated through debugger), lsass proces crashes.
- might had "idle ts session limit". had set 3 hours, led logoff when user idle 3 hours. led logoff of account, led closing of window, led reboot of server. but.. if analyse previous reboots of server, there might user of administrator had idle session of 3 hours.
- found in de sp2 hotfix list;
"the dns client service , nla service crash intermittently in windows vista sp1 , in windows server 2008" hotfix http://support.microsoft.com/kb/956721 kb article isn’t public available though, can’t find more details on issue.
- helpfull turn kerberos logging on? explained in http://support.microsoft.com/kb/262177/en-us. , should in logging?
- http://support.microsoft.com/kb/911185/en-us: looks problem, don’t use mit domain + @ windows 2008 server, not 2003.
the redirection of key in shim handled through uac (registry redirect), turned off , still had reboots.
- uninstalled act shim , still.. reboots.
- found there still redirection active, though uac , shim deinstalled. has fact uac active in past , has redirections active in user profile. can't sure, because lack details of how uac works, can't seem find technical detail on uac.
- tried delete userprofiles on servers, resulted in users temp profiles being loaded. deletion of userprofile reg key of 1 of users led problem users couldn't log on entirely..
the application deinstall not option, because servers without users don't reboot (there no problem when there no users logged on). , application has been running since last year , has not given problem yet stability. application used on 2003 farm , has not given stability problems yet.
the other odd thing while running debug session on lsass process , closing dbc process, creates lsass.exe failure, causes server reboot (initiated system account). should run debugger on debugger itself? ;-)
so far disabled "ts idle time" policy, users won't logged off after long idle session. seems trick, seems "ts idle time" logoff policy might create authentication failure in lsass.exe. far (7 hours ongoing now), still no reboots.
currently case filed ms support, can think of nothing else do. post people or experience same behaviour (let's share..)
please contact microsoft support. should able debug further , take appropriate closure.
thanks!
thanks!
Windows Server > Remote Desktop Services (Terminal Services)
Comments
Post a Comment