WSUS 3.2 replica server synchronization failed with error 'OutOfMemoryException'

hi,

problem have is, downstream servers cannot synchronize upstream server. every time gets synchronization 0%  error 'outofmemoryexception'…
nevertheless same problem exists version 3.2.7600.226 too.
automatic approvals activated several computer groups, during process occurred no error.
assume failure occured approved many updates 60 computer groups.

have 1 upstream w2k8 server , 22 downstream w2k3 server replica.
servers locating in different geographical locations , have problem.
master upstream server has no synchronization problem ms update server.

these steps have done without effect.
wsusutil reset
dbcc checkdb
reindexed db
reduced declined updates
restarted iis, bit, sql server, wsus service

upstream or master server 2008 sp2 x64:
ram 8gb
wsus version: 3.2.7600.251
updates: 5459
approved: 5459
unapproved: 0
declined: 0
computer groups: 60
internal database

downstream server 2003 r2 sp2 x86:
ram 4gb
wsus version: 3.2.7600.251
updates: 5459
approved: 5459
unapproved: 0
declined: 0
internal database

2012-12-30 16:18:39.967 utc info w3wp.10 threadentry isapiruntime.processrequest
2012-12-30 16:18:39.967 utc change w3wp.10 admindataaccess.startsubscriptionmanually synchronization manually started
2012-12-30 16:18:40.811 utc info w3wp.22 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:40.827 utc info w3wp.23 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:40.827 utc info w3wp.21 suseventdispatcher.registereventhandler registereventhandler called notificationeventname: configurationchange
2012-12-30 16:18:43.108 utc info w3wp.17 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:43.108 utc info w3wp.13 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:43.108 utc info w3wp.38 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:43.124 utc info w3wp.37 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:43.124 utc info w3wp.38 revisionidcachechangenotificationdispatcher.internaleventhandler get event configurationchange dispatchmanager
2012-12-30 16:18:43.608 utc info wsusservice.7 suseventdispatcher.triggerevent triggerevent called notificationeventname: catalogsyncagent, eventinfo:
2012-12-30 16:18:43.608 utc info wsusservice.16 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: catalogsyncagent
2012-12-30 16:18:43.608 utc info wsusservice.7 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:43.608 utc info wsusservice.16 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:43.624 utc info wsusservice.28 threadentry threadhelper.threadstart
2012-12-30 16:18:43.624 utc info wsusservice.28 eventlogeventreporter.reportevent eventid=382,type=information,category=synchronization,message=a manual synchronization started.
2012-12-30 16:18:43.655 utc info wsusservice.16 webservicecommunicationhelper.configchangedhandler update server configuration has changed. reporting service: stats.update.microsoft.com, syncfrommu: false
2012-12-30 16:18:43.671 utc info wsusservice.28 catalogsyncagentcore.executesyncprotocol server id da9aa6fb-ea7f-4929-862a-7d6e9eae1212
2012-12-30 16:18:44.530 utc info w3wp.25 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:44.546 utc info w3wp.36 threadentry threadhelper.threadstart
2012-12-30 16:18:44.546 utc info w3wp.36 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:44.546 utc info w3wp.36 changenotificationdispatcher.internaleventhandler get event configurationchange dispatchmanager
2012-12-30 16:18:44.546 utc info w3wp.26 client.onconfigurationchange creating new clientimplementation because db configuration changed
2012-12-30 16:18:44.546 utc info w3wp.26 clientimplementation..ctor initializing clientwebservice processid = 3700, process start time = 12/30/2012 1:54:47 pm, product version = 3.2.7600.251
2012-12-30 16:18:45.827 utc info w3wp.22 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:45.842 utc info w3wp.18 threadentry threadhelper.threadstart
2012-12-30 16:18:45.842 utc info w3wp.18 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:45.842 utc info w3wp.21 suseventdispatcher.registereventhandler registereventhandler called notificationeventname: configurationchange
2012-12-30 16:18:46.202 utc change wsusservice.28 admindataaccess.executespsetconfiguration wsus configuration has been changed
2012-12-30 16:18:46.202 utc info wsusservice.28 catalogsyncagentcore.syncconfigupdatesfromuss category sync: filter: <filter syncanchor="716667,2012-12-30 16:10:03.702" getconfig="1"></filter>
2012-12-30 16:18:46.764 utc info wsusservice.28 catalogsyncagentcore.syncconfigupdatesfromuss need 0 config updates, 0 new
2012-12-30 16:18:46.764 utc info wsusservice.28 catalogsyncagentcore.syncconfigupdatesfromuss category sync: new config anchor: 716667,2012-12-30 16:19:50.501
2012-12-30 16:18:46.796 utc info wsusservice.28 catalogsyncagentcore.executesyncprotocol catalog sync: filter: <filter syncanchor="0,2000-01-01 00:00:01.000"><languagelist><language delta="1" value="1033"> </languagelist></filter>
2012-12-30 16:18:48.124 utc info w3wp.17 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:48.124 utc info w3wp.13 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:48.139 utc info w3wp.40 threadentry threadhelper.threadstart
2012-12-30 16:18:48.139 utc info w3wp.40 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:48.139 utc info w3wp.41 threadentry threadhelper.threadstart
2012-12-30 16:18:48.139 utc info w3wp.40 revisionidcachechangenotificationdispatcher.internaleventhandler get event configurationchange dispatchmanager
2012-12-30 16:18:48.139 utc info w3wp.41 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:48.624 utc info wsusservice.7 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:48.624 utc info wsusservice.24 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:48.655 utc info wsusservice.24 webservicecommunicationhelper.configchangedhandler update server configuration has changed. reporting service: stats.update.microsoft.com, syncfrommu: false
2012-12-30 16:18:49.546 utc info w3wp.25 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:49.546 utc info w3wp.42 threadentry threadhelper.threadstart
2012-12-30 16:18:49.561 utc info w3wp.42 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:49.561 utc info w3wp.42 changenotificationdispatcher.internaleventhandler get event configurationchange dispatchmanager
2012-12-30 16:18:49.561 utc info w3wp.26 client.onconfigurationchange creating new clientimplementation because db configuration changed
2012-12-30 16:18:49.561 utc info w3wp.26 clientimplementation..ctor initializing clientwebservice processid = 3700, process start time = 12/30/2012 1:54:47 pm, product version = 3.2.7600.251
2012-12-30 16:18:50.842 utc info w3wp.22 suseventdispatcher.triggerevent triggerevent called notificationeventname: configurationchange, eventinfo: configurationchange
2012-12-30 16:18:50.858 utc info w3wp.43 threadentry threadhelper.threadstart
2012-12-30 16:18:50.858 utc info w3wp.43 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: configurationchange
2012-12-30 16:18:50.858 utc info w3wp.21 suseventdispatcher.registereventhandler registereventhandler called notificationeventname: configurationchange
2012-12-30 16:18:54.936 utc warning w3wp.7 soaputilities.createexception throwexception: actor = http://<server>/clientwebservice/client.asmx, id=8e430128-1b6b-4fb5-951c-de5424619d33, errorcode=configchanged, message=, client=415558c3-b27e-4bef-a596-7160f44e2ad7
2012-12-30 16:19:07.077 utc info wsusservice.28 catalogsyncagentcore.executesyncprotocol catalog sync: new anchor: 716667,2012-12-30 16:19:51.610
2012-12-30 16:19:07.561 utc info wsusservice.28 catalogsyncagentcore.executesyncprotocol need 48297 updates, 0 new
2012-12-30 16:19:07.561 utc info wsusservice.28 catalogsyncagentcore.executesyncprotocol total inserted updates: 0
2012-12-30 16:19:07.561 utc info wsusservice.28 catalogsyncagentcore.replicasync deploymentsync: starting full sync
2012-12-30 16:19:07.561 utc info wsusservice.28 catalogsyncagentcore.replicasync deploymentsync: anchors: "" "716667,2012-12-30 16:19:51.610"
2012-12-30 16:19:10.358 utc warning w3wp.5 soaputilities.createexception throwexception: actor = http://<server>/clientwebservice/client.asmx, id=c2b65e86-03c2-47c1-ab45-7e1d592d3915, errorcode=configchanged, message=, client=5ddd6843-b615-44c4-92a8-ff5761e6fb79
2012-12-30 16:19:18.139 utc info w3wp.8 threadentry _timercallback.performtimercallback
2012-12-30 16:19:18.139 utc info w3wp.8 serverimplementation.updatecache database change occured; check if need update cache.
2012-12-30 16:21:23.499 utc warning w3wp.1 soaputilities.createexception throwexception: actor = http://<server>/reportingwebservice/reportingwebservice.asmx, id=b59fe272-0909-42c6-81f9-de4d35696f57, errorcode=configchanged, message=, client=d2b2c050-2ebf-4d18-a5c7-540aaa77f929
2012-12-30 16:21:23.514 utc error w3wp.1 webservice.reporteventbatch exception occured in reporteventbatch: fault occurred
   @ microsoft.updateservices.internal.reporting.webservice.reporteventbatch(cookie cookie, datetime clienttime, reportingevent[] eventbatch)
   @ system.runtimemethodhandle._invokemethodfast(object target, object[] arguments, signaturestruct& sig, methodattributes methodattributes, runtimetypehandle typeowner)
   @ system.runtimemethodhandle.invokemethodfast(object target, object[] arguments, signature sig, methodattributes methodattributes, runtimetypehandle typeowner)
   @ system.reflection.runtimemethodinfo.invoke(object obj, bindingflags invokeattr, binder binder, object[] parameters, cultureinfo culture, boolean skipvisibilitychecks)
   @ system.reflection.runtimemethodinfo.invoke(object obj, bindingflags invokeattr, binder binder, object[] parameters, cultureinfo culture)
   @ system.web.services.protocols.logicalmethodinfo.invoke(object target, object[] values)
   @ system.web.services.protocols.webservicehandler.invoke()
   @ system.web.services.protocols.webservicehandler.coreprocessrequest()
   @ system.web.services.protocols.syncsessionlesshandler.processrequest(httpcontext context)
   @ system.web.httpapplication.callhandlerexecutionstep.system.web.httpapplication.iexecutionstep.execute()
   @ system.web.httpapplication.executestep(iexecutionstep step, boolean& completedsynchronously)
   @ system.web.httpapplication.applicationstepmanager.resumesteps(exception error)
   @ system.web.httpapplication.system.web.ihttpasynchandler.beginprocessrequest(httpcontext context, asynccallback cb, object extradata)
   @ system.web.httpruntime.processrequestinternal(httpworkerrequest wr)
   @ system.web.httpruntime.processrequestnodemand(httpworkerrequest wr)
   @ system.web.hosting.isapiruntime.processrequest(intptr ecb, int32 iwrtype)
2012-12-30 16:22:27.608 utc warning w3wp.7 soaputilities.createexception throwexception: actor = http://<server>/clientwebservice/client.asmx, id=a0f46442-1628-4693-840c-6ab9fe923558, errorcode=configchanged, message=, client=d2b2c050-2ebf-4d18-a5c7-540aaa77f929
2012-12-30 16:22:33.233 utc error wsusservice.28 serversynccompressionproxy.getwebresponse getwebresponse exception:
   @ microsoft.updateservices.serversync.serversynccompressionproxy.getwebresponse(webrequest webrequest)
   @ system.web.services.protocols.soaphttpclientprotocol.invoke(string methodname, object[] parameters)
   @ microsoft.updateservices.serversyncwebservices.serversync.serversyncproxy.getdeployments(cookie cookie, string deploymentanchor, string syncanchor)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.webservicegetdeployments(string anchormin, string anchormax)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.replicasync()
   @ microsoft.updateservices.serversync.catalogsyncagentcore.executesyncprotocol(boolean allowredirect)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.catalogsyncthreadprocess()
   @ system.threading.threadhelper.threadstart_context(object state)
   @ system.threading.executioncontext.runtrycode(object userdata)
   @ system.runtime.compilerservices.runtimehelpers.executecodewithguaranteedcleanup(trycode code, cleanupcode backoutcode, object userdata)
   @ system.threading.executioncontext.runinternal(executioncontext executioncontext, contextcallback callback, object state)
   @ system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state)
   @ system.threading.threadhelper.threadstart()
2012-12-30 16:22:33.233 utc error wsusservice.28 webservicecommunicationhelper.processwebserviceproxyexception processwebserviceproxyexception found exception enexpected type. action: no retry, fail. exception details: system.outofmemoryexception: exception of type 'system.outofmemoryexception' thrown.
   @ microsoft.updateservices.serversync.serversynccompressionproxy.getwebresponse(webrequest webrequest)
   @ system.web.services.protocols.soaphttpclientprotocol.invoke(string methodname, object[] parameters)
   @ microsoft.updateservices.serversyncwebservices.serversync.serversyncproxy.getdeployments(cookie cookie, string deploymentanchor, string syncanchor)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.webservicegetdeployments(string anchormin, string anchormax)
   @ microsoft.updateservices.internal.webservicecommunicationhelper.processwebserviceproxyexception(soaphttpclientprotocol& webserviceobject, exception exceptioninfo)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.webservicegetdeployments(string anchormin, string anchormax)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.replicasync()
   @ microsoft.updateservices.serversync.catalogsyncagentcore.executesyncprotocol(boolean allowredirect)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.catalogsyncthreadprocess()
   @ system.threading.threadhelper.threadstart_context(object state)
   @ system.threading.executioncontext.runtrycode(object userdata)
   @ system.runtime.compilerservices.runtimehelpers.executecodewithguaranteedcleanup(trycode code, cleanupcode backoutcode, object userdata)
   @ system.threading.executioncontext.runinternal(executioncontext executioncontext, contextcallback callback, object state)
   @ system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state)
   @ system.threading.threadhelper.threadstart()
2012-12-30 16:22:33.233 utc error wsusservice.28 catalogsyncagentcore.executesyncprotocol system.outofmemoryexception: exception of type 'system.outofmemoryexception' thrown.
   @ microsoft.updateservices.serversync.serversynccompressionproxy.getwebresponse(webrequest webrequest)
   @ system.web.services.protocols.soaphttpclientprotocol.invoke(string methodname, object[] parameters)
   @ microsoft.updateservices.serversyncwebservices.serversync.serversyncproxy.getdeployments(cookie cookie, string deploymentanchor, string syncanchor)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.webservicegetdeployments(string anchormin, string anchormax)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.replicasync()
   @ microsoft.updateservices.serversync.catalogsyncagentcore.executesyncprotocol(boolean allowredirect)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.executesyncprotocol(boolean allowredirect)
   @ microsoft.updateservices.serversync.catalogsyncagentcore.catalogsyncthreadprocess()
   @ system.threading.threadhelper.threadstart_context(object state)
   @ system.threading.executioncontext.runtrycode(object userdata)
   @ system.runtime.compilerservices.runtimehelpers.executecodewithguaranteedcleanup(trycode code, cleanupcode backoutcode, object userdata)
   @ system.threading.executioncontext.runinternal(executioncontext executioncontext, contextcallback callback, object state)
   @ system.threading.executioncontext.run(executioncontext executioncontext, contextcallback callback, object state)
   @ system.threading.threadhelper.threadstart()
2012-12-30 16:22:33.233 utc info wsusservice.28 catalogsyncagentcore.updatesyncresultandgeneratereportingevent catalogsyncthreadprocess: report subscription undefined error class or success
2012-12-30 16:22:33.233 utc info wsusservice.28 eventlogeventreporter.reportevent eventid=386,type=error,category=synchronization,message=synchronization failed. reason: exception of type 'system.outofmemoryexception' thrown..
2012-12-30 16:22:33.249 utc info wsusservice.17 catalogsyncagent.waituntilsyncfinishedorcancelled agent signalled done.
2012-12-30 16:22:33.249 utc info wsusservice.17 catalogsyncagent.setsubscriptionstatewithretry firing event syncfailtostart...
2012-12-30 16:22:33.249 utc info wsusservice.17 catalogsyncagent.wakeupworkerthreadproc found no more jobs. catalogsyncagent quits run rollup before terminating ...
2012-12-30 16:22:33.264 utc info wsusservice.17 catalogsyncagent.updateserverhealthstatusbasedonerror serverhealth: updating server health component: catalogsyncagent, marking not running
2012-12-30 16:22:34.577 utc info wsusservice.7 suseventdispatcher.triggerevent triggerevent called notificationeventname: rollupagent, eventinfo:
2012-12-30 16:22:34.577 utc info wsusservice.27 threadentry threadhelper.threadstart
2012-12-30 16:22:34.577 utc info wsusservice.27 suseventdispatcher.dispatchmanagerworkerthreadproc dispatchmanager worker thread processing notificationevent: rollupagent

hope me.

thanks,

kailar

the allowed classification updates of upstream sever „critical“ , „security updates“. these updates being approved automatically on patch day pilot computer groups.

okay. explains why have 5400+ approved updates.

now need go unapprove ~5000+ updates not need approved anymore.

i did decline 1008 updates.

that's start. guess there's @ least 4000 can changed. on wsus server security updates , critical updates there not more 400-500 updates need approved maintain current patching requirements.

of course..... converting of 4000+ updates approved declined, cause yet problem (the introduction of declines), , synchronization continue fail (as we've seen several times in recent months result of mass declinations of thousands of updates on upstream server).

at point, critical not decline updates, set them not approved. once done, try synchronization again; it's still possible simple removal of approvals may problematic, , may require rebuild downstream replica servers (which, actually, faster trying continue fight situation).

---

lawrence garvin, m.s., mcitp:ea, mcdba, mcsa
solarwinds head geek
microsoft mvp - software distribution (2005-2012)
mvp profile: http://mvp.support.microsoft.com/profile/lawrence.garvin
the views expressed on post mine , not reflect views of solarwinds.

Windows Server  >  WSUS