Search Service Failing at Client: “Internal server error exception: Correlation ID: 81edb1ab-e32f-4e

Configuration: SharePoint 2010 RTM running on Windows Server 2008 R2 Standard with SQL Server 2008
Initial install was a wizard install letting the wizard install and configure all services. I was not privy to the install but the server configuration intent was to have:

Database Server: D-MOSS-DB
Application Servers: D-MOSS-App-1 and D-MOSS-App-2
In fact only the database server, i.e., D-MOSS-DB, and the application server D-MOSS-App-1 wound up being in the installed environment. When search is configured – using only D-MOSS-App-1 – and the crawler runs and a search is attempted there are
intermittent errors returned by the search request: “Internal server error exception: Correlation ID: 81edb1ab-e32f-4e02-8030…”. The SharePoint log (see below) reflects communication by the Query component and subsequently Topology and Administration
components to the non-existant D-MOSS-App-2 server. This server is no where to be found – at least where I’ve been looking – in the search service topology or proxy connections. The error appears to be happening during search service initialization and can
be cleared by creating a Web Analytics | Number of Queries report. The initial request to Web Analytics also creates an exception again referencing a request to the non-existant server D-MOSS-App-2. Creating the Number of Queries report, however, works and
searches now work from the client. There appears to be a service idle timer of some sort so that this exception recurs if the search service is idle for a few minutes.

SharePoint Log
SharePoint Server Search       Query                          dka1 High     SearchServiceApplicationProxy::GetProxyInfo–Proxy
Name:Search Service Application EndPoint:
http://d-moss-app-2:32843/77f7302513f04a358b85fe3a8a35ca69/SearchService.svc
 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Query                        
 dk8z High     SearchServiceApplicationProxy::GetChannel–Channel Creation time: 0 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Foundation          Topology                     
 e5mc Medium   WcfSendRequest: RemoteAddress: ‘http://d-moss-app-2:32843/77f7302513f04a358b85fe3a8a35ca69/SearchService.svc’ Channel: ‘Microsoft.Office.Server.Search.Administration.ISearchServiceApplication’ Action: ‘http://tempuri.org/ISearchSiteAdministrationServiceApplication/GetProxyInfo’
MessageId: ‘urn:uuid:90b04b8f-38fb-4b29-acd1-54a5b3682429’ 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Foundation          Monitoring                   
 b4ly High     Leaving Monitored Scope (ExecuteWcfOperation:http://tempuri.org/ISearchSiteAdministrationServiceApplication/GetProxyInfo). Execution Time=2294.81428505578 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint
Foundation          Monitoring                     b4ly High     Leaving Monitored Scope
(GetProxyInfo). Execution Time=2297.5724568346 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Query                        
 dka4 High     SearchServiceApplicationProxy::GetProxyInfo–Error occured: System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at

http://d-moss-app-2:32843/77f7302513f04a358b85fe3a8a35ca69/SearchService.svc
that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> System.Net.WebException: The
remote name could not be resolved: ‘d-moss-app-2’     at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)     at System.Net.HttpWebRequest.GetRequestStream()     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()    
— End of inner exception stack trace —    Server stack trace:      at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.G… 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search     
 Query                          dka4 High     …etOutputStream()     at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan
timeout)     at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.RequestChannel.Request(Message
message, TimeSpan timeout)     at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message
message, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMe… 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint
Server Search       Query                          dka4 High     …ssage methodCall,
ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at Microsoft.Office.Server.Search.Administration.ISearchSiteAdministrationServiceApplication.GetProxyInfo()    
at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.b__f5(ISearchServiceApplication serviceApplication)     at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBackedOperation`1
webServiceCall, Int32 timeoutInMillise… 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Query                        
 dka4 High     …conds, Int32 wcfTimeoutInMilliseconds, String operationName) 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Administration               
 fv2l High     ‘Search Service Application’, location cache: Exception while fetching location configuration data. StackTrace: System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at

http://d-moss-app-2:32843/77f7302513f04a358b85fe3a8a35ca69/SearchService.svc
that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> System.Net.WebException: The
remote name could not be resolved: ‘d-moss-app-2’     at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)     at System.Net.HttpWebRequest.GetRequestStream()     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()    
— End of inner exception stack trace —    Server stack trace:      at System.Ser… 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Administration               
 fv2l High     …viceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()     at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)     at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message
message, TimeSpan timeout)     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message
message, TimeSpan timeout)     at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway,
ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Chan… 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Administration               
 fv2l High     …nels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)   
Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type)     at Microsoft.Office.Server.Search.Administration.ISearchSiteAdministrationServiceApplication.GetProxyInfo()     at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.b__f5(ISearchServiceApplication
serviceApplication)     at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoSpLoadBalancedUriWsOp[T](WebServiceBack… 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search     
 Administration                 fv2l High     …edOperation`1 webServiceCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String
operationName)     at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, WebServiceBackedOperation`1
webServiceCall)     at Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy.DoWebServiceBackedOperation[T](String operationName, WebServiceBackedOperation`1 webServiceCall)     at Microsoft.Office.Server.Search.Administration.LocationConfigurationCollection.PopulateCollection()    
at Microsoft.Office.Server.Search.Administration.LocationConfigurationCollection.Init()     at Microsoft.Office.Server.Search.Administration.LocationConfigu… 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search     
 Administration                 fv2l High     …rationCollection..ctor(SearchServiceApplicationProxy searchAdmin)     at Microsoft.Office.Server.Search.Administration.LocationCache.FetchLocationConfigurationData() 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint
Server              General                        0 Medium   Constructed
a new async cache named Search Proxy Cache 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Query                        
 dka1 High     SearchServiceApplicationProxy::GetSearchServiceApplicationInfo–Proxy Name:Search Service Application EndPoint:

http://d-moss-app-1:32843/77f7302513f04a358b85fe3a8a35ca69/SearchService.svc
 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Query                        
 dk8z High     SearchServiceApplicationProxy::GetChannel–Channel Creation time: 0 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Foundation          Topology                     
 e5mc Medium   WcfSendRequest: RemoteAddress: ‘http://d-moss-app-1:32843/77f7302513f04a358b85fe3a8a35ca69/SearchService.svc’ Channel: ‘Microsoft.Office.Server.Search.Administration.ISearchServiceApplication’ Action: ‘http://tempuri.org/ISearchQueryServiceApplication/GetSearchServiceApplicationInfo’
MessageId: ‘urn:uuid:970d6fa4-36d9-48ab-835f-446ebd563a03’ 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Foundation          Topology                     
 e5mb Medium   WcfReceiveRequest: LocalAddress: ‘http://d-moss-app-1.nwhsea.org:32843/77f7302513f04a358b85fe3a8a35ca69/SearchService.svc’ Channel: ‘System.ServiceModel.Channels.ServiceChannel’ Action: ‘http://tempuri.org/ISearchQueryServiceApplication/GetSearchServiceApplicationInfo’
MessageId: ‘urn:uuid:970d6fa4-36d9-48ab-835f-446ebd563a03’ 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Foundation          Monitoring                   
 nasq Medium   Entering monitored scope (ExecuteWcfServerOperation) 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Foundation          Monitoring                   
 b4ly Medium   Leaving Monitored Scope (ExecuteWcfServerOperation). Execution Time=0.650641352462394 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Query                        
 dka2 High     SearchServiceApplicationProxy::GetSearchServiceApplicationInfo–Id: Elapsed Time: 31.2522 Proxy Name/ID: Search Service Application/47e671b8-d631-4fde-a7b5-5ba9bbcc2790 EndPoint:

http://d-moss-app-1:32843/77f7302513f04a358b85fe3a8a35ca69/SearchService.svc
 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Query                        
 fm9a Unexpected CoreResultsWebPart::OnInit: Exception initializing: System.NullReferenceException: Object reference not set to an instance of an object.     at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.SetPropertiesOnQueryReader()    
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.OnInit(EventArgs e) 230bf6d0-f3ed-4360-a947-b32893d1f508 SharePoint Server Search       Query                        
 g1j9 Exception Internal server error exception: System.NullReferenceException: Object reference not set to an instance of an object.     at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.SetPropertiesOnQueryReader()    
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.OnInit(EventArgs e) System.NullReferenceException: Object reference not set to an instance of an object.     at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.SetPropertiesOnQueryReader()    
at Microsoft.Office.Server.Search.WebControls.CoreResultsWebPart.OnInit(EventArgs e) 230bf6d0-f3ed-4360-a947-b32893d1f508

Exception from Web Analytics

System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at

http://d-moss-app-2:32843/645ff35e9ce04429a8d542b918405b26/WebAnalyticsService.svc
that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> System.Net.WebException:
The remote name could not be resolved: ‘d-moss-app-2’ at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context) at System.Net.HttpWebRequest.GetRequestStream() at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()
— End of inner exception stack trace — Server stack trace: at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream() at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout) at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message
message, TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message
message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage
methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&
msgData, Int32 type) at Microsoft.Office.Server.WebAnalytics.Administration.IWebAnalyticsWebServiceApplication.GetData(AggregationContext aggregationContext, DataContext dataContext, GetDataOptions options) at Microsoft.Office.Server.WebAnalytics.Administration.WebAnalyticsServiceApplicationProxy.<>c__DisplayClass2.b__0(IWebAnalyticsWebServiceApplication
channel) at Microsoft.Office.Server.WebAnalytics.Administration.WebAnalyticsServiceApplicationProxy.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at Microsoft.Office.Server.WebAnalytics.Administration.WebAnalyticsServiceApplicationProxy.GetData(AggregationContext
aggregationContext, DataContext dataContext, GetDataOptions options) at Microsoft.Office.Server.WebAnalytics.Reporting.FrontEndDataRetriever.QueryData(AggregationContext context, List`1 viewProperties, String viewName, List`1 viewParameters, Condition whereCondition,
List`1 sortOrder, Int32 rowIndex, Int32 rowCount, Boolean bypassCache) at Microsoft.Office.Server.WebAnalytics.Reporting.Report.GetDataPacket(AggregationContext context, DateTime date, Dictionary`2 parameters, Boolean bypassCache)

Hi Donald,What Status your Query Component is showing in Search Service Application. You can check Status by Scrolling all the way Down on Search Administration Page in Search Service Application. /

From the Search Application Topology section of Search Administration the Administration Component, Crawl Component 0 and Query Component 0 are all showing Online. There are no other Admin, Crawl or Index Partition components (there is only one index partition).

This appears to be a defect within the Search Service. Depsite the faulty configuration the Search Service should recognize that one of the servers is not in the configuration and recover gracefully. Whatever is happening when the Web Analytics | Number