System Center Operations Manager 2007: Troubleshooting

Workaround to Operations Manager 2007 SP1 'Performance Module' Error

Thu, 20 Mar 2008 13:36:33 GMT

http://feeds.feedburner.com/~r/systemcenterforum/~3/244964343/ | Comments

The MOM Team Blog posted a temporary workaround for the “Performance Module could not find a performance counter” we’ve all been seeing. A long-term solution is in the works.

Instructions can be found HERE.

Troubleshooting event id 33333 logged by the DataAccessLayer

Tue, 18 Mar 2008 20:42:46 GMT

Courtesy of: Boris Yanushpolsky

Sometimes after installing an agent or importing a management pack you may see event number 33333 logged by the Data Access layer. In many situations this error is caused by the fact that proxying is not enabled for a particular agent. In OpsMgr SP1, we have added a new rule to pickup this event and generate an alert. The alert looks like the following:

You can figure out using powershell the name of the agent that requires proxying to be enabled using the following command:

Get-MonitoringObject -id:'2090A6E7-27D2-425F-5617-6EE7B2291970' | ft DisplayName

This will provide the name of the agent that needs to have proxying enabled.

Note: You will need to replace the GUID that you use as the parameter to the cmdlet with the first GUID that you see in the alert.

WMI Probe Module Failed Execution on W2k servers after SP1 RTM upgrade.

Tue, 18 Mar 2008 12:59:29 GMT

http://feeds.feedburner.com/~r/PontusOperationsManagementBlog/~3/253545818/wmi-probe-module-failed-execution-on.html

There is a know problem with the Microsoft.SystemCenter.DiscoverWindowsOSProperties discovery on w2k servers. On w2k3 servers it works OK.

The alert looks like this:

Object enumeration failed Query: 'SELECT
Version,Caption,BuildNumber,CSDVersion,ServicePackMajorVersion,ServicePackMinorVersion,SerialNumber,InstallDate,SystemDrive,WindowsDirectory,TotalVisibleMemorySize
FROM Win32_OperatingSystem' HRESULT: 0x80041017 Details: Invalid query One
or more workflows were affected by this. Workflow name:
Microsoft.SystemCenter.DiscoverWindowsOSProperties

Current work around is to make a override for this discovery for Windows 2000 Server group. Microsoft is looking in to this issue.

Citrix MP CitrixSessionDown.js script error

Tue, 18 Mar 2008 12:58:58 GMT

http://feeds.feedburner.com/~r/PontusOperationsManagementBlog/~3/253534450/citrix-mp-citrixsessiondownjs-script.html

Using the Citrix Presentation Server MP you get the following error from the
Citrix.PresentationServer.CitrixSessionDown.Monitor.

"Script or executable failed to run".
The description shows:
=====================================================
The process started at 1:24:51 failed to create System.PropertyBagData, no
errors detected in the output. The process exited with 0

Command executed: "C:\WINDOWS\system32\cscript.exe" //d "C:\Program
Files\System Center Operations Manager 2007\Health Service State\Monitoring
Host Temporary Files 2\477\CitrixSessionDown.js" 120
Working Directory: C:\Program Files\System Center Operations Manager
2007\Health Service State\Monitoring Host Temporary Files 2\477\

One or more workflows were affected by this.

Workflow name: Citrix.PresentationServer.CitrixSessionDown.Monitor

This is due a Citrix MP config error:

WriteAction ID="WA" RunAs="CtxLibrary!Citrix.PresentationServer.CitrixAdministratorAccount" TypeID="System!System.CommandExecuter">
%SystemDirectory%\net.exe
%Temp%
start CitrixWMIService
60
true

As you can see the MP uses a variable %SystemDirectory% wich default doesn´t exist.
If you create a Systemvariable on the Citrix servers where %SystemDirectory% = %systemroot%\system32 then the script works.

Thanks to Ross Williamson for this information found in Citrix support forums.

Why: 'Not Monitored' State for Management Servers or Agents?

Mon, 17 Mar 2008 02:28:20 GMT

Courtesy of: Operations Manager Product Team Blog

Every week I see one customer running into an issue where they either see the management servers or agents showing up as not monitored. While there a number of reasons why these roles may show up as being ‘not monitored’ I wanted to list out a couple of things to check to help you figure out what the problem maybe. Below are some steps to drill down to understanding why you are seeing a 'Not Monitored' state for Management Servers and Agents

1) If you are seeing Not Monitored on the “Root Management Server” log onto to the RMS and verify that the Configuration and Health Services are running. This can be done by opening the service snap-in. If the Management Server and Agent is showing up as being not monitored then verify that the Health Service is running on each of those individual roles.

2) If you are upgrading from RTM to SP1 make sure that you import the latest updated management packs. The core system MP’s are updated automatically after upgrade but the non-core MP’s do not get updated automatically and users need to manually re-import those MPs. Kevin Holman wrote a good blog about this which can be found here. If you are seeing unmonitored after you upgrade this could be another possible reason. All the SP1 MP’s should have version 6.0.6278.0 you can find the version of the MP’s by opening the OpsMgr console going to the Administration view and selecting the Management Packs node. The version number of the MP’s will be listed in the version column. If you see an MP with a version nuber that is below 6278 then you will need to re-import those MP’s. Kevin wrote this SQL script which will also list out the MP’s and the version number.

SELECT MPName, MPFriendlyName, MPVersion, MPIsSealed

FROM ManagementPack WITH(NOLOCK)

ORDER BY MPName

3) If users renames a server or shuts down a server that is being monitored then users will see that the server will show up as being un-monitored. This is only if they just deployed a MS or an agent and someone brings down the servers. In most cases you would see a grey icon with a checkbox which indicates that the server stopped heart beating and the last known state was healthy.

4) If there is no trust between the domain the agent is on and the management server then you are likely to get the ‘not monitored’ state. If there is only a one way trust between the Management server and agent then you would also see this issue. In OpsMgr 2007 agents initiate communication with the MS and the MS uses the same channel to communicate back to the agents. An easy way to figure out if this is the issue is to go to the event log on the agent and see if you see error events that state mutual authentication could not be established.

5) Gateway Servers are one of the big criminals of showing the ‘not monitored’ state. 99% of the times when they show that they are not monitored it usually means the users have not configured the certificates correctly, they not run the Gateway approval command line tool or they are not using the right Public Key Infrastructure (PKI).

6) When you deploy a management pack and the action account is configured as a low priv account, some workflows (monitors/rules/discoveries/tasks/diagnostics/recoveries) may not be able to execute because by default they will run under the low priv account and may not have sufficient rights to access the instrumentation they need in order to function properly. You can get more information on this from Boris’s blog.

7) If the agent action account does not have enough privileges some of the properties of the server will show up as being not monitored. Check the OpsMgr agent event log for event 1201. The agent health service normally will log that event for each management pack it downloads. Alternatively you can also go to the Heath State folder under the %Program files%\Microsoft System Center Operations Manager 2007\ (there should be a management pack folder) you can check to see if the agent downloaded the various MPs. If there are no P’s in the folder it is another sign that agents are not getting configuration from the Management Server.

While the above steps may not be the direct solution to the problem they should help put you on the right track to diagnose the root of the issue.

Manually Register the SPN of a Clustered RMS

Thu, 13 Mar 2008 19:08:54 GMT

The information below is provided courtesy of Charlie Chen, Consultant, Microsoft Consulting Services.

Overview

There are two usages of servicePrincipleName (SPN) in OpsMgr 2007. The first is for the Health Service, the second is for the SDK Service, both used for agent to server communication.

For the Health Service, the agent will request authentication by referencing to the registered SPN MSOMHSvc/<management server FQDN> and MSOMHSvc/<management server NetBIOS>, where for a clustered RMS, the management server FQDN and NetBIOS is the virtual name of the RMS. For a normal, non-clustered machine, the SPN of the Health Service is self-registered by the SDK at startup. For a clustered RMS, AD permissions don’t allow self-registration. Agent to server communications is only over Kerberos or with certificates so if the SPN is not properly registered, mutual authentication will not work.

With OpsMgr 2007 SP1, the SDK client will use the SPN MSOMSdkSvc/<RMS FQDN> as well as MSOMSdkSvc/<RMS NetBIOS> (when using a cluster for a RMS, each node will need to be registered instead of the virtual name of the RMS cluster). In the case of the SDK client, if Kerberos does not work, it will try to fall back to NTLM. The SDK service will try to self-register its SPN at startup. However, AD permissions will only allow this if the SDK service is running as local system or network service, and thus will not do so on a cluster, since the SDK service runs as a domain account. Like for the Health Service, the SPN must be manually registered for the SDK Service. The main case of needing Kerberos is if you are trying to have a remote web console server and want to use integrated authentication on it. The other case NTLM won’t work is if the client is a computer account, since computer accounts can only auth via Kerberos, and not NTLM.

Set the Values for the SPN

Install the Windows Server 2003 Service Pack 1 Support Tools on a computer that will be performing the following procedures, which does not need to be a root management server or a management server.

For the Health Service, use either ADSIEdit.msc or SetSpn.exe to add the following values to the servicePrincipleName attribute of computer object of the RMS cluster, RMS:

¾ MSOMHSvc/RMS.contoso.com

¾ MSOMHSvc/RMS

For the SDK Service use either ADSIEdit.msc or SetSpn.exe to add the following values to the servicePrincipleName attribute of user object of the SDK and Config Service Account, SDK_Account.:

¾ MSOMSdkSvc/RMSnode01.contoso.com

¾ MSOMSdkSvc/RMSnode01

¾ MSOMSdkSvc/RMSnode02.contoso.com

¾ MSOMSdkSvc/RMSnode02

Using ASDIEdit.msc

To use ASDIEdit.msc, browse to the object and go to its Properties. Go to the servicePrincipleName attribute and Add the values above.

Using SetSpn.exe

To use SetSpn.exe, use the following commands:

¾ SetSpn.exe -A MSOMHSvc/RMS.contoso.com

¾ SetSpn.exe -A MSOMHSvc/RMS

¾ SetSpn.exe -A MSOMSdkSvc/RMSnode01.contoso.com contoso.com\SDK_Account.

¾ SetSpn.exe -A MSOMSdkSvc/RMSnode01 contoso.com\SDK_Account.

¾ SetSpn.exe -A MSOMSdkSvc/RMSnode02.contoso.com contoso.com\SDK_Account.

¾ SetSpn.exe -A MSOMSdkSvc/RMSnode02 contoso.com\SDK_Account.

Verify the SPN Values

To verify that the values have been set, use ASDIEdit.msc or the follow LDIFDE commands:

¾ Ldifde -f c:\ldifde.txt -t 3268 -d DC=CONTOSO,DC=COM –r "(serviceprincipalname=MSOMHSvc/*)" -l serviceprincipalname -p subtree

¾ Ldifde -f c:\ldifde.txt -t 3268 -d DC=CONTOSO,DC=COM –r "(serviceprincipalname=MSOMSdkSvc/*)" -l serviceprincipalname -p subtree

Change the Active Directory Based Agent Assignment Account

When using a cluster for a RMS, the Active Directory Based Agent Assignment Account will also need to be changed from Local System Windows Account to the SDK domain account.

In the Operations Console, go to the Administration pane. Under Security, Run As Account, create an account called SDK and Config Service Account and use the values from the SDK domain account. Then change the Run As Profile for Active Directory Based Agent Assignment Account for the RMS from Local System Windows Account to SDK and Config Service Account.

Restart the SDK Service

In the Cluster Administrator, take the SDK Service offline, then bring it online.

Reference

After registering the SPN, changing the Run As Account, and stopping and starting the SDK Service resource on the cluster, you will get an Information Event ID 11470 in about 20-30 minutes that successfully adds computers to the management server security groups, if you have the assignments defined as detailed in How to Use Active Directory Domain Services to Assign Computers to Operations Manager 2007 Management Servers. Also see Walter's post on Understanding How Active Directory Integration(ADI) Feature Works in OpsMgr 2007.

Upgrading OpsMgr Evaluation Copies to SP1 RTM

Thu, 06 Mar 2008 13:20:38 GMT

Courtesy of: Clive Eastwood

Check out http://blogs.technet.com/cliveeastwood/archive/2008/02/25/upgrading-opsmgr-evaluation-copies-to-sp1-rtm.aspx if you are are having issues upgrading from the evaluation version of SP1. Clive does a fantastic job clearing up some of the confusion. Also, his MGINFO utility is very useful.

OpsMgr 2007 Reporting Installation not Supported on SQL Reporting Services with SharePoint Integration Enabled.

Wed, 12 Dec 2007 16:51:15 GMT

Courtesy of Clive Eastwood:

Summary
=============================
System Center Operations Manager 2007 (Opsmgr) and System Center Essentials 2007 (Essentials)reporting are not supported on SQL Server Reporting Services 2005 when configured in Sharepoint integration mode.

More Information
=============================
If SQL Reporting Services is configured with its server mode configured for SharePoint integration Operations Manager Reporting and Essentials Reporting features will not work correctly. Therefore Opsmgr or Esssentials reporting features cannot share a SQL Reporting Services installation with Microsoft Sharepoint Services. The Check Prerequisites tool in Opsmgr and Essentials does not explicitly check SQL Reporting services to determine if Sharepoint integration has been enabled and therefore will not indicate a failed status.
Although the Reporting feature will pass prerequsite checking and install, reports will fail to deploy to SQL Reporting Services and the Reporting node of the administration console will return an error.

The following Operations manager event will be generated in the Operations Manager event log if Sharepoint integration is enabled:

Event ID : 31569
Source: Health Service Modules
Category: Data Warehouse
Description: Report deployment process failed to request management pack list from SQL_RS Server. The operation will be retired.
Exception 'SoapException'. System.Web.Services.Protocols.SoapException" The option is not supported on a report server that is configured to run in SharePoint integrated mode. <remainder of event removed for clarity>

When attempting to view reports from the administration console the follow dialog will be displayed

Loading reporting hierarchy failed.
Exception 'SoapException'. System.Web.Services.Protocols.SoapException" The option is not supported on a report server that is configured to run in SharePoint integrated mode. <remainder of event dialog text removed for clarity>

If OpsMgr or Essentials reporting has already been installed on a SQL Reporting Services where Sharepoint integration is required, the Opsmgr reporting feature will need to be reinstalled. Essentials will need to be uninstalled and reinstalled in its entirety since the reporting feature is integrated into the Essentials installation.
If SQL Reporting Services Sharepoint integration mode was configured in error during reporting services installation and there is no intention of installing Microsoft Sharepoint Services itself, it is recommended that the Opsmgr reporting feature be uninstalled if present, that SQL Reporting Services be removed and reinstalled without Sharepoint integration.
System center Essentials should be uninstalled together with SQL Reporting services and then both reinstalled without the Sharepoint integration.

Targeting Series, Part 2: Why Targeting a Computer Group Fails

Sat, 17 Nov 2007 11:50:39 GMT

Courtesy of Jonobie Ford:

Targeting a computer group in Operations Manager 2007 produces unexpected results because the computer group target (that is, the class of objects that are computer groups), only exists on the root Management Server. Therefore, any rule that you target to this object class attempts to collect event information from the root Management Server. In some cases, the root Management Server might not have the instrumentation that the rule depends on. For example, a rule that depends on IIS metrics will not be able to collect any information on a root Management Server that has no IIS server on the computer. In other cases, you might collect information that is not what you expect because the rule is intended to collect data on a machine other than the root Management Server.

The only way to send rules to specific computers is to target the rule to an object that exists on the computer, such as a SQL Server object or an IIS Web site object. You can then use overrides to specify the places where the rule works.

For example, consider the scenario where you want to target a rule to all Windows Server 2003 servers that are in a particular computer group. To do this, complete the following steps:

Create a rule that is targeted to the Windows Server 2003 class.
Set the rule so it is not enabled by default.
Create an override that will enable the rule only for objects that are members of the desired group.

Installation Order: Distributed Environment

Tue, 23 Oct 2007 17:25:23 GMT

OpsMgr 2007 introduces two new components, the Root Management Server (RMS) and the gateway Server. When deploying OpsMgr 2007 in a distributed environment, the components can be installed in various orders. In an environment where multiple management servers are employed, I highly recommend that OpsMgr 2007 Reporting Services are installed after the RMS is installed. Once Reporting is installed, then install all subsequent Management Servers. In addition, due to a bug, you may have to inspect the

"Run As Profiles", there are 4 profiles for the Data Warehouse component. If you go into the properties of these profiles and then click the Run As Accounts tab, there should be a target computer entry for each of your management servers. For some
reason, all management server post RMS do not appear. Add them. Verify that all of your management servers are represented. Else, you will see the following errors:

Event data collection process unable to write data to the Data Warehouse
Performance Data Collection Process unable to write data to the Data Warehouse

Network Bandwidth Utilization and Console Performance

Tue, 23 Oct 2007 11:49:13 GMT

One of the biggest pain points in OpsMgr 2007 is console performance. Even with a properly resourced implementation the console may still be slow. In lieu of SP1, which increases console performance tremendously, check the network link configuration between the RMS and the node hosting the console, including all applicable switches. It is very common for the NICs to be configured for 'Auto' under 'Speed & Duplex' as well as their corresponding networked switch port. So if the network is a 100Mbps LAN, then the NIC on the servers, nodes hosting the Operations Console as well as the intermediary switches should be set to 100 Mb Full. At several clients, this increased console performance considerably.

I am not a network expert, but what we observed late one night, while working on something totally unrelated, that if there is any issue on the links, the ports auto-negotiate the speed down to reduce the errors. As a result, you may think you are running at 100 Mbps when in fact you are not. If you are launching the console locally on the RMS then this is not the issue. This is not true for every environment given the multitude of components that comprise networks but it did work in four environments.