Kåre Rude Andersen - Be a hero – optimize scom and present your services
Download as PPTX, PDF0 likes9,895 views
This document summarizes a presentation on optimizing System Center Operations Manager (SCOM) and presenting services. The presentation covers defining services and service level objectives, identifying performance bottlenecks at the database, management server, and agent layers, and using the SCOM SDK for automation including automatically changing alert resolution states and resending alerts. Code examples are provided for monitoring database performance, identifying SQL and I/O wait times, and automating tasks using PowerShell and the SCOM SDK.
![Layer 1 - Database
Memory – Page Life Expectancy
SELECT cntr_value AS [Page Life Expectancy]
FROM sys.dm_os_performance_counters
WHERE [object_name] = N'SQLServer:Buffer Manager’
AND counter_name = N'Page life expectancy' OPTION
RECOMPILE);](https://image.slidesharecdn.com/krerudeandersen-beaherooptimizescomandpresentyourservices-140128143047-phpapp02/85/Kare-Rude-Andersen-Be-a-hero-optimize-scom-and-present-your-services-11-320.jpg)
![Layer 1 - Database
Disks – Database Performance against SAN, DAS
SELECT DB_NAME(fs.database_id) AS [Database Name], mf.physical_name, io_stall_read_ms,
num_of_reads,
CAST(io_stall_read_ms/(1.0 + num_of_reads) AS NUMERIC(10,1)) AS
[avg_read_stall_ms],io_stall_write_ms,
num_of_writes,CAST(io_stall_write_ms/(1.0+num_of_writes) AS NUMERIC(10,1)) AS
[avg_write_stall_ms],
io_stall_read_ms + io_stall_write_ms AS [io_stalls], num_of_reads + num_of_writes AS
[total_io],
CAST((io_stall_read_ms + io_stall_write_ms)/(1.0 + num_of_reads + num_of_writes) AS
NUMERIC(10,1))
AS [avg_io_stall_ms]
FROM sys.dm_io_virtual_file_stats(null,null) AS fs
INNER JOIN sys.master_files AS mf
ON fs.database_id = mf.database_id
AND fs.[file_id] = mf.[file_id]
ORDER BY avg_io_stall_ms DESC OPTION (RECOMPILE);](https://image.slidesharecdn.com/krerudeandersen-beaherooptimizescomandpresentyourservices-140128143047-phpapp02/85/Kare-Rude-Andersen-Be-a-hero-optimize-scom-and-present-your-services-12-320.jpg)
![Layer 1 - Database
Disks – Database Performance against SAN, DAS
SELECT DB_NAME(fs.database_id) AS [Database Name], mf.physical_name, io_stall_read_ms,
num_of_reads,
CAST(io_stall_read_ms/(1.0 + num_of_reads) AS NUMERIC(10,1)) AS
[avg_read_stall_ms],io_stall_write_ms,
num_of_writes,CAST(io_stall_write_ms/(1.0+num_of_writes) AS NUMERIC(10,1)) AS
[avg_write_stall_ms],
io_stall_read_ms + io_stall_write_ms AS [io_stalls], num_of_reads + num_of_writes AS
[total_io],
CAST((io_stall_read_ms + io_stall_write_ms)/(1.0 + num_of_reads + num_of_writes) AS
NUMERIC(10,1))
AS [avg_io_stall_ms]
FROM sys.dm_io_virtual_file_stats(null,null) AS fs
INNER JOIN sys.master_files AS mf
ON fs.database_id = mf.database_id
AND fs.[file_id] = mf.[file_id]
ORDER BY avg_io_stall_ms DESC OPTION (RECOMPILE);](https://image.slidesharecdn.com/krerudeandersen-beaherooptimizescomandpresentyourservices-140128143047-phpapp02/85/Kare-Rude-Andersen-Be-a-hero-optimize-scom-and-present-your-services-13-320.jpg)
![Resend Alert again, again a…
$oldAlerts = Get-SCOMAlert | Where-Object {($_.LastModified -ge
[DateTime]::Now.AddHours(-4)) -and ($_.ResolutionState -eq 0)}
ForEach($alert in $oldAlerts)
{
$alert.Update("")
}](https://image.slidesharecdn.com/krerudeandersen-beaherooptimizescomandpresentyourservices-140128143047-phpapp02/85/Kare-Rude-Andersen-Be-a-hero-optimize-scom-and-present-your-services-28-320.jpg)
Kåre Rude Andersen - Be a hero – optimize scom and present your services
- 1. Kåre Rude Andersen Be a Hero – Optimize SCOM and present your Services
- 2. Who am I • • • Kåre Rude Andersen - @KraCoretech Consultant / Speaker / Trainer / Member of Microsoft SC Alliance Team Coretech A/S | mail: [email protected]
- 3. Agenda • • • • • • Introduction to Services Ready Steady – GO! Present SLO and SLA in a Dashboard Bottlenecks – SCOM Performance Whats New in R2 Use cases
- 4. Services Concept • ITEL/MOF Process • People, Process and IT • Service Alert Escalation • Service Owners (layer 1-2-3) • 1/3 have a defined service plan, and less downtime
- 5. Services Concept • • • • • Clear Goal Base Monitoring What is ”Normal” Marketing yourself – Do your colleagues know about the fantastic work you are doing? Get the Control and Console • • • • Find those SQL Report Servers installed on HP HW Find the objects and the error Security Services – Use the products (AEM/ACS/APM) Service Catalog (CMDB – Service Manager)
- 6. Ready, Steady - GO! • • • • • • • Define the Goal Define Service Owners and let them in.. Define which dashboard your organization will see Do a PoC, and Test Built-in Dashboard Visio Maps / Sharepoint 3. part solutions
- 7. SLA Dashboard • Create DA’s or Groups in SCOM • Define your Services SLA and SLO • Create Rollup and Present
- 8. Demo • Create a Group • Put SLA on it • Present it in varios ways
- 9. SBTOS
- 10. Bottleneck • Is it slow? • Identify the individual layers • What’s Normal???
- 11. Layer 1 - Database Memory – Page Life Expectancy SELECT cntr_value AS [Page Life Expectancy] FROM sys.dm_os_performance_counters WHERE [object_name] = N'SQLServer:Buffer Manager’ AND counter_name = N'Page life expectancy' OPTION RECOMPILE);
- 12. Layer 1 - Database Disks – Database Performance against SAN, DAS SELECT DB_NAME(fs.database_id) AS [Database Name], mf.physical_name, io_stall_read_ms, num_of_reads, CAST(io_stall_read_ms/(1.0 + num_of_reads) AS NUMERIC(10,1)) AS [avg_read_stall_ms],io_stall_write_ms, num_of_writes,CAST(io_stall_write_ms/(1.0+num_of_writes) AS NUMERIC(10,1)) AS [avg_write_stall_ms], io_stall_read_ms + io_stall_write_ms AS [io_stalls], num_of_reads + num_of_writes AS [total_io], CAST((io_stall_read_ms + io_stall_write_ms)/(1.0 + num_of_reads + num_of_writes) AS NUMERIC(10,1)) AS [avg_io_stall_ms] FROM sys.dm_io_virtual_file_stats(null,null) AS fs INNER JOIN sys.master_files AS mf ON fs.database_id = mf.database_id AND fs.[file_id] = mf.[file_id] ORDER BY avg_io_stall_ms DESC OPTION (RECOMPILE);
- 13. Layer 1 - Database Disks – Database Performance against SAN, DAS SELECT DB_NAME(fs.database_id) AS [Database Name], mf.physical_name, io_stall_read_ms, num_of_reads, CAST(io_stall_read_ms/(1.0 + num_of_reads) AS NUMERIC(10,1)) AS [avg_read_stall_ms],io_stall_write_ms, num_of_writes,CAST(io_stall_write_ms/(1.0+num_of_writes) AS NUMERIC(10,1)) AS [avg_write_stall_ms], io_stall_read_ms + io_stall_write_ms AS [io_stalls], num_of_reads + num_of_writes AS [total_io], CAST((io_stall_read_ms + io_stall_write_ms)/(1.0 + num_of_reads + num_of_writes) AS NUMERIC(10,1)) AS [avg_io_stall_ms] FROM sys.dm_io_virtual_file_stats(null,null) AS fs INNER JOIN sys.master_files AS mf ON fs.database_id = mf.database_id AND fs.[file_id] = mf.[file_id] ORDER BY avg_io_stall_ms DESC OPTION (RECOMPILE);
- 14. Layer 1 – SQL – Why Wait? WITH Waits AS (SELECT wait_type, wait_time_ms / 1000. AS wait_time_s, 100. * wait_time_ms / SUM(wait_time_ms) OVER() AS pct, ROW_NUMBER() OVER(ORDER BY wait_time_ms DESC) AS rn FROM sys.dm_os_wait_stats WHERE wait_type NOT IN ('CLR_SEMAPHORE','LAZYWRITER_SLEEP','RESOURCE_QUEUE','SLEEP_TASK', 'SLEEP_SYSTEMTASK','SQLTRACE_BUFFER_FLUSH','WAITFOR', 'LOGMGR_QUEUE','CHECKPOINT_QUEUE', 'REQUEST_FOR_DEADLOCK_SEARCH','XE_TIMER_EVENT','BROKER_TO_FLUSH','BROKER_TASK_STOP','CLR_MANUAL_EVENT', 'CLR_AUTO_EVENT','DISPATCHER_QUEUE_SEMAPHORE', 'FT_IFTS_SCHEDULER_IDLE_WAIT', 'XE_DISPATCHER_WAIT', 'XE_DISPATCHER_JOIN', 'SQLTRACE_INCREMENTAL_FLUSH_SLEEP', 'ONDEMAND_TASK_QUEUE', 'BROKER_EVENTHANDLER', 'SLEEP_BPOOL_FLUSH')) SELECT W1.wait_type, CAST(W1.wait_time_s AS DECIMAL(12, 2)) AS wait_time_s, CAST(W1.pct AS DECIMAL(12, 2)) AS pct, CAST(SUM(W2.pct) AS DECIMAL(12, 2)) AS running_pct FROM Waits AS W1 INNER JOIN Waits AS W2 ON W2.rn <= W1.rn GROUP BY W1.rn, W1.wait_type, W1.wait_time_s, W1.pct HAVING SUM(W2.pct) - W1.pct < 99 OPTION (RECOMPILE)
- 15. SQL – Database Performance • SQL Standard settings • • • • • MDOP Stripe the Database (Tempdb and OpsMgr) Max Memory Truncate log on Checkpoint Always On http://blogs.technet.com/b/manageabilityguys/archive/2013/ 09/20/quot-mdoping-quot-your-scom-console-performancewith-some-simple-sql-tips.aspx
- 16. Layer 2 – Management Server • • • • Disk, Disk, Disk, Disk, Disk Disk, Disk, Disk, Disk, Disk Disk, Disk, Disk, Disk, Disk CrystalDiskMark
- 17. Layer 2 – Management Server • Memory • Installed connectors on the Management Server • Distribute the Workload • Resource pools • GW servers if round time threshold > 20 ms • Processer: • HKLMSOFTWAREMicrosoftMicrosoft Operations Manager3.0GroupCalcPollingIntervalMilliseconds
- 18. Layer 3 – Agent, Linux, Network • Management Pack Design – discoveries • Scripts, Powershell timeout • MaximumQueuesizeKB • Network and Linux
- 19. Demo • SQL Server Performance
- 20. SDK • Get down to it • Good samples
- 21. SDK – Web Services param($ServiceNowUrl,$AlertXML) $ErrorActionPreference = "stop" Import-Module OperationsManager $Alert = Get-SCOMAlert -Id $AlertXML.AlertId #login SOAP $username = "MIDserver3M" $password = "password" | ConvertTo-SecureString -asPlainText -Force $credential = New-Object System.Management.Automation.PSCredential($username,$password) $url = 'https://MMMdev.service-now.com/u_scom.do?WSDL' $sn = New-WebServiceProxy -uri $url -Credential $credential #get autogenerated namespace $type = $sn.GetType().Namespace $insert = new-object ($type + '.insert') #Insert a new record with these values $insert.acknowledgesync = "" $insert.category = $Alert.Category $insert.customfield1 = $Alert.Custom1 $insert.customfield2 = $Alert.Custom2 $insert.description = $Alert.AlertDescription $insert.id = $Alert.AlertId $insert.managementgroup = $Alert.managementgroup #TODO $insert.monitoringobjectfullname = $Alert.ManagedEntityFullName $insert.monitoringobjectid = $Alert.ManagedEntity $insert.monitoringobjectpath = $Alert.ManagedEntityPath $insert.name = $Alert.AlertName $insert.netbioscomputername = $Alert.netbioscomputername #TODO $insert.owner = $Alert.AlertOwner $insert.parameters = $Alert.AlertParams $insert.principalname = $Alert.principalname #TODO $insert.priority = $Alert.priority $insert.repeatcount = $Alert.RepeatCount $insert.resolutionstate = $Alert.ResolutionStateName $insert.severity = $Alert.severity $insert.sitename = $Alert.sitename #TODO $insert.timeraised = $Alert.TimeRaised $insert.timeresolution_telastmodified = $Alert.TimeResolutionStateLastModified $result = $sn.insert($insert) $Alert | Set-SCOMAlert -TicketId $result.display_value
- 22. SDK – Get the Guids from Name
- 23. SDK – Get HealthState
- 24. Use-Cases • Automatic Maintenance Mode • Alert Robot • Resend Alert again
- 25. Automatic Maintenance Mode Use either: http://www.systemcentercentral.com/opsmgr-2012-groupmaintenance-mode-via-powershell-the-way-it-should-be or http://gallery.technet.microsoft.com/scriptcenter/Put-OM2012Computer-Group-43902672 Next slide
- 26. Automatic Maintenance Mode Create a Command Channel that runs a script like this:
- 27. Alert Robot Import-Module OperationsManager $Alerts = get-scomalert -ResolutionState 0 if ($Alerts) { foreach($Alert in $Alerts) { $newState = $null switch -wildcard ($Alert.Description) { "*Active directory*" { $newState = 10 } "*DNS*" { $newState = 30 } "*Cisco*" { $newState = 250} } switch -wildcard ($Alert.Name) { "ACME.Monitor.Event.Dummy.100" { $newState = 249 } "Another Alert Text" { $newState = 10 } } if($newState -ne $null) { $Alert.ResolutionState = $newState $Alert.Update(“Resolution State changed automatically by the QueueManager Robot”) } } }
- 28. Resend Alert again, again a… $oldAlerts = Get-SCOMAlert | Where-Object {($_.LastModified -ge [DateTime]::Now.AddHours(-4)) -and ($_.ResolutionState -eq 0)} ForEach($alert in $oldAlerts) { $alert.Update("") }
- 29. Summary • Present your Services • Find the bottlenecks • Implement Automation
- 30. Please evaluate the session before you leave