Admin Arsenal Blog

Then a miracle occurs (understanding vs. memorizing Systems Management)

It still amazes me how many system administrators don't feel the need to understand the under-pinnings of Systems Management. I push a button and software gets installed on a remote computer. I run an inventory scan and suddenly I have a list of installed software.

I'm constantly reminded of the Sidney Harris cartoon "Then a miracle occurs". How did that software get installed? Where does SCCM extract the list of installed software on each machine? How does Admin Arsenal know if a laptop has a smart-card reader?

If I have a prayer of performing any meaningful troubleshooting when problems occur, then it is essential to understand the How, Where, and What of the my environment.

You can, very quickly, determine if a person troubleshooting a problem has a good understanding of their environment. If they don't understand what's going on under the hood, then their troubleshooting amounts to little more than memorization. If "problem A" occurs, then "solution A" is the answer. "See, it's clearly written in this KB article." But what happens when the correct answer isn't, in fact, "solution A" ? What happens when the problem doesn't even have a description in the knowledge base? 

I know, for example, that Tivoli retrieves its Installed Software list from two sources: A) Data stored in the "Uninstall" key of the registry; and B) Signature data where software names, publishers and versions are mapped using a file name and its corresponding file size.  

If some software that I am expecting to see listed as being installed doesn't show up in an inventory report, then I should know exactly WHERE (on the target computer) that data is supposed to be found. I will be able, therefore, to troubleshoot more effectively. If I know HOW the scanned data is expected to move from the registry of the target computer to its ultimate entry in a database, then I can more effectively determine when and where any errors are occurring during transit.

My next few blog posts will be about some of the under-workings of common systems management tools. 

 

---

 

Follow me on Twitter @ShaneCorellian

 

7 mistakes windows administrators make with PC inventory (part II)

Last Friday I elaborated on the first four mistakes made by windows administrators in their collection of PC inventory. Today I conclude by jumping into more detail on numbers 5-7.

To refresh, here is the list.

1. Limiting inventory collection to entries in Add/Remove Programs
   
2. Excluding important registry entries
3. Giving them what they 'ask for'
4. Limiting inventory scans to .exe files
5. No master baseline table to compare against
   
6. Data is static
7. Poor reporting

Let's jump into number 5. If you scan for all files that are executable and do a count search on any random PC you'll discover that the average workstation will contain well over 10,000 of these types of files. Gathering this much information will make you data rich but information poor. Make your data meaningful by comparing what you collect against what you are expecting (or what constitutes a vulnerability if performing a security scan).This is done by storing an expected baseline in a database table.

Number 6 is the static problem. This usually results when your reporting (or collection) is stored on spreadsheets. This is especially true if data is manually gathered. Be careful with this method - remember that your decisions are only as good as the data that they were based on. A good automated scanning solution writing to a relational database is the most desired method.

Ahh, number 7. This last item is perhaps the most dangerous because it impacts organizations that have the best inventory scanning solutions as well as those who only collect manually.  It's the inability to report what you have.

SQL skills are essential when you're dealing with data stored in a relational database. I learned this years ago when I had a organization level manager ask for a report of all installed software on his computers. When I provided him the list he quickly stated that it was wrong. It was too granular (see item 3).  He didn't want to see every Microsoft hotfix listed.  He also only wanted to see one entry for Microsoft Office, rather than each component (Excel, Word, Powerpoint).

In short he wanted an inventory summary rather than an installed software report, even though he specifically asked for "what was installed on his computers." Remember, what they want and what they ask for are sometimes quite different.

SQL knowledge as well as a baseline of expected software (see item 5) would've done the trick. Instead of listing all the hotfixes installed I should have compared against a table listing required patches and reported the discrepencies.

On the Microsoft Office issue certain components discovered could be compared against a table to reflect the Office installation as a whole.

The frustrating outcome is that this manager concluded that we didn't know what was installed. Even though we were scanning registries, looking for many types of executable files, and even scanning for .exe header information, without the reporting skills I wasn't able to show what we knew. In a way he was right.  I resolved to make reporting a central feature of inventory collection. 

Afterall, if you can't show someone what you know, you might as well not know it.

 

7 PC inventory mistakes made by windows administrators (part I)

Windows administrators are often bearers of news. Sometimes good, sometimes bad. 

When the news is good, you look like a hero. When it's bad you offer suggestions on how to a) deal with repurcussions, and b) avoid the problem next time. While no one likes to give bad news there is something worse... not knowing.

When it comes to knowing what software is installed on the computers that you manage there are myriad ways to discover. Here are the 7 reasons a windows administrator might not know what is installed on their systems:

1. Limiting inventory collection to entries in Add/Remove Programs
   
2. Excluding important registry entries
3. Giving them what they 'ask for'
4. Limiting inventory scans to .exe files
5. No master baseline table to compare against
   
6. Data is static
7. Poor reporting ability
   

Number 1 is quite common. Software inventory is much more than what is listed in Add/Remove. While true that most of what you'll need to report on will be gleaned from this section, sometimes the data you need isn't a stand-alone program per se, but rather a file or group of files that together may cause a vulnerability.

The solution is to determine what you are looking for and then find the right solutions, be they over the counter or custom scanning ability. Keep reading to see some common mistakes made by admins on the quest for too much information.

Number 2 isn't answered easily. The registry is a big place and it's easy to get lost. You need to know what you're looking for before you create a custom scanner to glean registry data. That said, sometimes the registry is the only place to get the data, especially when you are looking for the configuration of specific applications. 

Number 3 is a fun one. We all know that what a customer asks for isn't necesarily what they want. I'll elaborate a bit more in the end of the artcile, but suffice it to say that gathering every piece of information on the computer won't help much if you can't put it into their perspective.

Number 4 is a common mistake made when an admin tries to venture beyond Add/Remove. Simply put, they scan for all .exe's while forgetting that there are many files that are executable.

Keep an eye out for some of the following (certainly not an exhaustive list):

• .exe
• .bat
• .cmd
• .dll
• .vbs
• .js

The next mistake is looking for too much, which takes us to number 5, which I will cover in next Friday's blog post.

IT Administrator Abuses

A recent slashdot article on IT Administrator abuse might more appropriately be titled IT Administrator Biases. It brings up an excellent point which highlights the known fact that, generally speaking, IT departments are horrible at eating their own dogfood.

The example in the article points to the blocking of forums or discussion groups, but leaving wide open the access for Digg, Fark, and Slashdot. 

My experience points to management deference. They either don't understand or don't want to understand the nuances of IT life. They will often accept point blank what their IT guys say. In a way this is beneficial. Afterall, most companies don't make money from their IT departments. 

While it could generate bad will among the rank and file within a company I think little is likely to change.

It goes much deeper than website blocking. IT admins will commonly install software on their systems that has not passed internal security checks or is not documented or even (gasp) licensed. 

I recall running a software inventory scan for a client only to find several sys admins who had sniffer applications installed. To say that 'it' hit the fan would be an understatement. 

All in all it will build a stronger environment if the IT department eats the same dogfood, but then again, how would we get anything done if that were the case?

Determining OS Architecture

 

    Photo by Boston Public Library

What is one of the problems you may encounter when a customer (or your boss) asks you to report how many computers have a 64 bit OS? One problem is that if you simply base the information on the CPU then you run the risk of getting false positives when a 32 bit OS is running on a 64 bit processor.

 

What you want to look for is the Architecture type of the OS. Since a 64 bit OS cannot run on a 32 bit processor you know that if you identify the current OS type then you've accomplished your goal.

I prefer to look in WMI. One problem that you can run into is Windows versions prior to Vista. When looking at Windows XP and Windows 2003 you need to look in the Win32_ComputerSystem class. The property that I have found useful is SystemType. I extract the value from this property and apply a regular expression to extract the first three characters found. Here is a function in Visual Basic .NET that I use.

Dim OSArchitecture As Integer = 32
Dim objCS As New ManagementObjectSearcher("SELECT * FROM Win32_ComputerSystem")
For Each objMgmt As ManagementObject In objCS.Get
Dim TypeMatch As New Regex("(x64)")
Dim CSystemType As String= objMgmt("SystemType")).ToString
If TypeMatch.Match(CSystemType).Success Then
Return 64
End If
Next

For Vista or later I use the Win32_OperatingSystem class and extract the value of the OSArchitecture property. I use a regular expression to match the first two digits encountered.

Dim objOS As New ManagementObjectSearcher("SELECT * FROM Win32_OperatingSystem")
Dim OSArchMatch As New Regex("(^\d\d)")
For Each objMgmt As ManagementObject In objOS.Get
If OSArchitecture = 32 Then
Dim OSArchitectureString As String = GetWmiValue(objMgmt, "OSArchitecture").ToString
If Not OSArchitectureString = "" Then
If OSArchMatch.Match(OSArchitectureString).Success Then
OSArchitecture = AsInt(OSArchMatch.Group(0))
Return OSArchitecture
End If
End If
End If
Return 32
Next

I'm always interested in more efficient methods for extracting this type of information. If you know of other methods I can use then, by all means, let me know.

Where can I find a device's firmware version in WMI?

Many hardware vendors store the firmware version of their devices in WMI. Extracting this value can give you some very important information when it comes to determining which devices may need to have their firmware updated.

In this example I am using the Microsoft WMI Object Browser (available free for download here).

Let's say that I want to script a solution that reports the Firmware version for all the Smart Card Readers in my organization. In the following screenshots you can see where this data can be viewed via the Device Manager:

By looking at the properties of the device I can see both the Device Instance ID (this is important when you need to extract this device via WMI) and the Firmware Version.

When you open the WMI Object Browser change the namespace from root\cimv2 to root\WMI.

The property that we are interested in is FirmwareRevision.

Here is an example of extracting this property in Visual Basic .NET.

Private Function GetFirmware(ByVal id As String) As String
Dim firmWare As String = ""
Try
Dim objFirmWare As New ManagementObjectSearcher("\\.\root\WMI", "SELECT * FROM MSDeviceUI_FirmwareRevision where InstanceName = '" & id.Replace("\", "\\") & "_0'")
For Each objMgmt As ManagementObject In objFirmWare.Get
firmWare = AsString(GetWmiValue(objMgmt, "FirmwareRevision"))
Next
Catch ex As Exception
' do nothing
End Try
Return firmWare
End Function

When this function is instantiated it processes an argument called ID. This ID is extracted in a previous Class via the Microsoft tool DevCon.exe. Devcon.Exe is a decent command line utility used to extract information available via Device Manager. While this article is not centered on Devcon.exe, I will include an example of the output of devcon.

As you can see from the example, the first line returned is the DeviceID that I need in my WMI Query.

As always, if you have questions, comments or smart-ass remarks, let me know.

Collections vs. Collection Data

Dynamic Collections are an extremely simple, yet powerful, tool that can be used to breakdown your environment into logical groupings. For instance, you can have a Collection that specifies that all member computers must have Symantec Antivirus installed, or must have at least 2 GB of RAM.

There are two tools within Admin Arsenal that you can use to help you be more effective with your Collections. The first is the Export / Import command.

Export / Import allows you to either share or receive Collections that have been previously defined, perhaps by another Administrator. If you have defined a great Collection go ahead and export it and offer it to other AA Administrators. This can save duplication of effort plus you can look like a bad ass.

To export a Collection (or a Collection Folder) go to Admin Arsenal, right click on a Collection and select Export...

An XML file will be created. You or any other AA Administrator can take that XML file and import it into another AA console.

In a nutshell, the Export command allows you to save or share the collection DEFINITION. If you want to export or extract the Collection data (such as member computers, etc.), then you will need to use a different command altogether.

With a Collection highlighted, simply go to your Computer menu and select Save Computers As...

Enter the name of your file and hit Save. You can now import this data into Excel or Lotus 123 or another reporting tool.

Service Pack Blues

It's important to know which of your computers don't have the latest Windows Service Packs. I'm going to show you a quick way to find this out using the inventory in Admin Arsenal.

Our goal is to create a collection for each operating system, and then roll them up into one collection folder which will give a birds-eye view of all the missing service packs. In the end, you should have collections that look like this:

The first step is to create a collection folder. Once you have the folder, right click on it and select Any Children under Collection Folder Rollup. This will show all of the computers that appear in any of the child collections we create.

Below this collection folder, create a dynamic collection for each operating system. Each collection will need two filters, one for the operating system and one for the service pack. For example, the XP collection looks like:

Once you're done, all you need to do is select the top folder to see all of your computers which are out of date. Collections are "live" in that when computers get re-scanned, the members of the collection will be updated automatically.

You can tweak any of the collections to meet your needs. You may have, for example, some computers which cannot or should not be updated. You could then add filters to the appropriate collections to keep them from showing up.

To save you time, I've exported the collections used in in the example (out-of-date-service-packs.xml) Import them and use them to your heart's content.

Heartbeat vs. Scan - Smackdown!

Admin Arsenal provides two methods for getting information about computers into the database: Heartbeat and Inventory Scan. The question in some peoples' minds is "Why two methods, what's the difference?" But, wait, that's two questions, you might say. Yes it is, but I'll forgive the imaginary person that I'm having a conversation with, because I don't want to make him mad. The answer to these questions revolves around performance.

It would be great if Admin Arsenal could grab all information from all your computers in an instant, but the reality is that it takes time. It's also reality that some data change more often than others and is therefore more time sensitive. Balancing these two realities in a way to give administrators the best experience, we decided to split the data gathering into two processes. Heartbeat is intended to run more often and to get information that is both quick to gather and changes more frequently. Typically, inventory scans run once a day, so heartbeats get information that is likely to change more often than that. To that end, heartbeat gathers the following:

• Online/Offline Status. This comes from a standard ICMP ping.
• Current User. Users log into and out of computers during the day, so this is quite volatile data.
• Boot Time/Uptime. Computers are also rebooted during the day, so their uptime may change frequently.

Inventory scans, on the other hand, gather the following:

• Everything else.

We'd love feedback from users if this is a good balance. Should more be added to the heartbeat? Should we make it so that you can move data back and forth between them? Should we add a third method somewhere between the two? Just who is my imaginary friend and why do I fear him?

You've got it (and they want it)

At some point nearly everyone in your organization will want it.

And they'll want it from you.

And they'll want it from you for one reason; they'll think that you have it.

The common thread that connects you to your manager, CFO, HR manager, facilities person, and coffee breath guy is data. Truth be told, data is the biggest reason that you have a job right now. After all, IT is nothing more than agateway to data.

So when they come asking, be sure to get two answers from them:

1. Why do they want it?
2. What will they be doing with it?

Your reasons for needing the answers have nothing to do with being a power-hungry Admin (there are other reasons for that). You need to know because each department has their own definition of data, as well as their own reason for needing it.

You have data in a ton of different locations. Databases, registries, flat files, bios', etc. Depending on what the answers are will tell you which place is the best location to retrieve the data.

Your manager wants to know how many memory dimms the company has. You provide the answer from your blazing fast database and she reports the numbers on up and everyone is fat, dumb, and happy. Unbeknownst to your boss is the fact that your data only includes the memory currently in use on the network. You've left out the older (but yet to be decommissioned) computers currently serving a role as door stops, or the 512MB dimms recently replaced by 2GB dimms in preparation for Vista. Oh, and there are the new laptops that came in last week but haven't yet been placed on the network...

Suddenly the data that your boss has is incomplete. But the problem wasn't the data. Your data was perfect. You were right. The problem was the scope of the data. This is why you need to know why they need what they say that they need.

OK, so your manager has lower than expected numbers to report to accounting. That's an inconvenience, but it probably won't lead to the death of the company. However, some errors are less forgiving.

Your security department believes that a disgruntled ex-employee may be responsible for recent sabotage. They may ask you for his old laptop. But is that enough? Does your security department know that this guy logged onto 37 different workstations and 5 different servers in the week before he was fired? If you have the data, and know the answers to their two questions, you'll be able to give them what they pay you for. Information.