Posted by Adam Ruth on Wed, Sep 08, 2010
Photo by Don Hankins
We all know that malware is evil, but like most evil things there are some valid uses for them (that's true, right? Evil things can have uses, even in a Time Bandits sort of way? I thought so.)
Well, in case you don't believe me, here are the top 10 uses for malware that don't require you to be evil.
10. As a little present for the Nigerian 419 scammers when you send them your computer password so they can get your bank account numbers.
9. Any prank involving that guy from sales who keeps making fun of your tradeshow t-shirts.
8. Keyboard logging on your dad's computer so you can see what he typed right before "it broke and I swear I didn't change anything!"
7. Creating an unscheduled downtime emergency to get excused from a boring meeting.
6. Watching for references to computers on Hollywood scriptwriting computers and making the necessary changes so that the plot is somewhat in touch with reality.
5. Infecting the BIOS of your uncle's 12 year old Packard Bell computer so you can finally convince him that it's time to upgrade.
4. Making OS X feel more familiar to Windows users.
3. Showing up that obnoxious jerk at the class reunion by taking over the slide projector and showing Photoshopped pictures of him in his underwear.
2. Shutting down a real estate developer's computers to prevent the destruction of a building housing a rag-tag group of lovable orphans.
1. Defcon groupies.
Follow me on Twitter
Posted by Shane Corellian on Mon, Sep 06, 2010
We were deploying Tivoli Management Agent (TMA) out to about 15,000 computers in 2001. There were about 170 administrators that were responsible for all these systems. There was A LOT of resistence from those administrators about adopting a centralized management system like Tivoli Framework. I will never forget one of the messages we received from an administrator who clearly had no concept of proper troubleshooting. The message, in part, read:
"We cannot install the TMA on our 230 computers because it breaks Oracle".
I replied with a request for her to be MUCH more specific. Her response was priceless:
"We feel that the TMA may be causing the Oracle client to lose connection to the Oracle database." (Italics added to demonstrate stupidity...I mean for emphasis.)
"May be causing ...?!" To paraphrase B.J. Honeycutt: "So far I have a definite possibility of three absolute maybes".
A co-worker of mine responded with the famous flea/cricket story used to demonstrate improper troubleshooting and to call her out on her text book use of the "post hoc ergo proctor hoc" logical fallacy.
A scientist taught a flea to jump on command. Out of curiosity he thought he would do some experiments with his trained flea. "Jump!" he yelled, and the flea jumped six inches into the air. The scientist then pulled off two of the flea's legs and yelled "Jump!". This jump was only four inches high. He ripped off two more legs and the jump was reduced to two inches. After the last two legs came off the flea didn't jump anymore. The scientist then wrote a paper explaining how if you pull all the legs off a flea, the flea goes deaf.
Needless to say, this email didn't go over well but that's a story for another time.
My co-worker and I wrote another email detailing some things we wanted her to document for us.
- Basic info: OS, Oracle client version etc.
- How consistent are the errors? Are these errors present on multiple machines?
- Please duplicate the error and send us the relevant Event Log entries.
- Can you duplicate the error with the TMA service turned off?
- Are there any articles on IBM or Oracle support sites documenting these problems?
There were a few more suggestions but you get the point. We wrapped it up with an invitation to come to their department and perform these tests ourselves.
We never heard back. The TMA was installed as planned.
I know, trust me, how painful it can be to have customers or end-users make blanket accusations or knee-jerk explainations of their problems. The most common is probably "The network is down!" because they can't get to a particular website or a print queue is backed up. The problem is that many of our management tools or configuration settings MAY be causing the problem that a user is experiencing. We have to remember that we have one or more swimmers in the public swimming pool that is a distributed computer environment. Maybe one of our swimmers is peeing in the pool. We can't have a knee-jerk reaction to what is perhaps the user's knee-jerk reaction because we may be at fault.
Keep the emotions in check, work the problem and save your frustrations and ranting for poker night or the occasional blog entry.
Follow me on Twitter @ShaneCorellian
Use PDQ Deploy to deploy software to your computers. It's fully functional, fully free and 99.6% urine free.
Posted by Adam Ruth on Mon, Aug 30, 2010
Photo by Frankie Roberto
IT Expert Voice has a great article detailing what's coming up with USB 3.0 and what to expect. It's very informative and answered a few questions I had (such as backward compatibility and the nature of the cables and connectors.) A good and quick read if you're interested in what's coming down the pike.
The final section talks about Intel's Light Peak as a competitor and whether USB 3.0 will have the juice to fight against it. It's certainly a compelling discussion and it brings to mind other classic technology battles. It most closely reminds me of USB 2.0 vs FireWire. FireWire had a lot going for it, and I was in its camp thinking that it would win dominance over USB 2.0. But, as things usually seem to go, I was way off. A lot of people were.
Why is it so hard to predict the direction of technology? I think this question is really just a part of the larger questions of trying to predict markets generally. Markets and technology work like evolution, it's the strongest that survive and it's not always clear what attributes make them strongest. USB 2.0 won out over FireWire for a number of reasons, some of which are obvious, and some of which are not quite clear. Even in retrospect it's not always easy to see the factors that lead to one "species" surviving in the great marketplace of ideas while other seemingly stronger contenders fall by the wayside.
The lesson for me is to never be too confident in backing a single horse in the race. It's prudent to hedge those bets a bit. I'd love to just focus on one programming language or platform, but it would be a problem if that platform is pulled out from under me. I've known programmers who put all their eggs in one basket and never spend any time learning anything else. That may work well for them in the present, but the future is too hazy to see how that will continue to work.
In short, I'm scared to stop learning new and different things. I don't want to end up a dinosaur fossil in a programming museum.
Follow me on Twitter @AdamRuth
Posted by Adam Ruth on Fri, Jul 30, 2010
I've been recently catching up on watching Red vs. Blue which is a web video series created by a company called Rooster Teeth. If you're already familiar with the show, then you can skip all of this. Otherwise, you're in for a treat.
Red vs. Blue is a comedy series set in the world of Microsoft's Halo video game. The footage is all from within the game itself, with conversations between characters as they get into various predicaments. The basic premise of the show is a satirical look at these types of first-person shooter games. The Red and Blue armies have outputs located in an isolated box canyon called Blood Gulch, which is just the kind of map that you might find yourself in while playing the game. No real reason for being there, other than to set up a battle.
The show's greatest strength is its characters, who are a collection of clueless misfits bumbling around their artificial world finding new and interesting ways to irritate each other. Plenty of nerd based humor makes its way into the dialog and situations, which is funny even if you've never played a computer game. Each episode is a short couple of minutes, and you can watch most of them on the RoosterTeeth YouTube channel.
Enjoy!
Follow me on Twitter @AdamRuth
Posted by Adam Ruth on Mon, Jul 19, 2010
Photo by kretyen
This article on Tech Republic showing a computer that was the victim of a lightening strike got me thinking. (WARNING: if you're squeamish about graphic computer carnage, don't view the photos.) What it got me thinking about was static and computers. Back when I first got into IT I constantly heard about the dangers of static electricity. It was only the most daring soul who would breach the hermetic seal of the computer case without guaranteeing they were properly grounded.
It didn't take me long to start thinking that there was more hype than substance to the claims of static damaging computer components. I have no doubt that are plenty of components lying in pauper's graves that were killed by an errant arc from someone's fingertip. But how many times was static the go-to scapegoat when something went wrong that no one wanted to bother explaining? I remember very early in my career when I accidentally dropped a screw onto the motherboard of a running computer, so I am certain why that ISA slot was broken (I did say it was early in my career.) However, there was a time when the mysterious death of a component almost required that static be the first straw grasped.
It hardly seems that way anymore. "You must have static-ized it," is very rare these days. Have we System Administrators gotten better at dealing with static or do we know better that static is usually just an easy excuse? I'm sure it's still possible to build up enough of a charge to fry some modern components, but how likely is it while standing on a tiled floor after touching metal racks and server casings? I'm sure that if your server room is located in the middle of a cotton-ball factory carpeted with 1970s era shag and your company uniform is a corduroy jumpsuit with fuzzy bunny slippers you may want to slip on a grounding strap. Otherwise, we've learned to be less paranoid.
So what does that tell us about modern computing concerns? Are there things today that are like the static of yesteryear where blame seems to automatically go in a knee-jerk fashion? It seems to be when some new, and not yet fully ripe, technology starts to get adopted it continues to be blamed for problems long after those problems have been ironed out. That makes sense, because there was a time when the problems were real, so the knee-jerk reaction was probably right.
Windows instability seems like one of those historical issues. There was a time when a BSOD was almost certainly a bug in the OS and a reboot was the fix. Nowadays, though, a BSOD is more likely to be a hardware problem that isn't going to go away with a simple reboot. I'm sure you can think of some others.
Follow me on Twitter @AdamRuth
Posted by Adam Ruth on Fri, Jul 16, 2010
Photo by dhepnar
I've been following the story of IPv6 for a while now, and I've been following it like I believe most system administrators have been. Which is the standard waiting-for-when-I-have-to-drop-everything-and-get-it-implemented way. I'm sure that this attitude makes some people unhappy, after-all we're running out of IPv4 addresses and I'm keeping us from solving the problem. If we all took the time now to get things switched over then we could avoid IP Addressageddon.
Well, it's not as bad as all that. In my, admittedly limited, experience I have learned a few things which makes me a little less worried. Less worried, but still a bit worried.
1. The death of IPv4 has been greatly exaggerated.
Available IPv4 addresses were supposed to run out this year, but that date has been pushed back to 2011 despite a (somewhat) mad rush on addresses. The date will probably be pushed back again, but each time it does, the cost of not transitioning will increase. No one wants to be left without a chair when the music stops, but on the flip side, no one wants to spend time or money now when there still seem to be quite a few chairs left.
Personally, I think that IPv4 will be with us long until after the IPv6 transition is essentially complete and that some form of technology will keep it alive in the same way that there are still people on dial-up. In the same vein, though, there are good reasons to get off of dial-up.
2. Vendors are already there, so there's no excuse!
One thing that I hear from IPv6 advocates (zealots?) is that vendors are supporting IPv6 in droves and so it should be simple to move our networks over. IPv6 has been in Windows, Linux, various UNIX, and OS X for years (with hardware for about as long) what's the hold-up?
Well, implementing IPv6 at the vendor level is (not to put too fine a point on it) a helluva lot easier than at the system administrator level. It's nice having all of the pieces of the puzzle, but it's the job of putting them together that is the hard work.
3. IPv4 mapping could have been better.
I think that the IPv6 standards committee dropped the ball here. I get the feeling (and this is based solely on my impression, I haven't done enough research to verify) that IPv6 was designed to be as minimally compatible with IPv4 as possible. The idea seems to have been that by making it harder to move to v6 piecemeal it would make the transition happen faster. If that is the case, then I think that the opposite is the result. By increasing the cost of a slow rollout of v6 it made it more attractive to stick with v4 as long as possible.
I may be completely wrong on this, I will admit. There may be technical reasons why a move to v6 couldn't have been very smooth. But I'm thinking of NAT here. When NAT came out it was a godsend for many reasons, even though it was a kludge that has a whole new set of problems. It hit where it needed to, by allowing new nodes to be added with very little disruption to existing infrastructure. IPv4 mapping seems to be trying to provide a seamless move to v6 but either it's not as good as it could be or it has really bad PR.
4. IPv6 readability.
IPv6 addresses are hard to read. There, I said it, you can now come to confiscate my nerd credentials.
Really, though, I understand that v6 addresses are 4 times longer than v4 addresses and there's no way that they can be as easy to remember and work with. But the 4 octet scheme of v4 is so ingrained in sys admin culture that it seems crazy to depart from it so much with v6. Sure, eventually we'll all be thinking in v6 subnets and addresses but it's a big jump. I can't help but think that the v6 representation came out of academia instead of from the trenches.
5. Subnetting is still important.
When the move to v6 finally comes, don't give up all of the knowledge you've built up around subnetting. Sure, even the smallest v6 address allocation provides for 65536 subnets each with billions and billions of nodes. But that only takes care of part of the issues with subnetting today. It's still necessary, although easier, to understand where to divide subnets relative to physical routing. It'll be much easier, but not automatic.
Summary
IPv6 is inevitable, so let's all keep an eye on the bouncing ball. But don't panic just yet, the tipping point is still a ways away. It would be nice if the move to v6 was as easy as rolling out a new OS (even though that certainly isn't easy, but it is easier.) But it isn't, so we sys admins will continue to do what we always have done: Make it all work somehow.
Follow me on Twitter
Posted by Shane Corellian on Wed, Jul 07, 2010
The fun side of studying
Part III of my "Understanding or Memorizing" series. In 2006 a friend and former co-worker came into our offices and saw Adam reading about some new Systems Management technologies that were in the early phases of acceptance. Our mutual friend went up to Adam and, interrupting his studies, said "Why are you studying? I thought you were a wizard!" I still love Adam's reply (paraphrased) "If I am a Wizard then it's only because I actually do study."
Touché. (By the way, Adam IS a wizard. Ask anyone who has ever worked with him...that dude is siiiick.)
Keep sharp. Be constantly learning about new and evolving technology, especially if it has to do with your career. Do you know where to go when you have questions? Have you signed up with the SysAdmin Network? What about ServerFault and The Nubby Admin?
I live and die by O'Reilly books. Mine are dog-eared, written on, highlighted, and some pages have tea or coffee stains.
Eric Sink wrote a fantastic book called "Eric Sink On The Business Of Software". This was one of the best books that I read in 2007. On Page 108 he gives some recaps related to hiring qualified developers (easily translated to Sys Admins). The first three points are of special interest to me:
- Can this candidate bring something to the team that nobody else has?
- Is this candidate constantly learning?
- Is this candidate aware of his/her weaknesses and comfortable discussing them?
I love being around people who are self-aware. Know your weaknesses as well as your strengths! One of the best statements made by Mr. Sink is found on page 125. "...Worrying about how others perceive your [aptitude/intelligence] is a waste of time. The key to a great career is to focus on [learning]..."
Learn or burn.
Follow me on Twitter @ShaneCorellian
Posted by Shane Corellian on Wed, Jun 30, 2010
To continue from my last article...
The difference between an entry level computer support technician and a seasoned, trained and efficient Systems Administrator can be summed up with three simple words:
Understanding vs. Memorizing
In 1995 my boss (and technical mentor), Tom Mann, asked me to network the office computers (running Windows 3.11 for Workgroups). He purchased Novell NetWare 3.12 and told me get the network up within two weeks. I had never done such a thing and I asked him, "Um, how do I do that?" His response will be forever etched in my mind.
"Figure it out."
I went right out to Barnes and Noble and purchased Using Netware 3.12. I started reading at work. Remember that episode of The Simpsons where Homer wants to keep his job at the bowling alley so he decided to come up with a brilliant marketing scheme? You see him reading a book called Advanced Marketing which then fades into him reading another book called "Beginning Marketing" which then fades into him reading the dictionary and looking up the word "Marketing". Well, that was how I felt. I didn't understand a lot of the concepts so I had to read and re-read. I ended up buying several books to answer new questions that I had. I built the 3.12 server at least 6 times before going into production with our small office. I began to understand via simple cause and effect. I was eventually able to correctly predict behavior if I changed configuration settings on the server or clients.
By building that little network I began to truly understand the "problems" that would eventually pop up. When Beth complained that the printer wasn't "working" I was able to quickly deduce if the problem was caused by the server, the network, the local computer configuration or, in this case, the user Beth.
Your "Beth" doesn't have to understand how a document on her computer gets successfully sent to a printer. She just knows that if she hits the Print button then her document will print. That is fine. But if it is your job to make sure the users can print and access important files on the network then you had damn well better understand what's going on under the hood.
Remember, Google is your friend. It is inevitable that many Sys Admins have encountered your same computer problems and very likely that some of them have documented what went wrong, why, and what can be done to fix it.
Also, whatever you do, don't say you won't "fix" something just because you don't understand it. I think we often fall into this trap. About 8 years ago I remember data calls would come down from my corporate office. These calls were usually accompanied by some SQL code that we (the local administrators) would run against our respective databases. I started realizing that I was simply memorizing the queries but that I truly didn't have an understanding how they worked. I didn't know a Union from Join. Normalization? I could spell it but that was as deep as my understanding went.
Figure it out.
It's OK to not know the answer. Just figure it out. So, I began studying and using SQL. I frequently picked Adam's brain. I googled and bought some books. Once I jumped on it I realized how easy it actually was to understand. I was hooked.
The best way to understand how something works is simply to USE it.
Posted by Shane Corellian on Fri, Jun 11, 2010
In the course of our duties as Sys Admins I'm sure we've all "pushed the wrong button" a few times. If you haven't yet, you probably haven't been a Sys Admin very long.
About 10 years ago while working on a project which was wrought with political and technical paranoia I decided to prepare my own "Lysine Contingency" in the event that the political hailstorm became too much for me to handle. I first heard the term Lysine Contingency (LC) from Jurassic Park. I was always intrigued by the idea that Samuel L.'s character introduced. Certain destruction to the park's dinosaurs if a problem occurred that was simply unmanageable.
A Lysine Contingency is not your normal Change Management Rollback plan. It is basically you screaming "MISSION COMPROMISED! ABORT ABORT".
So, I want to offer some things I have learned about Lysine Contingency plans.
Points to consider:
- How was the change implemented?
- Were there multiple execution points such as Group Policy, Software Deployment, Login Scripts, Manual Installations, cron jobs or other local schedulers?
- Which execution point can be used to produce the greatest yield on your rollback? If you can hit 80% of your computers in 20 minutes with Software Deployment vs. 1 day with Group Policies then it's obvious where you should place your focus.
- Can the change be rolled back or otherwise mitigated without affecting other production systems, users or applications?
- Does your change require a reboot of the target system or application?
- What, if any, are the dependent services or applications?
- Will any data be lost? Are there other methods of removing the change that may take longer but save the data?
- Do you need the cooperation of any external entity?
- Are you allowed to deploy software but you're not allowed to modify GPOs? Are the GPOs part of your LC?
- The best Lysine solutions are the solutions that don't require anyone else but you. In a serious do-or-die situation you don't want to rely on the guy who's always on a smoking break or the lady who has more drama in her life than the evening lineup on Lifetime.
I want to stress: I'm not talking about proper Change Management procedures here. You should always have a rollback plan for major changes in your computing environment. I have had to rollback many changes to my various environments over the years, however, I have only had to enact a Lysine Contingency once and this came over two years AFTER the changes were introduced! Remember, the issues that often require your Lysine Contingency are just as easily going to be political as they are technical.
In the case I mentioned earlier I had my entire LC whittled down to one script. When the alarm sounded I knew that I had prepared for this scenario and there was no need to script some uninstallation routine or quickly write a new GPO or even to hold a meeting. This script set one flag which my verification utility read as "get the hell out of here!". With this flag, no more new verifications or installations would take place via GPO. The next section of my script deployed a Tivoli Software Package which immediately uninstalled the utility on all Windows computers. The next section added some log files to a central server. These files detailed the actions taken and would, ultimately, record the success rate of the Lysine Contingency.
Within 30 minutes the only traces of the original changes were contained in logs which I promptly gave to my manager. The Lysine Contingency went off without a hitch so that when the musical chairs music stopped I wasn't the one left standing.
Follow me on Twitter @ShaneCorellian