Infrastructure Management

Ubuntu Landscape and MotD integration kills Gitlab SSH performance

Posted in Computing, Infrastructure Management, Systems Administration on June 15th, 2016 by Jeff – Be the first to comment

I had nothing to do with this discovery but my colleague Lance Johnston, who did, felt that we should share it because of a lack of information about the issue on the Internet.

I lead a team at work that, among other things, manages our version control system. We started to have some performance issues with our Gitlab instance as usage increased and it started to impacted users so we decided to restart the Gitlab services first.

We observed a pretty high load when we checked before stopping the services, around 10-12, which we expected to go down when we shutdown the services. However when the services were off, the load did not go down, which was very curious.

Lance investigates and as he watched ‘top’ he observed batches of inbound ssh connections, as one would expect. But when the connections happened, he immediately saw another batch of processes named ‘landscape-sysinfo’.

A little digging turned up some information indicating that whenever a shell is spawned, such as when there’s an ssh connection, the Message of the Day is presented. The MotD runs the ‘landscape-sysinfo’ program in order to collect metrics that are presented to users when they login. So we have literally hundreds of ssh connections at any given time as Jenkins and developers do their jobs so this program was producing a consistently high load average.

Since the vast majority of ssh connections are not interactive, we disabled the Message of the Day and the load dropped immediately to .01, with the Gitlab services off. When they were turned back on we stabilized around .7 and during the work day it doesn’t go over 5 during usage spikes.

Store Time Machine Backups on an Ubuntu Server

Posted in Infrastructure Management, Personal Computing, Systems Administration, Uncategorized on April 19th, 2014 by Jeff – Be the first to comment

I found this concise article (author’s claim verified) on setting up Mac OS X Time Machine backups on a network drive. I tried using SMB/CIFS to no avail but setting up a Netatalk share did the trick!

Note that I did not modify the Avahi configuration since it wasn’t necessary to make the share usable for backups.

Netalyzr – Network debugging tool

Posted in Infrastructure Management on January 1st, 2012 by Jeff – Be the first to comment

I’ve had a transient issue with my Internet access randomly “going away”. It’s annoying but generally clears up within a minute or two. I came across a tool called Netalyzr by a group within UC Berkeley. Netalyzr is a Java application available as either an in-browser Applet or a command line utility. It runs a number of network connectivity tests and provides a detailed report hosted on their web site that uses a simple red/yellow/green motif to show problems and their relative importance.

While Netalyzr didn’t clearly show what was going on with my Internet connection it did raise a red flag about network buffers that might be the issue. Unfortunately, that’s a router configuration issue on the part of my ISP so I’m not hopeful for a resolution. But I can always gather data then open a trouble ticket with the vendor.

Regardless, Netalyzr looks like a great tool for troubleshooting connectivity issues.

Lighting a fire under WordPress

Posted in Infrastructure Management on September 5th, 2011 by Jeff – Be the first to comment

Since I moved my personal web site from Roller to WordPress a couple of years ago, my web site had been a dog. After reading an article about a PHP-based web site configured to support 9 millions hits per day, and knowing through experience that my site should be significantly faster, I decided it was time to light a fire under WordPress.

(Note that I’ve included gists at the bottom of the article with the important configuration files.)
read more »

Ubuntu Making Cloud Computing Accessible

Posted in Computing, Infrastructure Management on February 23rd, 2009 by Jeff – Be the first to comment

It seems that Ubuntu, one of my favorite Linux distributions, is embracing Amazon EC2 styled cloud computing in an upcoming release.

Ubuntu has included the Eucapyptus project in the aptly named Karmic Koala release 9.10 project in order for mere mortals to be able to install and manage their own cloud. Considering that some important people recommend keeping your data on your own hardware to avoid vendor lock-in and privacy issues, this allows companies to avoid the issues while getting some of the advantages of the cloud. Of course, companies will still need a team, albeit much smaller, of crack infrastructure managers but they will have the advantages provided by the abstraction of hardware and it’s associated management tools.

Besides, what could possibly be cooler than having a compute cloud in your basement data center running on a bunch of cheap PC servers? I’m sure nothing can quite compare.