Systems Administration

Ubuntu Landscape and MotD integration kills Gitlab SSH performance

Posted in Computing, Infrastructure Management, Systems Administration on June 15th, 2016 by Jeff – Be the first to comment

I had nothing to do with this discovery but my colleague Lance Johnston, who did, felt that we should share it because of a lack of information about the issue on the Internet.

I lead a team at work that, among other things, manages our version control system. We started to have some performance issues with our Gitlab instance as usage increased and it started to impacted users so we decided to restart the Gitlab services first.

We observed a pretty high load when we checked before stopping the services, around 10-12, which we expected to go down when we shutdown the services. However when the services were off, the load did not go down, which was very curious.

Lance investigates and as he watched ‘top’ he observed batches of inbound ssh connections, as one would expect. But when the connections happened, he immediately saw another batch of processes named ‘landscape-sysinfo’.

A little digging turned up some information indicating that whenever a shell is spawned, such as when there’s an ssh connection, the Message of the Day is presented. The MotD runs the ‘landscape-sysinfo’ program in order to collect metrics that are presented to users when they login. So we have literally hundreds of ssh connections at any given time as Jenkins and developers do their jobs so this program was producing a consistently high load average.

Since the vast majority of ssh connections are not interactive, we disabled the Message of the Day and the load dropped immediately to .01, with the Gitlab services off. When they were turned back on we stabilized around .7 and during the work day it doesn’t go over 5 during usage spikes.

Store Time Machine Backups on an Ubuntu Server

Posted in Infrastructure Management, Personal Computing, Systems Administration, Uncategorized on April 19th, 2014 by Jeff – Be the first to comment

I found this concise article (author’s claim verified) on setting up Mac OS X Time Machine backups on a network drive. I tried using SMB/CIFS to no avail but setting up a Netatalk share did the trick!

Note that I did not modify the Avahi configuration since it wasn’t necessary to make the share usable for backups.

Intercept HTTP requests with Squid

Posted in Systems Administration on April 20th, 2011 by Jeff – Be the first to comment

On one of my projects we had some questions about how much bandwidth was being used by requests to a third party service but we didn’t have any a view beyond general traffic on the network interface. I hit upon the idea of using a transparent proxy to log requests then use log analysis to break out data transfer amounts per third party service. And since we already had squid as part of our infrastructure applications it seemed like a good choice.

The tricky part of this setup is that everything is hosted on the same hardware node and we also have some web services that needed to be left untouched. These requirements implied some network configuration using iptables to force outbound web requests through the proxy.
read more »

Unix CLI Command Repository

Posted in Computing, Systems Administration on February 5th, 2009 by Jeff – Be the first to comment

I just found the Command Line Fu command repository via reddit.com. Tons of very useful commands for a variety of tasks.

For those that might not be a *nix sysadmin, you’re missing the joy of firing off scripts from the command line that can do an extraordinary amount of work in a short amount of time. I was a veritable hero early in my career when I was able to help a client replace a copyright string in some 5000 files and create backups with this one:

find . -type f -name '*.html' | xargs grep -l '©' | xargs perl -pi.bak -e 's/©1997/©1999/g'

Or something like that.

Regardless, the client was amazed and happy that one could work such magic in just a few minutes and the content in the Command Line Fu repository looks chock full of opportunities to amaze folks with your wizardry.

New Web Site, Again

Posted in News, Systems Administration on January 31st, 2009 by Jeff – 2 Comments

I’ve setup a new blog on a new server for a variety of reasons, mostly having to do with it being a terrible idea to host on my home network. I’ve chosen a hosting service provider rather than hosting it on my own hardware. I’m using Slicehost for now. It’s fairly inexpensive (not the cheapest by far), has a good amount of positive buzz around it and they use operating system virtualization. This is cool since I get full control of my “slice” (Ubuntu) just like a regular server but upgrades are done through a web-based management console so I don’t have to drive to a Denver hosting facility to do maintenance. If I need to upgrade memory or storage I can do so with a click or two. Plus I can take snapshots of the entire slice which can be used to rebuild the whole VPS through a nifty administrative console. They don’t offer much storage but I plan to use Amazon S3 for storage on the cheap.

Anyway, so far so good.

Next up is moving my 25 web sites to the new server! (Who knew I was so rich in Internet property!)

Anyway, I’ll be posting up my old articles and files over the next couple of weeks but in the meantime here’s a nice picture:

The Flatirons