Posts Tagged ‘ubuntu’

Ubuntu Landscape and MotD integration kills Gitlab SSH performance

Posted in Computing, Infrastructure Management, Systems Administration on June 15th, 2016 by Jeff – Be the first to comment

I had nothing to do with this discovery but my colleague Lance Johnston, who did, felt that we should share it because of a lack of information about the issue on the Internet.

I lead a team at work that, among other things, manages our version control system. We started to have some performance issues with our Gitlab instance as usage increased and it started to impacted users so we decided to restart the Gitlab services first.

We observed a pretty high load when we checked before stopping the services, around 10-12, which we expected to go down when we shutdown the services. However when the services were off, the load did not go down, which was very curious.

Lance investigates and as he watched ‘top’ he observed batches of inbound ssh connections, as one would expect. But when the connections happened, he immediately saw another batch of processes named ‘landscape-sysinfo’.

A little digging turned up some information indicating that whenever a shell is spawned, such as when there’s an ssh connection, the Message of the Day is presented. The MotD runs the ‘landscape-sysinfo’ program in order to collect metrics that are presented to users when they login. So we have literally hundreds of ssh connections at any given time as Jenkins and developers do their jobs so this program was producing a consistently high load average.

Since the vast majority of ssh connections are not interactive, we disabled the Message of the Day and the load dropped immediately to .01, with the Gitlab services off. When they were turned back on we stabilized around .7 and during the work day it doesn’t go over 5 during usage spikes.

New Web Site, Again

Posted in News, Systems Administration on January 31st, 2009 by Jeff – 2 Comments

I’ve setup a new blog on a new server for a variety of reasons, mostly having to do with it being a terrible idea to host on my home network. I’ve chosen a hosting service provider rather than hosting it on my own hardware. I’m using Slicehost for now. It’s fairly inexpensive (not the cheapest by far), has a good amount of positive buzz around it and they use operating system virtualization. This is cool since I get full control of my “slice” (Ubuntu) just like a regular server but upgrades are done through a web-based management console so I don’t have to drive to a Denver hosting facility to do maintenance. If I need to upgrade memory or storage I can do so with a click or two. Plus I can take snapshots of the entire slice which can be used to rebuild the whole VPS through a nifty administrative console. They don’t offer much storage but I plan to use Amazon S3 for storage on the cheap.

Anyway, so far so good.

Next up is moving my 25 web sites to the new server! (Who knew I was so rich in Internet property!)

Anyway, I’ll be posting up my old articles and files over the next couple of weeks but in the meantime here’s a nice picture:

The Flatirons