Archive for June, 2016

Ubuntu Landscape and MotD integration kills Gitlab SSH performance

Posted in Computing, Infrastructure Management, Systems Administration on June 15th, 2016 by Jeff – Be the first to comment

I had nothing to do with this discovery but my colleague Lance Johnston, who did, felt that we should share it because of a lack of information about the issue on the Internet.

I lead a team at work that, among other things, manages our version control system. We started to have some performance issues with our Gitlab instance as usage increased and it started to impacted users so we decided to restart the Gitlab services first.

We observed a pretty high load when we checked before stopping the services, around 10-12, which we expected to go down when we shutdown the services. However when the services were off, the load did not go down, which was very curious.

Lance investigates and as he watched ‘top’ he observed batches of inbound ssh connections, as one would expect. But when the connections happened, he immediately saw another batch of processes named ‘landscape-sysinfo’.

A little digging turned up some information indicating that whenever a shell is spawned, such as when there’s an ssh connection, the Message of the Day is presented. The MotD runs the ‘landscape-sysinfo’ program in order to collect metrics that are presented to users when they login. So we have literally hundreds of ssh connections at any given time as Jenkins and developers do their jobs so this program was producing a consistently high load average.

Since the vast majority of ssh connections are not interactive, we disabled the Message of the Day and the load dropped immediately to .01, with the Gitlab services off. When they were turned back on we stabilized around .7 and during the work day it doesn’t go over 5 during usage spikes.