Gentoo In Production Because One-Size-Rarely-Fits-All

Who in their right mind would run Gentoo on a production server? I would!

Maybe I’m not in my right mind, but Gentoo is one of the best Linux distributions out there for several use cases from large server farms all the way down to embedded devices. One thing that is an issue with almost every Linux distribution – scratch that – one thing that is an issue with nearly every binary system is that someone else decides what is best for everyone. In a Linux distribution that means someone else, a package maintainer, decides what software and features are included in a particular package. Sometimes that just fine, other times it can be an issue. For example with openoffice and libreoffice there is very little that can be customized about the install and as such a binary install package is acceptable. However, with other software take Apache for example, I might not want every other feature enabled – in fact I rarely do! This is where Gentoo really shines brightest: the ability to choose what features are enabled system-wide and on a per-package basis.

Another area Gentoo excels is with the security tool glsa-check It is better than anything else I’ve ever seen or used on any distro. I can check for vulnerabilities, read about the vulnerability, test what the recommended fix is, and even have it apply the fix all within one utility.

I need enterprise support from the vendor! No, you don’t! If you think you do step aside and let someone else run things. If your IT team says they need enterprise support go ahead and fire them right now as they don’t know how to do their job. Microsoft, Oracle, etc. provide paid support services for their products. The thing is they don’t provide access to anything that couldn’t have been found with a Google search or reading related discussion forums. I don’t have time for that, well then hire an IT person/team that does because that’s part of their job!

Gentoo supports multilib and slots which makes it very flexible. Most modern distributions support mulilib so I won’t go into that, but you may be wonder what slots are. Slots allow many packages, particularly libraries, to have more than one version installed at the same time. On my dev box this is a life safer as I can have multiple versions of a library installed without having to resort to any trickery. For example, on my laptop I have webkit-gtk 2.4.11 and 2.14.5 installed at the same time.

In Gentoo software is installed, configured, and uninstalled with the tool named ’emerge’ which is part of the portage (software package) system. Emerge can handle dependencies, complication, binary packages, and tons of other abilities and features. You can use wildcards with emerge, in fact some use flags even support regular expression, which is very handy when you need to manipulate multiple packages with the same or similar names. For example, say I had gtk+ installed to use two different x11 applications that each required a different version of gtk+ but I no longer need them. With emerge, just like most package manager, I can name each package I wish to uninstall by listing them out – or – with emerge I can use a wildcard and have it remove all of them: ’emerge -Ca x11-libs/gtk+*’. This command will remove, ask before doing so, and search/remove any package that matches that pattern.

Another area where Gentoo is a level above the rest is with freedom of choice. My boss, whom I generally think is quite intelligent, thinks systemd is the cats meow and anyone who doesn’t get on board is an idiot. I, on the other hand, think it is a pile of s**t that is turning Linux into a binary blob operating system. If I wanted to run a binary blob operating system, I’d run the original aka Windows. Well with Gentoo we get the freedom to select what we want. You want to put all your eggs in one basket? Install systemd! You want a system that doesn’t try to assimilate everything like the Borg on Star Trek or destroys what it cannot? Install OpenRC. The choice is yours and all of the documentation is written to handle either path you take.

While on the topic of documentation, Gentoo has some of the best out there. I even refer to it often when working on other distributions because of the quality and detail often found in the guides, wiki, and forums. Regardless of the operating system used, if you are not willing to learn some basics then you have no business being an admin. The same argument could be made for using automated tools that hide what is going on and make assumptions on the best approach.

In Gentoo we use what are called USE flags to select the features to be used in the system or for a particular package. On any Gentoo system these can be found in /usr/portage/profile/use.desc for global to all packages and /usr/portage/profile/use.local.desc for USE flags that are specific/specialized to only one package. System-wide USE flags are set in /etc/portage/make.conf and per-package USE flags can be set in /etc/portage/package.use/foo (where foo is any filename you wish). For example, I may want to have postgresql set on a system-wide level but a specific package, say zabbix, I do not want to have postgresql support. To do this I would add ‘postgresql’ to my USE flag in /etc/portage/make.conf and then in /etc/portage/package.use/foo I would add a line that reads ‘net-analyzer/zabbix -postgresql’. Here you can see the minus sign in front ‘-postgresql’ which tells emerge that we do not want that feature enabled which overrides the system-wide setting that is in /etc/portage/make.conf.

Compiling everything from source takes too long! Well sometimes, but not always. We can also mitigate this to some degree with a combination of distcc and cross-compile if we have other machines at our disposal. Most source packages are quite small and do not take long to compile and install. There are a few exceptions such as glibc and gcc which can takes more than an hour to compile and install. If you have more than one machine available the compiles can be distributed to them with distcc an with cross-dev even different architectures can be used. I do this to speed things up on my Raspberry Pi systems by making use of my faster multi-CPU/multi-core machines. I mentioned previously that some packages, such as libreoffice, offer very little customization and can be time consuming to compile. For these infrequent cases there are binary packages available in the portage tree. I use a libreoffice binary package on my laptop because my laptop is slow and there is not much I wish to change about libreoffice to begin with. On my desktop I use the standard source package because that machines has 8 cores and compiles pretty quick.

Gentoo is a rolling release which means the software is always being updated. Many distributions have started to move to this model and I personally prefer it over the huge steps that seem to happen in the other major distros. There have been so many upgrades in Debian GNU/Linux that broke things due to the massive time between releases that it has become standard practice to use a staging server to test on before attempting the real upgrade. I’m all for testing things, but the fact that you are almost forced to does not sit well with me. In recent years Ubuntu has gotten nearly as bad not to mention they seem to willy-nilly decide what packages to include and drop between releases *cough*mediawiki*cough*. With a rolling release we avoid these big steps and have less time spent testing and fixing things in staging. One could argue the disadvantage is that you have to upgrade more often, but you don’t have to. You are in control here, not someone else. Given that there are almost weekly updates purely for security purposes in the major distributions, I feel like this is a moot issue anyway.

We also find that there is superior configuration management in Gentoo. There are two different tools that can be used depending on what you wish to do. If you want a quickly see if there is a difference between the installed configuration and a new one you can use ‘etc-update’ which will list the files that are different and will allow you to chose your existing one, the new one, attempt a simple merge, or exit without doing anything. For bigger changes or finer control there is a tool called ‘dispatch-conf’ that allows line-by-line comparison between configuration files. In all cases configuration files are NEVER overwritten automatically during a package upgrade if you have made changes to the original!

So when it comes down to is the initial install is time consuming and a bit labor intensive as you have to make a lot of decisions about your system and then implement them. Suck it up and get it done. Once the installation is finished and the system is up and running there is never a software reason to reinstall. I have one system that has been happily updated since 2008 with only kernel upgrades causing reboots.

Gentoo isn’t for everyone, but the older I get the more I know what I want and expect out of things. With Gentoo the system is setup how I want it, things are configured how I like it, and only those features I wish to use are present. The one-size-fits-most attitude of the other major distributions does not work for me. I want my cake and I’m gonna eat it too!