Private Gentoo Mirror For A Large LAN

I run Gentoo on some 30 or so devices including PCs, Raspberry Pis, Virtual Machines, rackmounted servers, and so forth. These devices are mostly housed in my home office with a few random ones scattered throughout the house. To me it seems like a waste of bandwidth to have each of them download packages from the Internet directly. This is especially apparent when doing updates and watching the same exact things get downloaded to multiple devices. There is also an issue where most mirrors, and this applies to all mirrors not just Gentoo, that they have limits on how many connections per day from the same IP they allowed. My solution to the problem is to run my own Gentoo mirror on one of the machines with my own local (LAN) copy of the portage tree and also the distfiles.

Originally I ran my Gentoo mirror on one of my servers, but recently I moved it to its own dedicated VM to make management a little easier. That allows me to move it between machines if needed as well as take advantage of the ZFS RAID array on one of my servers. The disk space required is currently 367GB for all of the files. I allocated 500GB for my setup to allow room for growth. Anyway, I’ll assume you have a base Gentoo system up and running ready to be turned into a mirror.

First step is to install ‘gentoo-rsync-mirror’ package. This will install a script to /opt/gentoo-rsync/rsync-gentoo-portage.sh which we will copy to /usr/local/bin/ and modify it to look like this:

#!/bin/bash

LOG=/var/log/rsync-gentoo.log
LOCKFILE=/tmp/gentoo-mirror-sync

source /etc/rsync/gentoo-mirror.conf

if [ -e $LOCKFILE ]; then
echo “sync still running, or stale lock file!”
logger -t rsync “sync still running, or stale lock file!”
else
touch $LOCKFILE
fi

echo “Started Gentoo Portage sync at” `date` >> $LOG 2>&1
logger -t rsync “re-rsyncing the gentoo-portage tree”
${RSYNC} ${OPTS} ${PORT_SRC} ${PORT_DST} >> $LOG 2>&1
logger -t rsync “deleting spurious Changelog files”
find ${PORT_DST} -iname “.ChangeLog*” | xargs rm -rf
echo “End of Gentoo Portage sync: “`date` >> $LOG 2>&1
#
echo “Started Gentoo main sync at” `date` >> $LOG 2>&1
logger -t rsync “re-rsyncing the gentoo main tree”
${RSYNC} ${OPTS} ${GEN_SRC} ${GEN_DST} >> $LOG 2>&1
logger -t rsync “deleting spurious Changelog files”
find ${GEN_DST} -iname “.ChangeLog*” | xargs rm -rf
echo “End of Gentoo main sync: “`date` >> $LOG 2>&1

rm -f $LOCKFILE

Now edit /etc/rsync/rsyncd.conf to look like:

# Copyright 1999-2004 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Id$

uid = nobody
gid = nobody
use chroot = yes
max connections = 20
pid file = /var/run/rsyncd.pid
log file = /var/log/rsync.log
motd file = /etc/rsync/rsyncd.motd
transfer logging = yes
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300

[gentoo-portage]
#modern versions of portage use this entry
path = /mirror/gentoo-portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles

[gentoo]
path = /mirror/gentoo
comment = Gentoo Linux mirror

You can change the path as needed. In my setup I have a mount point /mirror that is used to house the files. You can also edit /etc/rsync/rsyncd.motd if you want to display a custom message when a system syncs.

Now edit /etc/rsync/gentoo-mirror.conf to look like:

# Gentoo rsync mirror config

RSYNC=”/usr/bin/rsync”
OPTS=”–quiet –recursive –links –perms –times –devices –delete –timeout=300″
#Uncomment the following line only if you have been granted access to rsync1.us.gentoo.org
#SRC=”rsync://rsync1.us.gentoo.org/gentoo-portage”
#If you are waiting for access to our master mirror, select one of our mirrors to mirror from:
#SRC=”rsync://rsync.de.gentoo.org/gentoo-portage”
PORT_SRC=”rsync://mirrors.kernel.org/gentoo-portage”
GEN_SRC=”rsync://mirrors.kernel.org/gentoo”
PORT_DST=”/mirror/gentoo-portage/”
GEN_DST=”/mirror/gentoo/”

Again, change the path if needed and you can also change the mirror to a closer one if you wish.

Now we need to make a cron job to do the work:

crontab -e

0 */12 * * * /usr/local/bin/rsync-gentoo.sh

Here I am syncing every 12 hours, which technically is more than the once-per-day limit, but I figure I’m saving the mirrors a bunch of traffic/work as a trade off.

Now we need to set rsyncd to autostart and start it:

rc-update add rsyncd default

service rsyncd start

Now we should have rsync working. Next we need to provide either FTP or HTTP retrieval of distfiles. I prefer HTTP, so emerge apache and set it to autostart just like we did for rsyncd. The last step is to edit /etc/apache2/vhosts.d/default_vhost.include to point the document root to the mirror location.

The initial sync takes a while, but once it is completed keeping things up-to-date does not involve much bandwidth.  Once the first sync is finished, the last step is to configure each individual machine to use the local mirror.

Edit /etc/portage/repos.conf/gentoo.conf to use this line:

sync-uri = rsync://<YOUR MIRRORS IP>/gentoo-portage

Then edit /etc/portage/make.conf to include:

GENTOO_MIRRORS=”http://<YOUR MIRRORS IP>/gentoo”

Now you should be able to ’emerge –sync’ using your local mirror and the distfiles should be pulled from your mirror as well.

Advertisements