Config management at MadIRC

A week ago I've talked about config management of an InspIRCd instance in the #inspircd support channel. So today I'll provide you a little insight into the config management I build for MadIRC.

But first of all:

What is InspIRCd and IRC in general?

InspIRCd is an irc deamon written in C++ and one of the most popular irc deamons in the world. IRC allows users to communicate in Chatroom all over the world in multiple IRC networks. MadIRC is one of those networks, but a young one. MadIRC exists since 2 years but IRC itself started in 1988. 1993 the first RFC of the protocol was published.

The population of IRC networks was growing fast and until 2004 IRC was the most popular chat protocol ever seen. Even today it's a big player in chatting but not in its classical use. Twitch's whole chat system is based on IRC and also internal chatting of Minecraft and Counter Strike.

Why is IRC interesting? It's a really simple, text-based protocol. You can write a bot in less than 10 Minutes and there is no need to register an account or similar. You can use a plain TCP socket and send you commands.

But back to topic:

The idea behind the config structure

MadIRC as an IRC network needs to have some parts of its config shared. Means there are common parts inside our config which need to be identical on all servers but also some individual settings which can be different on each server. It's also important to have a history of our configs. And at least we want to be able to refresh the configs on all servers without the need to access the servers by using SSH or similar.

What we did:
At first we created a private git repository for each server on a central gogs instance. Those repositories are filled with the individual settings of the servers.

Every git repository has the same git submodule. This submodule is another repository named 'common' and stores all network wide settings.

What we covered now is:

  • history of configs
  • individual and common parts of our config

Updating the config using a script

To allow updating our config we added SSH keys without a password on each host and added the public key fingerprint as deployment key to the common git repository and to their individual one.

So we are ready to use git commands to update our local config repositories.

InspIRCd allows to run a script on rehash and startup if you place an <include executable>-tag inside the config.

<include executable="conf/common/update-config.sh &servername;">  

This line calls a shell script inside our common directory which updates our configs. And as long as we don't update the file which includes the line we use right now it's working on every reloading process.

Let's see the content of conf/common/update-config.sh:

#!/bin/sh
cd conf/  
git pull >> update.log 2>&1  
git submodule foreach git pull origin master >> update.log 2>&1  
rev=`git rev-parse HEAD`  
echo '<alias text="CONFREV" replace="NOTICE $nick :Configuration rev ID is '$rev' for '$1'" operonly="yes">'  
cd ..  

What we do is we enter the config directory, update the individual config and then update all submodules. At least we add an alias for our current config reference. Means the git commit hash. (Please notice this requires m_alias to be loaded)

Config handling

Handling round about 5000 lines of config isn't fun, especially if spread across 51 files. InspIRCd doesn't allow include-directories. So you need to add an <include>-tag for each file. Horror!

So we started with adding includes to the config files but we quickly ended up in a real jungle of includes where some hosts have individual files to override settings while others don't have and so on.

So we added a script to generate <include>-tags automatically to include files from the individual and common settings if they exist.

After a while we discovered that having config directories - like apache2 has - is much more useful, so we added this ability to our script, too. So today we have the following script running:

#!/bin/bash


function includeCommonFile {  
    if [ -z "$1" ]
        then 
            return
    fi
    echo "<include file=\"conf/common/$1\">"
}

function includeFile {  
    if [ -z "$1" ]
        then 
            return
    fi
    if [ -f "conf/$1" ]
    then
        echo "<include file=\"conf/$1\">"
    fi
}

function includeCommonFirst {  
    if [ -z "$1" ]
        then 
            return
    fi
    includeCommonFile "$1"
    includeFile "$1"
}
function includeCommonLast {  
    if [ -z "$1" ]
        then 
            return
    fi
    includeFile "$1"
    includeCommonFile "$1"
}

function includeConfFolder {  
    for moduleConfFile in $1
    do
      echo "<include file=\"$moduleConfFile\">"
    done;
}

includeCommonFile "general.conf"  
includeCommonFirst "binds.conf"  
#-#-#-#-#-#-#-#-#-#-  CONNECTIONS CONFIGURATION  -#-#-#-#-#-#-#-#-#-#-#
#Servers
includeCommonLast "connectblockserver.conf"  
includeCommonLast "connectblockexception.conf"  
includeCommonFile "connectblocksasl.conf"

#Users
includeFile "connectblock.conf"  
includeCommonFile "connectblock.conf"


#Opers.conf
includeCommonFirst "opers.conf"

#-#-#-#-#-#-#-#-#-#-#-#-#-#-  BAN OPTIONS  -#-#-#-#-#-#-#-#-#-#-#-#-#-#
includeCommonLast "badnicks.conf"

# exception: Hosts that are exempt from [kgz]lines.  
includeCommonLast "eline.conf"  
#other badhosts
includeCommonLast "badhosts.conf"

#-#-#-#-#-#-#-#-#-#-#-    Server Links     -#-#-#-#-#-#-#-#-#-#-#-#-#-#    
includeCommonLast "links.conf"

#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-# MODULES #-#-#-#-#-#-#-#-#-#-#-#-#-#-#
includeCommonFile "modules.conf"

includeConfFolder "conf/common/modules/*.conf"  
includeConfFolder "conf/modules/*.conf"  

This handles the whole config file jungle and is really really useful. The reason why we have the normal modules config and the modules directory includes: Well, there are a lot of modules which only need to be loaded. So having each of them in an own file would be a bit too much. In other words we were lazy ;)

Just a side-note: The reason for having includeCommonFirst and includeCommonLast is that sometimes the order is important. Especially for connect block config.

Message of the day

The next problem we were running into was synchronizing the MOTD of all servers. Because MOTD doesn't have includes, we had to catch this also using a script.

We just used cat and echo to generate individual MOTD files on each server using the common and the individual config directories.

#!/bin/bash
###############
## $1 = Servername
###############

rm -r conf/cache/ > /dev/null  
mkdir conf/cache/ > /dev/null  
for motdFile in conf/common/motd/*.motd  
do  
    for insertFile in conf/common/motd/*.txt
    do
        cat $insertFile >> conf/cache/`basename $motdFile`
        echo "" >> conf/cache/`basename $motdFile`
        echo "" >> conf/cache/`basename $motdFile`
    done;
    cat $motdFile >> conf/cache/`basename $motdFile`
    echo "" >> conf/cache/`basename $motdFile`
    echo "Generated at: "`date` >> conf/cache/`basename $motdFile`
done;  
if [ `ls -1 conf/motd | wc -l` -gt 0 ]  
    then 
        for motdFile in conf/motd/*.motd
        do
            for insertFile in conf/common/motd/*.txt
            do
                cat $insertFile >> conf/cache/`basename $motdFile`
                echo "" >> conf/cache/`basename $motdFile`
                echo "" >> conf/cache/`basename $motdFile`
            done;
            cat $motdFile >> conf/cache/`basename $motdFile`
            echo "" >> conf/cache/`basename $motdFile`
            echo "Generated at: "`date` >> conf/cache/`basename $motdFile`
        done;
fi  
sed -i 's/&servername;/'$1'/g' conf/cache/*.motd  
sed -i 's/&fingerprint;/'`openssl x509 -in conf/cert.pem -noout -fingerprint | cut -d'=' -f 2`'/g' conf/cache/*.motd  
sed -i 's/&fingerprint256;/'`openssl x509 -in conf/cert.pem -noout -sha256 -fingerprint | cut -d'=' -f 2`'/g' conf/cache/*.motd  
sed -i 's/&fingerprintmd5;/'`openssl x509 -in conf/cert.pem -noout -md5 -fingerprint | cut -d'=' -f 2`'/g' conf/cache/*.motd

echo "<files motd=\"conf/cache/common.motd\" rules=\"conf/common/rules.txt\""  
for finishedMotdFile in conf/cache/*.motd  
do  
    echo `basename $finishedMotdFile .motd`"=\""$finishedMotdFile"\""
done;  
echo ">"  

This is a lot of black magic for a few lines of MOTD but we have to say that we have multiple MOTD for different connect classes and we include our SSL Fingerprints and server name into the MOTD. So maybe you could cut it a bit to fit your needs.

For all of you interested in calling this file:

<include executable="conf/common/dynamic-motd.sh &servernetworkname;">  

So what we got now is nearly our whole config structure.

The basic configuration

So as we got the whole config thing now the last question is: How to use it?

Well, the answer is really simple. I'll show you our inspircd.conf:

<config format="xml">  
<include file="conf/common/inspircd.conf">  

Yeah, that's it. It's nearly empty. And now let's see conf/common/inspircd.conf:

<include file="conf/define.conf">  
<include executable="conf/common/update-config.sh &servername;">  
<include executable="conf/common/dynamic-motd.sh &servernetworkname;">  
<include executable="conf/common/dynamic-include.sh">  

So we add the local define config to get the servername, updating our configs, generate and include the MOTD files and load configs by using the dynamic include script.

Conclusion

That's it! Managing IRCd configs is so easy if you only know how to!

You may insist and say "Why aren't you using ansible or puppet to mange the config?" Yeah... To be honest: When we were writing those scripts I didn't know about ansible :D

On the other hand I now know ansible and really don't want to create something similar using templates. It's crazy.

And we use this setup for over 1½ year and it's so lovely because it just works. Since the last few month our network structure has changed and I have no access to some IRCd hostsystems by SSH but I can maintain their config without problems.

All I need to do is /rehash * and all servers in our network are updating their config and reloading it.

So I hope you enjoyed the read and learned something useful. If you have suggestions or questions send me an email, message me on twitter or write a comment :) Don't forget to share this post if you like it and tell your friends!


Further information:

MadIRC:

×

Stay in touch

By follow me on Twitter, follow RSS or sign up for my newsletter.