Marios Zindilis

ISO Date Format in Apache Logs

Apache's mod_log_config module (installed by default on CentOS 6) allows for the CustomLog directive, which in turn takes a log format specification. The default log format string is usually:

LogFormat "%h %l %u %t \"%r\" %>s %b" common

In the above line, common is the name of the log format, which is later references in a CustomLog directive, for example:

CustomLog logs/example.com-access_log common

The meaning of the % fields is specified at the Apache documentation for mod_log_config. What is interesting is that the %t parameter can take an optional date formatting string, in the form %{date format}t. The date format part should be strftime(3)-compatible.

For example, the following date format will produce timestamps like 2014-05-16 13:45:32:

LogFormat "%h %l %u %{%Y-%m-%d %H:%M:%S}t \"%r\" %>s %b" example

You will then need to reference the example name of this log format specification as:

CustomLog logs/example.com-access_log example.

The % fields of strftime are listed in the man page for strftime

How to get Google search suggestions in Firefox on Linux Mint

The Google search feature is not offered as an option in the version of Firefox that comes installed in Linux Mint. You can add it with the Manage Search Engines feature, but when you do, you will notice that there are no search suggestions when you start typing a search term.

To add Google search suggestions, you need to edit the file google.xml, located in the searchplugins directory, inside your Firefox user profile. In my case, the full path for this file is:

/home/marios/.mozilla/firefox/mwad0hks.default/searchplugins/google.xml

Yours should be similar, and the command locate google.xml can help you find the path.

Edit that file, and add the following line:

<Url type="application/x-suggestions+json" method="GET" template="https://www.google.com/complete/search?client=firefox&amp;q={searchTerms}"/>

...somewhere before the closing </SearchPlugin> tag. Restart your Firefox, and suggestions should now work.

Is bandwidth still relevant for web hosting?

The Numix Project received some web love lately, mentioned in last Sunday's Linux Action Show web episode, and later on It's F.O.S.S and probably elsewhere, for their announcement to release their own Linux distribution. As a result, their website is now down, with the server returning 500:

mariosz@super-mario:~$ curl -I numixproject.org
HTTP/1.0 500 Internal Server Error

...and displaying the message "Bandwidth Limit Exceeded". I would have guessed that with everything-as-a-service and the whole Cloud #!, bandwidth consumtion would have already become irrelevant.

I mean, there's enough BW to feed our amplification attacks ( 400Gbps on CloudFlare, 350Gbps on OVH), but not enough to host our websites? Does that not call for a revision of the model?

Contribute to an open source project on GitHub

I don't contribute to open source projects as much as I would like to. As a result, I forget the sequence of actions required to do so, when a project is hosted on GitHub. To overcome that, I wrote down the procedure in a how-to guide. I hope it will be useful to others as well.

Read the guide: How to contribute to open source projects on Github

Google Servers Migration Paper

This has to be the coolest paper I 've read in a long time. It explains how a few engineers working at Google, migrated thousands of servers from the ancient RedHat Linux 7.1 to a modern version of Debian, with minimum downtime, by replacing small bits of the operating system at a time, in many many iterations.

The paper is from a USENIX Conference, here you go, and you're very welcome: Live upgrading thousands of servers from an ancient Red Hat distribution to 10 year newer Debian based one.

Fedora Ask in Greek

With the noble efforts of the members of the Greek Fedora community, the Ask Fedora website now has a subsection for questions in Greek.

openDCIM Installation on CentOS 6

openDCIM is a free and open source solution for DataCenter Infrastructure Management. It is already used by a few organizations, and is quickly improving due to the efforts of its developers.

I have written a complete, bottom-up installation for openDCIM on CentOS. This will get you up to the first screen in the web interface, after which you can follow Scott Milliken's video tutorial for the initial settings of the application.

The guide is written for openDCIM 2.1, but should be fairly applicable to newer versions.

Read the full installation guide here.

Bash: Check that string is IP

Let's say that we have a Bash script that accepts a parameter that is supposed to be an IP Address. How do we verify that input, before proceeding with any further actions? Here's what I do so far.

Breakdown of the conditions

First, check that exactly three dots exist in the string:

if [ `echo $IsIP | grep -o '\.' | wc -l` -ne 3 ]; then
    echo "Parameter '$1' does not look like an IP Address.";
    exit 1;
fi

Then, check that exactly four parts exist if the string is broken down by dots:

if [ `echo $IsIP | tr '.' ' ' | wc -w` -ne 4 ]; then
    echo "Parameter '$1' does not look like an IP Address.";
    exit 1;
fi

Then, check that all four parts are numeric:

for OCTET in `echo $IsIP | tr '.' ' '; do
    if ! [[ $OCTET =~ ^[0-9]+$ ]]; then
        echo "Parameter '$1' does not look like an IP Address.";
        exit 1;
    fi
done

Finally check that all four parts are in the range 0 to 255:

for OCTET in `echo $IsIP | tr '.' ' '; do
    if [[ $OCTET -lt 0 || $OCTET -gt 255 ]]; then
        echo "Parameter '$IsIP' does not look like in IP Address (octet '$OCTET' in not in range 0-255).";
    fi
done

So the parameter passed to the script must fulfill the following four criteria:

  1. Contain exactly three dots.
  2. Contain exactly four parts, if broken down by dots.
  3. All four parts are numeric.
  4. All four parts are between 0 and 255.

A Bash function

Here's the entire thing in the form of a Bash function, with an additional comment on what went wrong:

# Verify that the parameter passed is an IP Address:
function is_IP() {
if [ `echo $1 | grep -o '\.' | wc -l` -ne 3 ]; then
        echo "Parameter '$1' does not look like an IP Address (does not contain 3 dots).";
        exit 1;
elif [ `echo $1 | tr '.' ' ' | wc -w` -ne 4 ]; then
        echo "Parameter '$1' does not look like an IP Address (does not contain 4 octets).";
        exit 1;
else
        for OCTET in `echo $1 | tr '.' ' '`; do
                if ! [[ $OCTET =~ ^[0-9]+$ ]]; then
                        echo "Parameter '$1' does not look like in IP Address (octet '$OCTET' is not numeric).";
                        exit 1;
                elif [[ $OCTET -lt 0 || $OCTET -gt 255 ]]; then
                        echo "Parameter '$1' does not look like in IP Address (octet '$OCTET' in not in range 0-255).";
                        exit 1;
                fi
        done
fi

return 0;
}

Example output of the function when saved as a standalone script:

marios@yovan ~ $ ./is_IP 1234
Parameter '1234' does not look like an IP Address (does not contain 3 dots).
marios@yovan ~ $ ./is_IP 1234...
Parameter '1234...' does not look like an IP Address (does not contain 4 octets).
marios@yovan ~ $ ./is_IP 1.2.3.A
Parameter '1.2.3.A' does not look like in IP Address (octet 'A' is not numeric).
marios@yovan ~ $ ./is_IP 1.2.3.300
Parameter '1.2.3.300' does not look like in IP Address (octet '300' in not in range 0-255).
marios@yovan ~ $ ./is_IP 1.2.3.4
marios@yovan ~ $

If someone has any better ways to check this, or improvements on the above, please step forward :)

Fork these scripts on GitHub.

Using Regular Expressions

In an older blog platform that I used, Efstathios Chatzikyriakidis commented and suggested the use of regular expressions, which in retrospect made all the above pretty useless. The expression that I found working best is the one at IP Address regular expression.

Έργα στο Public Domain από 1η Ιανουαρίου 2013

Από 1η Ιανουαρίου 2013, ελευθερώνονται ως Κοινό Κτήμα (Public Domain) όλα τα έργα δημιουργών που πέθαναν το 1942.

Η σύντομη λίστα των Ελλήνων δημιουργών συμπεριλαμβάνει:

Ποιητές - Λογοτέχνες

Ζωγράφοι - Γλύπτες

Μουσικοί

How to create a large MySQL database for tests

I wanted to create a fairly big MySQL database (~30GB on disk) for tests. I created a 'test' database, and a 'table2' table, with a 'content' field with datatype TEXT. Here's what I run from Bash:

marios@mysql-server:~$ time for i in `seq 1 500000`; \
do echo $i of 500000; \
mysql -u root -D test -e \
"INSERT INTO table2 (content) VALUES ('`cat /var/log/messages | tr -d \'`')"; \
done

This took a few hours to finish, while creating data at a rate of 100MBytes per minute on my test server. The size of the database was 30GB.

This did the job but it was too slow. The better way to do it, would be to run the above command for far less repetitions, say 100000, to create a smaller table, then copy that table in the same database as many times as required for the entire database to reach the desirable size, like:

mysql> create table2 like table;
mysql> insert into table2 select * from table;