I wanted to show off another python script today.  I think it’s pretty cool.  It’s kind of like a very rudimentary version of something you might see in Iron Man.  And, of course, anything in Iron Man is cool.

I dub it Log Reader 3000.  It’s purpose?  It helps me monitor logs.  How?  Well, sometimes I need to follow a log in real time as it is written, but I can quickly get bored.  The log scrolls by endlessly while, very often, little new information spits itself out.  I can quickly lose focus, or at the very least, damage my vision after staring at a screen intently for extended periods.

Ideally, I want the log to simply flow through me, and if my subconscious notices something odd, then I can act on it.  If the log is read aloud to me, then I can work on other tasks and let my auditory memory and auditory processing take note of oddities on which I need to act.

So, I made a python script to read the log out to me as it is written. It is my first Python 2.6 script.  I take advantage of the new multiprocessing module built into the standard library.  I also use the open-source festival text-to-speech tool.

First, install festival.  sudo apt-get install festival in Ubuntu.  You probably want to set it up to work with ALSA or ESD sound.  By default, festival uses /dev/dsp, which means that you can’t use festival and any other program that uses audio (like Skype) at the same time.   Fortunately, and as usual, Ubuntu provides detailed, simple instructions to set up festival with ALSA: https://help.ubuntu.com/community/TextToSpeech .

Finally, just find an appropriate use case.  Note that most log monitoring applications would not be improved with Log Reader 3000.  If you just want to be notified of errors, you should have a program email you when an error appears in a log.  If you want to understand the log output of a program that has already run, understand that Log Reader 3000 is meant for live-running programs.  Yes, Log Reader 3000 can be modified to read any text file line-by-line.  But, you will find that reading ends up being much faster than listening to a slow automated voice, so I recommend that you just try to skim a completed program’s output with VIM.

So then why ever use Log Reader 3000?  It is useful for applications which fit all of the following criteria:

1. you want to monitor a live running program
2. and the debugging information is nuanced and you need a human to interpret it (i.e. it cannot be filtered programmatically) and/or you want to be able to intervene while the program is running to keep it doing what it ought to be doing in real time.

Applications:

• Say that you are spidering the web, and what the spider should and should not be spidering is not yet well-defined, but a human knows, then the Log Reader 3000 can read aloud where the spider is, and the human can correct course as he or she notices the spider going astray.
• Or, say that you are working on some kind of artificial intelligence.  Perhaps, the AI program can reason aloud and a human can correct or redirect the machine’s reasoning as it goes along.  I have no idea how or why an AI would do that.
• Maybe you want to protect against bot attacks, but your aggressor is particularly clever and seems to avoid looking like a bot in all of the obvious ways.  You can pipe the output of your log into Log Reader 3000 and notice new kinds of suspicious patterns live while reclining in your chair or surfing the web.
• You run a securities trading program.  You have numerous checks and double-checks to ensure that everything works correctly.  Nevertheless, you need to have a human monitoring the system as a whole continuously anyway, so you have Log Reader 3000 read aloud total portfolio value, or live trades, or trading efficiency, or fast-moving securities, or all of the above.
• The lobby of your startup has a TV screen with graphs of user growth and interaction on the site.  You want to increase the coolness factor by having a computer voice read aloud some of the searches or conversations happening on your site live.
• You make a living by selling cool techy art projects which blend absurdity with electronics.  You read aloud live google searches, or live wikipedia edits, or inane YouTube comments out of what looks like a spinning vinyl record.  Passersby whisper of your genius.

Once you have the application, just tail -f the log, parse out the parts you want the log reader to read (you can use awk for that, for example, or maybe a simple python script), and pipe that into the Log Reader 3000.

tail -f output.log | awk “{ print $1 }” | ./log_reader_3000.py

How does Log Reader 3000 work?  The main process reads in one line at a time.  As it reads in each line from stdin, it sends it to the processing queue.  The child process reads the last item in the queue (it discards the items at the top of the queue because those are old and we need to catch up with the latest output line) and then calls a function to say() the line.  The say() function simply uses the subprocess module to call festival in a separate process and then blocks until it is done saying it aloud.

Because having a computer voice read aloud a sentence takes a while, the log probably outputs many more lines than can be read aloud.  That is why a multiprocess queue is needed, and that is why Log Reader 3000 only reads out the most recent line which has been output, which is why it is most useful for specific applications.

Here is the script, log_reader_3000.py:

#!/usr/bin/env python2.6

import sys

import subprocess

from multiprocessing import Process, Queue

def say(line):

    say = '(SayText "%s")' % line

    echo = subprocess.Popen(['echo', say], stdout=subprocess.PIPE)

    subprocess.call(['festival'], stdin=echo.stdout)

def listen_to_lines(queue):

    line = 'I am Log Reader 3000.  The world is beautiful.'

    while True:

        while not queue.empty():

            line = queue.get()

        say(line)

queue = Queue()

p = Process(target=listen_to_lines, args=(queue,))

p.start()

while True:

    line = sys.stdin.readline()

    sys.stdout.write(line)

    queue.put(line)