Web Scanning for Devices on your network using Nmap and Perl

Introduction

I wanted a way of keeping tabs on how much the kid's computer was being used during the day while we were out. I can look on the router to see what devices are connected to the network, but this only works while I'm at home.

As I run a web server at my home, I decided to put together a quick webpage that scans for devices and shows the result on a smartphone.

Scanning the Network

After a few web searches I found the best tool to use on my Linux server was nmap, so I installed it and started playing around scanning my network. Results were initially pretty patchy, either the scan took too long (in excess of a minute) or finished quickly but didn't pick everything up.

The results depend on what sort of scan you want to do, it will scan ports and even attempt to detect the device's operating system.

In it's simplest form just enter:-

  nmap 192.168.1.*

Which will scan for range of addresses using a wildcard, or you could just do the following for a smaller range (192.168.1.1 - 192.168.1.20):-

  nmap 192.168.1.1-20

You can perform a fast scan by using the F flag:-

  nmap -F 192.168.1.*

Or include a range of ports (say 80 to 200):-

  nmap -p 80-200 192.168.1.*

Anyway, it does loads of good stuff, and also generated a lot of errors like this..

  RTTVAR has grown to over 2.3 seconds, decreasing to 2.0

I couldn't find much help about this warning message, but it seems to be related to the scan taking longer than expected. Be patient.

Eventually, I settled on the following command which gave a rapid reply and a reasonable degree of accuracy. (The T5 flag sets the timing template so it times-out sooner.)

  nmap -F -T5 192.168.1.*

Building it into a Webpage

My main aim was to show the data in a webpage which could be viewed from anywhere. I chose to use Perl and the CGI library to do this, although I could have used PHP instead. (I'm just more familiar with Perl)

Here's the code I came up with, it also uses the Nmap_Parser library to make handling the nmap data easy.

#!/usr/bin/perl
use strict;
use Nmap::Parser;
use CGI ":all";

my $q = CGI->new();
print $q->header();
print $q->start_html(-title => 'Scan Network',
                     -head  => meta({-name    => 'viewport',
                                     -content => 'width=device-width, initial-scale=1'}));

my %known = ('Unknown-f0-b4-79-19-a8-ff.home'             => "Rick's Laptop",
                         'android-16f4d57ae478ea5.home'   => "Saul's Phone",
                         'Unknown-84-38-35-41-0d-56.home' => "Ruth's Laptop");

print "<h1>Devices on Network</h1>";
print "<FORM><INPUT TYPE='button' onClick='history.go(0)' VALUE='Refresh'></FORM></h1>";

my $np = new Nmap::Parser;
$np->parsescan('/usr/bin/nmap','-F -T5',"192.168.1.*");

for my $host ($np->all_hosts()){
   if ($host->hostname){
       my $hostname = $host->hostname;
       if (exists($known{$hostname})){ $hostname = $known{$hostname}; }     
       print "<p><span style='display:inline-block;width:7em'>".$host->addr."</span>: ".$hostname."</p>";
   }
}
print $q->end_html();

You may also notice I added a way of remapping 'Unknown' devices to something more helpful. It goes without saying that you should add your own devices to the '%known' hash array, if you find them being displayed with unhelpful names.

I also set the meta tag in the html head so that it behaves better on mobile devices.

Performance

In use it takes between 5 and 10 seconds to perform a scan. Sometimes devices aren't detected on the first scan (especially mobile devices), so refresh the page a few times to get a more accurate idea.

It picks up most devices including my Bluray player and Humax PVR, but for some reason it wouldn't detect the Nintendo Wii earlier. (not sure why)

Anyway, hope some of you find it useful.

Minecraft Server Log Web Interface

Introduction

If you run a Minecraft server in Linux then you'll probably like to keep track of what's going on in the server log. I found that it was a real pain having to remote shell into the server all the time and if it was available via a web browser I could keep track of it more conveniently, or by using my phone or tablet.

I decided that I would use PHP (even though PERL is generally my scripting weapon of choice) and once I'd found the correct commands and syntax it came together quite quickly. If you fancy giving it a try you just need a web server (such as Apache) and of course PHP running on your server.

How it works

It started out being a simple process of opening the server.log file and outputting each line as pre-formatted html.

Something like this..

<html>
<head><title>Minecraft Log</title></head>
<body><pre>
<?php
$filepath = "/opt/minecraft/server.log";
$file = file($filepath);
foreach($file as $line) {
    echo $line."<br/>\n";
}
?>
</pre></body>
</html>

This was great at first, but after we'd been running for a few weeks the server.log file gets too big and the web page gets slower and slower to load. What we needed was a way of only showing the current days log entries, and maybe a date selector so you can reference older stuff.

So I created version two with the following enhancements:-

• Default to showing today's log entries.
• A date selector is added to the top of the web page which sends the new value if altered.
• If page requested with a date parameter then use this instead of today's date.
• If no date has been passed then run a javascript function to scroll the log section to the bottom.

There's a nice feature in the log where text is coloured. This would typically be for chat or system warnings, but as it stood they left messy control codes littered about the log. Rather than remove these I decided to replace them and style the text to match these colours.

This enhancement required the following:-

• Create a series of regular expression replaces to match the control codes.
• Each colour code instance is replaced by a span tag which corresponds to a set of pre-defined styles (the css for these is defined in the head section).
• Put log into a scrollable div.

Here's the completed code..

<!DOCTYPE html>
<html>
<head>
<title>Minecraft Log</title>
<style>
    #log{
        height:400px;
        overflow-y: scroll;
        margin: 5px 0 0 1px;
        border: 1px solid;
    }
    .green{color: green;}
    .red{color: red;}
    .purple{color: purple;}
    .black{color: black;}
    .blue{color: darkblue;}
    .gold{color: gold;}
    .cyan{color: darkcyan;}
    .aqua{color: cadetblue;}
    .gray{color: #888;}
    .bold{ font-weight:bold;}
</style>
<script>
    function scroll(){
    //scroll div to bottom
    var objDiv = document.getElementById("log");
    objDiv.scrollTop = objDiv.scrollHeight;
    }
</script>
</head>
<?php
$pass = htmlspecialchars($_POST["date"]);
if (!$pass){
    echo "<body onLoad='scroll()'>";
}
else {
    echo "<body>";
}

$filepath = "/opt/minecraft/server.log";
$file = file($filepath);
$dates = array();
$last = "";


/* Get List Of Dates */
foreach($file as $line) {
    $date = substr($line, 0, 10);
    if ($date != $last){
        if(preg_match('/\d{4}\-\d{2}\-\d{2}/',$date)) {
            array_push($dates, substr($line, 0, 10));
            $last = $date;
        }
    }
}

/* If Date Provided Use This */
if(preg_match('/\d{4}\-\d{2}\-\d{2}/',$pass)) {
    $last = $pass;
}

/* Add Form Element and Date Selector */
echo "<form method='post'>Select date: <select name='date' onchange='this.form.submit()'>\n";
foreach($dates as $value){
    $select = "";
    if ($value == $last){ $select = " selected='selected'"; }
    echo "<option value='".$value."'".$select.">".$value."</option>\n";
}
echo "</select></form>\n";

/* Output Log For Required Date */
echo "<div id='log'>";
foreach($file as $line) {
    /* Remap Problem Characters */
    $line = preg_replace("/</","&lt;",$line);
    $line = preg_replace("/>/","</span>&gt;",$line);
    $date = substr($line, 0, 10);
    if ($date == $last) {
        /* Remove Unrequired Formatting Codes */
        $line = str_replace("[m","",$line);
        $line = str_replace("[21m","",$line);
        $line = str_replace("[3m","",$line);
        /* Split Log Line Into Sections to Using First Formatting Code Style */
        $segarray = preg_split( '/(\[0|\[m)/', $line );
        for ($i = 1; $i < count($segarray); ++$i){
            /* Do Replace to Add Styled Spans */
            if (preg_match('/;\d{2};\d+m/', $segarray[$i])) {
                $segarray[$i] = preg_replace("/;30/","<span class='black",$segarray[$i]);
                $segarray[$i] = preg_replace("/;31/","<span class='red",$segarray[$i]);
                $segarray[$i] = preg_replace("/;32/","<span class='green",$segarray[$i]);
                $segarray[$i] = preg_replace("/;33/","<span class='gold",$segarray[$i]);
                $segarray[$i] = preg_replace("/;34/","<span class='blue",$segarray[$i]);
            $segarray[$i] = preg_replace("/;35/","<span class='purple",$segarray[$i]);
                $segarray[$i] = preg_replace("/;36/","<span class='aqua",$segarray[$i]);
                $segarray[$i] = preg_replace("/;37/","<span class='gray",$segarray[$i]);
                $segarray[$i] = preg_replace("/;22m/","'>",$segarray[$i]);
                $segarray[$i] = preg_replace("/;1m/"," bold'>",$segarray[$i]);
                $segarray[$i] = $segarray[$i]."</span>";
            }
        }
        /* Rejoin Then Split Log Line Using Second Formatting Code Style */
        $line = join("",$segarray);
        $segarray = preg_split( '/§/', $line );
        for ($i = 1; $i < count($segarray); ++$i){
            /* Do Replace to Add Styled Spans */
            $segarray[$i] = preg_replace("/^0/","<span class='black'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^1/","<span class='blue'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^2/","<span class='green'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^3/","<span class='aqua'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^4/","<span class='red'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^5/","<span class='purple'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^6/","<span class='gold'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^7/","<span class='gray'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^8/","<span class='gray'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^9/","<span class='blue'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^a/","<span class='green'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^b/","<span class='aqua'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^c/","<span class='red'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^d/","<span class='purple'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^e/","<span class='gold'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^f/","<span class='black'>",$segarray[$i]);
            $segarray[$i] = preg_replace("/^r/","<span class='black'>",$segarray[$i]);
            $segarray[$i] = $segarray[$i]."</span>";
        }
        /* Rejoin and Output to Webpage */
        $line = join("",$segarray);
        echo $line."<br/>\n";
    }
}

echo "</div>";
?>
</body>
</html>

nb. Notice the log must now be read twice!

Installation Instructions

Copy the code into a file called index.php and drop it into a directory accessible to your web server and ensure you grant the file execution rights.

Alter the $filepath = "/opt/minecraft/server.log"; line so that it points to your server log and ensure the web server has read rights to this file.

How to Use

Just open the PHP code in your browser and the a page should open showing today's log entries, scrolled to the bottom of the log.

If you then select an earlier date the page will refresh and show this dates entries without the auto-scrolling.

Going Live

The State of Things

My four wire case fan arrived last week featuring low speed operation, rubberised frame and unique corrigated design blades. Originally I'd ordered the three wire version by mistake, but this was unsuitable if you want the system board to be able control the fan speed. Then towards the end of the week I worked out how to transfer my databases over for Davical and Mediawiki. I'd already transferred my other mySQL databases by copying the files from /var/lib/mysql, but the Wiki failed to work and Davical uses PostgreSQL. I knew next to nothing about PostgreSQL so it was a slog to work out how the security worked, how data might be transfered, and how to fix the annoying missing PHP library problems along the way. Nothing works without a battle and this was a fresh install, so it should have been easy?! Go-live was now or never, the other apps could wait, and I didn't want to go through the hassle of transferring data again.

On the hardware side, the new fan (an 80mm 'be quiet!' SHADOW WINGS SW1)
required its mounting plugs to be modified, as the plastic locking pins wouldn't go through the threaded mounting bracket. It would have been flimsy anyway, so I discarded the pins, shortened two of the mounting plugs and used a couple of long bolts and washers to secure it. The fan was then plugged into the system fan header on the motherboard and the computer started. It was immediately obvious that the fan was running very slowly, the BIOS reporting just 500 rpm.

Fan mounted using modified only the lower pegs.

Nb, always test things before you modify them, that way if it's DOA you can get it replaced!

That should be just enough to keep a small amount of air moving through the case, and if temperatures heat up, then the fan should speed up to a maximum of 2000 rpm, at a whisper quiet 16.6dB sound level.

Out With The Old

Going live essentially means moving the data and backup disks over to the new server. But before doing this I was keen to compare power usage and disk speeds, (before and after) ultimately reassuring myself that spending all that money was worthwhile.

With all drives running on the old Via C7 based machine the system was shown to draw a steady 32 watts. This dropped to 24 watts after the drives were removed, leaving just the IDE SSD and case fan as additional loads. That's higher than expected for an eleven watt system board, even allowing for power supply inefficiencies.

I used a utility called hdparm to measure the drive speeds. This is a general purpose drive tweaking tool that can perform a multitude of tasks such as optimising speeds and changing time-out parameters. The following command produces two performance measurements:

   pingu:=# hdparm -Tt /dev/sdx (where x is the drive letter to be measured)

Drive	Cached Reads	Buffered Reads
IDE SSD	225Mb/sec	28.4Mb/sec
Samsung Momentus HN-M101MBB 2½" sata disk (via IDE to SATA adapter)	225Mb/sec	29.5Mb/sec
Samsung Momentus ST1000LM024 2½" sata disk (sata port)	233Mb/sec	30.1Mb/sec
Western Digital Caviar Green 3½" sata disk (sata port)	234Mb/sec	111.5Mb/sec

It's interesting that the SSD wasn't out-performing the other drives in this test, though in practice it had certainly yielded a performance boost when I'd first installed it. I know they wear out, but I'm sure they don't go slower!

For Shiela.

In With The New

The disks where installed in the new Intel i3 based machine and mount points added to the fstab. These I added using their Block ID rather than their device name, which is a unique drive identifier rather than one that's tied to the port being used. You can find out which disks are connected by typing the following command:

   pingu:=# blkid

The address numbers are a little bit too long to write down and type in manually, so I piped the output of this file into my fstab and edited the result to the correct format.

   pingu:=# blkid >> /etc/fstab

This task completed, I re-tested the disks for speed.

Drive	Cached Reads	Buffered Reads
Intel 525 60Gb mSATA card	6470Mb/sec	247Mb/sec
Samsung Momentus HN-M101MBB 2½" sata disk	6500Mb/sec	103Mb/sec
Samsung Momentus ST1000LM024 2½" sata disk	6590Mb/sec	100Mb/sec
Western Digital Caviar Green 3½" sata disk (in SATA2 port)	6500Mb/sec	119.5Mb/sec

There's quite obviously a performance boost but it's surprising that the 3½ inch desktop drive connected to a faster sata2 port wasn't any quicker on buffered reads. To be fair this disk is designed for efficiency and low power with only a 5,400 rpm platter.

Two laptop drives on top and a desktop drive under the mounting plate.

With all disks running the power drawn was just 23 watts, a nine watt saving over the previous machine. That's like turning off a compact fluorescent light bulb that's been burning for the last four years.

A Game Of Two Halves

Building the Beast

It was the day of building the eponymous beast, we had all the bits, the house was cleared of small urchins with their touching fingers, and we were finally ready to go. I opened the system board box and carefully removed & inspected the contents. The Intel DQ77KB board comes with a poster sized "how-to-do" instruction sheet. It has good clear instructions, colour diagrams and each stage is split into simple numbered steps. My son Saul (who was new to computer building) had no problems following them under my watchful eye and with the odd word of assurance. I was like the sensei passing on noble skills to his apprentice.

Number one super guy!

We had a chat about static electricity and being careful about which bits you touch, and then kicked Off with Stage 1: processor installation. The Intel 1150 socket is a marvel of engineering and easier than it looked to use. I open and closed it a few times admiring the action, (it's a sort of tilt, slide and spring-lock). The retail boxed i3 processor came with a standard (four wire) cooling fan which just pressed into the four holes in the system board around the base of the socket. The fan header was close enough by to accommodate the short cable, and under operation it turned quite slowly and was almost silent.

nb. Inside the box I found a HDMI to DVI adaptor, which I'd gone out of my way to buy separately, but hey these things happen. The one I bought is far nicer :-)

Next the laptop style memory just slotted in like normal, though Saul was a little worried about how you had to joggle them a little before they would firmly snap down. The mSATA card fixes onto the full size PCI Express mini-card slot on the system board. I'd never seen these cards before and it really was small (see picture below with it in my son's hand). Resembling memory, it slid in at an angle and then had to be pushed flat against quite a springy force. Two tiny screws held it in place and there was some potential for this "liveliness" to catapult them (never to be seen again) if you didn't have your wits about you.

The mSATA card in Saul's hand.

Overall I am very impressed with the Intel system board, it's clear english instructions and the nifty processor socket and fan. Next step was to mount the board in the case and fire-it-up.

A Case of Beauty

The Streacom F7C EVO came exquisitely packed in a matt black cardboard box not unlike an Apple product, (we have a few of those in the house) protected by foam packaging and a white fabric bag. The comparison didn't end there, the look and quality of the case was equally top notch.

The Streacom F7C EVO Mini-ITX case.

I must say, I've very impressed by the solid feel and attention to detail. It's made of sand blasted thick anodised aluminium, laser engraved with large padded plastic feet and a minimalistic look. This does mean you don't get a reset button or a hard disk light. As a side project I started to contemplate adding an LED shining through the infra-red receiver window. The case is designed for a multimedia PC so an add-on receiver board for a remote control can be fitted behind the font panel, next to the two USB sockets.

Inside is also quite minimal (and no sharp edges anywhere, leaving fingers safe from cuts), there's a painted steel rack that bridges from front to back where you can bolt your hard disks (1 x 3.5" and 1 x 2.5") and a mount for a slot-loading CD/DVD drive. Then towards the rear there's a small bracket for mounting a standard 8" case fan.

Bagged Up Hardware.

Mounting hardware comes in individually labelled bags, but only sizes are stated & not what they're intended for. I took them out and by process of elimination worked out which ones where intended for mounting the system board. But I think they missed a trick here because there are no instructions in the box. Just adding 3.5" HDD or Motherboard to the appropriate label would have really helped, but if you've built a few machines before you'll soon work it out.

The motherboard matched the four mounting pillars exactly, but it does sit a little low in the case making some of the rear sockets slightly hard to access. Finally the power switch and power LED connectors were attached to the colour coded headers.

System Board Mounted Ready to Boot.

That went well, we had a full computer system on a board less than 9 inches square in under an hour.

And then the Fun Began

We both smiled as it booted first time, but then I couldn't get it into the BIOS settings. The splash-screen with instructions flicked by so quickly!! After a few restarts I figured out that F2 needs to be held down and then I was able to continue (surprising this wasn't on the quick-start sheet). I reduced the memory voltage down to 1.35 volts, and then spent a few moments looking at the various config pages. Each option has help text that displays over on the right of the screen, and it actually makes sense. This is a far cry from what I'm used to.

To take care of the disk cloning I downloaded a bootable CD image of Parted Magic. It took a while to start from a USB CD drive up on my old 1Ghz machine but once running it gives you a gnome based system to modify and manage your disks without having to get "down and dirty" with the command line. It actually does a whole lot more so I'd recommend checking it out if you run Linux.

Parted Magic Screen.

The plan was to clone the old IDE solid state disk onto a temporary disk, then clone that onto the mSATA drive. The first clone worked OK, taking about 10 minutes to do a 30Gb disk, but when I attempted the second stage (onto the mSATA card) it kept hanging. In one instance I left it running for over an hour, but it wasn't able to move on from 61% complete. I repeated this a few more times in vein, and then tried to manually format the mSATA from the command line. This rewarded me with a kernel panic and a "Buffer I/O error on device" message.

After a few more hours of trying different things I reverted back to my old server. I've raised an RMA with eBuyer to have it replaced, but it's left my son a little disappointed. He was really hoping to get the Minecraft server up and running. Perhaps it had been going too well, as those who have previously built computers will recognise, but as Saul's first experience at PC building it has actually been quite positive. He did tell me later that it was much easier than he'd imagined, and I think he enjoyed doing it.

So a result,.. though for the meanwhile only a minor one. (We'll try again after I've been through eBuyer's returns process)