Cronache di Sarvegia

...because every new challenge hides an opportunity

Subscribe to RSS feed

Posts tagged with "one liners"

Parsing promise_summary.log

Tuesday, May 7, 2013 1:08:00 PM

, , ,

In CFEngine's work directory there is a log file called promise_summary.log. Unsurprisingly, it contains a summary of how agent runs went in the past: how many promises were kept, how many repaired, and how many failed to be repaired. Some weeks ago a blog post on nanard.org showed how this file can be parsed to graph cfengine's activity, and I thought it could be a nice thing to do the same thing in Perl.

For those who never had a peek, the file looks like this:

bronto@murray:/var/cfengine$ tail promise_summary.log 
1367869877,1367869877: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367869877,1367869905: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870164,1367870164: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870164,1367870191: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870450,1367870450: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870450,1367870475: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870797,1367870797: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870797,1367870825: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367871083,1367871084: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367871084,1367871111: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%

The first thing, and I did it in some ten minutes, is to parse the file to extract the relevant information. Check this one-liner:

perl -F: -alne 'next if m{failsafe.cf}i ; my ($start,$finish) = split(",",$F[0],2) ; my $duration = $finish-$start ; my ($version) = m{version (.+) \(} ; my ($kept,$rep,$notrep) = m{Promises observed to be kept (\d+)%, Promises repaired (\d+)%, Promises not repaired (\d+)%} ; print join("\t",$start,$duration,$version,$kept,$rep,$notrep)' promise_summary.log

Or, reformatted:
#!/usr/bin/perl -F: -aln

next if m{failsafe.cf}i ;

my ($start,$finish) = split(",",$F[0],2) ;
my $duration = $finish-$start ;
my ($version) = m{version (.+) \(} ;
my ($kept,$rep,$notrep) = m{Promises observed to be kept (\d+)%, Promises repaired (\d+)%, Promises not repaired (\d+)%} ;

print join("\t",$start,$duration,$version,$kept,$rep,$notrep)

This parses the file and outputs the relevant information tab-separated.

The command line explained
  • -a turns on the "autosplit" feature, where the input is split (normally at spaces) and the chunks are put in the @F array; in this specific case however, the "-F:" switch will make Perl split at colons ":";
  • -l will make it so that newlines are added after each print;
  • -n will wrap the code around a while loop, reading from the file(s) given as argument, and put each line in the $_ "default" variable;
  • -e will execute the code given on the command line;

The code explained
  • we skip the lines generated by the failsafe policy
  • we split the first field (timestamps), and calculate the difference (that is: the duration of the run)
  • we match the version string
  • we match the percentage for kept, repaired, and not repaired promises;
  • we print the bunch in tab-separated format

This is just the start, of course. E.g.: rather than tab-separated values one could print the values in a format suitable to rrdtool, and then use rrdgraph to create the graphs... But that's for later, take care for now. Ciao!

Default paper size in Linux

Tuesday, April 30, 2013 1:36:28 PM

, ,

How does one change the default paper size in Linux? It's incredible how many complicated HOWTOs one finds by google'ing on the subject, and how actually simple it is. Want to default to A4? As superuser, do

echo a4 > /etc/papersize

done!

And if you need more information, you can always man papersize

An eye on the clock

Friday, December 14, 2012 12:26:37 PM

, , , ...

Recently I was trying to make sane a system clock that, for some reason, suddenly slowed down to a crawl. I started to fiddle with adjtimex, and I needed a way to verify the reaction of the clock itself, and ntpd's. On one window, I had a watch ntpq -c pe -c as running. On another, there was an ntpdate -q in a loop. In a third one, I wanted to monitor the changes in frequency (set by ntpd) and ticks (set by me). I found this one-liner pretty useful:

# while true ; do adjtimex -p | perl -alne '/frequency/ and $f=sprintf("%3.2f%%",100*$F[1]/32768000) ; /tick/ and $t=$F[1]-10000 ; END { print scalar(localtime),"\tf=$f\tt=$t" }' ; sleep 30 ; done

This snippet assumes that the frequency tolerance of the clock is 32768000, and that the normal value for ticks is 10000. Check the output of adjtimex -p and the man page to verify this fits your system, too.

Detecting runaway processes

Friday, July 27, 2012 10:37:51 AM

, ,

A simple one liner I used to take a stroll on a few systems, and see where I had a runaway backend process. This uses ps's command extended options (-o); I almost never use those, so I thought it was better to make a note to myself smile
That sort command could be crafted better, but it did its job in this case.

$ for SERVER in x{1..6} ; do echo $SERVER ; ssh -n $SERVER 'ps -C backend -o bsdtime,pid,comm | grep -v TIME | sort -rn | head -n 3' ; done

I'd like to investigate how to manage such a situation automatically using CFEngine. Will try that sooner or later wink

Faking a process name

Tuesday, November 22, 2011 2:51:52 PM

, , ,


Sometimes, for testing purposes, you may need to pretend you have more processes named, e.g., ntpd than you actually do. In my case, I was testing a cfengine policy that should kill all processes named ntpd if they are more than one, and then start a new ntpd from a clean state.

Doing that in perl is very easy. In fact, not only the $0 variable holds the name of the process, but it works also the other way round: if you assign $0, it will change the process name in the system's process table.

So, to fake an ntpd process, you can create a perl script called ntpd like this one:

#!/usr/bin/perl

$0 = q{ntpd} ;
sleep 600 ;

and you run it.

This trick allowed me to test if my policy worked as expected (it did, by the way smile

Can you do it with a one-liner? Of course you can:

perl -e '$0="ntpd" ; sleep 600'

Enjoy!

Say no to letter!

Tuesday, October 11, 2011 3:20:56 PM

, ,

Nothing against the postal service here. Rather, I was trying to turn a small PDF document, four A4 pages, into a booklet that could fit into a single A4 sheet, printed on both sides. Nothing frightful here, it's a simple command chain. Yet, some commands in the chain switch from the A4 format to letter, and I happen not to like their initiative smile

If you need to make such a booklet from file.pdf to booklet.pdf, and want it to stay A4 throughout the process, you need to do this:

pdf2ps file.pdf - | psbook | psnup -2 -pa4 | ps2pdf -sPAPERSIZE=a4 - booklet.pdf

For some reason, psnup and ps2pdf think they should turn A4 into letter. To force A4 in psnup, you use the -p option. ps2pdf uses the same options as ghostscript (gs), hence the -sPAPERSIZE=a4 there.
Also notice the dash in pdf2ps file.pdf -: if you don't use the dash, ps2pdf will output to file.ps, the rest of the chain will starve, leaving you with a funny empty booklet.pdf document.
ps2pdf is also a bit picky, and you need to explicitly tell it that it should read from the standard input (the dash) and write to booklet.pdf.

Have fun!

A more self-explanatory output for iptables logs

Thursday, March 24, 2011 7:00:00 AM

, , , ...

Months ago I wrote a post about how to extract information from iptables logs and get a summary of the packets being blocked.

These days I am playing with firewalls on Xen, and the physdev modules came into play. So, I needed a slight better output to keep into account the physical device the packet was coming from. In short, each single report line had to be like this:

source address > inward physical interface > inward interface > destination address:port:protocol > outward interface > outward physical interface

In my case, I could actually leave the last two out, since we are talking about filtered packets that are not going anywhere, but if you are logging, e.g., forwarded packets, then you are interested in those two fields, as well.

To get this result, I had to change the onliner from the original post slightly, this way:

perl -alne 'my %field ; foreach $token (@F) { next unless $token =~ /=/ ; my ($k,$v) = split(/=/,$token,2) ; $field{$k} = $v } ; print qq{ $field{SRC} > $field{PHYSIN} > $field{IN} > $field{DST}:$field{DPT}:$field{PROTO} > $field{OUT} > $field{PHYSOUT} }' ./selected.rule5.log | sort | uniq -c | sort -nr

Notice I am not using the "if" at the beginning of the one-liner, since the parsed file only contains the lines I am interested in. And the print has changed, of course wink

Flying in the night with puppet

Friday, February 11, 2011 10:48:53 AM

, ,

Following the inheritance mess I talked about earlier, I had to work hard to make the structure sane. Of course, it is quite difficult to implement such a change in steps; rather, you have to change everything in a single step. Nonetheless, it is possible to go the cautious way.

Read more...

Drawing puppet class relationships with graphviz

Tuesday, February 8, 2011 2:38:21 PM

, , , ...

In the last few days, I got the impression that my puppet class hierarchy was growing a bit out of control. In particular, it looked like "glue" classes aimed to simplify stuff were actually causing more harm than benefit. How I found out? Well, it was getting too complicated to understand what included what, and walking the classes by hand quickly became a terrible task.

So, I decided to go for something that could do the work for me: go through my modules, get the include statements, and draw the relationships. Luckily, I remembered I saw that graphviz was designed explicitly to ease that latest task.

Read more...

Quickly graphing RRD data

Friday, December 10, 2010 8:00:00 AM

, , ,

Still fiddling with RRD, and I'm about at the end of the tunnel smile I finally collected a significant amount of data, and I have scripts to aggregate different sources. What I was missing was a quick way to generate graphs, so that I could visually check if my aggregated data "looks" OK.

As usual, the syntax of rrdtool graph is quite verbose and cryptic, not exactly what you hope when all you need is a quick one-shot graph of some RRD files.

As always, Perl comes to the rescue, this time with RRD::Simple. RRD::Simple is by default able to generate nice graphs with pre-loaded color schemes --if you suck at choosing colors like me, this is a really appreciable feature. It has a set of pre-defined graphs that it can generate, as well, but since it accepts native RRD options (beside its own set), it's actually easy to bend it at your need, and generating a graph just needs a one-liner:

perl -MRRD::Simple -e 'RRD::Simple->graph("aggr_root_storage.rrd", destination => "/tmp", sources => [ qw{perc_used worst_perc_free} ], width => 1024, height => 768, start => 1288566000, end => 1291158000, periods => [ "daily" ] )'


Or, reindented:

perl -MRRD::Simple -e \
  'RRD::Simple->graph("aggr_root_storage.rrd", \
    destination => "/tmp", \
    sources => [ qw{perc_used worst_perc_free} ], \
    width => 1024, height => 768, \
    start => 1288566000, end => 1291158000, \
    periods => [ "daily" ] )'

The periods options, in this case, has no purpose but to generate only one graph (otherwise you would get many graphs, all equal; why? go and find out yourself, if you really care wink

And what about plotting a collection of RRDs? It could be something like:

$ for FILE in aggr*.rrd ; do export FILE ; perl -MRRD::Simple -e 'RRD::Simple->graph($ENV{FILE}, destination => "/tmp", width => 1024, height => 768, start => 1288566000, end => 1291158000, periods => [ "daily" ] )' ; done

or, clearer:

$ for FILE in aggr*.rrd ; \
do \
  export FILE ; \
  perl -MRRD::Simple -e \
    'RRD::Simple->graph($ENV{FILE}, \
      destination => "/tmp", \
      width => 1024, height => 768, \
      start => 1288566000, end => 1291158000, \
      periods => [ "daily" ] )' ; \
done

1 2 Next »