Wednesday, July 14, 2010

Catalyst $c->req->params sucks

Catalyst has added parameters() to its Catalyst::Request object and it allows you to get values in an array ref if there are multiple.



my $form = $c->request->parameters;

# ?a=b&b=c
# $form = { a => 'b', b => 'c' }
# ?a=b&a=c&b=c
# $form = { a => [ 'b', 'c' ], b => 'c' };



This might look intuitive but wait a minute. The data structure gets different per user input rather than how you code it, and that sucks. This means you have to always check if the value is an array ref or not, since:



my $v = $c->request->parameters;
my $query = $v->{query};
my @names = @{$v->{name}};



$query might become ARRAY(0xabcdef) if there are multiple query= parameters in the query. @names line might cause Can't use string as an ARRAY ref error if there's only one (or zero) name parameter. This causes horrible issues when using standard HTML elements like option or checkbox forms, or tools like jQuery's serialize().

The correct way to write that would be:



my $v = $c->request->parameters;
my $query = ref $v->{query} eq 'ARRAY' ? $v->{query}->[0] : $v->{query};
my @names = ref $v->{name} eq 'ARRAY' ? @{$v->{name}} : ($v->{name});



and it is tedious and gross.

Thursday, July 8, 2010

Real private method in Perl

For a truly private method, you would usually write the following:


my $_private_method = sub { my ($self, ...) = @_; ... };
...
$self->$_private_method($some, $arguments);
...


which makes the method scoped to the block it was defined in, or the file if it’s not within braces.

This is Perl.

The reason is $_private_method is not special, it's just a "my" variable. So it will be "invisible" outside block or file scope.

Wednesday, June 23, 2010

perlipc in CGI

Sometimes we need start a background process from CGI. When leave that CGI by closing page or window we wish to clean up what we have put in the background. This is perlipc get into role.

There is an example. I have a page displaying a list of network interfaces. I want to get a real time throughput(I/O) chart window when I click on one of these interfaces. Certainly it should be "UP". My solution is forking a poller in background. Poller keeps polling, calculating, and writing data to a shared memory segment. CGI reads from shared memory segment then draws chart for user.



+----------+ fork +------+
|chart page| ----------------> |poller|
+----------+ +------+
| ^
| |
|request |Shared Memory
| |
| +--------+ |
+------> | charter| <------+
+--------+


When click "Close" button on chart page, I need to clean up shared memory and kill the poller. This works fine.

This is the chart page:


...
...

defined (my $pid = fork) or die "Can't fork: $!";

if (!$pid) {
setsid;
open STDIN, '>/dev/null' or die "Can't read from /dev/null: $!";
open STDOUT, '>/dev/null' or die "Can't write to /dev/null: $!";
open STDERR, '>/dev/null' or die "Can't write to /dev/null: $!";
exec '/usr/local/NPMS/bin/if_poller.pl', $val->{ip}, $val->{commstring}, $val->{if};
}
...
...

print <<ENDOFFORM;
<script type="text/javascript">
function clean_up() {
document.getElementById('form1').submit();
window.close();
}
</script>
<form id="form1" method="post" action="clean_poller.cgi">
<input name="pid" value="$pid" type="hidden">
<input name="close" value="Close" onclick="clean_up();" type="button">
</form>
ENDOFFORM



This is the clean_poller.cgi:


...
...
kill 'INT' => $in{'pid'};
...
...


This is the poller which trap SIGINT to gracefully exit after clean up resources:


...
...
my $EXIT = 0;
$SIG{INT} = sub {
$EXIT = 1;
};
...
...
while (1) {
...
...
do polling stuff
do {
IPC::Shareable->clean_up_all;
exit;
} if $EXIT;
...
...
}



Let me explain this whole thing:
1. User click one of the active interfaces("UP" state)
2. Open a new window
3. Before display chart, fork a poller
4. poller start to work in background
5. poller write data sample to shared memory segment
6. chart page request charter to draw a chart
7. charter read data sample from shared memory segment and draw
8. User click "Close" button
9. chart page request clean_poller.cgi before closing
10. clean_poller.cgi get pid and send a INT to it
11. clean_poller.cgi close window
12. poller get INT signal, clean up shared memory segment he created
13. poller exit
14. all resources get cleaned up

That's it!

Detail information refer to perlipc, IPC::Shareable, signal, fork, ...

Monday, June 21, 2010

SNMP read IfInOctets => great trend But what about the absolute throughput value?

Original Link

If you have ever tried to read the throughput value of an ethernet interface in and out using SNMP you may notice that it’s quite easy to get a nice trend graph using your favorite plotting tool (MRTG, zabbix,…) but when you try to get the actual throughput value the amount just never seems to be correct.

During this article I’ll try to explain how the Cisco IfInOctets and IfOutOctets work and what you need to do to get the right value.


1) What is the IfInOctets and IfOutOctets value:

The first thing you need to know is that a Cisco router holds it’s interface value in two tables IfTable and IfXTable, these are fully described in RFC1213/RFC2233.

- ifTable defines 32-bit counters for inbound and outbound octets

-ifXTable provides similar 64-bit counters, also called high capacity (HC) counters

The values we are interested in are IfInOctest, IfHCInOctest, IfOutOctest and IfHCOutOctests. For the sake of this article we will focus on the In values but know that the exact same logic also holds for the Out counters. So lets have a look at the two in Counters. Let’s have a look at what the Cisco documentation tell’s us about these two counters:

- IfInOctets: "The total number of octets received on the interface,
including framing characters. The reference OID is: 1.3.6.1.2.1.2.2.1.10

Lets query this value of interface eth0 of a CentOS Linux Server and see the result:

[root@buildbox55-64bit Perl]# snmpwalk -Os -c public -v2c 192.168.1.230 .1.3.6.1.2.1.2.2.1.10.2
ifInOctets.2 = Counter32: 77043026


- IfHCInOctets: "The total number of octets received on the interface,
including framing characters. This object is a 64-bit
version of ifInOctets. The reference OID is: 1.3.6.1.2.1.31.1.1.1.6

Lets query this value of interface eth0 of a CentOS Linux Server and see the result:

[root@buildbox55-64bit Perl]# snmpwalk -Os -c public -v2c localhost 1.3.6.1.2.1.31.1.1.1.6.2
ifHCInOctets.2 = Counter64: 78219248


As you can read, both values actually return the same data “Total number of octest received”, but we are faced with a first dilemma, you have two counters to poll, each returning a different absolute value and in some bizarre way both are giving you the Total number of octets received.

- IfInOctets 1.3.6.1.2.1.2.2.1.10

For us to understand this we need to know what the value is that is being returned. SNMP is a very basic protocol that runs on just about any network device,… The core idea behind SNMP is simplicity, generic usable and low footprint. To ensure the low footprint SNMP has very little to no intelligence built in. It just returns values you would like to monitor and relies on your toolset to harvest this data en make it usable for you.

The counter we are reading returns the amount of octets received since

- the boot of the device

- since the last rollover period

There are two concepts here that we need to explain be for the puzzle will start to fall together for you:

1) # of octets received => that means if you want to know the data throughput you will have to read the counter twice at time_slot1_value and then lets say 1 second later at time_slot2_value. To know the throughput of octets now subtract

time_slot2_value – time_slot1_value = total # of octets send

2) since the last rollover => as you can imagine the amount of octet bytes sent is just an increasing value and this number can grow really really fast. And this is where the 32bit / 64bit values come into play. In the old days with slow speed networks a 32bit value was used to store the # of octets send, once this 32bit value fill’s to it’s maximum the counter resets to Null and restarts it’s count until it reaches the maximum value again. As you can imagine on a slow speed network this 32bit value fill’s up quite gradually and rollover does not occur all that often. However on a high speed gigabit network a lot of packets are passing through the interface and a 32bit value in memory fill’s up much faster. The net problem with roll over is that at a certain point in time you will subtract time_slot1_value from time_slot2_value but time_slot2_value will be smaller than time_slot1_value thus giving you a negative net value. This is alright for trend,… analysis as long as it does not happen to often.

To give you an idea of how fast this rollover occurs:

- a 10 Mbps stream of back-to-back, full-size packets causes ifInOctets to wrap in just over 57 minutes.

- At 100 Mbps, the minimum wrap time is 5.7 minutes

- At 1 Gbps, the minimum is 34 seconds.

=> this means the 32bit value is just not good enough for modern high speed networks and you will almost always ant to resort back to the 64bit ifHCInOctets counter value.

To follow Cisco documenation: “

For interfaces that operate at 20,000,000 (20 million) bits per second or less, you must use 32-bit byte and packet counters. For interfaces that operate faster than 20 million bits per second, and slower than 650,000,000 bits per second, you must use 32-bit packet counters and 64-bit octet counters. For interfaces that operate at 650,000,000 bits/second or faster, 64-bit packet and octet counters must be used.

Correspondingly, Cisco IOS® Software does not support 64-bit counters for interface speeds of less than 20 Mbps. This means that 64-bit counters are not supported on 10 Mb Ethernet ports, only 100 Mb Fast-Ethernet and other high speed ports support 64-bit counters. “

3) Converting Octets to bits is the last part we need to understand if we want to know the bits per tick that pass through our network interface. To convert the amount of transmitted octets on the Ethernet network to bits we must multiply by 8.

The ending formula to know the #of bits being transferred between two ticks would thus be:

Value2-Value1 * 8 = # bits transferred

Sunday, June 20, 2010

Permission denied of shmget

When using IPC::Shareable as a non-root user, sometimes got this:


IPC::Shareable::SharedMem: shmget: Permission denied


Just because shared memory is same as regular file. If someone has already created this "file", you can not create it with the same name ($glue) unless your are root.

So, whoever your are, please clean up this shm after you done with it.

(tied VARIABLE)->remove;
IPC::Shareable->clean_up;
IPC::Shareable->clean_up_all;


Refer to: perldoc IPC::Shareable

find tips - about the last char

Packaging Moose binary need some CPAN source. I am using cpan2rpm. On my build box, I just use cpan to install Moose, then I knew that all the dependcies are ready. :)

Next I need to convert Moose to binary RPM, for all the source, I want to just get from ~/.cpan, but I always can not remember the -exec syntax for find. Here:

# find ~/.cpan/sources/authors/id -regex '.*\.tar\.gz$' -exec cp {} . \;

Thursday, June 17, 2010

Change IF-MIB ifTable update frequency

CentOS 5.5 x86_64
net-snmp-5.3.2.2-9

When I run this command to try to get the updated data for InOctets:

[root@buildbox55-64bit SPECS]# while [ 1 ]; do snmpwalk -Os -c MYCOMMUNITY -v2c localhost .1.3.6.1.2.1.2.2.1.10.2; sleep 1; done


It seems that the data does not change in a 30 seconds interval:

ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7925145
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505
ifInOctets.2 = Counter32: 7927505


That's ok for "Standard" SNMP poller with a minimum polling interval > 60 seconds. Now if I need a very short interval, say 3 seconds or even 1 seconds. This does not work well.

The solution is to change the IFTABLE_CACHE_TIMEOUT in net-snmp source, here is mine:

/usr/src/redhat/SOURCES/net-snmp-5.3.2.2/agent/mibgroup/if-mib/ifTable/ifTable_data_access.h

Change:

#define IFTABLE_CACHE_TIMEOUT 30

to:

#define IFTABLE_CACHE_TIMEOUT 1


then rebuild net-snmp and upgrade it. now it's working as expected:

[root@buildbox55-64bit SPECS]# while [ 1 ]; do snmpwalk -Os -c MYCOMMUNITY -v2c localhost .1.3.6.1.2.1.2.2.1.10.2; sleep 1; done
ifInOctets.2 = Counter32: 8087029
ifInOctets.2 = Counter32: 8087421
ifInOctets.2 = Counter32: 8087519
ifInOctets.2 = Counter32: 8087911
ifInOctets.2 = Counter32: 8088219
ifInOctets.2 = Counter32: 8088475
ifInOctets.2 = Counter32: 8088633
ifInOctets.2 = Counter32: 8088731
ifInOctets.2 = Counter32: 8088889
ifInOctets.2 = Counter32: 8088987
ifInOctets.2 = Counter32: 8089145
ifInOctets.2 = Counter32: 8089243
ifInOctets.2 = Counter32: 8089401
ifInOctets.2 = Counter32: 8089499
ifInOctets.2 = Counter32: 8089657
ifInOctets.2 = Counter32: 8090085
ifInOctets.2 = Counter32: 8090670
ifInOctets.2 = Counter32: 8090768
ifInOctets.2 = Counter32: 8091268
ifInOctets.2 = Counter32: 8091426
ifInOctets.2 = Counter32: 8091524
ifInOctets.2 = Counter32: 8091682
ifInOctets.2 = Counter32: 8091780
ifInOctets.2 = Counter32: 8091938
ifInOctets.2 = Counter32: 8092036
ifInOctets.2 = Counter32: 8092194
ifInOctets.2 = Counter32: 8092382
ifInOctets.2 = Counter32: 8092604
ifInOctets.2 = Counter32: 8092702
ifInOctets.2 = Counter32: 8092924
ifInOctets.2 = Counter32: 8093022
ifInOctets.2 = Counter32: 8093180
ifInOctets.2 = Counter32: 8093278
ifInOctets.2 = Counter32: 8093436
ifInOctets.2 = Counter32: 8093692
ifInOctets.2 = Counter32: 8093790
ifInOctets.2 = Counter32: 8093948
ifInOctets.2 = Counter32: 8094046
ifInOctets.2 = Counter32: 8094452
ifInOctets.2 = Counter32: 8094550
ifInOctets.2 = Counter32: 8094708
ifInOctets.2 = Counter32: 8094806
ifInOctets.2 = Counter32: 8094964
ifInOctets.2 = Counter32: 8095062
ifInOctets.2 = Counter32: 8095220
ifInOctets.2 = Counter32: 8095318
ifInOctets.2 = Counter32: 8095476
ifInOctets.2 = Counter32: 8095818
ifInOctets.2 = Counter32: 8095976