I published an SQLite3 storage back-end to NIST::NVD on the CPAN. It’s pretty quick. About as fast as the DB_File one, but without the down side of being tied to DB_File. It shouldn’t be too difficult to re-factor this code to any DBI-based database. MariaDB anyone?

I know it works on Debian. The nightly CPAN test results should come back shortly and I’ll find out how well it works on other platforms.

Dear Lazyweb,

  I'd like a video of the 90s commercial "That's what daffodils do".  If you do that, I'll release a basic library for the iPhone to use SPDY[1].

[1] I said basic!

Readers who look at our blog itself (rather than one of the lovely sites that reprint our articles) may have noticed that you can now comment in either the usual WordPress way (Name/Email/Link) or by logging in with a social media profile from one of a large range of providers, including WordPress, Livejournal, Yahoo, Google and many more.

This uses the broadly-cooperative openID system. If you run a website that accepts reader contributions, you should allow comments with openid because it helps people to use their existing social media membership without you having to surrender any control to facebook, twitter, or anyone else (unless you choose to). You also don’t have to ask your readers to weaken their security settings like with disqus (which requires javascript and third-party cookies).

The comment form on our site is powered by the openid plugin, together with our co-op’s version of the comments-with-openid plugin which can be downloaded from our site. Please download them if you’d find them useful for your WordPress site. (I’d love to adopt the official comments-with-openid at wordpress.org because the previous maintainer doesn’t answer – anyone know how to do that? I’m surprised it’s not in the FAQ.)

Do you use some other platform? What tools have let you add openid logins to it? For example, Drupal has some openID support in its core distribution: what else is out there?

• Michael Meeks gave awesome talks at FOSDEM, so Kate was inspired to hack on LibreOffice. I was inspired to write this blog entry in a list. She probably wins.
• Building LibreOffice master on Debian stable failed for her with a segmentation fault in GNU Make. A bit of searching threw up Savannah bug #20033, which is hitting everyone on the upstream mailing list.
• Bumped severity and offered to NMU Debian bug #622644.
• Then actually tried building make-dfsg in cowbuilder, and aclocal fails in the clean environment because /usr/share/aclocal does not exist. I think it's related to Debian bug #565663, but I'm still poking it.

Matthew Aslett wrote about how the proportion of projects released under GPL-like licenses appears to be declining, at least as far as various sets of figures go. But what does that actually mean? In absolute terms, GPL use has increased - any change isn't down to GPL projects transitioning over to liberal licenses. But an increasing number of new projects are being released under liberal licenses. Why is that?

The figures from Black Duck aren't a great help here, because they tell us very little about the software they're looking at. FLOSSmole is rather more interesting. I pulled the license figures from a few sites and found the following proportion of GPLed projects:

RubyForge: ~30%
Google Code: ~50%
Launchpad: ~70%

I've left the numbers rough because there's various uncertainties - should proprietary licenses be included in the numbers, is CC Sharealike enough like the GPL to count it there, that kind of thing. But what's clear is that these three sites have massively different levels of GPL use, and it's not hard to imagine why. They all attract different types of developer. The RubyForge figures are obviously going to be heavily influenced by Ruby developers, and that (handwavily) implies more of a bias towards web developers than the general developer population. Launchpad, on the other hand, is going to have a closer association with people with an Ubuntu background - it's probably more representative of Linux developers. Google Code? The 50% figure is the closest to the 56.8% figure that Black Duck give, so it's probably representative of the more general development community.

The impression gained from this is that the probability of you using one of the GPL licenses is influenced by the community that you're part of. And it's not a huge leap to believe that an increasing number of developers are targeting the web, and the web development community has never been especially attached to the GPL. It's not hard to see why - the benefits of the GPL vanish pretty much entirely when you're never actually obliged to distribute the code, and while Affero attempts to compensate from that it also constrains your UI and deployment model. No matter how strong a believer in Copyleft you are, the web makes it difficult for users to take any advantage of the freedoms you'd want to offer. It's as easy not to bother.
So it's pretty unsurprising that an increase in web development would be associated with a decrease in the proportion of projects licensed under the GPL.

This obviously isn't a rigorous analysis. I have very little hard evidence to back up my assumptions. But nor does anyone who claims that the change is because the FSF alienated the community during GPLv3 development. I'd be fascinated to see someone spend some time comparing project type with license use and trying to come up with a more convincing argument.

(Raw data from FLOSSmole: Howison, J., Conklin, M., & Crowston, K. (2006). FLOSSmole: A collaborative repository for FLOSS research data and analyses. International Journal of Information Technology and Web Engineering, 1(3), 17–26.)

comments

After not attending FOSDEM for a few years, this year I decided to attend and also give a talk about “Debian packaging for beginners”, a replay of a talk given by Gergely Nagy in Debconf11 (video). As the per distribution devrooms were replaced a few years ago with the cross-distributions rooms, I thought it might be a good chance to have this kind of introduction talk to help people started contributing to Debian (or derived distributions).

The talk wasn’t meant to replace reading the  documentation (new maintainer’s guide, developer’s references and the debian policy), but it’s a good start for those who want some hands-on experience. The idea is to start with a just the upstream directory and progress trough the various errors and warnings we get during the build process using dpkg-buildpackage. The outcome is of course very basic, but enough to get people ready to do things on their own, including various QA tests on the package (e.g. lintian and debdiff).

The presentation covers the important points of the 3 main files in the debian directory: control, changelog and rules. It addition it holds some information about the various tools one can used to test the packages. I hope to make another version of the presentation to be more standalone than just having the main points during the talk itself.

Filed under: Debian GNU/Linux, FOSDEM

SNMP stands for Simple Network Management Protocol. It allows a manager to query information from an agent. A popular use of SNMP is to retrieve network interface counters to plot a bandwidth graph. While the “S” in SNMP stands for simple, SNMP can be quite difficult to deal with. However, it is still the de facto standard for retrieving metrics in an heterogeneous environment. Net-SNMP is a suite of SNMP applications including an agent. Out of the box, this agent exports a lot of information but if something is missing, there are several ways to extend it. Which ones?

tl;dr: in my opinion, extending Net-SNMP should be done with either of those methods:

• using extend directive;
• using pass_persist directive;
• using AgentX protocol.

• SNMP in a nutshell
• Extending Net-SNMP
  • With arbitrary commands
  • With pass-through scripts
  • With AgentX protocol
    • Perl and Python
    • C with “new” API
    • C with “traditional” API
  • Other solutions

SNMP in a nutshell

The variables accessible with SNMP through the agent are organized in a tree. Each variable is typed and associated to an object identifier (or OID). For example, .1.3.6.1.2.1.31.1.1.1.1.2 is the OID of the name of the second network card.

Usually, SNMP is operated over UDP, port 161. For the purpose of this article, we consider a manager can issue three kinds of requests:

GET

retrieve a variable.

GETNEXT

retrieve the next variable, in lexical order of their OID. This operation enables an SNMP manager to walk through all available variables, for example to discover the list of interfaces of an agent. This operation is what makes an SNMP agent difficult to implement but this is also a critical feature.

SET

request the modification of a variable.

Since such an OID may be difficult to handle by humans, available variables are described by a MIB module which is a file that should be parsed to get a mapping between OID and variable names and types. For example, the above OID can also be referred as IF-MIB::ifName.2. Those MIB modules are described using a subset of ASN.1 defined in RFC 2578.

A MIB module allows to group several variables into a conceptual table. For example, IF-MIB::ifDescr, IF-MIB::ifType, IF-MIB::ifSpeed are columns of IF-MIB::ifTable. Each row is associated to an index. In the case of IF-MIB::ifTable, it is the interface index. Net-SNMP includes snmptable allowing one to display such a table in a natural manner1:

$ snmptable localhost IF-MIB::ifTable SNMP table: IF-MIB::ifTable ifIndex ifDescr ifType ifMtu ifSpeed 1 lo softwareLoopback 16436 10000000 2 eth0 ethernetCsmacd 1500 100000000 3 eth1 ethernetCsmacd 1500 10000000 4 br0 ethernetCsmacd 1500 0

Variables inside a table are called columnar objects while those outside are scalar objects. SNMP, as a protocol, does not care of this distinction since it does not know the concept of MIB modules.

Other useful tools from Net-SNMP includes snmpget to get a specific variable, snmpwalk to discover available variables (with GETNEXT operation) from an OID and snmptranslate to translate between object names and OID.

$ snmpget localhost IF-MIB::ifDescr.2 IF-MIB::ifDescr.2 = STRING: eth0 $ snmpwalk localhost IF-MIB::ifDescr IF-MIB::ifDescr.1 = STRING: lo IF-MIB::ifDescr.2 = STRING: eth0 IF-MIB::ifDescr.3 = STRING: eth1 IF-MIB::ifDescr.4 = STRING: br0 $ snmpwalk -On public localhost .1.3.6.1.2.1.2.2.1.2 .1.3.6.1.2.1.2.2.1.2.1 = STRING: lo .1.3.6.1.2.1.2.2.1.2.2 = STRING: eth0 .1.3.6.1.2.1.2.2.1.2.3 = STRING: eth1 .1.3.6.1.2.1.2.2.1.2.4 = STRING: br0 $ snmptranslate -On IF-MIB::ifDescr.3 .1.3.6.1.2.1.2.2.1.2.3 Extending Net-SNMP

ethtool -S allows us to access various statistics from a network card, like the number of octets received and transmitted or the number of collisions. While some of those statistics are defined in IF-MIB, in RMON-MIB and in EtherLike-MIB, some of them are not available and very specific. Therefore, we would like to export those statistics, unmodified, through SNMP.

To the best of my knowledge, no MIB module exists for such an usage. The first task is to write one. It can be called ETHTOOL-MIB. It only contains one table indexed by the interface index and the name of the statistic. Here is the expected output:

$ snmpwalk localhost ETHTOOL-MIB::ethtoolStat.2 ethtoolStat.2.'align_errors' = Counter64: 0 ethtoolStat.2.'broadcast' = Counter64: 25 ethtoolStat.2.'multicast' = Counter64: 345 ethtoolStat.2.'rx_errors' = Counter64: 0 ethtoolStat.2.'rx_missed' = Counter64: 0 ethtoolStat.2.'rx_packets' = Counter64: 262011 ethtoolStat.2.'tx_aborted' = Counter64: 0 ethtoolStat.2.'tx_errors' = Counter64: 0 ethtoolStat.2.'tx_multi_collisions' = Counter64: 0 ethtoolStat.2.'tx_packets' = Counter64: 296170 ethtoolStat.2.'tx_single_collisions' = Counter64: 0 ethtoolStat.2.'tx_underrun' = Counter64: 0 ethtoolStat.2.'unicast' = Counter64: 261641

I have coded various implementations of this MIB module and made them available in a GitHub repository.

With arbitrary commands

The extend directive allows the agent to execute an arbitrary command and to provide its output and status through NET-SNMP-EXTEND-MIB module. If we want to export the statistics for eth0, we could add something like this in snmpd.conf, using a custom script to format the output of ethtool -S in a way that is more suitable for our use:

# Export eth0 statistics extend eth0-names /full/path/to/ethtool-stats eth0 names extend eth0-values /full/path/to/ethtool-stats eth0 values

The result can be retrieved through NET-SNMP-EXTEND-MIB::‌nsExtendOutput2Table:

$ snmpwalk localhost NET-SNMP-EXTEND-MIB::nsExtendOutput2Table nsExtendOutLine."eth0-names".1 = STRING: tx_packets nsExtendOutLine."eth0-names".2 = STRING: rx_packets nsExtendOutLine."eth0-names".3 = STRING: tx_errors nsExtendOutLine."eth0-names".4 = STRING: rx_errors nsExtendOutLine."eth0-names".5 = STRING: rx_missed nsExtendOutLine."eth0-names".6 = STRING: align_errors nsExtendOutLine."eth0-names".7 = STRING: tx_single_collisions nsExtendOutLine."eth0-names".8 = STRING: tx_multi_collisions nsExtendOutLine."eth0-values".1 = STRING: 246309 nsExtendOutLine."eth0-values".2 = STRING: 223941 nsExtendOutLine."eth0-values".3 = STRING: 0 nsExtendOutLine."eth0-values".4 = STRING: 0 nsExtendOutLine."eth0-values".5 = STRING: 0 nsExtendOutLine."eth0-values".6 = STRING: 0 nsExtendOutLine."eth0-values".7 = STRING: 0 nsExtendOutLine."eth0-values".8 = STRING: 0

It is also possible to configure such a command using SET requests. For example, if we want to add statistics for eth2, we could issue the following command:

$ snmpset -m +NET-SNMP-EXTEND-MIB localhost \ > 'nsExtendStatus."eth2-names"' = createAndGo \ > 'nsExtendCommand."eth2-names"' = /full/path/to/ethtool-stats \ > 'nsExtendArgs."eth2-names"' = 'eth2 names' \ > 'nsExtendStatus."eth2-values"' = createAndGo \ > 'nsExtendCommand."eth2-values"' = /full/path/to/ethtool-stats \ > 'nsExtendArgs."eth2-values"' = 'eth2 values' nsExtendStatus."eth2-names" = INTEGER: createAndGo(4) nsExtendCommand."eth2-names" = STRING: /full/path/to/ethtool-stats nsExtendArgs."eth2-names" = STRING: eth2 names nsExtendStatus."eth2-values" = INTEGER: createAndGo(4) nsExtendCommand."eth2-values" = STRING: /full/path/to/ethtool-stats nsExtendArgs."eth2-values" = STRING: eth2 values $ snmpgetnext -m +NET-SNMP-EXTEND-MIB localhost \ > 'nsExtendOutLine."eth2-names"' nsExtendOutLine."eth2-names".1 = STRING: tx_packets

This can be very convenient but also a huge security risk. You should disable this functionality or enable it only for some SNMPv3 user.

Using extend does not allow to implement an arbitrary MIB module. If the cache is not disabled, extend is a good solution for various simple needs, like providing the content of a one-line file or the output of some Nagios plugin. However, exporting anything tabular, like interface statistics, is not a good fit.

With pass-through scripts

Net-SNMP allows one to extend the agent by delegating some sub-tree to a script defined by the pass_persist directive. Such a script will receive requests on its standard input with a very simple line-oriented protocol described in the manual page of snmpd.conf. Here is an example of interaction:

→ PING ← PONG → get → .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116 ← .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116 ← counter64 ← 223941 → getnext → .1.3.6.1.4.1.39178.100.1.1.1.2.2 ← .1.3.6.1.4.1.39178.100.1.1.1.2.2.97.108.105.103.110.95.101.114 ← counter64 ← 0

The protocol is simple enough to implement it from scratch in any scripting language. However, in Perl, you may prefer to use SNMP::‌ExtensionSNMP::‌PassPersist extension from Sébastien Aperghis-Tramoni. You can find the complete example on Github.

use SNMP::Extension::PassPersist; my $extsnmp = SNMP::Extension::PassPersist->new( backend_collect => \&update_tree ); $extsnmp->run; sub update_tree { my @interfaces = </sys/class/net/*>; foreach my $interface (@interfaces) { $interface =~ s/^.*\///; open(ETHTOOL, "ethtool -S $interface 2>/dev/null |") or next; while (<ETHTOOL>) { /^\s+(\w+): (\d+)$/ or next; my $name = $1; my $value = int($2); my $oid = oid_compute($interface, $name); $extsnmp->add_oid_entry($oid, "counter", $value); } close(ETHTOOL); } }

With Python, snmp-passpersist module, from Nicolas Agius, provides a similar interface. Look at the complete example on GitHub.

pass_persist allows you to implement an arbitrary MIB module with honest performances. Here are some important things to know about this directive:

• Unless you have a recent version of Net-SNMP, 64bit types are converted to 32bit equivalents. The counters may overflow quickly.
• While handling a request, you are not allowed to pause (for example, to fetch a remote information): the agent would become unresponsive. If you need slow-to-get data or data from the agent, grab them in a separate thread and ensure you have them beforehand.

With AgentX protocol

AgentX is a protocol defined in RFC 2741 allowing a master agent to be extended by independent sub-agents. For snmpd to become a master agent, you just need to add master agentx in snmpd.conf. Here are some of the benefits of an AgentX sub-agent over pass_persist:

1. No configuration is needed for the master agent to accept an additional sub-agent. A sub-agent registers to the master agent the MIB modules (or part of them) it wants to take care of.
2. A sub-agent is decoupled from the master agent. It can run with a different identity or be integrated into another daemon to export its internal metrics, send traps or allow remote configuration through SNMP.
3. AgentX protocol can be carried over TCP. Sub-agents can therefore run on a foreign host or in a jailed environment.
4. 64bit types are fully supported. Traps are also supported.

Perl and Python

Net-SNMP provides a Perl module, NetSNMP::agent, to write a sub-agent using AgentX protocol. This module is not as convenient as the module for pass_persist described above since it does not provide a convenient layer to put required information into some datastore. Basically, it mimics the available C API. Here are a few examples of its use:

• An implementation of ETHTOOL-MIB I have done in the same way as the pass_persist version. My Perl skills are a bit rusty and the code has many rough edges.
• An implementation of a sub-agent for Redis servers.
• An implementation of BRIDGE-MIB shipped with Net-SNMP.

There is no similar binding for Python but there is an agentx extension using ctypes. It provides a more high-level view. I did not write an example for this one but you can look at the example provided by the author.

C with “new” API

To write a sub-agent in C, you have to choose between two API:

1. the API inherited from UCD-SNMP also known as the “traditional” API;
2. the new API developed for Net-SNMP 5.x.

With both API, an handler is registered to take care of one or several sub-trees. With the traditional API, the handler is invoked with very little information. It is then up to you to locate the appropriate variable by mapping the index part of the OID to the objects you want to export. It is quite simple for scalar objects but columnar objects with complex indexes are a pain with such an API.

The new API provides various helpers. Some of them allow you to copy objects into an array or a list and let Net-SNMP library do the hard work of serving them. Another helper will provide methods to serve objects without putting them in a special structure but by providing an iterator. Some of those helpers can also help you to cache some variables.

On top of this new API, Net-SNMP comes with mib2c, a tool that will help you convert a MIB module into C code: after a few questions, you will get a skeleton C code you will have to complete with the logic to retrieve real objects. A generated file also provides directions on what to do (with some magic command to have a step-by-step cookbook).

I was able to implement ETHTOOL-MIB by just running it through mib2c, choosing the table container helper with cache and adding some code in ethtoolStatTable_data_access.c. You can look at the result on Github. Here is a stripped-down version2 of what has been modified:

/* Socket for ioctl */ skfd = socket(AF_INET, SOCK_DGRAM, 0); /* Iterate through all interfaces */ getifaddrs(&ifap); for (ifa = ifap; ifa != NULL; ifa = ifa->ifa_next) { /* Grab statistics name and values */ strcpy(ifr.ifr_name, ifa->ifa_name); drvinfo.cmd = ETHTOOL_GDRVINFO; ifr.ifr_data = (caddr_t) &drvinfo; ioctl(skfd, SIOCETHTOOL, &ifr); n_stats = drvinfo.n_stats; strings->cmd = ETHTOOL_GSTRINGS; strings->string_set = ETH_SS_STATS; strings->len = n_stats; ifr.ifr_data = (caddr_t) strings; ioctl(skfd, SIOCETHTOOL, &ifr); stats->cmd = ETHTOOL_GSTATS; stats->n_stats = n_stats; ifr.ifr_data = (caddr_t) stats; ioctl(skfd, SIOCETHTOOL, &ifr); ifIndex = if_nametoindex(ifa->ifa_name); /* Iterate through statistics */ for (i = 0; i < n_stats; i++) { /* Declare the appropriate index */ strncpy(ethtoolStatName, (char *)&strings->data[i * ETH_GSTRING_LEN], ETH_GSTRING_LEN); ethtoolStatName[sizeof(ethtoolStatName) - 1] = '\0'; ethtoolStatName_len = strlen(ethtoolStatName); /* Append this new variable to the container. */ rowreq_ctx = ethtoolStatTable_allocate_rowreq_ctx(); ethtoolStatTable_indexes_set(rowreq_ctx, ifIndex, ethtoolStatName, ethtoolStatName_len); rowreq_ctx->data.ethtoolStat.high = stats->data[i] >> 32; rowreq_ctx->data.ethtoolStat.low = (uint32_t)stats->data[i]; CONTAINER_INSERT(container, rowreq_ctx); } }

See? It is pretty easy. I have only enumerated all the objects I wanted to expose. I didn’t even have to convert statistics names to an OID, mib2c built a ethtoolStatTable_indexes_set() for this purpose. OpenHPI project provides an extensive documentation to write a sub-agent this way.

Now, on the downside, to implement a single table, I have 14 generated files whose code style is usually unfit to integrate into an existing project : some people, me included, do not like automatically generated code, all the more when it is so verbose. The solution would be to use the new API without mib2c; unfortunately, while the documentation exists, all provided examples rely exclusively on mib2c tool. Unless you are able to grasp the new API without it, I would restrict its use to two cases:

• Writing a new module for inclusion into Net-SNMP. Since most modules are now written with the help of mib2c, there is no coding style problem.
• Writing a standalone sub-agent for some MIB. As a standalone project, you won’t run into coding style problems either.

For other uses, let’s have a look at the traditional API.

C with “traditional” API

The traditional API is way simpler and even if it is quite old, it is easier to find associated examples. Here is how to initialize the agent, again, in a stripped-down version (the complete version is on GitHub):

static oid ethtool_oid[] = {1, 3, 6, 1, 4, 1, 39178, 100, 1}; static struct variable3 ethtool_vars[] = { {1, ASN_COUNTER64, RONLY, ethtool_stat, 3, {1, 1, 2}} }; int main() { netsnmp_enable_subagent(); snmp_disable_log(); snmp_enable_stderrlog(); init_agent("ethtoolAgent"); REGISTER_MIB("ethtoolStatTable", ethtool_vars, variable3, ethtool_oid); init_snmp("ethtoolAgent"); while (1) agent_check_and_process(1); }

The REGISTER_MIB() macro is used to register the provided root OID (ethtool_oid) to the master agent. It also registers handlers associated to various sub-OID through the ethtool_vars[] array. Each member of this array features the following fields:

• magic is an integer allowing one to discriminate OID handled by the same handler. For a table, this allows to handle several columns with the same handler since they share a common index.
• type is the type of the variable that will be returned.
• acl tells if the object is read-only (RONLY) or read-write (RWRITE).
• findVar is the handler associated to the object. Its task will be to find the appropriate object and return its value.
• The two last fields are the relative OID this entry is responsible for along with its length.

The prototype of ethtool_stat() handler function is the following:

static u_char* ethtool_stat(struct variable *vp, oid *name, size_t *length, int exact, size_t *var_len, WriteMethod **write_method);

This function should return the value of the requested variable or NULL if no appropriate object was found. The value can be statically stored since it will be copied. Here is the description of the arguments:

• struct variable *vp is the structure the handle was associated with, except that vp->name is now a full OID, not a relative one. You can access vp->magic to determine which column or which scalar to handle if this handler is common to several objects.
• oid *name and size_t *length describe the OID requested. If you want to return another OID (for a GETNEXT request for example), you can modify them. name points to an OID array large enough to fit any OID you want to return.
• int exact tells if the requested OID should be exactly matched (case of GET or SET) or not (GETNEXT).
• size_t *var_len is used to tell the size of the returned object.
• WriteMethod **write_method is only used if the variable is modifiable. When a SET request is received, the API will call the handler with exact set to 1 and expects to get a pointer to the function that will handle the write operation3.

To implement this function, you are on your own. When relying on externally fetched data, you need to handle a cache yourself. I have used a red-black tree for this purpose (with code stolen from OpenBSD). This also enables an easy implementation of the GETNEXT operation.

Here are three projects using this API to provide SNMP support:

• Keepalived is a VRRP daemon and a monitoring daemon for LVS clusters. I have added a complete SNMP support. The most interesting files are core/snmp.c, check/check_snmp.c and vrrp/vrrp_snmp.c. It features a tight integration into Keepalived event loop that should be reusable by other projects. It includes read-only support of all data structures (configuration, state and statistics), ability to send traps on events and a write support for some values.
• lldpd is an implementation of 802.1AB (LLDP) which is a protocol allowing an equipment to advertise itself to its neighbors on Ethernet level. 802.1AB comes with a huge MIB module and many extensions. lldpd provides a fairly complete read-only implementation of this MIB module. Some tables have complex indexes. Have a look at agent.c. There is currently no write support and no traps but lldpd features privilege separation and some code allows it to still use an Unix socket to speak with the master agent.
• Asterisk, an open source PBX, includes an AgentX sub-agent allowing one to grab some metrics. Look at res/snmp/agent.c.

Other solutions

The manual page for snmpd.conf explores other ways to extend Net-SNMP agent under the section “Extending agent functionality“:

• exec and sh are deprecated. Use extend instead.
• pass is a bit simpler than pass_persist. The command is run once for each request. Such a command can easily be turned into a pass_persist script. Moreover, Net-SNMP will not answer any request while a pass command is running.
• proxy redirects requests to another SNMP agent. Unless you have some code that can only act as a complete SNMP agent, it is better to use AgentX instead. For programs using Net-SNMP API, turning an agent into an AgentX sub-agent is just one line of code. Since a sub-agent involves less code than a complete agent, it is more efficient and more reliable.
• smux has been superseeded by AgentX.
• perl and dlmod allow one to load and execute external code inside the running agent. Unless you have important performance requirements, it is better to use a sub-agent instead: if some bug happens, only the sub-agent will be affected, not the master agent. A sub-agent sticks more closely to the Unix philosophy: “write programs that do one thing and do it well”. However, Net-SNMP FAQ says:

Implementing the module in C within the main agent (directly or via dlmod) is probably the most efficient and reliable, closely followed by embedded perl (or python) extensions. These have the advantage of minimal overheads between the code implementing the MIB module, and the agent framework, and no inter-process communication issues.

1. Additional arguments for snmp* are usually required to specify a version and a community (for SNMPv1 and SNMPv2c) or a user (for SNMPv3). In all examples, I do not specify anything. I assume that ~/.snmp/snmp.conf contains the appropriate bits. Moreover, some outputs are truncated. ↩

2. No allocations, no variable declarations and no error checkings. Do not do this at home! ↩

3. If you want to look at an example of how to handle a SET request, look at the
   SNMP support for Keepalived. A SET operation is more complex because SNMP ensures that the
   SET operation is atomic across all provided variables. It uses a three-staged commit. The traditional API is very inefficient for this kind of operation because the object has to be located again for each step. ↩

Since infamous erasing of factory calibration in my ColorHug device and restoring calibration matrix, I noticed it did screen calibration wrong. However I did not find time to properly investigate the issue. Yesterdays mail from Richard was actually trigger for me so I've opened up this topic.

In the end it turned out to be caused by Little CMS wrongly parsing CCMX in case you are using locales which use something else than . as decimal point.

After lot of googling, I've realized there is probably no good way of parsing floats independent on current locales, so I used one of hacks I found and I think it's less intrusive - get current decimal point by printing float string using printf and then convert the string to it. I know it looks ugly, but including own implementation of strtod is also not nice and playing with locales is definitely something not thread safe to do within widely used library.

Anyway I've asked upstream to merge my patches, so let's see what they think of it.

Filed under: Debian English Linux Photography Suse | 8 comments | Flattr this!

Dear Lazyweb,

is it possible to tell aptitude to show older versions of a package next to the currently available one if it is still present in /var/cache/apt/archives? Like it does when you use unstable and experimental side by side? I know that aptitude does not really support downgrades of packages, but showing those packages directly in aptitude if available in the cache is a lot easier than searching them in the file system and installing them manually, especially if you don’t know where they hide.

Software in the Public Interest, the mass-membership association that supports some great Free and Open Source Software projects, will hold a public board of directors meeting today, Thursday 9th February 2012 at 21:00 UTC. The day and time of SPI meetings has changed recently, so maybe different people can get to them now.

They’re held online, on irc.spi-inc.org (the OFTC network). The agenda for the meeting is open and available at http://www.spi-inc.org/meetings/agendas/2012/2012-02-09/ and there’s been a bit of discussion of back office support on the SPI email list.

I’ll link to a meeting summary from the comments in this blog post after it happens.

Paul Wayper writes about the merits of using toll transponders to pay for parking.

I can report that I'm able to use my FasTrak tag to pay for parking at San Francisco International Airport, and it does indeed rock.

I'm unaware of anywhere else accepting it as a form of payment though.

So, Collier Technologies LLC needs to pass annual audits to operate a certification authority recognized by the SoS. To this end, I’m working with the fine group of developers who maintain SELinux. It seems that the configuration of Xorg that I’m using while typing this here blog post does not have a policy set up for it in the Debian packages. Or if it does, I don’t know enough about it to figure it out.

I’ve been keeping logs and publishing them here:

http://www.colliertech.org/federal/nsa/

I’ll update this post as progress is made.

[edit 20120608T1042]

It looks like loading all .pp files (except alsa) makes X run:

cjac@foxtrot:/usr/share/selinux/default$ time sudo \ semodule -i `ls *.pp | grep -v -e 'base.pp' -e 'alsa.pp'` real 0m24.148s user 0m23.249s sys 0m0.628s

I had to boot into single user mode and load the policies before switching to runlevel 2. To get the kernel args added to the grub command line, I modified /etc/default/grub to include this line:

cjac@foxtrot:/usr/share/selinux/default$ grep -i selinux /etc/default/grub GRUB_CMDLINE_LINUX=" selinux=1 security=selinux"

Next steps:

• get the policies loaded at boot time
• get seinfo working

[edit 20120208T1305]

It looks like the seinfo package has not been updated in the last 18 months.

cjac@foxtrot:/usr/src/git/debian/setools$ grep url .git/config url = git://git.debian.org/git/users/srivasta/debian/setools.git cjac@foxtrot:/usr/src/git/debian/setools$ git log | head -4 commit 22a5d3e451d8a1e60a3c746466c865e63089a92a Merge: fa238f0 149e283 Author: Manoj Srivastava <srivasta> Date: Tue Jul 20 23:10:06 2010 -0700

[edit 20120208T1346]

Stephen tells me that the modules are persistent across re-boots.

> What's the best way to do this at boot? You just do it once and it remains until/unless you remove it with semodule -r. No need to do it on each boot. Normally it is done when you install the policy package, but since your policy package apparently didn't install all modules, I'm suggesting that you do so manually.

I’ve previously written Some Notes on DRBD [1] and a post about DRBD Benchmarking [2].

Previously I had determined that replication protocol C gives the best performance for DRBD, that the batch-time parameters for Ext4 aren’t worth touching for a single IDE disk, that barrier=0 gives a massive performance boost, and that DRBD gives a significant performance hit even when the secondary is not connected. Below are the results of some more tests of delivering mail from my Postal benchmark to my LMTP server which uses the Dovecot delivery agent to write it to disk, the rates are in messages per minute where each message is an average of 70K in size. The ext4 filesystem is used for all tests and the filesystem features list is “has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize“.

p4-2.8 Default Ext4 1663 barrier=0 2875 DRBD no secondary al-extents=7 645 DRBD no secondary default 2409 DRBD no secondary al-extents=1024 2513 DRBD no secondary al-extents=3389 2650 DRBD connected 1575 DRBD connected al-extents=1024 1560 DRBD connected al-extents=1024 Gig-E 1544

The al-extents option determines the size of the dirty areas that need to be resynced when a failed node rejoins the cluster. The default is 127 extents of 4M each for a block size of 508MB to be synchronised. The maximum is 3389 for a synchronisation block size of just over 13G. Even with fast disks and gigabit Ethernet it’s going to take a while to synchronise things if dirty zones are 13GB in size. In my tests using the maximum size of al-extents gives a 10% performance benefit in disconnected mode while a size of 1024 gives a 4% performance boost. Changing the al-extents size seems to make no significant difference for a connected DRBD device.

All the tests on connected DRBD devices were done with 100baseT apart from the last one which was a separate Gigabit Ethernet cable connecting the two systems.

Conclusions

For the level of traffic that I’m using it seems that Gigabit Ethernet provides no performance benefit, the fact that it gave a slightly lower result is not relevant as the difference is within the margin of error.

Increasing the al-extents value helps with disconnected performance, a value of 1024 gives a 4% performance boost. I’m not sure that a value of 3389 is a good idea though.

The ext4 barriers are disabled by DRBD so a disconnected DRBD device gives performance that is closer to a barrier=0 mount than a regular ext4 mount. With the significant performance difference between connected and disconnected modes it seems possible that for some usage scenarios it could be useful to disable the DRBD secondary at times of peak load – it depends on whether DRBD is used as a really current backup or a strict mirror.

Future Tests

I plan to do some tests of DRBD over Linux software RAID-1 and tests to compare RAID-1 with and without bitmap support. I also plan to do some tests with the BTRFS filesystem, I know it’s not ready for production but it would still be nice to know what the performance is like.

But I won’t use the same systems, they don’t have enough CPU power. In my previous tests I established that a 1.5GHz P4 isn’t capable of driving the 20G IDE disk to it’s maximum capacity and I’m not sure that the 2.8GHz P4 is capable of running a RAID to it’s capacity. So I will use a dual-core 64bit system with a pair of SATA disks for future tests. The difference in performance between 20G IDE disks and 160G SATA disks should be a lot less than the performance difference between a 2.8GHz P4 and a dual-core 64bit CPU.

• [1] http://etbe.coker.com.au/2011/12/17/drbd-notes/
• [2] http://etbe.coker.com.au/2012/01/05/drbd-benchmarking/

Related posts:

1. DRBD Benchmarking I’ve got some performance problems with a mail server that’s...
2. Some Notes on DRBD DRBD is a system for replicating a block device across...
3. Ethernet bonding Bonding is one of the terms used to describe multiple...

As you might have noticed from my previous posts, we've moved phpMyAdmin code to github. Now I'm going to describe some things which might be useful for you if you are about do to similar switch.

While using git, moving to another hosting provider should be pretty straightforward. Just add another remote, push code there and it should be done. On the other side you probably have dozens of things in your infrastructure which you need to adjust. So the first thing to do is write down what all services are connected to your current git repositories. Let me name some which might be worth checking:

• continuous integration server
• snapshots generating
• demo server (in case you're providing something like we do)
• commit notifications
• various statistics services such as cia.vc
• website generating
• references in wiki, website and documetation

Once you think you have remembered all important things (the less important will probably show up anyway, but majority of things should work), you're ready to make the move.

I've decided to make the move in few steps. First push all data to new location, what can take some time. I'll get in more details to that later. In the same time I asked all contributors to give me their login information, so that I can allow them access to new repositories. Once all recently active developers were migrated, it was time to push all remaining commits to new git repositories and make the switch for real.

Pushing git repo to another location, should be pretty easy. On the other side if you have many branches, it get's slightly more complex, I've ended up with following shell snippet (pushing all branches present in origin to github remote):

git branch -r | grep origin/ | grep -v HEAD | sed 's@.*/@@' | while read b ; do git checkout $b ; git push github $b:$b ; done

Please ensure that you check output of this, because you may hit network problems somewhere in the middle and you end up with few branches than you expect. As the code is pretty much idempotent, you can safely run it several times until there is nothing to push. You should also push all tags to new location:

git push --tags github

Okay, we've all data on right place, so let's switch all our users to new location:

git remote set-url origin git@github.com:phpmyadmin/phpmyadmin.git # read/write git remote set-url origin git://github.com/phpmyadmin/phpmyadmin.git # read only

Of course everybody has to do this manually.

Next good thing is to let people know when they are using wrong repo (which will stay there for some time). Unfortunately there is AFAIK no way to warn them on pull, so let's warn at least on push:

$ cat > hooks/update #!/bin/sh echo "phpMyAdmin git repositories have moved to https://github.com/phpmyadmin" exit 1

I think this is pretty much all. You can find some more bits in our Git migration wiki page.

PS: Thanks to github for offering us hosting and sorry for breaking their branch displaying page by too many divergent branches.

Filed under: Coding English Phpmyadmin | 4 comments | Flattr this!

Recently I've discovered a subtle consequence of how the order in which dependencies are specified in debian actually matters. While re-factoring the code of dose3, I changed the order in which dependencies are considered by our sat solver (of edos-fame) . I witnessed a twofold performance loss just by randomizing how variables were presented to our sat solver. This highlights, on one hand, how our solver is strongly dependent on the structure of the problem and, on the other hand the standard practice of debian maintainers to assign an implicit priority in the disjunctive dependencies where the first is the most preferred packages (and maybe the most tested, at least dependency-wise).

The basic idea of distcheck is to encode the dependencies information contained in a Packages file in CNF format and then to feed them to a sat solver to find out if a package has broken dependencies or if its dependencies are such that no matter what, it would be impossible to install this package on a user machine.

Conflicts are encoded as binary clauses. So if package A conflicts with package B, I add a constraint they says "not (A and B)" , that is A and B cannot be considered together. The dependencies encoding associates to each disjunction of the depends field a clause that says "A implies B". If a package foo depends on A,B | C,D , I'll add "foo implies A and B" & "foo implies C and D" . This encoding is pretty standard and it is easy to understand.

The problem is how the sat solver will search for a solution to the problem "Is is possible to install package foo in an empty repository". The solver we use is very efficient and can easily deal with 100K packages or more. But in general is not very good at dealing with random CNF instances. The reason because edos-debcheck is so efficient lies in the way it exploits the structure of the sat problems.

The goal of a sat solver is to find a model (that is a variable assignment list) that is compatible with the given set of constraints. So if my encoding of the debian repository is a set of constraints R, the installability problem boils down to add an additional constraint to R imposing that the variable associated to the package foo must be true, and then ask the solver to find a model to make this possible. This installation, in sat terms, would be just an array of variables that must be true in order to satisfy the given set of constraints.

If you look at the logic problem as a truth table, the idea is to find a row in this table. This is the solution of your problem. Brute force of course is not an option and modern sat solvers use a number of strategies and heuristic to guide the search in the most intelligent way possible. Some of them try to learn from previous attempts, some of them, when they are lost try to pick a random variable to proceed.

If we consider problems that have a lot of structure, award winning sat solver do not back track very much. By exploiting the structure of the problem, their algorithm allows them to considerably narrow down the search only to those variables that are really important to find a solution.

All this long introduction was to talk about the solver that is currently used in edos-debcheck and distcheck (that is a rewrite of the edos-debcheck).

So why dependency order matters ? If we consider any package, even if the policy does not specify any order in the dependencies, it's common practice to write disjunctive dependencies specifying the most probable and tested alternative first and all other, more esoteric choices later. Moreover real packages are considered *before* virtual packages. Since every developer seems be doing the same, some kind of structure might be hidden in the order in which dependencies are specified.

Part of the efficiency of the the solver used in our tools is actually due to the fact that its search strategy is strongly dependent on the order in which literal are specified in each clause. Saying the package foo depends on A and B is "different" then saying it depends on B and A, even if semantically equivalent.

In my tests, I found about a twofold performance loss if the order of literals is either randomized or inverted. This is clearly a specific problem related to our solvers, while other solvers might not be susceptible to such small structural changes. Sat competitions often employs some form of obfuscation strategies of well known problems with well known structures in order to make useless to encode a search strategy that exploits the specific structure of a problem.

Since here we're not trying to win a sat competition, but to provide tool to solve a specific problem, we are of course very happy to exploit this structure.

This is the second entry in my series about squeeze release critical bug squashing. In response to my last blog post it was asked whether this is proper release critical bug squashing. Indeed there haven't been any patches or upload involved in this, only BTS handling, but this doesn't mean that these bugs weren't considered to be affecting squeeze. You can see this effort currently as weeding out the "wrong" bugs so that the list gets more useful and actually be able to ask maintainers to address the real issues.

You can at least see in this graph that the blue line is going down constantly since the year change instead of rising up like before. And I hope I will be able to keep it below the green line for a while still. Also thanks to the release-team and ftpmasters that it was possible to keep the massbugs about waf binary blob not being preferred source for modification out of affecting squeeze and ignore it for the current stable release—the required changes for those would rather be a fair bit intrusive for a stable update.

• 637213: doesn't fail in squeeze
• 648989: doesn't fail in squeeze
• 584568: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 624464: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632106: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632108: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632262: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632264: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632265: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 632266: most stuff (BTS, PTS, ...) use maintainer information from unstable only
• 595298: squeeze has linux 2.6.32
• 613732: doesn't fail in squeeze
• 616694: doesn't fail in squeeze
• 620264: doesn't fail in squeeze
• 620573: doesn't fail in squeeze
• 621440: libdb4.8 not obsolete in squeeze
• 624593: doesn't fail in squeeze
• 626868: doesn't fail in squeeze
• 628475: libdb4.8 not obsolete in squeeze
• 629838: boost1.46 not in squeeze
• 628475: boost1.46 not in squeeze
• 631623: doesn't fail in squeeze
• 639990: doesn't fail in squeeze
• 632763: no libevent 2.0 in squeeze
• 627000: not affecting squeeze
• 658216: doesn't fail in squeeze
• 654870: doesn't fail in squeeze
• 655115: this is explicitly about testing
• 657092: working in squeeze
• 655710: doesn't fail in squeeze
• 644691: no nautilus 3 in squeeze

I am glad that I managed to keep it up and even have a nice margin in case I can't put any effort into it some day but still have more bugs squashed than days there are in the year so far. Currently I am at 60 bugs in 39 days. This gives a warm feeling. :)

Enjoy!

/debian | permanent link | Comments: 3 | Flattr this

Something interesting I discovered about Python and --prefix that I can't see a lot of documentation on...

When you build Python you can use the standard --prefix flag to configure to home the installation as you require. You might expect that this would hard-code the location to look for the support libraries to the value you gave; however in reality it doesn't quite work like that.

Python will only look in the directory specified by prefix after it first searches relative to the path of the executing binary. Specifically, it looks at argv[0] and works through a few steps — is argv[0] a symlink? then dereference it. Does argv[0] have any slashes in it? if not, then search the $PATH for the binary. After this, it starts searching for dirname(argv[0])/lib/pythonX.Y/os.py, then dirname(argv[0])/../lib and so on, until it reaches the root. Only after these searches fail does the interpreter then fall back to the hard-coded path specified in the --prefix when configured.

What is the practical implications? It means you can move around a python installation tree and have it all "just work", which is nice. In my situation, I noticed this because we have a completely self-encapsulated build toolchain, but we wish to ship the same interpreter on the thing that we're building (during the build, we run the interpreter to create .pyc files for distribution, and we need to be sure that when we did this we didn't accidentally pick up any of the build hosts python; only the toolchain python).

The PYTHONHOME environment variable does override this behaviour; if it is set then the search stops there. Another interesting thing is that sys.prefix is therefore not the value passed in by --prefix during configure, but the value of the current dynamically determined prefix value.

If you run an strace, you can see this in operation.

readlink("/usr/bin/python", "python2.7", 4096) = 9 readlink("/usr/bin/python2.7", 0xbf8b014c, 4096) = -1 EINVAL (Invalid argument) stat64("/usr/bin/Modules/Setup", 0xbf8af0a0) = -1 ENOENT (No such file or directory) stat64("/usr/bin/lib/python2.7/os.py", 0xbf8af090) = -1 ENOENT (No such file or directory) stat64("/usr/bin/lib/python2.7/os.pyc", 0xbf8af090) = -1 ENOENT (No such file or directory) stat64("/usr/lib/python2.7/os.py", {st_mode=S_IFREG|0644, st_size=26300, ...}) = 0 stat64("/usr/bin/Modules/Setup", 0xbf8af0a0) = -1 ENOENT (No such file or directory) stat64("/usr/bin/lib/python2.7/lib-dynload", 0xbf8af0a0) = -1 ENOENT (No such file or directory) stat64("/usr/lib/python2.7/lib-dynload", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

Firstly it dereferences symlinks. Then it looks for Modules/Setup to see if it is running out of the build tree. Then it starts looking for os.py, walking its way upwards. One interesting thing that may either be a bug or a feature, I haven't decided, is that if you set the prefix to / then the interpreter will not go back to the root and then look in /lib. This is probably pretty obscure usage though!

All this is implemented in Modules/getpath.c which has a nice big comment at the top explaining the rules in detail.

FOSDEM 2012 was very nice, as expected.

This year, I decided to stop consuming only and getting more involved. I helped staff the token sale and held two Lightning Talks. This turned out to be an awesome experience.

During the beer event, we sold 4600 beer tokens (FOSDEM does not earn anything from this; all proceeds go to Cafe Delirium). The sale was staffed with one to three people at all times which made for a constant but mostly bearable workload. While the others made good-natured fun of me and my German efficiency, I like to think that I helped make things run a tad more smoothly. Along those lines, we will have a sign asking people for exact change and/or to buy stacks that can be paid in bills if possible, next year; that should help streamline the whole thing even more. All in all, it turned out to be quite stressful, but tons of fun. I am looking forward to chuck in again next year.

The two Lighting Talks (vcsh on Saturday and git-annex on Sunday) I held very rather well received. Feedback after the talks and online was very positive which is always nice. There's been a slight increase in help requests and generic questions, so I assume people are trying vcsh and git-annex as a consequence of those talks, which is even nicer.

Also, I met a lot of people. Most of the meetings were pre-arranged, some by chance. Carrying a name tag at all times, I was approached by several people who knew me online but didn't know my face. I will try to always carry a name tag on conferences now and I suggest others do the same.

Finally, a huge thank you to everyone who helped make FOSDEM happen; you rock!

If you ever get this cryptic error when loading an Android native app:

FATAL EXCEPTION: main java.lang.RuntimeException: Unable to start activity ComponentInfo{org.wikibooks.OpenGL/android.app.NativeActivity}: java.lang.IllegalArgumentException: Unable to load native library: /data/data/org.wikibooks.OpenGL/lib/libnative-activity.so at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1768) at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:1784) at android.app.ActivityThread.access$1500(ActivityThread.java:123) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:939) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:130) at android.app.ActivityThread.main(ActivityThread.java:3835) at java.lang.reflect.Method.invokeNative(Native Method) at java.lang.reflect.Method.invoke(Method.java:507) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:847) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:605) at dalvik.system.NativeStart.main(Native Method) Caused by: java.lang.IllegalArgumentException: Unable to load native library: /data/data/org.wikibooks.OpenGL/lib/libnative-activity.so at android.app.NativeActivity.onCreate(NativeActivity.java:199) at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1047) at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1722) ... 11 more

This may mean that Java couldn't find the ANativeActivity_onCreate function in your code, because it was stripped by the compiler.

If you use the native_app_glue NDK module, you may have noticed this strange code:

// Make sure glue isn't stripped. app_dummy();

Let's experiment what happens with and without this line:

Calling app_dummy:

$ arm-linux-androideabi-objdump -T libs/armeabi/libnative-activity.so | grep ANativeActivity_onCreate 000067fc g DF .text 000000f8 ANativeActivity_onCreate

Not calling app_dummy :

$ arm-linux-androideabi-objdump -T libs/armeabi/libnative-activity.so | grep ANativeActivity_onCreate $ # nothing

native_app_glue mainly defines Android callbacks. Since none of them are called directly by your code, the compiler strips the android_native_app_glue.o module entirely. If you use app_dummy however, it embeds it. Fortunately the compiler cannot strip the module on a per-function basis

That's why you need to call app_dummy when using the native_app_glue NDK module.

This looks like a ugly work-around though - isn't there a cleaner way?

Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this.

Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don’t want to know about what’s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path.

So I created a little tool called fatrace (“file access trace”, not “fat race” ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace rsyslogd(967): W /var/log/auth.log notify-osd(2264): O /usr/share/pixmaps/weechat.xpm compiz(2001): R device 8:2 inode 658203 [...]

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its’ not possible to determine a path (usually because it’s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace.

If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p `pidof gnome-terminal`).

So to investigate which programs are keeping your disk spinning, run something like

$ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes.

My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now.

The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.