Last week I discovered The Fan Club’s Experiments page. It reminds me of the Debian Experiments community on Google+. I like the concept of trying out all kinds of different things and reporting back to the community on how it works. I’ve been meaning to do more of that myself so this is me jumping in and reporting back on something I’ve been poking at this weekend.Introducing Squashfs
Squashfs is a read-only compressed filesystem commonly used on embedded devices, Linux installation media and remote file systems (as is done in LTSP). Typically, a system like tmpfs, unionfs or aufs is mounted over this read-only system to make it usable as a root filesystem. It has plenty of other use cases too but for the purposes of this entry we’ll stick with those use cases in mind. It supports gzip, lzo and xz(lzma) as compression back-ends. It also supports block sizes from 4K up to 1M.
Compression technique as well as block size can have major effects on both performance and file size. In most cases the defaults will probably be sufficient, but if you want to find a good balance between performance and space saving, then you’ll need some more insight.My Experiment: Test effects of squashfs compression methods and block sizes
I’m not the first person to have done some tests on squashfs performance and reported on it. Bernhard Wiedemann and the Squashfs LZMA project have posted some results before, and while very useful I want more information (especially compress/uncompress times). I was surprised to not find a more complete table elsewhere either. Even if such a table existed, I probably wouldn’t be satisfied with it. Each squashfs is different and it makes a big difference whether it contains already compressed information or largely uncompressed information like clear text. I’d rather be able to gather compression ratio/times for a specific image rather than for one that was used for testing purposes once-off.
So, I put together a quick script that takes a squashfs image, extracts it to tmpfs, and then re-compressing it using it all the specified compression techniques and block sizes… and then uncompressing those same images for their read speeds.My Testing Environment
For this post, I will try out my script on the Ubuntu Desktop 14.04.2 LTS squashfs image. It’s a complex image that contains a large mix of different kinds of files. I’m extracting it to RAM since I want to avoid having disk performance as a significant factor. I’m compressing the data back to SSD and extracting from there for read speed tests. The SSD seems fast enough not to have any significant effect on the tests. If you have a slow storage, the results of the larger images (with smaller block sizes) may be skewed unfavourably.
As Bernhard mentioned on his post, testing the speed of your memory can also be useful, especially when testing on different kinds of systems and comparing the results:# dd if=/dev/zero of=/dev/null bs=1M count=100000 104857600000 bytes (105 GB) copied, 4.90059 s, 21.4 GB/s
CPU is likely to be your biggest bottleneck by far when compressing. mksquashfs is SMP aware and will use all available cores by default. I’m testing this on a dual core Core i7 laptop with hyper-threading (so squashfs will use 4 threads) and with 16GB RAM apparently transferring around 21GB/s. The results of the squashfs testing script will differ greatly based on the CPU cores, core speed, memory speed and storage speed of the computer you’re running it on, so it shouldn’t come as a surprise if you get different results than I did. If you don’t have any significant bottleneck (like slow disks, slow CPU, running out of RAM, etc) then your results should more or less correspond in scale to mine for the same image.How to Run It
Create a directory and place the filesystem you’d like to test as filesystem.squashfs, then:$ apt-get install squashfs-tools $ wget https://raw.githubusercontent.com/highvoltage/squashfs-experiments/master/test-mksquashfs.sh $ bash test-mksquashfs.sh
With the default values in that file, you’ll end up with 18 squashfs images taking up about 18GB of disk space. I keep all the results for inspection, but I’ll probably adapt/fix the script to be more friendly to disk space usage some time.
You should see output that look something like this, with all the resulting data in the ./results directory.* Setting up... - Testing gzip * Running a squashfs using compression gzip, blocksize 4096 * Running a squashfs using compression gzip, blocksize 8192 * Running a squashfs using compression gzip, blocksize 16384 ... - Testing lzo * Running a squashfs using compression lzo, blocksize 4096 * Running a squashfs using compression lzo, blocksize 8192 * Running a squashfs using compression lzo, blocksize 16384 ... - Testing xz * Running a squashfs using compression xz, blocksize 4096 * Running a squashfs using compression xz, blocksize 8192 * Running a squashfs using compression xz, blocksize 16384 ... * Testing uncompressing times... * Reading results/squashfs-gzip-131072.squashfs... * Reading results/squashfs-gzip-16384.squashfs... * Reading results/squashfs-gzip-32768.squashfs... ... * Cleaning up... On to the Results
The report script will output the results into CSV.
Here’s the table with my results. Ratio is percentage of the size of the original uncompressed data, CTIME and UTIME is compression time and uncompress time for the entire image.Filename Size Ratio CTIME UTIME squashfs-gzip-4096.squashfs 1137016 39.66% 0m46.167s 0m37.220s squashfs-gzip-8192.squashfs 1079596 37.67% 0m53.155s 0m35.508s squashfs-gzip-16384.squashfs 1039076 36.27% 1m9.558s 0m26.988s squashfs-gzip-32768.squashfs 1008268 35.20% 1m30.056s 0m30.599s squashfs-gzip-65536.squashfs 987024 34.46% 1m51.281s 0m35.223s squashfs-gzip-131072.squashfs 975708 34.07% 1m59.663s 0m22.878s squashfs-gzip-262144.squashfs 970280 33.88% 2m13.246s 0m23.321s squashfs-gzip-524288.squashfs 967704 33.79% 2m11.515s 0m24.865s squashfs-gzip-1048576.squashfs 966580 33.75% 2m14.558s 0m28.029s squashfs-lzo-4096.squashfs 1286776 44.88% 1m36.025s 0m22.179s squashfs-lzo-8192.squashfs 1221920 42.64% 1m49.862s 0m21.690s squashfs-lzo-16384.squashfs 1170636 40.86% 2m5.008s 0m20.831s squashfs-lzo-32768.squashfs 1127432 39.36% 2m23.616s 0m20.597s squashfs-lzo-65536.squashfs 1092788 38.15% 2m48.817s 0m21.164s squashfs-lzo-131072.squashfs 1072208 37.43% 3m4.990s 0m20.563s squashfs-lzo-262144.squashfs 1062544 37.10% 3m26.816s 0m15.708s squashfs-lzo-524288.squashfs 1057780 36.93% 3m32.189s 0m16.166s squashfs-lzo-1048576.squashfs 1055532 36.85% 3m42.566s 0m17.507s squashfs-xz-4096.squashfs 1094880 38.19% 5m28.104s 2m21.373s squashfs-xz-8192.squashfs 1002876 34.99% 5m15.148s 2m1.780s squashfs-xz-16384.squashfs 937748 32.73% 5m11.683s 1m47.878s squashfs-xz-32768.squashfs 888908 31.03% 5m17.207s 1m43.399s squashfs-xz-65536.squashfs 852048 29.75% 5m27.819s 1m38.211s squashfs-xz-131072.squashfs 823216 28.74% 5m42.993s 1m29.708s squashfs-xz-262144.squashfs 799336 27.91% 6m30.575s 1m16.502s squashfs-xz-524288.squashfs 778140 27.17% 6m58.455s 1m20.234s squashfs-xz-1048576.squashfs 759244 26.51% 7m19.205s 1m28.721s
- Even though images with larger block sizes uncompress faster as a whole, they may introduce more latency on live media since a whole block will need to be uncompressed even if you’re just reading just 1 byte from a file.
- Ubuntu uses gzip with a block size of 131072 bytes on it’s official images. If you’re doing a custom spin, you can get improved performance on live media by using a 16384 block size with a sacrifice of around 3% more squashfs image space.
- I didn’t experiment with Xdict-size (dictionary size) for xz compression yet, might be worth while sacrificing some memory for better performance / compression ratio.
- I also want stats for random byte reads on a squashfs, and typical per-block decompression for compressed and uncompressed files. That will give better insights on what might work best on embedded devices, live environments and netboot images (the above table is more useful for large complete reads, which is useful for installer images but not much else), but that will have to wait for another day.
- In the meantime, please experiment on your own images and feel free to submit patches.
This makes it very useful if one needs to network boot a system since you can have the TFTP and DHCP part of the setup done easily, and only add NFS for a complete network boot.
Add to that that
One extra nice thing dnsmasq has is that it can mark specific hosts, addresses or ranges with some internal markers, then use those markers as symbolic names to apply settings based for classes of devices.
In the configuration snippet below, there is a rule I set up to make sure I would apply the 'netbsd' label to any system connecting through specific ethernet interfaces (one is the interface of the system, the other is a USB NIC I use from time to time):
#You will need a range for static IPs in the main file
# give the name 'kinder' to any machine connecting through the given ethernet nics and apply 'netbsd' label
dhcp-host=00:1a:70:99:60:BB,00:06:4F:0D:B1:95, kinder, 192.168.77.251, set:netbsd
# Machines tagged 'netbsd' shall use the given NFS root path
# Enable dnsmasq's built-in TFTP server
# Set the root directory for files available via FTP.
tftp-root=/srv/tftpSaving this configuration file in /etc/dnsmasq.d/kinder-netboot will enable this to be used by dnsmasq if this line is present in /etc/dnsmasq.conf
conf-dir=/etc/dnsmasq.dCommenting it will disable the netbsd part easily.
Sometimes you happen to realize how old you are when listening to the radio where they announce hits from your days of youth as "oldies". Or when todays youth asking you what a music casette or a 5.25" floppy disk is. You're even older than that when you still know 8" disks. We have one of those on our pin board.
Or you may realize your age when your favorite computer of your youth will celebrate its 30th anniversary this year!
In 1985 the Amiga was introduced to the public - and 30 years later Amigas around the world are still running! Some are still operated under AmigaOS, some are running maybe NetBSD, but some are running the Debian m68k port - even though the port was expelled from Debian years ago! But some people still take care of this port and it is keeping up with building packages, although mostly due to emulators doing the most work.
But anyway, the Amiga is turning 30 this year! And what a great machine this was! It brought multitasking, colorful graphics and multi-channel audio to the masses. It was years ahead of its competitors, but it was doomed to fail because of management failures by the producing company Commodore, which went bankrupt in 1994.
The "30th Anniversary Event" will take place on July 25th at the Computer History Museum in Mountain View, California, USA - if the kickstarting the event will be successful during the next two weeks. So, when you earned your first merits in computing on an Amiga as well, you might want to give your Amiga history another kickstart! Not the Kickstart floppy as the A1000 needed to boot up, but fundraising the event on kickstart.org!
I think this event is not only important to old Amiga users from the good old days, but for the rememberance of computing history in general. Without doubt the Amiga set new standards in computer history and is an important part of industrial heritage.Kategorie: DebianTags: Debian68kAmiga
In my opinion, creating and maintaining Debian copyright file is the most boring task required to create a Debian package. Unfortunately, this file is also one of the most important of a package: it specifies some legal aspect regarding the use of the software.
Debian copyright file is scrutinized by ftp masters gatekeepers when accepting a new package in Debian project: this file must accurately describe the copyright and licenses of all files of a source package, preferably using a specific syntax. (Kudos to the ftp-masters team: reading copyright files must be even more boring than writing them).
The content of the copyright file must reflect accurately the license of all files. This license is often specified in the comments of a source files. The licencecheck command is able to scan sources files and reports the copyright and licenses declared in there. But it does not summarize this information: a copyright line is generated for each file of a package.
licensecheck2dep5 (provided by cdbs package as /usr/lib/cdbs/licensecheck2dep5) does better: the output of licensecheck is consolidated and translated in Debian copyright format. The result is better, but must be heavily edited to be reviewable by ftp-masters team.
The new update subcommand of cme (available with libconfig-model-dpkg-perl 2.061 currently in experimental) goes further than licensecheck2deb:
- copyright are coalesced when possible (i.e. 2001,2002,2003-2005 is changed to 2001-2005)
- file entries same copyright owner and license are grouped, group of files may be represented with a wild card (‘*’)
- license text is filled with actual text for the most popular licenses
For instance, here’s the (slightly edited) output of cme run for pan package starting without debian/copyright file:$ cme update dpkg-copyright -quiet Adding dummy license text for license public-domain for path pan/general/sorted-vector.h Adding dummy license text for license BSD-2-clause for path pan/usenet-utils/MersenneTwister.h $ cat debian/copyright Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Files: * Copyright: 1994-2001, Frank Pilhofer. The author may License: GPL-2+ Files: pan/* Copyright: 2002-2007, Charles Kerr License: GPL-2 Files: pan/data/cert-store.cc pan/data/cert-store.h Copyright: 2011, Heinrich Müller 2002-2006, Charles Kerr License: GPL-2 Files: pan/general/e-util.cc pan/general/e-util.h Copyright: 2000, 2001, Ximian, Inc License: LGPL-2 Files: pan/general/locking.h pan/general/worker-pool.cc pan/general/worker-pool.h Copyright: 2007, Calin Culianu 2002-2007, Charles Kerr License: LGPL-2+ Files: pan/general/sorted-vector.h Copyright: 2002, Martin Holzherr (email@example.com). License: public-domain Please fill license public-domain from header of pan/general/sorted-vector.h [ about 100 more lines including license text for Zlib and several GPL licenses ]
This is a good start, but some modifications must be applied to get a correct license file:
- add missing upstream information (Upstream-Name, Upstream-Contact and Source items)
- remove irrelevant text from some copyright owner (e.g. remove “The author may” from Files: * entry).
- add some missing license text (e.g. include text from sorted-vector.h comments to specify upstream author’s version of public-domain)
These modifications can be done:
- directly after update by running cme update dpkg-copyright --edit
- after update by running cme edit dpkg-copyright
- with your favorite editor
This post has mentioned creation of Debian copyright file, but does not address the problem of updating an existing copyright file when packaging a new version of a software. This will be the subject of a next post.
I hope this new feature of cme will save hours of work for Debian packagers. As usual comments and suggestions are welcome
All the best
Here are some of the animals we have seen so far during our trip to Australia.Kookaburra
It is easy to find koala bears in many of the trees around Kennett River:Python
At the Moonlit Sanctuary, we were introduced to a python:BBQ
My cousin Hagen prepares the BBQ for lunch at the sanctuary:More koala bears
It was also possible to get up close to one of the koala bears at the sanctuary. Be careful not to get so close to them in the wild.Wallaby
The wallaby is like a small kangaroo:Kangaroo
Carla was able to get up close to a family of kangaroos and feed them some snacks:Tasmanian Devil
Saving the best for last, the Tasmanian devil is not just a cartoon character. His jaw is strong enough to crush your finger as if you had been run over by a train and he can run faster than just about anybody except for a trained sprinter. Consequently, we didn't jump in the enclosure to pose with him for this photo:
Of things I didn't plan for this Easter: Shooting PR photos for Datarock up under the roof in Vikingskipet.
Also, I'm getting way too old for this “go to bed at 5am” crap.
During eastern I had some time to continue working on the Norwegian docbook version of the 2004 book Free Culture by Lawrence Lessig. At the moment I am proof reading the finished text, looking for typos, inconsistent wordings and sentences that do not flow as they should. I'm more than two thirds done with the text, and welcome others to check the text up to chapter 13. The current status is available on the github project pages. You can also check out the PDF, EPUB and HTML version available in the archive directory.
Please report typos, bugs and improvements to the github project if you find any.
A couple of days ago this appeared in my system logs
Mar 31 23:59:59 kernel: Clock: inserting leap second 23:59:60 UTC
my first reaction of course was "great! they gave us one second more of sleep! MY PRECIOUSSSS", but then I realized that yes, this year there was supposed to be a leap second, but it should have been in June, not in March.
Other people I know noticed the message, but nobody knew anything else about it, and duckduckgoing didn't find anything, so I'm asking the lazyweb: does anybody know what happened?
Update: it seems that this has been traced to a single layer1 ntp server.
Recently the NEW queue grew due to lots of uploads of new KDE software and several smaller node-packages. The KDE-stuff will be processed one after another, but the node-stuff seems to be rather strange. After the last discussion I was told that all those small packages can be accumulated into bigger chunks. I hope this discussion doesn’t need to be repeated again …
Anyway, this month I marked 117 packages for accept and rejected 51 packages.
This month I got assigned a workload of 15.25h and I spent these hours to upload new versions of:
- [DLA 163-1] bind9 security update
- [DLA 166-1] libarchive security update
- [DLA 167-1] redcloth security update
- [DLA 170-1] mod-gnutls security update
- [DLA 171-1] libssh2 security update
- [DLA 181-1] xerces-c security update
- [DLA 182-1] batik security update
- [DLA 183-1] libxfont security update
- [DLA 184-1] binutils security update
Finally I was also able to upload the binutils package. Up to now, I got no complaints that something is not working anymore, so yeah, I seem to make it. The next big adventure will be a new upload of PHP. I already started with some patches, but it is still a good piece of work.
I also uploaded update for DLA 164-1] unace security update, [DLA 168-1] konversation security update and [DLA 172-1] libextlib-ruby security update although no LTS sponsor indicated any interest.
This month the severity of one bug in greylistd had been raised from normal to severe and such I had to upload a new version. Thanks to Andreas Beckmann for raising and for providing a patch.
I also uploaded a new version of dict-elements and closed a bug related to reproducible builds.
As I am the maintainer of libkeepalive, I got an email from Andreas Florath. He wanted to persuade me to create a package for his library libdontdie, which is rather similar to libkeepalive but has some improvements. As I promised to do some more packaging work, he didn’t have to argue much and voila, there now is a new package libdontdie available. As the cooperation with him is really pleasant, I also created a package for his other project: pipexec.
Thanks alot to all donors, this month I got 30€ in total. I really appreciate this and hope that everybody is pleased with my commitment. Don’t hesitate to make suggestions for improvements.
I use my Linode VPS as a VPN endpoint for my laptop when I'm using untrusted networks and I wanted to do the same on my Android 5 (Lollipop) phone.
It turns out that it's quite easy to do (doesn't require rooting your phone) and that it works very well.Install OpenVPN
From the easy-rsa directory you created while generating the server keys, create a new keypair for your phone:./build-key nexus6 # "nexus6" as Name, no password
and then copy the following files onto your phone:
If you configured your server as per my instructions, these are the settings you'll need to use on your phone:
- LZO Compression: YES
- Type: Certificates
- CA Certificate: ca.crt
- Client Certificate: nexus6.crt
- Client Certificate Key: nexus6.key
- Server address: hafnarfjordur.fmarier.org
- Port: 1194
- Protocol: UDP
- Custom Options: NO
- Expect TLS server certificate: YES
- Certificate hostname check: YES
- Remote certificate subject: server
- Use TLS Authentication: YES
- TLS Auth File: ta.key
- TLS Direction: 1
- Encryption cipher: AES-256-CBC
- Packet authentication: SHA384 (not SHA-384)
That's it. Everything else should work with the defaults.
I find interesting when faced a taken-for-granted situation, in particular in the tech world. This time’s history is about an Internet ADSL connection that allows all traffic but SSH. Yes, you read it correctly. It was very bizarre confirming such event was a real-life situation.
I don’t claim to be a networking expert but at least I want to think I’m well educated. After few minutes I’ve focused my efforts on dealing with the ADSL router/modem’s networking configuration. The device is provided by Movistar (formerly Telefonica) and it runs OpenRG. I’ve discovered that other people have experienced the same issue and what Movistar did was basically replacing the device. Of course the problem is gone after that.
So, this post is dedicated to those who don’t give up. Following the steps below will allow SSH outbound traffic for a OpenRG-based device.OpenRG device specs Software Version: 22.214.171.124.110.1.52 Upgrade Release Date: Oct 7 2014 Diagnostic
When you do the command below, it shows nothing but timeout. Even when you SSH the router it doesn’t establish connection to it.ssh -vv host.somewhere.com Solution Change router’s SSH service port.
This step will allow you to access the console-based configuration for the router (since I haven’t found any way to do the steps described below from the web management interface).
To do so, go to System > Management > SSH. Update the service port to something else than 22, for instance 2222.Connect to the SSH interface
Once you have changed the SSH service port, you can access it from a SSH client.ssh -p 2222 firstname.lastname@example.org email@example.com's password: OpenRG>
Once you have the console prompt, issue the following commands to allow SSH outbound traffic coming from the LAN and Wifi networks. After the last command, which saves and updates the device’s configuration, you should be able to do SSH from any computer in your network to the Internet (thanks to tips from inkhorn).OpenRG> conf set fw/policy/0/chain/fw_br0_in/rule/0/enabled 0 Returned 0 OpenRG> conf set fw/policy/0/chain/fw_br1_in/rule/0/enabled 0 Returned 0 OpenRG> conf reconf 1 Returned 0
- Getting Movistar Peru ZTE MF193 work in Debian GNU/Linux How to get Movistar Peru (Internet Móvil) ZTE 3G MF193...
- Subversion auth using SSH external key-pair Usually, when using Subversion’s SSH authentication facility, Subversion’s client will...
- s3tools – Simple Storage Service for non-Amazon providers Using s3tools to interact with a S3-compatible storage service for...
This was my fourth month working on Debian LTS. I was assigned 14.5 hours by Freexian's Debian LTS initiative, but I only worked 11.5 as I had a week's holiday and then was ill for part of this week.eglibc
My first task was to complete the eglibc update that I began at the end of February. There were a few unexpected failures in the regression test suite, but on inspection these turned out to be quirks of the build environment:
- A test of the nice(2) function failed because I ran the build itself with nice(1). I should probably use a cgroup to reduce its priority instead; this time I re-ran without using either.
- Several tests of cancellation failed because they rely on a socket write blocking due to filling up the buffer, while the kernel's minimum socket buffer size has increased such that the write isn't big enough to block. (The host kernel was Linux 3.16 from jessie.) I found and applied a fix for this from the (glibc) upstream repository.
With those changes, the regression test results are the same as for previous package versions. (There is still one 'unexpected' failure, but I didn't investigate because it is not a regression within squeeze.)
I reviewed the backported patches again to check as well as I could that they did not depend on other upstream changes, and they did not seem to. I also received positive feedback from further testing - I can't find the message now, but I think it was that the Univention application test suite passed with the updated eglibc package installed.
With that confirmation and no regressions reported, I uploaded eglibc 2.11.3-4+deb6u5 and issued DLA 165-1.freetype
A large number of vulnerabilities in font file parsing in Freetype were reported by Mateusz Jurczyk. I think these were less critical in squeeze than in wheezy, because while current web browsers use Freetype to render untrusted 'web fonts' we don't support any web browsers in squeeze LTS. But it seemed like they ought to be fixed anyway, and it was not too hard to backport the patches included in the wheezy security update.
Freetype doesn't have a regression test suite and I didn't have samples of the broken fonts, so I came up with some ad hoc tests for regressions. I viewed several of each of the affected font formats with the ftview demo application. I then wrote a script to match up the dependencies of a package (in this case, libfreetype6) with those used by Freexian's customers; the top results were fontconfig, xfonts-utils and imagemagick. So I ran some basic tests against each of the affected formats with each of these (fc-list, mkfontscale, and the simple text label recipe for ImageMagick) and found no regression. Uploaded freetype 2.4.2-2.1+squeeze5 and issued DLA 185-1.
- ¾ cups soylent
- 1 ½ cups rolled oats
- ½ cup sugar (white & dark brown)
- ¼ cup flour
- ¾ cup raisins
- ½ tsp baking soda & powder
- ½ tsp salt
- 1 stick butter (roomtemp - NOT melted. Don’t even try that. Stop. You. I see you.)
- 1 egg
- 1 tsp vanilla
As usual, I look at the application stats for phpMyAdmin just after student application period of Google Summer of Code is over.
First of all we got way more proposals than in last years, but also number of bogus proposals went up (you can see them as ignored in the chart).
Same as in past years, people leave the submission to the last moment, even though we encourage them to submit early so that they can adjust the application based on our feedback. But still we got more than half of the proposals in last three days.
Anyway we're just working on evaluation and will finalize it in upcoming days. Of course you will know the results from Google on April 27th.
So three weeks ago it occurred to me that it would be useful to test the reproducibility of Jessie too (until then we only tested unstable and experimental), as this will give us a nice data point to compare future developments against and also because we wanted to test "testing" in future anyway, so we could as well start while "testing" is still jessie... (because then we will have to test less once stretch development has started, as in the beginning stretch will be jessie anyway.)
Quite quickly I then realized that building all packages twice on the current, quite heavily loaded jenkins.debian.net machine, would take 4-6 weeks and that it might well happen that we'll release Jessie before this has finished. (And I do want a Jessie release rather sooner than later!)
So I've asked Profitbricks, who have been sponsoring the jenkins.debian.net setup since October 2012, for some more resources temporarily, and once again they quickly helped out and the next day I could add eight cores and 25 GB of RAM to the existing VM, for a total of 23 Cores and 75 GB RAM.
So currently jenkins.d.n runs eight reproducible build jobs simultaneously, each of them is pbuilding packages in tmpfs in RAM. Thus these builds are really fast: for small packages without additional build depends build times as low as 40 seconds can be seen, which doesn't sound impressive at first, but this includes the source download and building twice and it includes untar'ing the base.tgz twice as well as running debbindiff on the resulting binary packages. (And all this is while quite many other jobs continue to hammer the machine as well.)
(BTW: I doubt using LVM snapshots is faster then running this in tmpfs, but if you think LVM is faster I'd be curious to see some numbers.)
Interestingly this rebuild also discovered a rather unexpected result as we found an RC bug in jessie, which caused build failures for >20 packages: "#781076 cdbs: perlmodule-vars.mk LDDLFLAGS quoting problem".
And then there were some unexpected but really expectable results: testing turned out to "only" give reproducible results for 79.4% of all packages in main, while unstable has 82.6% reproducible packages. This was unexpected, as our (few) fixed packages in our repo are also used for testing testing and as testing is smaller than sid, I at first believed the reproducible percentage should be higher, as testing has 1000 packages less.
So why was the lower percentage to be expected? This is due to two factors: since FOSDEM we have introduced a number of new variations for the second build, that is we now build with a.) a different domainname, b.) with a different umask, c.) a different timezone and d.) a different kernel version (using 'linux64 --uname-2.6'). And secondly, most packages in sid haven't been rebuild since FOSDEM. Thus we'll now reschedule all packages in testing which are unreproducible in unstable, which should decrease the current reproducibility of sid a bit...
For those wondering about the 622 packages failing to build in testing: at least 453 are due to our test setup, categorized in (currently) three issues: timestamps_from_cpp_macros, ftbfs_werror_equals and ocaml_configure_not_as_root. The ones in this third issue category are rather trivial to fix, so this leaves 622-453=169 minus those fixed by #781076, so roughly 150 packeges which fail to build in our setup which need to be investigated...
And for those wondering about other missing variations, there is at least one big one missing: changing the build date. Current guestimate is that this will make 1-2000 packages unreprdoucible again but we'll only know for sure once we tested them.
So I would like to once again thank Profitbricks for supporting jenkins.d.n in the last 2.5 years and making reproducible.d.n possible so smoothly. Being able to painlessly add more resources when we needed them was incredible useful and I hope we can count on their support in the future. (I have some more ideas how to burn resources usefully in future. Stay tuned and btw, if you know how to put more hours in the day, please do tell me.
And, of course: thanks for your work on Debian, no matter whether you've been working on Jessie, reproducibility or something totally different!
To finish this post I'd like to remind everyone that currently all this is just about the prospects of reproducible builds for Debian. Debian is not 80% reproducible yet - but it easily could be! And I certainly hope it will be "soon", and hopefully "soon" will only mean a few months. We will rebuild and see.
The SFLC is hiring: idealist job posting
This is not actually an April 1 joke.
This seems as good a day as any to mention that I am a founding member of ArchiveTeam.
Way back, when Geocities was closing down, I was one of a small rag-tag group who saved a copy of most of it. That snapshot has since generated more publicity than most other projects I've worked on. I've heard many heartwarning stories of it being the only remaining copy of baby pictures and writings of deceased friends, and so on. It's even been the subject of serious academic study as outlined in this talk, which is pretty awesome.
I'm happy to let this guy be the public face of ArchiveTeam in internet meme-land. It's a 0.1% project for me, and has grown into a well-oiled machine, albeit one that shouldn't need to exist. I only get involved these days when there's another crazy internet silo fire drill and/or I'm bored.
(Rumors of me being the hand model for ArchiveTeam are, however, unsubstantiated.)
Today, Aaron Shaw and I are pleased to announce a new startup. The startup is based around an app we are building called RomancR that will bring the sharing economy directly into your bedrooms and romantic lives.
When launched, RomancR will bring the kind of market-driven convenience and efficiency that Uber has brought to ride sharing, and that AirBnB has brought to room sharing, directly into the most frustrating and inefficient domain of our personal lives. RomancR is Uber for romance and sex.
Here’s how it will work:
- Users will view profiles of nearby RomancR users that match any number of user-specified criteria for romantic matches (e.g., sexual orientation, gender, age, etc).
- When a user finds a nearby match who they are interested in meeting, they can send a request to meet in person. If they choose, users initiating these requests can attach an optional monetary donation to their request.
- When a user receives a request, they can accept or reject the request with a simple swipe to the left or right. Of course, they can take the donation offer into account when making this decision or “counter-offer” with a request for a higher donation. Larger donations will increase the likelihood of an affirmative answer.
- If a user agrees to meet in person, and if the couple then subsequently spends the night together — RomancR will measure this automatically by ensuring that the geolocation of both users’ phones match the same physical space for at least 8 hours — the donation will be transferred from the requester to the user who responded affirmatively.
- Users will be able to rate each other in ways that are similar to other sharing economy platforms.
Of course, there are many existing applications like Tinder and Grindr that help facilitate romance, dating, and hookups. Unfortunately, each of these still relies on old-fashion “intrinsic” ways of motivating people to participate in romantic endeavors. The sharing economy has shown us that systems that rely on these non-monetary motivations are ineffective and limiting! For example, many altruistic and socially-driven ride-sharing systems existed on platforms like Craigslist or Ridejoy before Uber. Similarly, volunteer-based communities like Couchsurfing and Hospitality Club existed for many years before AirBnB. None of those older systems took off in the way that their sharing economy counterparts were able to!
The reason that Uber and AirBnB exploded where previous efforts stalled is that this new generation of sharing economy startups brings the power of markets to bear on the problems they are trying to solve. Money both encourages more people to participate in providing a service and also makes it socially easier for people to take that service up without feeling like they are socially “in debt” to the person providing the service for free. The result has been more reliable and effective systems for proving rides and rooms! The reason that the sharing economy works, fundamentally, is that it has nothing to do with sharing at all! Systems that rely on people’s social desire to share without money — projects like Couchsurfing — are relics of the previous century.
RomancR, which we plan to launch later this year, will bring the power and efficiency of markets to our romantic lives. You will leave your pitiful dating life where it belongs in the dustbin of history! Go beyond antiquated non-market systems for finding lovers. Why should we rely on people’s fickle sense of taste and attractiveness, their complicated ideas of interpersonal compatibility, or their sense of altruism, when we can rely on the power of prices? With RomancR, we won’t have to!
Note: Thanks to Yochai Benkler whose example of how leaving a $100 bill on the bedside table of a person with whom you spent the night can change the nature of the a romantic interaction inspired the idea for this startup.
The event has yet to appear in the MiniDebconf agenda. Rumor has it that the hurds of fans that tend to attend his events pose a logistics and safety challenge, reason why the event might only appear in the agenda a couple of days before the event.
Lyon being such an easy place to get by train, flight, car, and even by boat, it is understandable and should be expected that a great number of people attend the minidebconf only for his very session.
If you have not yet registered, I'd recommend you to do so now and to book your travel and accommodation ASAP, before everything is overpriced due to high demand.
See you in ten days!
P.S. kudos to the organizers!