Planet Debian

Syndicate content
Planet Debian - http://planet.debian.org/
Updated: 22 min 13 sec ago

Zlatan Todorić: Home, is that you?

Tue, 2015-04-28 09:45

Today I found out (by mail from my German friend) that there is a place called Longo Mai. Still need to explore it but it made my day currently.

Categories: FLOSS Project Planets

Craig Small: Backporting and git-buildpackage

Tue, 2015-04-28 07:49

For working with Debian packages, one method of maintaining them is to put them in git and use git-buildpackage to build them right out of the git repository.  There are a few pitfalls with it, notably around if you forget to import the upstream you get this strange treeish related error which still throws me at first when I see it.

Part of maintaining packages is to be able to fix security bugs in older versions of them that are found in stable and even sometimes old stable (jessie and wheezy respectively at the time of writing).  At first I used to do this outside git because to me there wasn’t a clear way of doing it within it.  This is not too satisfactory because it means you lose the benefits of using git in the first place, and for distributions you are more likely to need collaboration with, such as working with the security team or help with backporting.

So because I don’t think the details are easily found and also I’ll probably lose them again and need a central spot to find what I did last time.

Building the environment

The first step for a new distribution is to create the building environment.  You really only need to do this once when there is a new distribution with then possibly some updates from time to time. The command will create the environment in /var/cache/pbuilder/base-DIST.cow/

DIST=wheezy git-pbuilder create

You will find all the files in /var/cache/pbuilder/base-wheezy.cow/ which is then ready to do be used.

For more information about this useful tool, look at the git-builder entry on the Debian wiki.

Creating the branch

The master branch is generally where sid will located. For the other distributions, I use branches. For the initial setup you will need to create the branch and start it off from the right point.  For example, the last non-security update for wordpress in jessie was version 4.1+dfsg-1.  We then want the jessie branch to start from this point.

git branch jessie debian/4.1+dfsg-1 git checkout jessie

This will then create and put you on the jessie branch. You only need the first line once. You can switch between sid (master branch) and jessie (jessie branch).

At this point you have a working place to make all the updates you need to do. Nothing is terribly different from the usual workflow.

Building the package

Make sure you have checked in all your changes and now it is time to build your package!

git-buildpackage --git-pbuilder --git-dist=jessie --git-debian-branch=jessie

You may need two additional flags:

  • -sa Use this for the first security update so there is also the source included on the security servers.
  • –git-tag Once you are sure this is the latest changes for this version this will tag in git with the debian tag, such as debian/4.1+dfsg-1+deb8u1

 

Categories: FLOSS Project Planets

Thomas Goirand: @Erich Schubert: why not trying to package Hadoop in Debian?

Tue, 2015-04-28 07:30

Erich,

 

As a follow-up on your blog post, where you complain about the state of Hadoop. First, I couldn’t agree more with all you wrote. All of it! But why not trying to get Hadoop in Debian, rather than only complaining about the state of things?

 

I have recently packaged and uploaded Sahara, which is OpenStack big data as a service (in other words: running Hadoop as a service on an OpenStack cloud). Its working well, though it was a bit frustrating to discover exactly what you complained about: the operating system cloud image needed to run within Sahara can only be downloaded as a pre-built image, which is impossible to check. It would have been so much work to package Hadoop that I just gave up (and frankly, packaging all of OpenStack in Debian is enough work for a single person doing the job… so no, I don’t have time to do it myself).

OpenStack Sahara already provides the reproducible deployment system which you seem to wish. We “only” need Hadoop itself.

Categories: FLOSS Project Planets

Michael Prokop: GLT15: Slides of my “Debian 8 aka jessie, what’s new” talk

Tue, 2015-04-28 05:55

I wasn’t sure whether I would make it to Linuxdays Graz (GLT15) this year so I didn’t participate in its call for lectures. But when meeting folks on the evening before the main event I came up with the idea of giving a lightning talk as special kind of celebrating the Debian jessie release.

So I gave a lightning talk about "Debian 8 aka jessie, what’s new" on 25th of April (couldn’t have been a better date :)) and the slides of my talk turned up to be more useful than expected (>3000 downloads within the first 48 hours and I received lots of great feedback), so maybe it’s worth mentioning them here as well: "Debian 8 aka jessie, what’s new" (PDF, 450KB)

PS: please join the #newinjessie game, also see #newinjessie on twitter

Categories: FLOSS Project Planets

Raphael Geissert: Updates to the Debian sources editor

Tue, 2015-04-28 01:00
Since the announcement of the chrome/chromium and firefox/iceweasel extensions that add in-browser editing of Debian sources there have been a few changes:

  • Update to Ace 1.1.9, which brings quite some fixes
  • Downloads with proper file names! this should work with recent browsers. E.g for a patch it will propose package-version_path_file.patch
  • Changing the order of additions and removals in the generated patches (it was strange to see additions first)
  • Improvements to the compatibility with debsources' URL parameters
  • Allow the extension to be enabled in private browsing mode (firefox/iceweasel)
  • Loading of the extension when browsing debsources via https
  • Internal extension packaging cleanup, including the generation of a versioned copy of the web (remote) files, so that users of version 0.0.x do use the remote code version 0.0.x

If your browser performs automatic updates of the extensions (the default), you should soon be upgraded to version 0.0.10 or later, bringing all those changes to your browser.

Want to see more? multi-file editing? emacs and vim editing modes? in-browser storage of the modified files? that and more can be done, so feel free to join me and contribute to the Debian sources editor!
Categories: FLOSS Project Planets

Gunnar Wolf: Bestest birthday ever

Mon, 2015-04-27 22:24

That's all I need to enjoy the best best party ever.

Oh! Shall I mention that we got a beautiful present for the kids from our very dear DebConf official Laminatrix! Photos not yet available, but will provide soon.

Categories: FLOSS Project Planets

Matthew Garrett: Reducing power consumption on Haswell and Broadwell systems

Mon, 2015-04-27 13:33
Edit to add: These patches on their own won't enable this functionality, they just give us a better set of options. Once they're merged we can look at changing the defaults so people get the benefit of this out of the box.

Haswell and Broadwell (Intel's previous and current generations of x86) both introduced a range of new power saving states that promised significant improvements in battery life. Unfortunately, the typical experience on Linux was an increase in power consumption. The reasons why are kind of complicated and distinctly unfortunate, and I'm at something of a loss as to why none of the companies who get paid to care about this kind of thing seemed to actually be caring until I got a Broadwell and looked unhappy, but here we are so let's make things better.

Recent Intel mobile parts have the Platform Controller Hub (Intel's term for the Southbridge, the chipset component responsible for most system i/o like SATA and USB) integrated onto the same package as the CPU. This makes it easier to implement aggressive power saving - the CPU package already has a bunch of hardware for turning various clock and power domains on and off, and these can be shared between the CPU, the GPU and the PCH. But that also introduces additional constraints, since if any component within a power management domain is active then the entire domain has to be enabled. We've pretty much been ignoring that.

The tldr is that Haswell and Broadwell are only able to get into deeper package power saving states if several different components are in their own power saving states. If the CPU is active, you'll stay in a higher-power state. If the GPU is active, you'll stay in a higher-power state. And if the PCH is active, you'll stay in a higher-power state. The last one is the killer here. Having a SATA link in a full-power state is sufficient to keep the PCH active, and that constrains the deepest package power savings state you can enter.

SATA power management on Linux is in a kind of odd state. We support it, but we don't enable it by default. In fact, right now we even remove any existing SATA power management configuration that the firmware has initialised. Distributions don't enable it by default because there are horror stories about some combinations of disk and controller and power management configuration resulting in corruption and data loss and apparently nobody had time to investigate the problem.

I did some digging and it turns out that our approach isn't entirely inconsistent with the industry. The default behaviour on Windows is pretty much the same as ours. But vendors don't tend to ship with the Windows AHCI driver, they replace it with the Intel Rapid Storage Technology driver - and it turns out that that has a default-on policy. But to make things even more awkwad, the policy implemented by Intel doesn't match any of the policies that Linux provides.

In an attempt to address this, I've written some patches. The aim here is to provide two new policies. The first simply inherits whichever configuration the firmware has provided, on the assumption that the system vendor probably didn't configure their system to corrupt data out of the box[1]. The second implements the policy that Intel use in IRST. With luck we'll be able to use the firmware settings by default and switch to the IRST settings on Intel mobile devices.

This change alone drops my idle power consumption from around 8.5W to about 5W. One reason we'd pretty much ignored this in the past was that SATA power management simply wasn't that big a win. Even at its most aggressive, we'd struggle to see 0.5W of saving. But on these new parts, the SATA link state is the difference between going to PC2 and going to PC7, and the difference between those states is a large part of the CPU package being powered up.

But this isn't the full story. There's still work to be done on other components, especially the GPU. Keeping the link between the GPU and an internal display panel active is both a power suck and requires additional chipset components to be powered up. Embedded Displayport 1.3 introduced a new feature called Panel Self-Refresh that permits the GPU and the screen to negotiate dropping the link, leaving it up to the screen to maintain its contents. There's patches to enable this on Intel systems, but it's still not turned on by default. Doing so increases the amount of time spent in PC7 and brings corresponding improvements to battery life.

This trend is likely to continue. As systems become more integrated we're going to have to pay more attention to the interdependencies in order to obtain the best possible power consumption, and that means that distribution vendors are going to have to spend some time figuring out what these dependencies are and what the appropriate default policy is for their users. Intel's done the work to add kernel support for most of these features, but they're not the ones shipping it to end-users. Let's figure out how to make this right out of the box.

[1] This is not necessarily a good assumption, but hey, let's see

comments
Categories: FLOSS Project Planets

Scott Kitterman: Enabling DNSSEC Support For OpenDKIM

Mon, 2015-04-27 09:43

If you are using DNSSEC you can now use it to verify DKIM keys with opendkim.

This does require a bit of configuration.

Opendkim uses unbound for DNSSEC support.

You have to:

  • Install the unbound package (not just the library, which is already pulled in as an opendkim dependency)
  • Configure the DNSSEC trust anchor for unbound ( either in /etc/unbound/unbound.conf or by adding a configuration snippet to /etc/unbound/unbound.conf.d – the latter makes it much less likely you’ll have to resolve conflicts in the configuration file if the default file is changed on later package upgrades)
  • Update /etc/opendkim.conf and add:

ResolverConfiguration     /etc/unbound/unbound.conf

Once that’s done, restart opendkim and your DKIM key queries are DNSSEC protected (you can verify this in your mail logs since opendkim annotates unprotected keys when it logs).

Note:  This should also apply to Ubuntu 14.04, 14.10, and 15.04.

Update: In Wheezy (and Squeeze, at least the version in backports, I didn’t check the release version) and Ubuntu 10.04 (similarly with backports) this was possible too.  The opendkim.conf parameter was called UnboundConfigFile.  You may have to update your local configuration to use the new name when you upgrade.

Categories: FLOSS Project Planets

Zlatan Todorić: Hell just froze

Mon, 2015-04-27 07:14

It's at least an interesting story that Microsoft celebrated Debian release. Once arch enemy of Linux world, now obviously on their knees when they must do such things to sustain their business model. Next stop, Microsoft goes open source, and still nobody cares. Then they go out of business - and no one cares. Maybe they survive as open hardware producer? ^_^

First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi

Categories: FLOSS Project Planets

Gunnar Wolf: Worthy weekend

Sun, 2015-04-26 23:34

(No, I'm not talking about a future Ubuntu release... After all, what kind of weird animal would a weekend be?)

This weekend we took the kids outside for the first time (not counting, of course, visits to the pediatrician). We were quite anxious... Of course, they were born somewhat under weight and at 7½ months of gestation. But this Saturday we feelt adventurous, and took the kids out for a day among people!

It might not sound like a big deal, but... Well, we took a not such beautiful or scenic route: We took them to the supermarket, and had a small lunch out. For the first time in the already two months they have been with us.

Dinner with friends at home, having a very good time, and –as expected– a... Very hard night for us. All that excitement had the babies very nervous.

Today –again, for the first time– we took the children out to visit some friends of ours. Again, it was great, they behaved very nicely, and were lovely all around.

Lets see what this night holds in place for us.

Anyway, with them growing slowly but steadily... We are very happy, thankful parents. For the first time since Regina is with me in Mexico, this time we decided we would not have a birthday party (yes, I'm 30 minutes away from being 39 year old). I cannot imagine a fuller, better celebration than what we are having. This two babies are the real event in our lives.

Oh... And by the way, this weekend also saw the release of a great new Debian release: Debian 8, codenamed Jessie. Thanks, folks, for such a great birthday present ;-) For reasons that should by now be obvious, I wasn't able to go to either of the release parties I knew of in Mexico City (even one of them was ~500m from home!)

Categories: FLOSS Project Planets

Russ Allbery: lbcd 3.5.2

Sun, 2015-04-26 19:56

This is a fairly minor patch release of the lbcd daemon, which is a daemon that listens for and responds to a simple UDP protocol to request information about system load. It's used in conjunction with lbnamed for dyanmic DNS, and can also be used as a lightweight way to remotely query load.

The only real change in this version is to support linking with libsystemd instead of libsystemd-daemon, since systemd upstream has merged the various small support libraries into one. I also did my normal merge of changes from C TAP Harness and rra-c-util.

NOTE: This package is actually orphaned. No one else has picked it up, and I still maintain the Debian package, so I went ahead and did a new release with this fix. But I'm not planning on doing any significant work on it, and am happy to hand it off to another maintainer.

You can get the latest release from the lbcd distribution page.

Categories: FLOSS Project Planets

Russ Allbery: rra-c-util 5.7

Sun, 2015-04-26 19:44

This release has a somewhat random collection of accumulated fixes.

A couple of them are for the PAM libraries: new support for the Mac OS X PAM implementation, which doesn't use the same options and error codes as the rest of the world. It also uses a different pattern of const declaration, which required some additional Autoconf probing for the fake PAM library for testing.

There are also a few fixes for the systemd probe framework: libsystemd-daemon has been rolled into libsystemd in current versions, and the probe was using $(), which doesn't work on Solaris 10.

The Kerberos Autoconf macros should now hopefully work with the version of Kerberos bundled with Solaris 10.

Finally, this release supports checking for Clang as a compiler and choosing compiler warning flags accordingly, although rra-c-util isn't warning-free with Clang -Weverything yet.

You can get the latest release from the rra-c-util distribution page.

Categories: FLOSS Project Planets

Russ Allbery: C TAP Harness 3.3

Sun, 2015-04-26 19:32

The primary purpose of this release is to merge work by D. Brashear on making it easier to show verbose test success and failure output. Now, if runtests is run with the -v option, or the C_TAP_VERBOSE environment variable is set, the full output of each test case will be printed. The output is currently pretty messy, and I hope to improve it in the future, but it's a start.

This release also supports compilation with Clang with all warnings enabled, and resolves all warnings shown by -Weverything (except the ones about padding, because that's not actually a useful warning). It includes new machinery to detect the compiler and switch warning flags. I will hopefully get a chance to go through my other projects and make them build cleanly with Clang, but it requires adding casts for any conversion from unsigned to signed or vice versa, so that may take a while.

You can get the latest version from the C TAP Harness distribution page.

Categories: FLOSS Project Planets

Steve Kemp: Validating puppet manifests via git hooks.

Sun, 2015-04-26 19:00

It looks like I'll be spending a lot of time working with puppet over the coming weeks.

I've setup some toy deployments on virtual machines, and have converted several of my own hosts to using it, rather than my own slaughter system.

When it comes to puppet some things are good, and some things are bad, as exected, and as any similar tool (even my own). At the moment I'm just aiming for consistency and making sure I can control all the systems - BSD, Debian GNU/Linux, Ubuntu, Microsoft Windows, etc.

Little changes are making me happy though - rather than using a local git pre-commit hook to validate puppet manifests I'm now doing that checking on the server-side via a git pre-receive hook.

Doing it on the server-side means that I can never forget to add the local hook and future-colleagues can similarly never make this mistake, and commit malformed puppetry.

It is almost a shame there isn't a decent collection of example git-hooks, for doing things like this puppet-validation. Maybe there is and I've missed it.

It only crossed my mind because I've had to write several of these recently - a hook to rebuild a static website when the repository has a new markdown file pushed to it, a hook to validate syntax when pushes are attempted, and another hook to deny updates if the C-code fails to compile.

Categories: FLOSS Project Planets

Erich Schubert: Your big data toolchain is a big security risk!

Sun, 2015-04-26 09:41
This post is a follow-up to my earlier post on the "sad state of sysadmin in the age of containers". While I was drafting this post, that story got picked up by HackerNews, Reddit and Twitter, sending a lot of comments and emails my way. Surprisingly many of the comments are supportive of my impression - I would have expected to see much more insults along the lines "you just don't like my-favorite-tool, so you rant against using it". But a lot of people seem to share my concerns. Thanks, you surprised me! Here is the new rant post, in the slightly different context of big data: Everybody is doing "big data" these days. Or at least, pretending to do so to upper management. A lot of the time, there is no big data. People do more data anylsis than before, and therefore stick the "big data" label on them to promote themselves and get green light from management, isn't it? "Big data" is not a technical term. It is a business term, referring to any attempt to get more value out of your business by analyzing data you did not use before. From this point of view, most of such projects are indeed "big data" as in "data-driven revenue generation" projects. It may be unsatisfactory to those interested in the challenges of volume and the other "V's", but this is the reality how the term is used. But even in those cases where the volume and complexity of the data would warrant the use of all the new toys tools, people overlook a major problem: security of their systems and of their data. The currently offered "big data technology stack" is all but secure. Sure, companies try to earn money with security add-ons such as Kerberos authentication to sell multi-tenancy, and with offering their version of Hadoop (their "Hadoop distribution"). The security problem is deep inside the "stack". It comes from the way this world ticks: the world of people that constantly follow the latest tool-of-the-day. In many of the projects, you no longer have mostly Linux developers that co-function as system administrators, but you see a lot of Apple iFanboys now. They live in a world where technology is outdated after half a year, so you will not need to support product longer than that. They love reinstalling their development environment frequently - because each time, they get to change something. They also live in a world where you would simply get a new model if your machine breaks down at some point. (Note that this will not work well for your big data project, restarting it from scratch every half year...)
And while Mac users have recently been surprisingly unaffected by various attacks (and unconcerned about e.g. GoToFail, or the fail to fix the rootpipe exploit) the operating system is not considered to be very secure. Combining this with users who do not care is an explosive mixture... This type of developer, who is good at getting a prototype website for a startup kicking in a short amount of time, rolling out new features every day to beta test on the live users is what currently makes the Dotcom 2.0 bubble grow. It's also this type of user that mainstream products aim at - he has already forgotten what was half a year ago, but is looking for the next tech product to announced soon, and willing to buy it as soon as it is available... This attitude causes a problem at the very heart of the stack: in the way packages are built, upgrades (and safety updates) are handled etc. - nobody is interested in consistency or reproducability anymore. Someone commented on my blog that all these tools "seem to be written by 20 year old" kids. He probably is right. It wouldn't be so bad if we had some experienced sysadmins with a cluebat around. People that have experience on how to build systems that can be maintained for 10 years, and securely deployed automatically, instead of relying on puppet hacks, wget and unzipping of unsigned binary code. I know that a lot of people don't want to hear this, but: Your Hadoop system contains unsigned binary code in a number of places, that people downloaded, uploaded and redownloaded a countless number of times. There is no guarantee that .jar ever was what people think it is. Hadoop has a huge set of dependencies, and little of this has been seriously audited for security - and in particular not in a way that would allow you to check that your binaries are built from this audited code anyway. There might be functionality hidden in the code that just sits there and waits for a system with a hostname somewhat like "yourcompany.com" to start looking for its command and control server to steal some key data from your company. The way your systems are built they probably do not have much of a firewall guarding against such. Much of the software may be constantly calling home, and your DevOps would not notice (nor would they care, anyway). The mentality of "big data stacks" these days is that of Windows Shareware in the 90s. People downloading random binaries from the Internet, not adequately checked for security (ever heard of anybody running an AntiVirus on his Hadoop cluster?) and installing them everywhere. And worse: not even keeping track of what they installed over time, or how. Because the tools change every year. But what if that developer leaves? You may never be able to get his stuff running properly again! Fire-and-forget. I predict that within the next 5 years, we will have a number of security incidents in various major companies. This is industrial espionage heaven. A lot of companies will cover it up, but some leaks will reach mass media, and there will be a major backlash against this hipster way of stringing together random components.
There is a big "Hadoop bubble" growing, that will eventually burst. In order to get into a trustworthy state, the big data toolchain needs to:
  • Consolidate. There are too many tools for every job. There are even too many tools to manage your too many tools, and frontends for your frontends.
  • Lose weight. Every project depends on way too many other projects, each of which only contributes a tiny fragment for a very specific use case. Get rid of most dependencies!
  • Modularize. If you can't get rid of a dependency, but it is still only of interest to a small group of users, make it an optional extension module that the user only has to install if he needs this particular functionality.
  • Buildable. Make sure that everybody can build everything from scratch, without having to rely on Maven or Ivy or SBT downloading something automagically in the background. Test your builds offline, with a clean build directory, and document them! Everything must be rebuildable by any sysadmin in a reproducible way, so he can ensure a bug fix is really applied.
  • Distribute. Do not rely on binary downloads from your CDN as sole distribution channel. Instead, encourage and support alternate means of distribution, such as the proper integration in existing and trusted Linux distributions.
  • Maintain compatibility. successful big data projects will not be fire-and-forget. Eventually, they will need to go into production and then it will be necessary to run them over years. It will be necessary to migrate them to newer, larger clusters. And you must not lose all the data while doing so.
  • Sign. Code needs to be signed, end-of-story.
  • Authenticate. All downloads need to come with a way of checking the downloaded files agree with what you uploaded.
  • Integrate. The key feature that makes Linux systems so very good at servers is the all-round integrated software management. When you tell the system to update - and you have different update channels available, such as a more conservative "stable/LTS" channel, a channel that gets you the latest version after basic QA, and a channel that gives you the latest versions shortly after their upload to help with QA. It covers almost all software on your system, so it does not matter whether the security fix is in your kernel, web server, library, auxillary service, extension module, scripting language etc. - it will pull this fix and update you in no time.
Now you may argue that Hortonworks, Cloudera, Bigtop etc. already provide packages. Well ... they provide crap. They have something they call a "package", but it fails by any quality standards. Technically, a Wartburg is a car; but not one that would pass todays safety regulations...
For example, they only support Ubuntu 12.04 - a three year old Ubuntu is the latest version they support... Furthermore, these packages are roughly the same. Cloudera eventually handed over their efforts to "the community" (in other words, they gave up on doing it themselves, and hoped that someone else would clean up their mess); and Hortonworks HDP (any maybe Pivotal HD, too) is derived from these efforts, too. Much of what they do is offering some extra documentation and training for the packages they built using Bigtop with minimal effort.
The "spark" .deb packages of Bigtop, for example, are empty. They forgot to include the .jars in the package. Do I really need to give more examples of bad packaging decisions? All bigtop packages now depend on their own version of groovy - for a single script. Instead of rewriting this script in an already required language - or in a way that it would run on the distribution-provided groovy version - they decided to make yet another package, bigtop-groovy. When I read about Hortonworks and IBM announcing their "Open Data Platform", I could not care less. As far as I can tell, they are only sticking their label on the existing tools anyway. Thus, I'm also not surprised that Cloudera and MapR do not join this rebranding effort - given the low divergence of Hadoop, who would need such a label anyway? So why does this matter? Essentially, if anything does not work, you are currently toast. Say there is a bug in Hadoop that makes it fail to process your data. Your business is belly-up because of that, no data is processed anymore, your are vegetable. Who is going to fix it? All these "distributions" are built from the same, messy, branch. There is probably only a dozen of people around the world who have figured this out well enough to be able to fully build this toolchain. Apparently, none of the "Hadoop" companies are able to support a newer Ubuntu than 2012.04 - are you sure they have really understood what they are selling? I have doubts. All the freelancers out there, they know how to download and use Hadoop. But can they get that business-critical bug fix into the toolchain to get you up and running again? This is much worse than with Linux distributions. They have build daemons - servers that continuously check they can compile all the software that is there. You need to type two well-documented lines to rebuild a typical Linux package from scratch on your workstation - any experienced developer can follow the manual, and get a fix into the package. There are even people who try to recompile complete distributions with a different compiler to discover compatibility issues early that may arise in the future. In other words, the "Hadoop distribution" they are selling you is not code they compiled themselves. It is mostly .jar files they downloaded from unsigned, unencrypted, unverified sources on the internet. They have no idea how to rebuild these parts, who compiled that, and how it was built. At most, they know for the very last layer. You can figure out how to recompile the Hadoop .jar. But when doing so, your computer will download a lot of binaries. It will not warn you of that, and they are included in the Hadoop distributions, too. As is, I can not recommend to trust your business data into Hadoop.
It is probably okay to copy the data into HDFS and play with it - in particular if you keep your cluster and development machines isolated with strong firewalls - but be prepared to toss everything and restart from scratch. It's not ready yet for prime time, and as they keep on adding more and more unneeded cruft, it does not look like it will be ready anytime soon. One more examples of the immaturity of the toolchain:
The scala package from scala-lang.org cannot be cleanly installed as an upgrade to the old scala package that already exists in Ubuntu and Debian (and the distributions seem to have given up on compiling a newer Scala due to a stupid Catch-22 build process, making it very hacky to bootstrap scala and sbt compilation).
And the "upstream" package also cannot be easily fixed, because it is not built with standard packaging tools, but with an automagic sbt helper that lacks important functionality (in particular, access to the Replaces: field, or even cleaner: a way of splitting the package properly into components) instead - obviously written by someone with 0 experience in packaging for Ubuntu or Debian; and instead of using the proven tools, he decided to hack some wrapper that tries to automatically do things the wrong way... I'm convinced that most "big data" projects will turn out to be a miserable failure. Either due to overmanagement or undermanagement, and due to lack of experience with the data, tools, and project management... Except that - of course - nobody will be willing to admit these failures. Since all these projects are political projects, they by definition must be successful, even if they never go into production, and never earn a single dollar.
Categories: FLOSS Project Planets

Joachim Breitner: Fifth place in Godingame World Cup

Sun, 2015-04-26 08:30

Last evening, Codingame held a “Programming World Cup” titled “There is no Spoon”. The format is that within four hours, you get to write a program that solves a given task. Submissions are first rated by completeness (there are 13 test inputs that you can check your code again, and further hidden tests that will only be checked after submission) and then by time of submission. You can only submit your code once.

What I like about Codingame is that they support a great number of programming languages in their Web-“IDE”, including Haskell. I had nothing better to do yesterday, so I joined. I was aiming for a good position in the Haskell-specific ranking.

After nearly two hours my code completed all the visible test cases and I submitted. I figured that this was a reasonable time to do so, as it was half-time and there are supposed to be two challenges. I turned out that the first, quite small task, which felt like a warm-up or qualification puzzle, was the first of those two, and that therefore I was done, and indeed the 5th fastest to complete a 100% solution! With only less than 5 minutes difference to the 3rd, money-winning place – if I had known I had such a chance, I had started on time...

Having submitted the highest ranked Haskell code, I will get a T-Shirt. I also defended Haskell’s reputation as an efficient programming language, ranked third in the contest, after C++ (rank 1) and Java (rank 2), but before PHP (9), C# (10) and Python (11), listing only those that had a 100% solution.

The task, solving a Bridges puzzle, did not feel like a great fit for Haskell at first. I was juggling Data.Maps around where otherwise I’d simple attach attributes to object, and a recursive function simulated nothing but a plain loop. But it played off the moment I had to implement guessing parts of the solution, trying what happens and backtracking when it did not work: With all state in parameters and pure code it was very simple to get a complete solution.

My code is of course not very polished, and having the main loop live in the IO monad just to be able to print diagnostic commands is a bit ugly.

The next, Lord of the Ring-themed world cup will be on June 27th. Maybe we will see more than 18 Haskell entries then?

Categories: FLOSS Project Planets

Petter Reinholdtsen: First Jessie based Debian Edu beta release

Sun, 2015-04-26 07:10

I am happy to report that the Debian Edu team sent out this announcement today:

the Debian Edu / Skolelinux project is pleased to announce the first *beta* release of Debian Edu "Jessie" 8.0+edu0~b1, which for the first time is composed entirely of packages from the current Debian stable release, Debian 8 "Jessie". (As most reading this will know, Debian "Jessie" hasn't actually been released by now. The release is still in progress but should finish later today ;) We expect to make a final release of Debian Edu "Jessie" in the coming weeks, timed with the first point release of Debian Jessie. Upgrades from this beta release of Debian Edu Jessie to the final release will be possible and encouraged! Please report feedback to debian-edu@lists.debian.org and/or submit bugs: http://wiki.debian.org/DebianEdu/HowTo/ReportBugs Debian Edu - sometimes also known as "Skolelinux" - is a complete operating system for schools, universities and other organisations. Through its pre- prepared installation profiles administrators can install servers, workstations and laptops which will work in harmony on the school network. With Debian Edu, the teachers themselves or their technical support staff can roll out a complete multi-user, multi-machine study environment within hours or days. Debian Edu is already in use at several hundred schools all over the world, particularly in Germany, Spain and Norway. Installations come with hundreds of applications pre-installed, plus the whole Debian archive of thousands of compatible packages within easy reach. For those who want to give Debian Edu Jessie a try, download and installation instructions are available, including detailed instructions in the manual explaining the first steps, such as setting up a network or adding users. Please note that the password for the user your prompted for during installation must have a length of at least 5 characters! == Where to download == A multi-architecture CD / usbstick image (649 MiB) for network booting can be downloaded at the following locations: http://ftp.skolelinux.org/skolelinux-cd/debian-edu-8.0+edu0~b1-CD.iso rsync -avzP ftp.skolelinux.org::skolelinux-cd/debian-edu-8.0+edu0~b1-CD.iso . The SHA1SUM of this image is: 54a524d16246cddd8d2cfd6ea52f2dd78c47ee0a Alternatively an extended DVD / usbstick image (4.9 GiB) is also available, with more software included (saving additional download time): http://ftp.skolelinux.org/skolelinux-cd/debian-edu-8.0+edu0~b1-USB.iso rsync -avzP ftp.skolelinux.org::skolelinux-cd/debian-edu-8.0+edu0~b1-USB.iso The SHA1SUM of this image is: fb1f1504a490c077a48653898f9d6a461cb3c636 Sources are available from the Debian archive, see http://ftp.debian.org/debian-cd/8.0.0/source/ for some download options. == Debian Edu Jessie manual in seven languages == Please see https://wiki.debian.org/DebianEdu/Documentation/Jessie/ for the English version of the Debian Edu jessie manual. This manual has been fully translated to German, French, Italian, Danish, Dutch and Norwegian Bokmål. A partly translated version exists for Spanish. See http://maintainer.skolelinux.org/debian-edu-doc/ for online version of the translated manual. More information about Debian 8 "Jessie" itself is provided in the release notes and the installation manual: - http://www.debian.org/releases/jessie/releasenotes - http://www.debian.org/releases/jessie/installmanual == Errata / known problems == It takes up to 15 minutes for a changed hostname to be updated via DHCP (#780461). The hostname script fails to update LTSP server hostname (#783087). Workaround: run update-hostname-from-ip on the client to update the hostname immediately. Check https://wiki.debian.org/DebianEdu/Status/Jessie for a possibly more current and complete list. == Some more details about Debian Edu 8.0+edu0~b1 Codename Jessie released 2015-04-25 == === Software updates === Everything which is new in Debian 8 Jessie, e.g.: * Linux kernel 3.16.7-ctk9; for the i386 architecture, support for i486 processors has been dropped; oldest supported ones: i586 (like Intel Pentium and AMD K5). * Desktop environments KDE Plasma Workspaces 4.11.13, GNOME 3.14, Xfce 4.12, LXDE 0.5.6 * new optional desktop environment: MATE 1.8 * KDE Plasma Workspaces is installed by default; to choose one of the others see the manual. * the browsers Iceweasel 31 ESR and Chromium 41 * LibreOffice 4.3.3 * GOsa 2.7.4 * LTSP 5.5.4 * CUPS print system 1.7.5 * new boot framework: systemd * Educational toolbox GCompris 14.12 * Music creator Rosegarden 14.02 * Image editor Gimp 2.8.14 * Virtual stargazer Stellarium 0.13.1 * golearn 0.9 * tuxpaint 0.9.22 * New version of debian-installer from Debian Jessie. * Debian Jessie includes about 43000 packages available for installation. * More information about Debian 8 Jessie is provided in its release notes and the installation manual, see the link above. === Installation changes === Installations done via PXE now also install firmware automatically for the hardware present. === Fixed bugs === A number of bugs have been fixed in this release; the most noticeable from a user perspective: * Inserting incorrect DNS information in Gosa will no longer break DNS completely, but instead stop DNS updates until the incorrect information is corrected (710362) * shutdown-at-night now shuts the system down if gdm3 is used (775608). === Sugar desktop removed === As the Sugar desktop was removed from Debian Jessie, it is also not available in Debian Edu jessie. == About Debian Edu / Skolelinux == Debian Edu, also known as Skolelinux, is a Linux distribution based on Debian providing an out-of-the box environment of a completely configured school network. Directly after installation a school server running all services needed for a school network is set up just waiting for users and machines being added via GOsa², a comfortable Web-UI. A netbooting environment is prepared using PXE, so after initial installation of the main server from CD or USB stick all other machines can be installed via the network. The provided school server provides LDAP database and Kerberos authentication service, centralized home directories, DHCP server, web proxy and many other services. The desktop contains more than 60 educational software packages and more are available from the Debian archive, and schools can choose between KDE, GNOME, LXDE, Xfce and MATE desktop environment. == About Debian == The Debian Project was founded in 1993 by Ian Murdock to be a truly free community project. Since then the project has grown to be one of the largest and most influential open source projects. Thousands of volunteers from all over the world work together to create and maintain Debian software. Available in 70 languages, and supporting a huge range of computer types, Debian calls itself the universal operating system. == Thanks == Thanks to everyone making Debian and Debian Edu / Skolelinux happen! You rock.
Categories: FLOSS Project Planets

Russell Coker: Anti-Systemd People

Sun, 2015-04-26 06:59
For the Technical People

This post isn’t really about technology, I’ll cover the technology briefly skip to the next section if you aren’t interested in Linux programming or system administration.

I’ve been using the Systemd init system for a long time, I first tested it in 2010 [1]. I use Systemd on most of my systems that run Debian/Wheezy (which means most of the Linux systems I run which aren’t embedded systems). Currently the only systems where I’m not running Systemd are some systems on which I don’t have console access, while Systemd works reasonably well it wasn’t a standard init system for Debian/Wheezy so I don’t run it everywhere. That said I haven’t had any problems with Systemd in Wheezy, so I might have been too paranoid.

I recently wrote a blog post about systemd, just some basic information on how to use it and why it’s not a big deal [2]. I’ve been playing with Systemd for almost 5 years and using it in production for almost 2 years and it’s performed well. The most serious bug I’ve found in systemd is Bug #774153 which causes a Wheezy->Jessie upgrade to hang until you run “systemctl daemon-reexec” [3].

I know that some people have had problems with systemd, but any piece of significant software will cause problems for some people, there are bugs in all software that is complex enough to be useful. However the fact that it has worked so well for me on so many systems suggests that it’s not going to cause huge problems, it should be covered in the routine testing that is needed for a significant deployment of any new version of a distribution.

I’ve been using Debian for a long time. The transitions from libc4 to libc5 and then libc6 were complex but didn’t break much. The use of devfs in Debian caused some issues and then the removal of devfs caused other issues. The introduction of udev probably caused problems for some people too. Doing major updates to Debian systems isn’t something that is new or which will necessarily cause significant problems, I don’t think that the change to systemd by default compares to changing from a.out binaries to ELF binaries (which required replacing all shared objects and executables).

The Social Issue of the Default Init

Recently the Debian technical committee determined that Systemd was the best choice for the default init system in Debian/Jessie (the next release of Debian which will come out soon). Decisions about which programs should be in the default install are made periodically and it’s usually not a big deal. Even when the choice is between options that directly involve the user (such as the KDE and GNOME desktop environments) it’s not really a big deal because you can just install a non-default option.

One of the strengths of Debian has always been the fact that any Debian Developer (DD) can just add any new package to the archive if they maintain it to a suitable technical standard and if copyright and all other relevant laws are respected. Any DD who doesn’t like any of the current init systems can just package a new one and upload it. Obviously the default option will get more testing, so the non-default options will need more testing by the maintainer. This is particularly difficult for programs that have significant interaction with other parts of the system, I’ve had difficulties with this over the course of 14 years of SE Linux development but I’ve also found that it’s not an impossible problem to solve.

It’s generally accepted that making demands of other people’s volunteer work is a bad thing, which to some extent is a reasonable position. There is a problem when this is taken to extremes, Debian has over 1000 developers who have to work together so sometimes it’s a question of who gets to do the extra work to make the parts of the distribution fit together. The issue of who gets to do the work is often based on what parts are the defaults or most commonly used options. For my work on SE Linux I often have to do a lot of extra work because it’s not part of the default install and I have to make my requests for changes to other packages be as small and simple as possible.

So part of the decision to make Systemd be the default init is essentially a decision to impose slightly more development effort on the people who maintain SysVInit if they are to provide the same level of support – of course given the lack of overall development on SysVInit the level of support provided may decrease. It also means slightly less development effort for the people who maintain Systemd as developers of daemon packages MUST make them work with it. Another part of this issue is the fact that DDs who maintain daemon packages need to maintain init.d scripts (for SysVInit) and systemd scripts, presumably most DDs will have a preference for one init system and do less testing for the other one. Therefore the choice of systemd as the default means that slightly less developer effort will go into init.d scripts. On average this will slightly increase the amount of sysadmin effort that will be required to run systems with SysVInit as the scripts will on average be less well tested. This isn’t going to be a problem in the short term as the current scripts are working reasonably well, but over the course of years bugs may creep in and a proposed solution to this is to have SysVInit scripts generated from systemd config files.

We did have a long debate within Debian about the issue of default init systems and many Debian Developers disagree about this. But there is a big difference between volunteers debating about their work and external people who don’t contribute but believe that they are entitled to tell us what to do. Especially when the non-contributors abuse the people who do the work.

The Crowd Reaction

In a world filled with reasonable people who aren’t assholes there wouldn’t be any more reaction to this than there has been to decisions such as which desktop environment should be the default (which has caused some debate but nothing serious). The issue of which desktop environment (or which version of a desktop environment) to support has a significant affect on users that can’t be avoided, I could understand people being a little upset about that. But the init system isn’t something that most users will notice – apart from the boot time.

For some reason the men in the Linux community who hate women the most seem to have taken a dislike to systemd. I understand that being “conservative” might mean not wanting changes to software as well as not wanting changes to inequality in society but even so this surprised me. My last blog post about systemd has probably set a personal record for the amount of misogynistic and homophobic abuse I received in the comments. More gender and sexuality related abuse than I usually receive when posting about the issues of gender and sexuality in the context of the FOSS community! For the record this doesn’t bother me, when I get such abuse I’m just going to write more about the topic in question.

While the issue of which init system to use by default in Debian was being discussed we had a lot of hostility from unimportant people who for some reason thought that they might get their way by being abusive and threatening people. As expected that didn’t give the result they desired, but it did result in a small trend towards people who are less concerned about the reactions of users taking on development work related to init systems.

The next thing that they did was to announce a “fork” of Debian. Forking software means maintaining a separate version due to a serious disagreement about how it should be maintained. Doing that requires a significant amount of work in compiling all the source code and testing the results. The sensible option would be to just maintain a separate repository of modified packages as has been done many times before. One of the most well known repositories was the Debian Multimedia repository, it was controversial due to flouting legal issues (the developer produced code that was legal where they lived) and due to confusion among users. But it demonstrated that you can make a repository containing many modified packages. In my work on SE Linux I’ve always had a repository of packages containing changes that haven’t been accepted into Debian, which included changes to SysVInit in about 2001.

The latest news on the fork-Debian front seems to be the call for donations [4]. Apparently most of the money that was spent went to accounting fees and buying a laptop for a developer. The amount of money involved is fairly small, Forbes has an article about how awful people can use “controversy” to get crowd-funding windfalls [5].

MikeeUSA is an evil person who hates systemd [6]. This isn’t any sort of evidence that systemd is great (I’m sure that evil people make reasonable choices about software on occasion). But it is a significant factor in support for non-systemd variants of Debian (and other Linux distributions). Decent people don’t want to be associated with people like MikeeUSA, the fact that the anti-systemd people seem happy to associate with him isn’t going to help their cause.

Conclusion

Forking Debian is not the correct technical solution to any problem you might have with a few packages. Filing bug reports and possibly forking those packages in an external repository is the right thing to do.

Sending homophobic and sexist abuse is going to make you as popular as the GamerGate and GodHatesAmerica.com people. It’s not going to convince anyone to change their mind about technical decisions.

Abusing volunteers who might consider donating some of their time to projects that you like is generally a bad idea. If you abuse them enough you might get them to volunteer less of their time, but the most likely result is that they just don’t volunteer on anything associated with you.

Abusing people who write technical blog posts isn’t going to convince them that they made an error. Abuse is evidence of the absence of technical errors.

Related posts:

  1. systemd – a Replacement for init etc The systemd projecct is an interesting concept for replacing init...
  2. Systemd Notes A few months ago I gave a lecture about systemd...
  3. more on anti-spam In response to my last entry about anti-spam measures and...
Categories: FLOSS Project Planets

Patrick Schoenfeld: Sharing code between puppet providers

Sun, 2015-04-26 03:26

So you’ve written that custom puppet type for something and start working on another puppet type in the same module. What if you needed to share some code between this types? Is there a way of code-reuse that works with the plugin sync mechanism?

Yes, there is.

Puppet even has two possible ways of sharing code between types.

Option #1: a shared base provider

A provider in puppet is basically a class associated with a certain (puppet) type and there can be a lot providers for a single type (just look at the package provider!). It seems quiet natural, that it’s possible to define a parent class for those providers. So natural, that even the official puppet documentation writes about it.

Option #2: Shared libraries

The second option is a shared library, shipped in a certain namespace in the lib directory of the module, whereas the idea is mostly sketched in the feature ticket #14149. Basically one defines a class in the special Puppetx namespace, using the author- and module-name in the class name, in order to avoid conflicts with other modules.

require 'puppetx' module Puppetx::<Author> module Puppetx::<Author>::<Modulename> ... your helper functions go here ... end end

This example would be saved to

lib/<author>/<modulename>

in your module’s folder and be included in your provider with something along the following:

require File.expand_path(File.join(File.dirname(__FILE__), "..", "..", "..", "puppet_x", "<author>", "<modulename>.rb"))

Compatibility with Puppet 4:

In puppet 4 the name of the namespace has changed slightly. It’s now called ‘PuppetX’ instead of ‘Puppetx’ and is stored in a file ‘puppet_x.rb’, which means that the require and the module name itself need to be changed:

require 'puppet_x' module PuppetX::<authorname> module PuppetX::<authorname>::<modulename>

For backward compatibility with puppet 3 you could instead add something like this, according to my co-worker mxey, who knows way more about ruby then I do:

module PuppetX   module <Author>     <Modulename> = Puppetx::<Authorname>::<Modulename>   end end

Apart from this you’d need to change the require to be conditional on the puppet-version and refer to the module by the aliased version (which is left as an exercise for the reader ;))

Categories: FLOSS Project Planets

Zlatan Todorić: Geekdom Happiness Arrived

Sat, 2015-04-25 20:24

Debian 8 aka Jessie has just been released and the celebrations are being held all around the world. So grab a cake or a beer because today your precious server is so much happy and you love your server. Debian will take care of the rest. Cheers!

Categories: FLOSS Project Planets